911 services went down at two different points so far in 2024. The first (written about in this blog) involved an AT&T outage caused by human error during a software upgrade. Landlines were an available alternative during the outage. The second is more recent: four states lost 911 capabilities – both cellular and landline – for 2.5 hours due to a light pole installation for three of those states (the responsible party does not serve one of the states).
I’ll go a little into the specifics of the recent outage first. All of South Dakota was impacted and only parts of Nebraska and Nevada - including Las Vegas - were. Texas’s outage was a cellular network issue.
In South Dakota, they used social media to inform the public they can still text and to call anyway, they’ll call back, which they did. The staff were also able to shift calls to the administrative (read: non-emergency) lines. This allowed first responders to receive the same dispatch services they normally receive. Nevada used the same workaround although their system could take landline calls.
Now let’s talk about the cause. Lumen is responsible for the 3-state outage. Lumen is an emergency communications services provider. The installation of the light pole was being done by a third party. Third party reliance is an issue. Ask Boeing. It shows, also, that services can be interconnected. So, where is the issue?
911 services are part of critical infrastructure and it exposes a massive vulnerability. Like a ship colliding with a bridge in Baltimore did last month (there are 16 sectors of critical infrastructure, these are just two). How many times does the canary need to sing in the coal mine before we invest in infrastructure?
A big take away is someone during the first AT&T-caused outage of 911 said, “Thank goodness we still have landlines!” To which they thought, or someone else replied, “Imagine if we didn’t…” and made moves to figure out what to do. Even better, during the outage, one state shared how they were successfully handling calls so other states could do it, too!
How true is this? I don’t know – but the point is at some time, someone thought about it and solved for it. Just in case. And all workable solutions need to be shared across an industry to has the same goal in service for the communities they serve. There is no competition here. This is how best practices are created. Actually I’d say the truth is probably closer to they’ve done business continuity workarounds ages ago because they are providing a critical service that needs to be around.
Never assume you’ll always have access to things, even if you are the thing everyone calls in an emergency, because someone, somewhere has never thought about it, hasn’t invested in it, or decided other things are more important. How many times does the canary need to sing in the coal mine before you make sure the ecosystem that supports your business is reliable?
People get very comfortable with assumptions of continued service. You can only offer that if you’ve done business continuity.
Comments