NATs has reported that the system is now back up and running. It is possible that the secondary system took over with data syncing having to take place before the switchover?
My original post was about my interest in the technical aspects of redundancy and failover mechanisms in something as mission-critical as air traffic control.
I come from a background in broadcasting, where failover is critical, and redundancy is built into any broadcast chain. e have multiple backups and jumping-off points to deal with any issues that arise. It's pretty rare we could ever go to “black”.
I am genuinely convinced that the UK is in collapse. The nhs is flattened, policing is meh, the infrastructure is pretty banged up, bills are through the roof, inflation and interest rates are mad high, and now this. Not to mention a noticeable drop in service quality throughout the private sector, with everyone blaming “staff shortage” and a government desperate to suppress the very same wages that are meant to attract more qualified staff. Seriously this is madness.
Most of these are global issues though in the developed world, even healthcare to an extent due to ageing population. I'm not sure what this has to do with an air traffic control outage either?
It does seem that a lot of infrastructure that was put in place during a golden age cannot be adequately maintained, but AFAIK this seems to be the case in USA and Europe too.
You'd hope an air traffic control centre would have a big box of popsicle sticks and black pen markers to cover the fallover of power | backup power | digital systems.
Because large systems are complex and have very complex and hard to detect failure modes.
What is more impressive is how rare these events are, given the complexity of the underlying systems. Redundancy is not without its own problems (source of truth, for instance).
The IT is just poor. Tech firms routinely change much more massive and complex systems at a far faster pace compared to the stagnation found in airline IT, and yet total failures are not more frequent.
Im not sure that's true. If something makes the news it's likely to be of interest to a large amount of people, that doesn't immediately qualify it as something rare.
A lot of times large scale outages like this are because of the redundancy. The whole system is interlinked with automatic failover. then it hits a corner case that was not engineered into the fail model and you get cascading system failure where each node starts bringing down other nodes automatically. basically the lesson is: In highly interlinked systems you get highly interlinked failures.
And then after a lot of angry words and finger pointing this new failure gets added to the failure model.
My personal takeaway after chasing the long tail of automatic failover on a few projects, is quite often it is better to drop a few 9's from your service goal, decouple some of the systems, and accept that while some parts of the system may go down, it should not bring everything down with it.
Complexity tends to hide problems. An over-engineered system is going to be less stable than a simple one where you know how things can break and how to bring them online pretty quickly.
The US air traffic control crash at the start of the year turned out to be a decades old system which had not been upgraded, I think the UK one is a little bit more recent, but like alot of software, people dont want to pay for the real costs of software developments.
They agiled software to the point where it’s written by product managers and wanna be engineers that cargo cult “good practice” without understanding what it’s meant to do. This types of issues are common in british made software.
Lots of US bound traffic took very southerly routings today to avoid UK airspace. My guess is that this will continue until the outage is completely resolved.
According to unconfirmed reports, a French airline entered a flight plan incorrectly, resulting in some form of data corruption. https://www.spectator.co.uk/article/is-one-badly-filed-fligh...
NATs has reported that the system is now back up and running. It is possible that the secondary system took over with data syncing having to take place before the switchover?
My original post was about my interest in the technical aspects of redundancy and failover mechanisms in something as mission-critical as air traffic control.
I come from a background in broadcasting, where failover is critical, and redundancy is built into any broadcast chain. e have multiple backups and jumping-off points to deal with any issues that arise. It's pretty rare we could ever go to “black”.
Russian attack, similar to the one in Poland last week? https://www.bbc.com/news/world-europe-66630260.amp
Unlikely, they’ve already arrested suspects following the attack
https://www.wprost.pl/amp/11364934/atak-na-polska-infrastruk...
No British Security Services, protecting the economy, by getting people and businesses to spend their money whether they like it or not.
I am genuinely convinced that the UK is in collapse. The nhs is flattened, policing is meh, the infrastructure is pretty banged up, bills are through the roof, inflation and interest rates are mad high, and now this. Not to mention a noticeable drop in service quality throughout the private sector, with everyone blaming “staff shortage” and a government desperate to suppress the very same wages that are meant to attract more qualified staff. Seriously this is madness.
Most of these are global issues though in the developed world, even healthcare to an extent due to ageing population. I'm not sure what this has to do with an air traffic control outage either?
It does seem that a lot of infrastructure that was put in place during a golden age cannot be adequately maintained, but AFAIK this seems to be the case in USA and Europe too.
How is this even possible in this day and age? The Redundancy for this must be crazy
System redundancy rarely covers software faults.
You'd hope an air traffic control centre would have a big box of popsicle sticks and black pen markers to cover the fallover of power | backup power | digital systems.
Well, they do (figuratively). That is why the entire airspace is not totally shut down, but capacity "just" substantially reduced.
I believe that's literally what they fail over to, yes.
Per aircraft physical batons were literally the way a number of pre digital air traffic control systems worked.
Whoever has the stick is responsible for the aircraft - it was a literal "unique token".
Because large systems are complex and have very complex and hard to detect failure modes.
What is more impressive is how rare these events are, given the complexity of the underlying systems. Redundancy is not without its own problems (source of truth, for instance).
They aren't that rare. Last UK ATC crash was in 2014 I think:
https://www.ft.com/content/65544730-8216-11e4-b9d0-00144feab...
And that's just the UK. Airport systems crashing is a regular occurrence. Paris crashed due to still relying on a Windows 3.1 system:
https://www.zdnet.com/article/a-23-year-old-windows-3-1-syst...
The IT is just poor. Tech firms routinely change much more massive and complex systems at a far faster pace compared to the stagnation found in airline IT, and yet total failures are not more frequent.
If it makes the news it is almost by definition rare.
Im not sure that's true. If something makes the news it's likely to be of interest to a large amount of people, that doesn't immediately qualify it as something rare.
Musk's foolish actions while owner of Twitter are not rare yet always on the news
Not to forget the failure of Frankfurt airport systems earlier this year due to the breakage of a single glass-fibre cable:
https://www.dw.com/en/lufthansa-system-failure-causes-massiv...
Though this is not due to outdated IT, but lack of redundancy of a critical component of the system instead.
A lot of times large scale outages like this are because of the redundancy. The whole system is interlinked with automatic failover. then it hits a corner case that was not engineered into the fail model and you get cascading system failure where each node starts bringing down other nodes automatically. basically the lesson is: In highly interlinked systems you get highly interlinked failures.
And then after a lot of angry words and finger pointing this new failure gets added to the failure model.
My personal takeaway after chasing the long tail of automatic failover on a few projects, is quite often it is better to drop a few 9's from your service goal, decouple some of the systems, and accept that while some parts of the system may go down, it should not bring everything down with it.
Complexity tends to hide problems. An over-engineered system is going to be less stable than a simple one where you know how things can break and how to bring them online pretty quickly.
The US air traffic control crash at the start of the year turned out to be a decades old system which had not been upgraded, I think the UK one is a little bit more recent, but like alot of software, people dont want to pay for the real costs of software developments.
They agiled software to the point where it’s written by product managers and wanna be engineers that cargo cult “good practice” without understanding what it’s meant to do. This types of issues are common in british made software.
Lots of US bound traffic took very southerly routings today to avoid UK airspace. My guess is that this will continue until the outage is completely resolved.