Saturday, 18 June 2011
Lessons for travellers from the United Airlines computer outage
It's not yet clear what caused the 5-hour network connectivity problem that disrupted United Airlines flights Friday, and has left many travellers still stranded or facing flight delays and cancellations. But this isn't the first time something like this has happened (the last time I wrote about a similar incident, in 2006, it also involved United) and it won't be the last.
What happened? And what could airlines and travellers each do to reduce the damage when it happens again?
Most major airlines outsource hosting of their reservations or Passenger Name Record (PNR) databases to one or another of the big four Computerized Reservation Systems (CRS's). In the case of United, that host CRS is the the Galileo/Apollo service of Travelport. The Galileo/Apollo CRS was originally developed in-house by United, but was sold off some years ago as part of United's (unsuccessful) efforts to raise enough cash to stave off bankruptcy.
Knowing that United is hosted in Galileo/Apollo by Travelport isn't enough, though, to identify who's responsible for yesterday's outage. There are many layers of "network connectivity", and many potential points of failure. Twenty years ago, most of those messages were routed over dedicated "leased line" data connections or proprietary networks developed and operated exclusively for the airline industry by AIRINC and SITA. Today, most of them travel over VPN's layered on top of the public Internet. That's cheaper for airlines than maintaining their own dedicated network, and airlines have been obsessed with reducing CRS and other networking costs. But unlike the SITA airline messaging network, the Internet wasn't designed for mission-critical real-time reliability. Today, any sort of problem with the public Internet could cause airlines to lose connectivity between their own ticketing and check-in counters and kiosks, their operations and aircraft and crew dispatch offices, and the outsourced, remotely hosted reservation databases -- in the case of United, the Galileo data centers in Denver and Chicago -- on which they have come to rely.
Like the general public, however, airlines in the USA have come to take the Internet and other information technology for granted even while relying on it more and more. As independent airline consultant Bob Mann noted in a story about United's problems, there are relatively few United employees who still know how to do things like issue boarding passes manually. That's one of the direct negative consequences for travellers of the pay cuts and layoffs that accompanied United's bankruptcy, and the de-skilling of front-line operational and "customer service" jobs. Is it worth saving a couple of dollars on each airline ticket you buy, if that is achieved by replacing the most skilled and experienced staff at the airport -- those best able to find ways to keep flights operating when things go wrong -- with lower-paid new hires whose only response is, "There's nothing we can do if the computers are down"?
By way of comparison to United, I've been on flights on Ethiopian Airlines that operated on schedule even though we were travelling with e-tickets and there were no computers or boarding pass printers operating at the departure airport and, in fact, the power had been out throughout the town for at least two days. Perhaps it's unfair to compare United to Ethiopian -- few, if any, airlines in the world can match Ethiopian's unflappable operational efficiency and reliability, especially in difficult conditions -- but an inability to find workarounds or cope with interruptions in connectivity isn't something inevitable. It's a result of deliberate decisions and attitudes of United's management toward the relative priorities to place on short-term profits, employee compensation and retention, and reliability and fault tolerance of operations. (By way of full disclosure, I've received discounted tickets in the past on both United and Ethiopian.)
United is offering to "waive penalties" for changes to reservations on flights that were cancelled or rescheduled Friday or Saturday. But it would be more accurate to describe that as a contractual obligation rather than an "offer". As discussed in my FAQ About Changes to Flights and Tickets, airlines' own conditions of carriage require them to give a full and unconditional refund (not, for example, merely a voucher or credit towards other tickets) -- even if your ticket was otherwise completely nonchangeable and nonrefundable -- if your flight is cancelled or you decline to accept even what they consider a "minor" schedule change. If they tell you, "This flight will be leaving an hour later than originally scheduled", you are fully entitled to say the magic words, "I do not accept that schedule change. I want my money back, in full, now."
I was travelling with e-tickets on Ethiopian, but both I and the airline had -- as a matter of that airline's routine -- paper backups. US Federal law and Department of Transportation (DOT) regulations require airlines to provide each passenger with a "ticket", but the DOT has chosen not to enforce this regulation with respect to e-tickets. Most US airlines have stopped providing paper tickets. "E-ticket confirmations" in their present form aren't an adequate replacement for paper tickets: They contain only a fraction of the information actually contained in the e-ticket record in the CRS (and formerly encoded on paper tickets) which both the traveller and the airline need if the e-ticket record is, for whatever reason, unavailable.
I raised exactly this issue with the DOT last year, but my comments (which include a list of the critical information included on tickets but missing from typical e-ticket confirmations) were ignored. Friday's incident makes clear the consequences of DOT regulatory nonfeasance, and why the DOT needs to start enforcing its rules requiring airlines to provide each passenger with a ticket, and why passengers need to insist on a complete ticket and not just a summary of their reservations. (More on the other problems with e-tickets and the other "consumer protection issues" that the DOT needs to address in its ongoing series of rulemakings on airline business practices.) I had already been hoping to meet with the DOT consumer protection office Friday, while I'm in Washington. In the event, I wasn't able to meet with the DOT on this visit, but I'll be reminding them of this incident in future comments in the next round of their rulemaking.
The final issue in the ability of airlines to operate without network connectivity is their obligation under secret implementing directives for Secure Flight to transmit personal information about each passenger to the TSA, and receive an affirmative per-flight, per-passenger "cleared" permission message, before issuing them a boarding pass. Airlines, CRS's, and other airline industry IT companies have spent more than US$2 billion, by DHS under-estimate, building those messaging and DHS control links into the airline and CRS infrastructure. The default is "no", but it remains a secret whether there is any exception in case of loss of connectivity between TSA and an airline (or, more precisely, its host CRS). The TSA continues to stonewall my FOIA requests and appeals for their procedures. Their latest message was that they are unable to say when, if ever, they will make an initial determination on some of my FOIA requests that were supposed to be acted on almost two years ago, despite specific requirements in the FOIA law that they act within at most 30 days of an initial request, and advise requesters of the date when they will do so.Link | Posted by Edward on Saturday, 18 June 2011, 18:27 ( 6:27 PM)