Microsoft says Delta’s ancient IT explains long outage after CrowdStrike snafu

submitted a month ago by some_guy

arstechnica.com/tech-policy/2024/08/microsoft-s…

195

Log in to comment

30 Comments

Bob Robertson IX a month ago

The letter said that after one Microsoft outreach on July 22, a "Delta employee replied, saying 'all good. Cool will let you know and thank you.' Despite this assessment that things were 'all good,' public reports indicate that Delta canceled more than 1,100 flights on July 22 and more than 500 flights on July 23."

whodatdair a month ago

michael_jackson_eating_popcorn.gif

FenrirIII a month ago

Munchin'

reddig33 a month ago

That’s strange. Southwest Airline’s ancient IT actually saved them from crowdstrike.

https://www.digitaltrends.com/computing/southwest-cloudstrike-windows-3-1/

some_guy [OP] a month ago
Optional a month ago

Ironically it debunks it by saying, yes, Southwest has key scheduling applications running on 3.1 and 95.

FiskFisk33 a month ago

No it doesn't, nowhere does it say that.

SkySolver and Crew Web Access, *look* “historic like they were designed on Windows 95”. The fact that they are also available as mobile applications should further make it clear that no, these applications are not running on Windows 3.1 or Windows 95.

Optional a month ago

The fact that they are also available as mobile applications should further make it clear that no, these applications are not running on Windows 3.1 or Windows 95.

That kind of language will get you kicked in the balls by engineers. Sure. It *should* make it "clear" that they're not running on this* OS *or* **that OS.

And what should also be made clear is that statement is an assumption. A probable one, IMO, a reasonable one, but *an assumption* nonetheless and therefore no one can call it a fact unless they just want to pretend to be right.

I ojbect to using language like "it looks like a thing so it's OBVIOUSLY a thing, you morons" being presented as irrefutable evidence of some sort.

The fact that it's an assumption should further make it clear that no, this is not a fact, and stating it as a fact is bullshit or deliberate misrepresentation.

kalleboo a month ago

Where does it say that? It says that the source says that they are mobile apps (so obviously NOT Windows) that "look like they were designed for Windows 95".

Optional a month ago

Southwest uses internally built and maintained systems called SkySolver and Crew Web Access for pilots and flight attendants. They can sign on to those systems to pick flights and then make changes when flights are canceled or delayed or when there is an illness.

“Southwest has generated systems internally themselves instead of using more standard programs that others have used,” Montgomery said. “Some systems even look historic like they were designed on Windows 95.”

SkySolver and Crew Web Access are both available as mobile apps, but those systems often break down during even mild weather events, and employees end up making phone calls to Southwest’s crew scheduling help desk to find better routes. During periods of heavy operational trouble, the system gets bogged down with too much demand.

I don't know what "look historic" is supposed to mean, but if it looks like it was developed on Windows 95 that's 99% of the time because it *was* developed on Windows 95. Mobile apps "are available" wasn't as definitive as perhaps the author intended - meaning what, exactly? It's an option?

If it's a homegrown app (and good for them if so - every weasel IT manager in the world has been trying to bring them down for it since day one I'll bet), and it was written originally for Win95 and it's still in use, the bet would be it's run inside a VM on whatever they use now. Should whatever they use now go into a boot loop - *theoretically* - they could run it natively if they had to.

All speculation of course.

TeoTwawki a month ago, edited a month ago

-removed-

It was what crowdstrike themselves told us to do!!!! but I get bad faith questions assumptions and exaggerations out of people allegedly in my field here on Lemmy. Bullshit. You clowns belonged back on reddit. You are the worst kind of people.

Optional a month ago

Complexity increases exponentially in large organizations, for a number of reasons.

TeoTwawki a month ago, edited a month ago

I know my industry thanks. I've over a decade of experiance in sizable complex organizations. You know who really likes to cling to outdated hardware and software? Hopefully this scares you because it should: medical organizations.

What the IT people did (and what others claimed was done) during this fiasco was objectively worse than the actual fix but everyone in this thread just isn't happy that I didn't join them in shitting on microsoft. This is where lemmy shows that its users are becoming more like reddit users every day. Or don't know what /s means.

Hopefully none of those systems were exposed to anything internet facing for obvious reasons, but given the shear incompetance observed I wouldn't be surprised.

lennivelkant a month ago

...and just how many PCs do you intend to "reboot into safemode delete one bad file and then reboot again"? Manually, or do you have some remote access tool that doesn't require a running system?

TeoTwawki a month ago, edited a month ago

Deleted by author

FiskFisk33 a month ago, edited a month ago

Yes, yes it is if you run bitlocker with external verification.

It's even harder if the server you use for the verification itself is down.

TimeSquirrel a month ago, edited a month ago

*Ta-da and chime noises*

I hope they still have Skifree on them.

_NetNomad a month ago

so Delta's non-Windows machines were the ones that suffered the most from a Windows software malfunction? that makes sense

Buelldozer a month ago

It didn't say "non-windows" it said "served by other providers like IBM". It could easily be Windows servers in IBM's cloud and wouldn't ya' know...IBM uses Crowdstrike.

_NetNomad a month ago

i'm gonna level with you, i completely forgot IBM cloud was a thing and just thought this was MS pointing fingers at system Z or system . thanks for catching that!

some_guy [OP] a month ago

Having to reset or recalibrate other old systems that were disrupted by newer ones going offline makes sense to me. If servers were providing Network Time Protocol and older clients drifted without it, that could cause them to be unable to rejoin a domain. I'm speculating wildly, but it's an example of how losing important infra can cause issues even after it's restored.

Optional a month ago

Here’s me believing a single fucking thing Micro$oft says

Avg a month ago

They are not lying though, they fired the guy who was really good with excel years ago and are now too afraid to change the excel file he created containing all bookings ever.

ಠ_ಠ

I'll just assume you're joking, even if it sounds oddly specific

lennivelkant a month ago

Having seen Excel used creatively, I think it's an exaggeration. It would make collaboration entirely impossible. I assume they have several smaller ones, with more or less - but not exactly - the same layout as it has been adapted for new use cases, and the only way to transfer records from one to the other is to manually copy and paste the info to the relevant cells, but mind the order you do it in and double check, or the Frankenstein's Macro running half the logic will crash.

EtherWhack a month ago

I know from working in manufacturing, at least, that people like to abuse excel and try to use it as a DB client

Wooki a month ago, edited a month ago

What was that? I couldn't hear over the all that Microsoft cloud hacking. But by golly that silence before sunburst was deafening...

technocrit a month ago

You can tell their infrastructure is outdated and insecure by the fact they're using Microsoft.