Microsoft outage

[amanda]

Global outage, affecting banks and supermarkets. Not believed to be cyber security related, but more like a  tech issue related to 'bad software' update from a third party. 

[amanda]

5 pm news update: Affecting airports too, America affected first.

Crowdstrike is the company affected, based in Texas. It seems like one of these servers that has gone down, and when that happens, they shut everything down in case it is a real cyber attack.

[tetsabb]

Nothing to do with a recent CME from the sun?

[barbados]

It’s screwed my morning up slightly getting dragged into meetings, but fortunately everyone in my office listens to me, so had their laptops shut down (bar a handful of people)
But the most obvious thing that springs to mind is don’t make changes on a Friday”

[barbados]

Jul 19, 2024 10:43:13 GMT tetsabb said:

Nothing to do with a recent CME from the sun?

No, crowdstrike pushed out an update with an “undocumented feature”

[jenny]

I experienced an inability to access the internet via cellular data yesterday, and the streaming on our TV was a bit unreliable later in the day too - I assume that was all connected.

[Dix]

Let's just say that some people at work had an extremely busy day today while most others were unable to do anything at all for most of the day. Cards were played and selfies were taken featuring the quite nice view we have on one side of the building.

[barbados]

I was in the first subset, mostly down to the peter principle kicking in with the people I was ignoring while busy fixing the issue.

[Bondee]

I heard several announcements that the train information boards may be showing the wrong departure times on my journey home, which I assume (yeah, I know!) had something to do with it.

[amanda]

Jul 19, 2024 21:10:57 GMT Bondee said:

I heard several announcements that the train information boards may be showing the wrong departure times on my journey home, which I assume (yeah, I know!) had something to do with it.

My Australian city's country train boards were shut down and subsequent delays occurred for those trains. I didn't check the metro trains so am unsure. 

Several shops and supermarkets systems also went down, affecting the flow on into Friday night shopping, a friend of mine took an item back to supermarket here but that was 7.30pm when she came and it seemed to be ok then. 

My Windows 11 laptop wasn't affected at all. 

[Bondee]

My mum said that the local Co-op was only accepting cash payments because their card system had gone down.

[efros]

12-hour time-lapse of American Airlines, Delta & United Airlines plane traffic after the IT outage

[amanda]

Stuffed up Australian airports too.

[suze]

A day on, what do we know?

Early reports had it that Microsoft Azure had been part of the problem, but that now seems to have been a red herring.  Azure did go down for a bit, but not worldwide and not for all that long.  Those who contribute to Wikipedia are not usually shy about an opportunity for a spot of Microsoft bashing, and even they have not chosen to go there.  That outage didn't help, but was not critical of itself.

These Crowdstrike folk, on the other hand, absolutely did manage to break Windows.  Precisely how they did that, well we don't know yet - even they don't know yet - and I probably wouldn't understand the explanation anyway.  Will it turn out to be something stupidly trivial, like a - sign that should have been a + sign? 

But how could this have been allowed to happen?  I'd rather imagined that when Crowdstrike's people have written a new version of their code, they then try it out on all the computers in their special Trying Stuff Out Room before they release it into the wild.  Are we to suppose that they didn't bother on this occasion?

Given the way that Corporate America works, how comes Crowdstrike's CEO still to have a job?  One has to question whether the company can survive the lawsuits that will surely be heading its way very soon, and the CEO is usually long gone by then.

[amanda]

With other major crashes, it's happened due to an error along the way, often/usually caused by an incorrect or 'bad' code with a planned update. Unless I see any of the three techy people who assist at the different libraries in the next week, I am unlikely to get a better explanation.

[barbados]

Jul 20, 2024 7:20:17 GMT suze said:

A day on, what do we know?

Early reports had it that Microsoft Azure had been part of the problem, but that now seems to have been a red herring.  Azure did go down for a bit, but not worldwide and not for all that long.  Those who contribute to Wikipedia are not usually shy about an opportunity for a spot of Microsoft bashing, and even they have not chosen to go there.  That outage didn't help, but was not critical of itself.

N

These Crowdstrike folk, on the other hand, absolutely did manage to break Windows.  Precisely how they did that, well we don't know yet - even they don't know yet - and I probably wouldn't understand the explanation anyway.  Will it turn out to be something stupidly trivial, like a - sign that should have been a + sign? 

But how could this have been allowed to happen?  I'd rather imagined that when Crowdstrike's people have written a new version of their code, they then try it out on all the computers in their special Trying Stuff Out Room before they release it into the wild.  Are we to suppose that they didn't bother on this occasion?

Given the way that Corporate America works, how comes Crowdstrike's CEO still to have a job?  One has to question whether the company can survive the lawsuits that will surely be heading its way very soon, and the CEO is usually long gone by then.

 

There’s a lot of stuff there,
Microsoft “Azure” (now called Entra) has outtages regularly, the advantage they have is it will generally only be “out” in a restricted area (of the product and location) this is because of the resilience in the infrastructure. If Teams fails on the Medway hub, freggsample, they just redirect the traffic to the Maidstone hub, it’s a bit degraded but no one notices that much.

Crowdstrike, however, is much more localised. Let’s say you were at work, and your school opted for Crowdstrike as their anti virus solution of choice, that impacts right down to the end of your fingers, so the impact when something goes wrong is going to be huge (both to you, and more widespread). What went wrong? They suspect a null pointer exception, basically a line of code that is looking for a variable that doesn’t exist, thus breaking the code. Why hasn’t the CEO lost his job? Well simply because the priority is to work out what went wrong, and that could be anything from someone not deleting a character, or deleting an additional character when moving from test to production, or it coukd be that the file just became corrupted at the point of pushing the file out (I suggested upthread that the issue wasn’t going to be because of the recent solar activity, but that is very much something that could cause that to happen - at the time there was no indication of what had actually happened).
Who knows, maybe he might lose his job for the 15pc drop in stock value if it doesn’t recover, but the priority with businesses this is to find out why it broke and ensure it doesn’t happen again.

[efros]

I think also the release regime may have to be looked at. The sacrificial job terminations will probably come after the post mortem has decided that Cthulhu was to blame.

[jenny]

It does raise the spectre of how dependent we all are on the functioning of the internet though and I wonder whether cyber criminals of some foreign power could efficiently cripple a country by a few such strikes in the right places. The American power grid is pretty vulnerable, for example.

[amanda]

It's already happened here with two separate hackings/data breaches of a phone/internet company Optus and a private health insurance company. I am with the former but have a prepaid phone though had the internet contract then and my data wasn't breached.

While the country wasn't crippled as such, it laid bare how vulnerable our data is.

Just done my passport application online and there's an extra question for is this renewal requested due to a data breach?

[barbados]

Copied from a closed facebook group

Attachments:

[amanda]

Ha! Reminds me of this meme:

[tetsabb]

Jul 21, 2024 12:03:57 GMT barbados said:

Copied from a closed facebook group

Someone had a cartoon of a guy mentioning the Y2k bug, and having to explain to a young colleague.

[Dix]

I brought a couple of floppy disks to work last year when I found out that some of our young 'uns had never seen a physical version of the "save" icon.
:-)

[jenny]

It's so funny that's still used as an icon - rather like the telephone icon, which I doubt many younger people have seen in real life except in very old movies.

[amanda]

During lockdown here, a local teacher was trying to collect a few of these items together to show her class what we had 30 years ago. How they may not have been able to have home schooling because the internet wasn't that good etc, had the pandemic occurred a few decades ago.

[alexanderhoward]

I believe the Vic20 was thrown out decades ago. These days it might be priceless as an antique.

It used BASIC, as did all those computers. I once tried to use VBA on Excel to reproduce a computer game I wrote for the ancient Vic20. It would not accept key-press commands during the macro runtime, but otherwise would have worked. It is funny how all the old techniques came back to me.

[barbados]

Talking of things coming back to you.

I wonder if George Kurtz is having that deja vu feeling?
His previous employer - MacAfee took out a load of Windows XP devices by classing the executable file that pretty much runs everything on Windows.
He was a senior VP, and the update was pushed out by his team, without adequate testing, and unlike the latest outage this one needed IT techs to resolve the issue.

[alexanderhoward]

I was once helping to run a website, and found it was being spammed by Google+ accounts (which shows how long ago it was....) and so I spamblocked 'google+' without realising that this blocked any mention of 'google'; on a site which had extensive Google links. I of course am an ignorant amateur, not a highly paid professional IT wizard.

The spam blocker has calmed down somewhat since the days it used to block academic papers on Roman Luguvalium, the plant Kniphofia (red-hot poker), or the Nympheaum in Kourion.

I hired a reasonable paid IT wizard once to set something up for our company, and he inadvertently left a "." out or added one that knocked out the whole IT system to the outside world. I had to find the error and ask him to fix it. He was so red-faced

I expect that when Kurtz wakes to his work, he will cry "The Horror! The Horror!"

[amanda]

alexanderhoward wrote:

The spam blocker), or the Nympheaum in Kourion.has calmed down somewhat since the days it used to block academic papers on Roman Luguvalium, the plant Kniphofia (red-hot poker)

Likewise with places in the UK ending in SEX - in the early days of the internet, school filters blocked this word, so denying students any access to materials on those places. 

[suze]

This issue used to be known facetiously as "The Scunthorpe Problem".

At this point I shall swerve TGH's mention of the former professional golfist Tony Jacklin.  Mr Jacklin comes from Scunthorpe, and at one time had a reputation as something of a Ladies' Man.  The notion that the town was renamed Shorpe after he moved away is, I am sure, entirely unworthy.

It wasn't only in the early days of the Internet that the issue arose on school software, though.  My school's chat network (still in existence albeit by now little used, and based on Pidgin for anyone whose memory is long enough) implemented a rude word filter within the last decade.  Pupils soon discovered that only two words were actually blocked.  If they abbreviated one of them to fk, well everyone knew what was meant - and the other was sex.  Quite apart from featuring in the names of two of our neighbouring counties, these three letters are also part of the curriculum.  The rude word filter was abandoned within a couple of weeks.