Google Cloud outage: A summary
Time to recap…
A power outage at a Google Cloud data centre near London has caused serious failures to services today, affecting operations for several UK banks.
Problems were first detected just after 7am this morning, disrupting “multiple services” in Google’s europe-west2 region (London).
Google blamed the problem on a ‘power issue’ – just afternoon today it said this event had been mitigated. By 1pm UK time services were recovering, and at 3pm Google said services were back online for most customers, with only “a very small number” still affected.
But by that time, millions of customers had been left unable to access and send cash at banks including Lloyds, HSBC and Revolut.
Revolut said:
A spokesperson for Lloyds added:
Legal proceedings were disrupted, because the video system used by defendants, lawyers and other parties to attend court hearings remotely across England and Wales which failed to work throughout the morning.
The City regulator, the Financial Conduct Authority, confirmed it was in touch with Google and seeking more information about the outage.
This outage will be the first major test for the FCA and Bank of England, since the City watchdogs were handed powers to regulate Google and other rival services that have been supplying “critical” cloud services to UK financial firms this summer.
Tech experts said the outage was another reminder of the dangers of relying on a small number of technology providers for cloud services, a year after problems at Amazon Cloud caused more widespread disruption.
Reynolds: Chinese EV decision will depend on UK interests
Business secretary Johnny Reynolds has insisted any decision to impose tariffs on Chinese EVs will be about British, not EU interests.
He told the business select committee that some were under the misapprehension that the EU wanted the UK to put tariffs on EVs because they feared the Chinese could use Britain as a “backdoor” to the bloc, circumventing trade defence measures including tariffs on EVs and quota on electric hybrid cars.
He point out that this Chinese cars exported from the UK “would not qualify” for duty free exports to the EU under the rules of origin clauses in the Brexit trade and cooperation agreement”.
He was also asked about last week’s Franco-German paper proposing that the UK would be included in the upcoming Made in Europe legislation on condition London aligned with EU trade defence measures.
But he said the “detail” of that proposal, which has yet to be adopted by the EU in any new drafting of the Industrial Accelerator Act, “has not been surfaced to the degree to which we’d be able to definitively say that that is what the ask is of the UK.”
Google Cloud outage: A summary
Time to recap…
A power outage at a Google Cloud data centre near London has caused serious failures to services today, affecting operations for several UK banks.
Problems were first detected just after 7am this morning, disrupting “multiple services” in Google’s europe-west2 region (London).
Google blamed the problem on a ‘power issue’ – just afternoon today it said this event had been mitigated. By 1pm UK time services were recovering, and at 3pm Google said services were back online for most customers, with only “a very small number” still affected.
But by that time, millions of customers had been left unable to access and send cash at banks including Lloyds, HSBC and Revolut.
Revolut said:
A spokesperson for Lloyds added:
Legal proceedings were disrupted, because the video system used by defendants, lawyers and other parties to attend court hearings remotely across England and Wales which failed to work throughout the morning.
The City regulator, the Financial Conduct Authority, confirmed it was in touch with Google and seeking more information about the outage.
This outage will be the first major test for the FCA and Bank of England, since the City watchdogs were handed powers to regulate Google and other rival services that have been supplying “critical” cloud services to UK financial firms this summer.
Tech experts said the outage was another reminder of the dangers of relying on a small number of technology providers for cloud services, a year after problems at Amazon Cloud caused more widespread disruption.
Google: We'll publish an analysis of what went wrong
Google add that it does not plan to provide further updates about the outage on its Service Health page – another sign that it thinks it has pretty much resolved the problem with its cloud services today.
It adds:
Google: issue has been mitigated for most customers
Newsflash: Google has declared that it has won, pretty much, its battle against today’s tech outage at its London-based cloud computing systems.
In an updated status message, Google says:
The shift to the past tense is significant – until now, the messages said that services “are” experiencing issues.
Google then declares that all but a few customers have now been helped, saying:
Royal Mail has said it was aware of the issues caused to its services by the Google Cloud outage, and added that there was no effect on letter and parcel deliveries.
Not that those deliveries are always good enough!
Businesss secretary Johnny Reynolds has said he has summoned Royal Mail bosses in for talks to discuss changes to improve the services, my colleague Lisa O’Carroll reports.
He told a parliamentary business committee that the Royal Mail is not delivering “the level of service people expect”.
Royal Mail have not met their first or second class standards for almost 10 years, the committee heard.
Reynolds said the government had negotiated the take over by Czech billionaire Daniel Křetínský last year which included a curb on dividends “until performance improved”.
“I know for parliamentarians this in oral questions on Thursday and other correspondence with members of Parliament that it is one of the most significant issues they’re facing,” Reynolds told the committee.
He said the government “recognised the need for regulatory reform” but Ofcom was taking a lead at the moment on this subject.
Lloyds: expect services to get back to normal soon
Lloyds Banking Group, which includes Halifax, is hopeful that services will be restored soon.
A spokesperson for Lloyds tells us:
UK regulators want answers from Google about Cloud outage
The Financial Conduct Authority and Bank of England are pushing Google for details of today’s cloud services outage, as they face their first major test as new regulators of “critical” tech suppliers to UK banks.
Both the FCA and Bank of England were handed powers in July to start directly overseeing the local arms of Google Cloud and three other Silicon Valley rivals – Amazon Web Services, Oracle and Microsoft – after being designated as critical third parties by the UK government, as their services become a crucial part of increasingly digital banking services.
A spokesperson for the FCA said they were in touch with Google and seeking more information about the outage. The Bank of England’s Prudential Regulation Authority is also understood to be speaking with the tech firm about the service outage which began this morning (details start here).
The watchdogs are in charge of ensuring that the cloud providers are resilient and actively reducing the risk of cyber-attacks and major outages like the one experienced today, which has disrupted services for millions of people and businesses across the UK.
For its part, Google is meant to prove it is carrying out proper stress testing, including how it would respond to imagined emergency scenarios that would put their operations under severe strain. They’re also meant to have reported any major incidents, including this power outages, to the two regulators (though it’s unclear in what time frame).
Low-paid workers and hospitality firms face disruption
The failure of banking apps due to the Google Cloud outage could disproportionately affect those most struggling with the cost of living, argues audit, tax and business advisory firm Blick Rothenberg.
Blick Rothenberg partner Andrew Sanford explains:
Affected companies coming back online again
There are signs that the companies affected by Google’s outage are getting online again.
Emma, the personal finance, budgeting, and money management app which was taken down by the Google Cloud outage (see 10.11am), has posted on X that it is operating again.
Emma says:
Google: We are seeing the recovery of services for most workloads
Newsflash: Google says service are being restored in its London-based cloud region after today’s outage.
The company says:
This isn’t the end of the story, though – Google are promising another update by 3pm UK time.
Google added that they will provide an update by 06:00 PDT (2pm BST) with “current details” of the situation.
Google: The power event is mitigated
Newsflash: Google says the ‘power event’ which affected its London-based cloud computing operating and caused widepread disruption to services in the UK has been ‘mitigated’.
In a new statement on its Service Health page, Google reports seeing signs that services are being restored.
It says:
[CORRECTED: 6am PDT is 2pm UK time.]
The Google Cloud outage could result in some companies losing sales, and they might struggle to get much compensation, suggests Ryan Gracey, partner at law firm Gordons.
Google: power event disrupted our Cloud services
Google are blaming a ‘power event’ for the disruption at its London-based Cloud computing operations today.
In an update on its status page, Google says:
PA: Court hearings delayed as nationwide video system goes down
The video system used by courts across England and Wales went down on Tuesday in an issue believed to be linked to the wider Google outage, the Press Association report.
The Cloud Video Platform (CVP), which is used by defendants, lawyers and other parties to attend court hearings remotely, was not functioning throughout the morning.
Users expecting to attend court on video link were met with a message saying there was an internal server error.
The issue is expected to have left many hearings across the country delayed or unable to go ahead as the system is often used for defendants who are in custody to attend court.
The judge in the Russell Brand trial, Mr Justice Bennathan, decided to press on with proceedings at Southwark Crown Court on Tuesday morning despite the link being broken.
Some journalists who would have been watching online were squeezed into extra seats in court, but many were not able to follow the proceedings as the fourth complainant started her evidence.
Staff at the court said there was a national outage and did not know when it would be resolved.
It is understood to be linked to an outage affecting Google’s Cloud network, which affected businesses including online bank Revolut, delivery service DPD and Royal Mail.
The nature of cloud ecosystems mean that a problem in one region can ripple out and cause widesread disruption.
Dr Amro Al-Said Ahmad, lecturer in computer science at Keele University, explains:
A Lloyds spokesperson has confirmed the bank (which includes Halifax) has experienced problems today:
Google’s latest update on its Cloud outage reiterates what we knew already.
It says:
