I heard that it was a "mandatory dependency" to mitigate "insider risk" or something. There's no way it's going anywhere. Odds are they'll just enforce even slower rollouts "to catch things early".
> Workers KV is in the process of being transitioned to significantly more resilient infrastructure for its central store: regrettably, we had a gap in coverage which was exposed during this incident.
Where did you see this information? Was it on a social media channel? I do see the IAM services in the list of affected services in the incident report.
The comment was self explanatory, and no, it wasn't a widespread GCP outage. Most everything was up except for GCS and firebase, and later on identity stuff started causing cascading issues but not when this was posted.
Incident affecting API Gateway, Agent Assist, AlloyDB for PostgreSQL, Apigee, Apigee Edge Private Cloud, Apigee Edge Public Cloud, Apigee Hybrid, Cloud Data Fusion, Cloud Firestore, Cloud Logging, Cloud Memorystore, Cloud Monitoring, Cloud Run, Cloud Security Command Center, Cloud Shell, Cloud Spanner, Cloud Workstations, Contact Center AI Platform, Contact Center Insights, Data Catalog, Database Migration Service, Dataform, Dataplex, Dataproc Metastore, Datastream, Dialogflow CX, Dialogflow ES, Google App Engine, Google BigQuery, Google Cloud Bigtable, Google Cloud Composer, Google Cloud Console, Google Cloud DNS, Google Cloud Dataflow, Google Cloud Dataproc, Google Cloud Pub/Sub, Google Cloud SQL, Google Cloud Storage, Google Compute Engine, Identity Platform, Identity and Access Management, Looker Studio, Managed Service for Apache Kafka, Memorystore for Memcached, Memorystore for Redis, Memorystore for Redis Cluster, Persistent Disk, Personalized Service Health, Pub/Sub Lite, Speech-to-Text, Text-to-Speech, Vertex AI Search
Our entire infra in GCP stayed up just fine, we just couldn't manage anything. IDK what to tell you. Many of the things you list here were not down at all.
That it wasn’t down for you does not mean it wasn’t down for others or even almost everyone. Certainly, Google wouldn’t have listed the services as having an outage if nobody was impacted. You can’t extrapolate from “works for me” to “it must have been working for everyone”.
I don't understand your argument? Wasn't GCP's own status page calling them outages? Some of our upstream providers (who use GCP) were definitely affected and down.
As a former SRE there, is "widespread outage" a specific, special kind of classification that's not obvious to the public just by looking at the status page...? Or what do you mean?
I gotta say, it's kinda nice when that happens... work just kinda pauses for everyone, from providers to customers. It kinda feels like a national holiday, and everyone downstream from the affected cloud can just kinda sit back and relax cuz there's nothing they can do anyway except wait.
When it's your own outage, it's all-hands-on-deck panic mode. When it's half the internet down, it's no longer your problem, lol
I guess it depends on what your company's acceptable level of downtime is. If you're like Cloudflare (who handled this well), you take this as a sign to build fault tolerance around your 3rd party providers.
If your application is mission-critical, downtime is anything but a holiday.
Yep, KV is broken too. Any worker that depends on KV is throwing exceptions. I was able to get into the dash, but it's very slow. Error rates started to go up significantly around 18:00 UTC.
It's not much of a reach to go from "discussion about impact on human-verification dialogs" to 'discussion about human-verification dialog policy". This isn't an incident-management channel, it's a discussion forum - tangents are fine!
I complained in the apnews.com thread, because the apnews.com verification, which is annoying by itself, did not work at all this time. That is hardly unrelated.
I really do appreciate the transparency and ownership that comes with these. We all fuck up, but a lot of companies would rather hide their mistakes than own up to them. Cloudflare's approach makes me trust them more.
Unrelated, they have a few services that rely on GCP which is down. Still, I imagine the people working on the maintenance for Tokyo turned white during that job worried it was caused by them...
So both Cloudflare authentication as well as Google's identity systems suffered major dowtime yesterday. Are there technical dependecies between these?
Cloudflare doesn't say this directly but in their blog they've written
> The cause of this outage was due to a failure in the underlying storage infrastructure used by our Workers KV service, which is a critical dependency for many Cloudflare products and relied upon for configuration, authentication and asset delivery across the affected services. Part of this infrastructure is backed by a third-party cloud provider, which experienced an outage today and directly impacted availability of our KV service.
proxy seems available in general, must just be local to workers because only one of my sites going thru ZT tunnel with identity access rules is affected
> Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency.
So they depend on GCP for (some of) their services
wrote a similar comment - good to know for the future.
If that is true, and there is no other BGP shenanigans, then I suspect this dependency will not be around for long
My WAG is it comprises 95% of the company infrastructure
I heard that it was a "mandatory dependency" to mitigate "insider risk" or something. There's no way it's going anywhere. Odds are they'll just enforce even slower rollouts "to catch things early".
WAG = wild-assed guess?
Maybe beastman82 is wagging his tail?
ceo just said not for long
From the article:
> Workers KV is in the process of being transitioned to significantly more resilient infrastructure for its central store: regrettably, we had a gap in coverage which was exposed during this incident.
Sub-processor pages are an easy way to verify that sort of thing.
https://www.cloudflare.com/gdpr/subprocessors/cloudflare-ser...
> So they depend on GCP for (some of) their services
Google denies they had any outages.
https://x.com/Google/status/1933246051512644069
https://nitter.net/Google/status/1933246051512644069
They can say that, but any one of their customers knows it's not true.
Is this the ol' "100% down for 3% of your customers" scenario?
Weaseling out of SLA/SLO payments.
Come on, linking four hour old tweets instead of their actual service dashboard where they clearly state there was an outage.
https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
https://downdetector.com/ is showing outages at many major companies including Google, CloudFlare, AWS and more.
Word on the street is that there are large BGP routing issues behind all of this.
Would make sense. I think the last time I saw this sort of thing it was BGP causing a bunch of traffic to route through Iran or china IIRC.
I vaguely recall that incident. But it did not feel like it affected this many services.
At the same time I have not noticed anything being down firsthand. I am in Europe.
Here's the case [1]. Looks like they targeted a single /24 so that's likely why it wasn't a bigger issue.
[1] https://bishopfox.com/blog/bgp-hijacking-technical-post-mort...
so this is related to Israel's escalation that everyone is expecting?
The Pentagon Pizza Report has been having a lot of activity the past 24 hours. Maybe just a coincidence
There was also an older instance with China https://www.cyberdefensemagazine.com/experts-detailed-how-ch...
Anthropic down/degraded as well. Time to go for a walk.
Internet Health Report is reporting "No data to show".
[1] https://www.ihr.live/
GCP is also down https://news.ycombinator.com/item?id=44260810
Odd coincidence. Wonder if Cloudflare uses GCP?
It's likely their auth infra based on what the Google outage is
What do you mean by this? The Google outage is a widespread outage of most GCP services.
Google is claiming the root cause is with some of their central IAM services, which would have a cascading effect to the rest of their services.
Where did you see this information? Was it on a social media channel? I do see the IAM services in the list of affected services in the incident report.
check https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
> Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue
Scroll up. Its literally in this HN comment section highly upvoted.
It was not when I posted the reply that you are replying to 2 hours after I posted it.
The comment was self explanatory, and no, it wasn't a widespread GCP outage. Most everything was up except for GCS and firebase, and later on identity stuff started causing cascading issues but not when this was posted.
> it wasn't a widespread GCP outage.
If this wasn't widespread, what is?
Incident affecting API Gateway, Agent Assist, AlloyDB for PostgreSQL, Apigee, Apigee Edge Private Cloud, Apigee Edge Public Cloud, Apigee Hybrid, Cloud Data Fusion, Cloud Firestore, Cloud Logging, Cloud Memorystore, Cloud Monitoring, Cloud Run, Cloud Security Command Center, Cloud Shell, Cloud Spanner, Cloud Workstations, Contact Center AI Platform, Contact Center Insights, Data Catalog, Database Migration Service, Dataform, Dataplex, Dataproc Metastore, Datastream, Dialogflow CX, Dialogflow ES, Google App Engine, Google BigQuery, Google Cloud Bigtable, Google Cloud Composer, Google Cloud Console, Google Cloud DNS, Google Cloud Dataflow, Google Cloud Dataproc, Google Cloud Pub/Sub, Google Cloud SQL, Google Cloud Storage, Google Compute Engine, Identity Platform, Identity and Access Management, Looker Studio, Managed Service for Apache Kafka, Memorystore for Memcached, Memorystore for Redis, Memorystore for Redis Cluster, Persistent Disk, Personalized Service Health, Pub/Sub Lite, Speech-to-Text, Text-to-Speech, Vertex AI Search
Our entire infra in GCP stayed up just fine, we just couldn't manage anything. IDK what to tell you. Many of the things you list here were not down at all.
That it wasn’t down for you does not mean it wasn’t down for others or even almost everyone. Certainly, Google wouldn’t have listed the services as having an outage if nobody was impacted. You can’t extrapolate from “works for me” to “it must have been working for everyone”.
Dude, I literally was an SRE there. I'm well aware of how this stuff works.
If some of those things listed had actual widespread outages, it would have been much much worse.
I don't understand your argument? Wasn't GCP's own status page calling them outages? Some of our upstream providers (who use GCP) were definitely affected and down.
As a former SRE there, is "widespread outage" a specific, special kind of classification that's not obvious to the public just by looking at the status page...? Or what do you mean?
So weird to argue when google themselves listed these as having an outage.
Their KV store was definitely down.
When being down scales. :D
Yeah this is going to be a problem. I haven't seen an issue this widespread across so many services in a while.
Seems to be semi regular now that everyone puts all their eggs in only a few baskets.
I gotta say, it's kinda nice when that happens... work just kinda pauses for everyone, from providers to customers. It kinda feels like a national holiday, and everyone downstream from the affected cloud can just kinda sit back and relax cuz there's nothing they can do anyway except wait.
When it's your own outage, it's all-hands-on-deck panic mode. When it's half the internet down, it's no longer your problem, lol
I guess it depends on what your company's acceptable level of downtime is. If you're like Cloudflare (who handled this well), you take this as a sign to build fault tolerance around your 3rd party providers.
If your application is mission-critical, downtime is anything but a holiday.
Let me guess, someone pushed out a bad BGP config?
For an outage this large and widespread that would have to be the main culprit.
Big blog post about how they saved the internet upcoming. ;)
Currently down, but reference: https://blog.cloudflare.com/the-ddos-that-almost-broke-the-i...
Seems to be affecting functionality of their "Verify you are human" dialogs as well as Workers.
Yep, KV is broken too. Any worker that depends on KV is throwing exceptions. I was able to get into the dash, but it's very slow. Error rates started to go up significantly around 18:00 UTC.
Edit: The CF status page has acknowledged it's a broad outage across many services: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp
After many tries I also got into the dashboard, but it's not that usable, constant error pop-ups.
It does. Another question is why do we get these dialogues always from Cloudflare and never from Akamai in the first place?
Downvoting this comment and flagging the submission does not address the serious issue. These verification dialogues make the Internet unusable.
Nor does venting about it in unrelated threads, or asserting your opinion as fact.
It's not much of a reach to go from "discussion about impact on human-verification dialogs" to 'discussion about human-verification dialog policy". This isn't an incident-management channel, it's a discussion forum - tangents are fine!
I complained in the apnews.com thread, because the apnews.com verification, which is annoying by itself, did not work at all this time. That is hardly unrelated.
They've changed the title to "Broad Cloudflare service outages"
Is it coincidence that there's a Scheduled Maintenance in Tokyo for 18:00 UTC in progress, and the problems started at 18:19 UTC?
Guess we'll find out from the postmortem. Always the silver lining with these, get to learn from and enjoy a good writeup.
Do these get posted publicly?
> Do these get posted publicly?
Yes.
Yeah. Cloudflare writes some of the best ones in the industry, and they're very enjoyable to read: https://blog.cloudflare.com/tag/post-mortem/
I really do appreciate the transparency and ownership that comes with these. We all fuck up, but a lot of companies would rather hide their mistakes than own up to them. Cloudflare's approach makes me trust them more.
Probably
There is always scheduled maintenance on that page, so that's not much of a signal in my experience
Unrelated, they have a few services that rely on GCP which is down. Still, I imagine the people working on the maintenance for Tokyo turned white during that job worried it was caused by them...
Cloudflare's lava lamps are dimming.
Can’t wait to read this post-mortem. Seems odd that a Google Cloud outage would bring down Cloudflare services.
They updated the incident noting that it's not just authentication affected.
Our Workers apps are up again
edit:
It works in the US but EU customers are still reporting our services as down.
edit:
EU customers are reporting ok
Workers KV has been down for like +30mins. This is impacting us seriously.
Their API is down too.
Amazing that something can impact their whole infrastructure like this given how much redundance they have.
The same is true for Google.
>can impact their whole infrastructure
CDN and WAF seem to be working fine. I think CF rushed a lot of newer services out without the reliability some of their older/core services enjoy
From their incident page (https://www.cloudflarestatus.com/incidents/25r9t0vz99rp):
> Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency.
I bet that 3rd party service is GCP.
I would be pretty pissed if I were a CF customer that used Workers KV for redundancy because it was heavily marketed as running on CF data centers.
So both Cloudflare authentication as well as Google's identity systems suffered major dowtime yesterday. Are there technical dependecies between these?
Cloudflare doesn't say this directly but in their blog they've written
> The cause of this outage was due to a failure in the underlying storage infrastructure used by our Workers KV service, which is a critical dependency for many Cloudflare products and relied upon for configuration, authentication and asset delivery across the affected services. Part of this infrastructure is backed by a third-party cloud provider, which experienced an outage today and directly impacted availability of our KV service.
proxy seems available in general, must just be local to workers because only one of my sites going thru ZT tunnel with identity access rules is affected
"All locations except us-central1 have fully recovered. us-central1 is mostly recovered. We do not have an ETA for full recovery in us-central1."
solar flare?
No, Cloudflare.
Hopefully they also publish the prompt that did this.
i was thinking about this too
They should make the AI lead the postmortem.
They're just moving fast and breaking things 100x faster. Who cares what code does just vibe it all away /s
lmao