Three down detectors walk into a bar. The bartender asks them if they're all up. The first says "I don't know". The second says "I don't know". The third says "Yes".
Those are all much smaller. Smaller providers have a much stronger incentive to be reliable, as they will lose customers if they are not. In a corporate settings management will say "this would not have happened if you had gone with AWS". its the current version of "no one ever got fired for buying IBM" (we had MS and others in between).
Hetzner provides a much simpler set of services than AWS. Less complexity to go wrong.
A lot of people want the brand recognition too. Its also become the standard way of doing things and is part of the business culture. I have sometimes been told its unprofessional or looks bad to run things yourself instead of using a managed service.
> Smaller providers have a much stronger incentive to be reliable, as they will lose customers if they are not.
Hard disagree. A smaller provider will think twice about whether they use a Tier 1 data center versus a Tier IV data center because the cost difference is substantial and in many cases prohibitively expensive.
Not to mention the familiarity of the company, its services and expectations. You can hire people with experience with AWS, Azure or GCP, but the more niche you go, the higher the possibility that some people you hire might not know how to work with those systems and their nuances, which is fine they can learn as they work, but that adds to ramp up time and could lead to inadvertent mistakes happening.
This could also be an anti-pattern for hiring - getting people with Amazing Web Service (tm) certification and missing out on candidates with a solid understanding of the foundational principles these services are built on
I agree, though the industry does this all the time by hiring someone with a degree vs someone who built key infrastructure and has no degree, solely because they have a degree. Remember, the creator of brew couldn't get past a Google interview because they asked him to hand craft some algorithm, I probably would have not done well with those either. Does that make him or me worse developers? Doubtful. Does it mean Google missed out on hiring someone who loves his craft? Yes.
The good news is that we're just living in a perfect natural experiment:
Cloudflare just caused a massive internet outage costing millions of dollars worldwide, in part due to a very sloppy mistake that definitely ought to have been prevented (using Rust's “unwrap” in production ). Let's see how many customers they lose because of that and we'll see how big are their incentives. (If you look at the evolution of their share value, it doesn't look like the incident terrified their shareholders at least…)
>I have sometimes been told its unprofessional or looks bad to run things yourself instead of using a managed service.
That's an incredibly bad take lol.
There are times where "The Cloud" makes sense, sure. But in my experience the majority of the time companies over-use the cloud. On Prem is GOOD. It's cheaper, arguably more secure if you configure it right (a challenge, I know, but hear me out) and gives you data sovereignty.
I don't quite think companies realize how bad it would be if EG AWS was hacked.
Any Data you have on the cloud is no longer your data. Not really. It's Amazon, Microsoft, Apple, whoevers.
> I don't quite think companies realize how bad it would be if EG AWS was hacked.
I don't think they'd care. Companies only care about one thing: stock price. Everything rolls up into that. If AWS got hacked and said company was affected by it, it wouldn't be a big deal because they'd be one of many and they'd be lost in the crowd. Any hit to their stock/profits would be minimal and easily forgotten about.
Now, if they were on prem or hosted with Bob's Cloud and got hacked? Different story altogether.
> Companies only care about one thing: stock price.
Its rarely affected in any case. Take a look at the Crowdstrike price chart (or revenue or profits). I think most people (including investors) just take it for granted that systems are unreliable and regard it as something you live with.
I think that's more of a indicator that it hasn't effected their business. They lost nearly 1/5 of their stock price after that incident (obviously not accounting for other factors; I'm not a stock analyst). Investors thought they'd lose customers and reacted in obvious fashion.
But it's since been restored. According to the news, they lost very little customers over the incident. That is why their stock came back. If they continued having problems, I doubt it would have been so rosy. So yes, to your point, a blip here or there happens.
It does mean that you get fewer services, you have to do more sysadmin internally or use other providers for those which a lot of people are very reluctant to do.
When forced to use AWS I only use the extra features I am specifically told to or that are already in use in order to make the system less tied to AWS and easier for me to manage (I am not an AWS specialist so its easier for me to just run stuff like I would on any server or VPS). I particularly dislike RDS (of things I have used). I like Lightsail because its reasonably priced and very like just getting a VPS.
S3 is something of an exception, but it does not tie you down (everyone provides block storage now, and you can use S3 even if everything else is somewhere else) for me if storing lots of large files that are not accessed very much (so egress fees are low).
They've recently introduced bunny.net Shield to add a security layer. I've not made use of it yet so I don't know what the coverage is like or how effective it is: https://bunny.net/shield/
I've done something similar, it's worth noting Scaleway in the same space, for people looking for an AWS replacement more like managed services (equivalents to fargate/lambda/sqs/s3/etc) instead of just bare instance hosting.
+1 for Scaleway. I also use Hetzner for most of my compute. But some stuff just really profits from using managed services. I‘ve used Scaleway‘s Serverless compute offers and managed DBs an been quite happy with them.
well they're not comparable to hetzner anymore, both in terms of features and price. only their dedibox brand could compare, as it's the classic hosting approach vs cloud.
for the hobby crowd it's a shame, for a corporation it's still cheaper than aws with the extra bonus of not having any tie to the us.
I know you were joking, but responding in seriousness - while in general it's worthwhile asking "Quis custodiet ipsos custodes?", in this particular case, I don't see any issue with Down Detector detecting the Down Detector Down Detector. Assuming they are in different availability zones, using different code, with a different deployment cadence, this approach works quite well in practice.
"To serve the Emperor. To protect His domains. To judge and stand guard over His subjects. To carry the Emperor's law to all worlds under His blessed protection. To pursue and punish those who trespassed against His word."
I randomly started vibe coding a website monitoring tool last week knowing full well about the mature competitors in this space and questioning myself along the way. Doesn't seem so crazy now.
Had to check, but that is actually beyond what DNS allows. Labels (the part between dots) are limited to 63 characters. We could sneakily drop an s somewhere in there and then it would fit.
It's a centralization vs decentralisation vs distributed system question.
Since down detectors serve to detect failures of centralized (and decentralized systems) the idea would be to at least get that right: a distributed system to detect outages.
You basically run detectors that heartbeat each others. Just a few suffice.
Once you start to see clusters of detectors go silent, you can assume things are falling apart, which is fine so long as a few remain.
Self healing also helps to make the web of nodes resilient to inevitable infrastructure failures.
I have similar project like this: https://hostbeat.info/
More like t uptime robot and sure, I was really surprised yesterday how many alerts I have got and how many notifications were sent yesterday for this system users. Good work anyway
Thank you for your service! Now, for an even bigger challenge: since it seems the increased demand for the Cloudflare status page brought down Amazon CloudFront for a bit as well, build a new CDN capable of handling that load as well...
But CDNs are made for static content so your comment means I can't run a dynamic website unless I have unlimited file descriptors and flawless connectivity.
"Need" is a strong word. But I think the point is that if you expect wildly spikey traffic/don't want the site to go down if it receives a very sudden influx of requests, going static is a very good answer, much cheaper than "serverless" or over-provisioning.
I feel like the classic East Dakota reply would be that cloud flare CDN does not host your data and merely proxies it (bonus points if he uses the words "mere conduit" in his reply and therefore cloud flare can't be held responsible yada yada).
I wonder though where is it hosted? Digital Ocean? :)
As the Web becomes more and more entangled, I don't know if there is any guarantee of what is really independent. We should make a diagram of this. Hopefully no cyclic dependencies there yet.
Three down detectors walk into a bar. The bartender asks them if they're all up. The first says "I don't know". The second says "I don't know". The third says "Yes".
Crying. I’m stealing this.
As a European solo developer, I’ve switched entirely to European alternatives for all my infrastructure since the beginning of the year.
Cloudflare > Bunny.net
AWS > Hetzner
Business email > Infomaniak
Not a single client site has experienced downtime, and it feels great to finally decouple from U.S. services.
Those are all much smaller. Smaller providers have a much stronger incentive to be reliable, as they will lose customers if they are not. In a corporate settings management will say "this would not have happened if you had gone with AWS". its the current version of "no one ever got fired for buying IBM" (we had MS and others in between).
Hetzner provides a much simpler set of services than AWS. Less complexity to go wrong.
A lot of people want the brand recognition too. Its also become the standard way of doing things and is part of the business culture. I have sometimes been told its unprofessional or looks bad to run things yourself instead of using a managed service.
> Smaller providers have a much stronger incentive to be reliable, as they will lose customers if they are not.
Hard disagree. A smaller provider will think twice about whether they use a Tier 1 data center versus a Tier IV data center because the cost difference is substantial and in many cases prohibitively expensive.
> A lot of people want the brand recognition too.
Not to mention the familiarity of the company, its services and expectations. You can hire people with experience with AWS, Azure or GCP, but the more niche you go, the higher the possibility that some people you hire might not know how to work with those systems and their nuances, which is fine they can learn as they work, but that adds to ramp up time and could lead to inadvertent mistakes happening.
This could also be an anti-pattern for hiring - getting people with Amazing Web Service (tm) certification and missing out on candidates with a solid understanding of the foundational principles these services are built on
I agree, though the industry does this all the time by hiring someone with a degree vs someone who built key infrastructure and has no degree, solely because they have a degree. Remember, the creator of brew couldn't get past a Google interview because they asked him to hand craft some algorithm, I probably would have not done well with those either. Does that make him or me worse developers? Doubtful. Does it mean Google missed out on hiring someone who loves his craft? Yes.
I think cloudflare has billions worth of incentives to be reliable however they can slip up, it happens and that's why centralization is bad.
That is true.
However, I would say that the effect of this outage on customer retention will be (relatively) smaller than it would be for a smaller CDN.
Maybe? Maybe not? It depends on the nature of the outage and how motivated their customers are to switch over to a new service.
The good news is that we're just living in a perfect natural experiment:
Cloudflare just caused a massive internet outage costing millions of dollars worldwide, in part due to a very sloppy mistake that definitely ought to have been prevented (using Rust's “unwrap” in production ). Let's see how many customers they lose because of that and we'll see how big are their incentives. (If you look at the evolution of their share value, it doesn't look like the incident terrified their shareholders at least…)
That experiment already happened last year with Crowdstrike. Nothing detrimental happened. Their revenue actually increased and stock went up
>I have sometimes been told its unprofessional or looks bad to run things yourself instead of using a managed service.
That's an incredibly bad take lol.
There are times where "The Cloud" makes sense, sure. But in my experience the majority of the time companies over-use the cloud. On Prem is GOOD. It's cheaper, arguably more secure if you configure it right (a challenge, I know, but hear me out) and gives you data sovereignty.
I don't quite think companies realize how bad it would be if EG AWS was hacked.
Any Data you have on the cloud is no longer your data. Not really. It's Amazon, Microsoft, Apple, whoevers.
> I don't quite think companies realize how bad it would be if EG AWS was hacked.
I don't think they'd care. Companies only care about one thing: stock price. Everything rolls up into that. If AWS got hacked and said company was affected by it, it wouldn't be a big deal because they'd be one of many and they'd be lost in the crowd. Any hit to their stock/profits would be minimal and easily forgotten about.
Now, if they were on prem or hosted with Bob's Cloud and got hacked? Different story altogether.
> Companies only care about one thing: stock price.
Its rarely affected in any case. Take a look at the Crowdstrike price chart (or revenue or profits). I think most people (including investors) just take it for granted that systems are unreliable and regard it as something you live with.
I think that's more of a indicator that it hasn't effected their business. They lost nearly 1/5 of their stock price after that incident (obviously not accounting for other factors; I'm not a stock analyst). Investors thought they'd lose customers and reacted in obvious fashion.
But it's since been restored. According to the news, they lost very little customers over the incident. That is why their stock came back. If they continued having problems, I doubt it would have been so rosy. So yes, to your point, a blip here or there happens.
> Less complexity to go wrong.
This sounds like a good thing.
It is, in itself.
It does mean that you get fewer services, you have to do more sysadmin internally or use other providers for those which a lot of people are very reluctant to do.
I bet most people don't even need the extra features.
When forced to use AWS I only use the extra features I am specifically told to or that are already in use in order to make the system less tied to AWS and easier for me to manage (I am not an AWS specialist so its easier for me to just run stuff like I would on any server or VPS). I particularly dislike RDS (of things I have used). I like Lightsail because its reasonably priced and very like just getting a VPS.
S3 is something of an exception, but it does not tie you down (everyone provides block storage now, and you can use S3 even if everything else is somewhere else) for me if storing lots of large files that are not accessed very much (so egress fees are low).
Looking forward to the Show HN: I built a web site that uses all of AWS services.
That would be an expensive Show HN.
As an American solo developer, I am close to doing the same. These mega-corps are out of control.
How does Infomaniak compare to Proton? I see they have more office productivity products, but regarding mail and drive?
Big fan of bunny.net as CDN, however Cloudflare is my "smart" filter for all kind of attacks, AI scrapers, malicious traffic, etc.
Am I missing something or is bunny.net not actually a replacement for that?
They've recently introduced bunny.net Shield to add a security layer. I've not made use of it yet so I don't know what the coverage is like or how effective it is: https://bunny.net/shield/
This is very interesting. Thank you for making me aware!
Are you using a US-based transactional email service like Twilio? Curious about EU-based alternatives.
Hello, You can test Sweego - https://www.sweego.io/ We (I'm the CTO) are fully European Bye Pierre-Yves
nice, im looking to ditch SES, one of the last services i have running on AWS
There are self hostable alts to twillio
I've done something similar, it's worth noting Scaleway in the same space, for people looking for an AWS replacement more like managed services (equivalents to fargate/lambda/sqs/s3/etc) instead of just bare instance hosting.
+1 for Scaleway. I also use Hetzner for most of my compute. But some stuff just really profits from using managed services. I‘ve used Scaleway‘s Serverless compute offers and managed DBs an been quite happy with them.
-1 for Scaleway, they were a really good deal years ago but have become expensive af
well they're not comparable to hetzner anymore, both in terms of features and price. only their dedibox brand could compare, as it's the classic hosting approach vs cloud.
for the hobby crowd it's a shame, for a corporation it's still cheaper than aws with the extra bonus of not having any tie to the us.
This is worth its own post.
Same here! I also got a nice peak in my traffic, because so many sites were down.
> Bunny.net
Ah yes, the place for RabbitMQ endpoints.
I had this same idea when I got the "Unblock challenges.cloudflare.com" error while trying to access downdetector, lol!
It looks really nice, good job!
This is GOLD Jerry, Gold.
but who detects the down detector detecting the down detector detecting the down detector
You're on that site right now!
I think the original down detectors do
Mutually assured down-detection.
Time for the META Down Detector - detecting which of the three is down
Can down detector not detect whether down detector detector is down or not?
Maybe distributed down detection?
I know there are people here perfectly capable of running with that idea and we might just see a distributed down detector announced on HN :)
I know you were joking, but responding in seriousness - while in general it's worthwhile asking "Quis custodiet ipsos custodes?", in this particular case, I don't see any issue with Down Detector detecting the Down Detector Down Detector. Assuming they are in different availability zones, using different code, with a different deployment cadence, this approach works quite well in practice.
> Quis custodiet ipsos Custodes?
Arbites.
"To serve the Emperor. To protect His domains. To judge and stand guard over His subjects. To carry the Emperor's law to all worlds under His blessed protection. To pursue and punish those who trespassed against His word."
It's detectors all the way down.
Or "Quis custodiet ipsos custodes?"
See, that's the joke, all of them are on cloudflare/us-west-1 so they all go down together anyway
pervs.
There's always another asking, "Are you down?" It's a bit of a bop.
https://youtu.be/DpMfP6qUSBo
Yeah we had a good laugh when Downdetector was down during the Cloudflare outage yesterday. So this is appropriate. +1
It's stuff like this that makes me still love the Internet.
I randomly started vibe coding a website monitoring tool last week knowing full well about the mature competitors in this space and questioning myself along the way. Doesn't seem so crazy now.
But we need another one to detect whether yours is still up.
It's downdetectorsdown all the way down.
https://downdetectorsdowndetectorsdowndetector.com/
https://downdetectorsdowndetectorsdowndetectorsdowndetector....
It says all systems operational yet Los Angeles, USA is down. :(
It says down now correctly :D
who is going to throw $10 at
https://downdetectorsdowndetectorsdowndetectorsdowndetectors...
Had to check, but that is actually beyond what DNS allows. Labels (the part between dots) are limited to 63 characters. We could sneakily drop an s somewhere in there and then it would fit.
https://datatracker.ietf.org/doc/html/rfc1035
Also I think I triggered a nice error log in domaintools just now. https://whois.domaintools.com/downdetectorsdowndetectorsdown...
Could we monitor all of these with downdetector?
I don't know if I'm the only one, but I keep coming back to check. :-)
4xDowndetector lol
It was worth the laugh, thanks!
It's a centralization vs decentralisation vs distributed system question.
Since down detectors serve to detect failures of centralized (and decentralized systems) the idea would be to at least get that right: a distributed system to detect outages.
You basically run detectors that heartbeat each others. Just a few suffice.
Once you start to see clusters of detectors go silent, you can assume things are falling apart, which is fine so long as a few remain.
Self healing also helps to make the web of nodes resilient to inevitable infrastructure failures.
Given enough of them, some fraction will always be down. It would be helpful if we had a site that could track that ratio.
We could create a linked list of these and just refer to the N’th one as N-down detector.
It's down detectors all the way down
here's a page that monitors that page: https://onlineornot.com/website-down-checker?requestId=jCfaD...
Looks like it's hosted in London?
Downdetection can be thought of as a directed graph, or digraph*.
From there, the "who's watching who?" can become mathematically interesting.
* https://en.wikipedia.org/wiki/Directed_Graph
Things might soon get bad enough that we will start calling them "up detectors."
How does it detect up-ness?
Downdetector was indeed down during the cf outage, but I think the index page was still returning 200 (although I didn't check).
Running a headless browser to take a screenshot to check would probably get you blocked by cf...
I have similar project like this: https://hostbeat.info/ More like t uptime robot and sure, I was really surprised yesterday how many alerts I have got and how many notifications were sent yesterday for this system users. Good work anyway
Sup dawg, I heard you like down detectors.
Thank you for your service! Now, for an even bigger challenge: since it seems the increased demand for the Cloudflare status page brought down Amazon CloudFront for a bit as well, build a new CDN capable of handling that load as well...
Do you need a CDN for a static html, no images? I would guess no, even if you.are being bombarded with requests
I would guess yes, unless you have a server with unlimited file descriptors and flawless connectivity to every other AS...
But CDNs are made for static content so your comment means I can't run a dynamic website unless I have unlimited file descriptors and flawless connectivity.
"Need" is a strong word. But I think the point is that if you expect wildly spikey traffic/don't want the site to go down if it receives a very sudden influx of requests, going static is a very good answer, much cheaper than "serverless" or over-provisioning.
Semi-related: Datadog recently created https://updog.ai
Obligatory: https://youtu.be/ihlN5nf1qew
Duplicate: https://news.ycombinator.com/item?id=45976670
Is it hosted on Cloudflare?
I feel like the classic East Dakota reply would be that cloud flare CDN does not host your data and merely proxies it (bonus points if he uses the words "mere conduit" in his reply and therefore cloud flare can't be held responsible yada yada).
Jokes aside, as far as I can tell, https://downdetectorsdowndetector.com/ is NOT using Cloudflare CDN/Proxy
https://downdetectorsdowndetector.com/ is NOT using Cloudflare SSL
However, selesti reports it uses cloudflare DNS?
https://checkforcloudflare.selesti.com/?q=https://downdetect...
https://downdetectorsdowndetector.com/ is using Cloudflare DNS!
Checked 8 global locations, found DNS entries for Cloudflare in 3
Found in: England, Russia, USA
Not found in: China, Denmark, Germany, Spain, Netherlands
That won’t be an issue though - as we all know, DNS is rarely related to cloud failures
of course, adding to the joke
and i still can't find any feathers
Nice! Who doesn’t like a good recursion? Fingers crossed that the down detector for down detector won’t be down, when down detector might be down
Use the original down detector to monitor the down detector for down detector for down detector. Complete the circle!
Niiice! Thank you for the laugh.
I wonder though where is it hosted? Digital Ocean? :)
As the Web becomes more and more entangled, I don't know if there is any guarantee of what is really independent. We should make a diagram of this. Hopefully no cyclic dependencies there yet.
isisitdowndown.com is still free
Make sure to host it at us-east-1 and proxy by cloudflare for good measure.
If my checks are correct, this site uses Cloudflare for DNS and AWS for hosting.
So if any of the things you want to know is down is down, chances are this site will be too ;)
Slippery slope- just matter of time before someone makes a downdetector for the downdetector for downdetector. Ad nauseum.
What are you, an LLM? You point the first one at the second one and create a loop instead of an infinite "one more" chain
quid custodiet ipso custodes, amirite?
Now if you make one for isup.me, you could call it isisupup.me
Ah, now we know that the answer to "who watches the watchers?" is "@gusowen". :D
But who is going to watch him?!
his cat. at least when he's on toilet
“Well, who’s gonna monitor the monitors of the monitors?”
Would it be a good idea to have a second instance of this watching the first one? /s