by ckozlowski 9 years ago

(Disclaimer: I work for AWS.)

The dashboard is not changing color due to the S3 issue. We're updating the banner in place of that.

Edit: Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

http://status.aws.amazon.com/

Ph4nt0m 9 years ago

For some reason, reading about "believe we understand root cause" made me think of: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."

perlgeek 9 years ago

Maybe you could encourage your colleagues to host the status page outside of AWS?

  • johansch 9 years ago

    Maybe with GCP? :)

  • oxguy3 9 years ago

    We'll have to wait for the postmortem, but I bet it was an unintentional dependency on S3 that no one realized had come into place until S3 went down -- especially considering how fast they were able to remove the dependency and fix it.

    • ckozlowski 9 years ago

      S3 gets used to store a lot of static content. Can't speak for that team, but I'm sure they'll take that feedback. Happy the banner functionality remained unimpeded.

    • idlewords 9 years ago

      It took them ~30 minutes.

      • LunaSea 9 years ago

        It took them 2 hours actually.

    • koolba 9 years ago

      This reminds me of a GitHub outage from them having a build dependency on GitHub. IIRC, they tried to roll back to building a prior version but since the site was offline, the build failed.

    • all_usernames 9 years ago

      Possibly AWS status page wisely relied on a third party, which relied on a fourth party, which relied on S3.

  • apapli 9 years ago

    I'm happy to offer up some spare space on my godaddy hosted Linux plan if that helps...

danappelxx 9 years ago

Could you go more in depth? What does S3 have anything to do with it?

  • unfunco 9 years ago

    It's where they store the error icon.

    • danappelxx 9 years ago

      So the "working" icon works, but the "not working" doesn't? I'm not sure that's right.

  • johansch 9 years ago

    I think the most reasonable guess is that they have some backend system that continously pushes some status json/xml file to an S3 bucket.

    Then there's the frontend, that apparently periodically reads this file from S3 and caches the results.

    I guess the comment they added on the top after two hours of being in the dark was likely manually added to the web frontend.

    Obviously all of this would be hilariously badly designed if it was made this way. Still...