> Please note that your account administrator will need to manually set up each individual user on your account. Once your users have been set up and all appropriate DNS records are configured, their email access will be reactivated, and they will start receiving emails and can send emails. Please note, that DNS changes take approximately 30 minutes to provision and in rare cases can take up to 24 hours
My sympathies to anyone involved in this tire fire, and on a Friday no less :-(
One thing I’ve never enjoyed about systems administration: trying to recover from catastrophic failure with money on the line and users breathing down your neck. I don’t envy anyone involved with this.
The trick is to design things such as "roll back to the last backup" is always good enough. Easier said then done, and there are often other priorities, but it's usually doable.
Vendors also rarely have to tools to the aid in recovering from catastrophic failure. We had an incident a couple of years ago where someone tested a policy in production that wiped out all email older than six months. We have tens of thousands of accounts. Microsoft had no answer for how to recover from something like that.
Rackspace just updated the status page[0], saying in part:
"After further analysis, we have determined that this is a security incident. ... We are working through the environment with our security teams and partners to determine the full scope and impact. ... We currently do not have an ETA for resolution. We are actively working with our support teams and anticipate our work may take several days. ... [W]e are encouraging admins to configure and set up their users accounts on Microsoft 365 so they can begin sending and receiving mail immediately."
I have a number of mailboxes on the services that are down. The big issue is that they aren't really communicating what is actually wrong so it is hard to tell if we should set email up on a different service. Also if we switch to O365 that they are offering as a work around, it isn't clear how that would be merged back with the existing mailboxes. Or maybe the suggestion of O365 indicates that they don't think the other mailboxes are coming back.
The fact that they aren't communicating makes me wonder if they got hit with ransomware of some type.
Ransomware (or at least a breach) is exceedingly likely.
> #Rackspace’s managed Exchange documentation points to Exchange clusters which are all offline. When viewed on Shodan, a few days ago they have the Exchange build number from August 2022 - ie before #ProxyNotShell patches.
With the mind-numbing level of incompetence that they are displaying here, would you actually consider availing yourself of their services? As the saying goes, fool me once, shame on you, fool me twice,…
They were already ms customers. One spla license for hosted exchange has a higher wholesale cost paid to Microsoft than the exchange online e1 they are now supplying.
The hosted exchange model has been a bad business idea for a while, before you even consider events like this.
This exact same thing happened at my Uni. Reason for it was the exchange server was compromised by ransomware. Solution was to abandon it and fast track O365 migration for all users.
Huh. I use Rackspace for email, not Hosted Exchange, just email. Haven't had a burp. Been using it for well over a decade with no problem.
I didn't know that the CEO got replaced a couple of months ago. Given the way they're handling this particular problem I may just go looking for someone else, just in case.
Got bought by private equity Apollo in 2016, usual squeeze story, ceo replaced by someone pure finance 2 months ago. Dying cow being milked by its owner.
I looked up their numbers and was surprised. DigitalOcean got $533M in revenue over the past year while Rackspace did $3.1B. It looks like Rackspace's revenue is 5.8x that of DigitalOcean. Cloudflare's revenue is $894M. Even Akamai is only $3.6B in revenue.
AWS is much larger (in the $70-80B per year range, it was $71B for 2021), but it looks like Rackspace is still pulling in a sizable amount of revenue compared to a lot of places.
I remember being on Slicehost in the pre-Rackspace days. When Slicehost got sold and became (or merged with?) Rackspace I jumped ship. In my next two professional roles in the 2010s I was brought on to improve process on the engineering team. Both of those companies were Rackspace customers (both cloud and servers). I found Rackspace terrible to deal with and their offerings weren’t very competitive or stable. First big thing I did was move as many sites away from Rackspace as possible. Maybe there are compelling reasons to be on Rackspace today (and I know this is Exchange hosting and not server hosting), but IMO there are so many better companies to deal with.
Microsoft is getting targeted so all of their flaws are being compromised. I had to not provide hosted exchange email services because Microsoft wasn't fixing the security holes in exchange. Moving to 365 lol would be moving from one flaw to another. Fyi everyone, finding an email provider now days without any issues is very scarce. Look into all this and you will see all said is no bs. Unfortunately Rackspaces company is dependent on MS Exchange
If after witnessing a Rackspace meltdown someone is still a glutton for guaranteed pain and wants to go down with the Hosted Exchange ship, Intermedia has been around doing it for a long time.
Otherwise, SMEs should find a good value-add licensing partner and move to O365. Use a proper migration service to make it easy.
Wowzers, this whole thing is just filled with "oh, fuck you then:"
The bitly link shorter in their tweet redirects to: https://status.apps.rackspace.com/index/viewincidents?group=...
And therein is this:
> Please note that your account administrator will need to manually set up each individual user on your account. Once your users have been set up and all appropriate DNS records are configured, their email access will be reactivated, and they will start receiving emails and can send emails. Please note, that DNS changes take approximately 30 minutes to provision and in rare cases can take up to 24 hours
My sympathies to anyone involved in this tire fire, and on a Friday no less :-(
One thing I’ve never enjoyed about systems administration: trying to recover from catastrophic failure with money on the line and users breathing down your neck. I don’t envy anyone involved with this.
Agreed. Not a job I would ever take. If "roll back to the last backup" isn't good enough, I want no part of it.
The trick is to design things such as "roll back to the last backup" is always good enough. Easier said then done, and there are often other priorities, but it's usually doable.
> The trick is to design things such as "roll back to the last backup" is always good enough.
...or to just make sure the option even exists. That's a great start by itself.
Vendors also rarely have to tools to the aid in recovering from catastrophic failure. We had an incident a couple of years ago where someone tested a policy in production that wiped out all email older than six months. We have tens of thousands of accounts. Microsoft had no answer for how to recover from something like that.
Rackspace just updated the status page[0], saying in part:
"After further analysis, we have determined that this is a security incident. ... We are working through the environment with our security teams and partners to determine the full scope and impact. ... We currently do not have an ETA for resolution. We are actively working with our support teams and anticipate our work may take several days. ... [W]e are encouraging admins to configure and set up their users accounts on Microsoft 365 so they can begin sending and receiving mail immediately."
[0] https://status.apps.rackspace.com/index/viewincidents?group=...
I have a number of mailboxes on the services that are down. The big issue is that they aren't really communicating what is actually wrong so it is hard to tell if we should set email up on a different service. Also if we switch to O365 that they are offering as a work around, it isn't clear how that would be merged back with the existing mailboxes. Or maybe the suggestion of O365 indicates that they don't think the other mailboxes are coming back.
The fact that they aren't communicating makes me wonder if they got hit with ransomware of some type.
Ransomware (or at least a breach) is exceedingly likely.
> #Rackspace’s managed Exchange documentation points to Exchange clusters which are all offline. When viewed on Shodan, a few days ago they have the Exchange build number from August 2022 - ie before #ProxyNotShell patches.
(Quote via https://web.archive.org/web/20221203050944/https://cyberplac...)
With the mind-numbing level of incompetence that they are displaying here, would you actually consider availing yourself of their services? As the saying goes, fool me once, shame on you, fool me twice,…
Sounds like somebody's about to lose some money.
https://simplywall.st/stocks/us/software/nasdaq-rxt/rackspac...
Pour out some coffee for the engineers at Rackspace & their impacted customers who are losing their Friday evening to this.
The whole weekend. F
And the next one, this requires extensive manual actions per mailbox. Microsoft is about to get gifted a ton of new customers.
They were already ms customers. One spla license for hosted exchange has a higher wholesale cost paid to Microsoft than the exchange online e1 they are now supplying.
The hosted exchange model has been a bad business idea for a while, before you even consider events like this.
Did they mention the cause of the outage?
This exact same thing happened at my Uni. Reason for it was the exchange server was compromised by ransomware. Solution was to abandon it and fast track O365 migration for all users.
Ransomware was actually my first guess as well.
Huh. I use Rackspace for email, not Hosted Exchange, just email. Haven't had a burp. Been using it for well over a decade with no problem.
I didn't know that the CEO got replaced a couple of months ago. Given the way they're handling this particular problem I may just go looking for someone else, just in case.
The last ceo wasn’t a good leader nor a good engineer. Source: other c levels that worked directly with him.
Rackspace has gone from one of the best in the industry to one of the worst.
They completed that transition several years ago. :(
Former Racker here. Once we got bought by Private Equity, we went downhill very fast.
The first small startup I worked for used Rackspace and I really liked their service, but I thought they died off over a decade ago.
Got bought by private equity Apollo in 2016, usual squeeze story, ceo replaced by someone pure finance 2 months ago. Dying cow being milked by its owner.
Sources:
https://www.crn.com/news/cloud/rackspace-again-a-public-comp...
https://www.marketwatch.com/story/apollo-backed-rackspace-pr...
Under the prior CEO they were also considering selling off parts of their business: https://www.computerweekly.com/news/252518110/Rackspace-mull...
I looked up their numbers and was surprised. DigitalOcean got $533M in revenue over the past year while Rackspace did $3.1B. It looks like Rackspace's revenue is 5.8x that of DigitalOcean. Cloudflare's revenue is $894M. Even Akamai is only $3.6B in revenue.
AWS is much larger (in the $70-80B per year range, it was $71B for 2021), but it looks like Rackspace is still pulling in a sizable amount of revenue compared to a lot of places.
But then again, revenue isn't everything.
They missed the cloud train 12 years ago or so.
they have been buying cloud services companies like Onica for a while though
I remember being on Slicehost in the pre-Rackspace days. When Slicehost got sold and became (or merged with?) Rackspace I jumped ship. In my next two professional roles in the 2010s I was brought on to improve process on the engineering team. Both of those companies were Rackspace customers (both cloud and servers). I found Rackspace terrible to deal with and their offerings weren’t very competitive or stable. First big thing I did was move as many sites away from Rackspace as possible. Maybe there are compelling reasons to be on Rackspace today (and I know this is Exchange hosting and not server hosting), but IMO there are so many better companies to deal with.
Microsoft is getting targeted so all of their flaws are being compromised. I had to not provide hosted exchange email services because Microsoft wasn't fixing the security holes in exchange. Moving to 365 lol would be moving from one flaw to another. Fyi everyone, finding an email provider now days without any issues is very scarce. Look into all this and you will see all said is no bs. Unfortunately Rackspaces company is dependent on MS Exchange
I have a bout 30 exchange accounts and still waiting for what to do.
These are my options at the moment: https://www.jucra.com/whmcs/announcements/108/
For the former DataPipe employee’s at RackSpace: “KetanKetan” .
RIP Rackspace
What’s a good alternative to switch to if someone wants to move off of Rackspace?
If after witnessing a Rackspace meltdown someone is still a glutton for guaranteed pain and wants to go down with the Hosted Exchange ship, Intermedia has been around doing it for a long time.
Otherwise, SMEs should find a good value-add licensing partner and move to O365. Use a proper migration service to make it easy.
If you're open to non O365 things, then potentially something like OX Cloud:
https://www.open-xchange.com/products/ox-app-suite/
I haven't used them though, just became aware of them a few years ago when they picked up PowerDNS (an OSS project I used to help out a little bit).
Funnily enough there's a Rackspace logo in the 'TRUSTED BY' section: https://www.open-xchange.com/products/ox-cloud