Keybase CEO here. Let me tell a quick story. January 2019. I was loading the car to leave for a short family ski vacation when I got a truly horrifying email: that my slack account had been accessed from a distant land (that I hadn't been to).
There goes my weekend!
When we first started Keybase, we used Slack as other teams did, but were gradually moving all Slack-based workflows over to Keybase. As such, we didn't use it for anything beyond communicating when Keybase was down. But I was very worried. I knew I used a good, random, one-time password for Slack, so it couldn't have been that the password was stolen from somewhere else. Had my computer been rooted? Had my side-hustle password manager been compromised (oneshallpass.com). I immediately contacted Slack security and asked them if this issue was on their side, and they neglected to point me to the relevant blog article from 2015 (which didn't detail the extent of the compromise, we now know). They just said they take security very seriously and hinted I was at fault.
In the subsequent few weeks, I reset all of my passwords, threw away all my computers, bought new computers, factory-reset my phone, rotated all of my Keybase devices, and reestablished everything from the ground up. It cost me a lot of time, money and stress. In the end, I was pretty sure but not 100% convinced that if I had been rooted, that the attackers couldn't follow me to my new setup. But with these things, you can really never know for sure.
I got the email today that my account might have been compromised in the attack. I would say for sure that it was compromised, and I can breathe a big sigh of relief, that was the explanation I wanted to hear all along.
It was great to know throughout this ordeal that the product we're building --- Keybase --- solves this problem in a fundamental way, not with adding further workarounds (2FA while better than just password alone, reminds me a bit of the 3-digit verification code on the back of your credit card; and if Slack's credential database is compromised again, 2FA won't help at all). With Keybase, all of your data is E2E encrypted, and your encryption keys never leave your device. A would-be attacker who compromised our database would have no ability to access any important user data.
Summary: estimated cost to me:
- $5000 worth of hardware
- 60 hours of labor
- 25 hours of lost sleep
- 10 additional hours of team effort
Fortunately:
* Keybase does not communicate sensitive information in slack such as cap tables, financials, employment decisions or compensation discussions, team reviews, company devops secrets, stupid memes that could be taken out of context, or private DM'ing. Basically we just use a `#breaking` channel in Slack, for when we break Keybase.
* Keybase itself is immune from this kind of break-in.
Note that Slack has only sent the email about account access since 2018:
>Additional security features: As of January 2018, we began sending an email every time your account is accessed from a new device; this is a simpler and more immediate way for you to be aware of new logins to your account than periodically reviewing your access logs.
(Quote from the "Slack password reset" email they recently sent out to affected users.)
I think if this happened to me I would just assume the company was hacked due to poor security practices. It seems much more likely than my password being stolen from a device of mine considering that a very significant percentage of companies I have account with seem to have had breaches at some point. But maybe I am just naive.
I think it’s different depending on who you are and what systems you have access to. The Keybase CEO is much more vulnerable to targeted attacks where 0-days would be potentially worth the cost.
Why throw away your computers? Why not remove/disconnect the batteries (if portable) and just store them somewhere in case you eventually no longer suspect a hardware hack, as in this case?
Further: Keybase is a security product and it wasn't deemed worth the risk for the CEO. And while Keybase isn't made of money, the $5k was roughly irrelevant compared to the other costs mentioned here and the _magnitude of the risk_.
If you haven't been through this kind of thing, it's hard to understand how scary it is to have a break-in of unknown origin. If you use strong, unique passwords as Max did, then you're almost certain it's a server break in (and again, this is why Slack is scary for sensitive info)...but being 99% certain isn't enough. Removing that computer permanently from the team gave peace of mind.
I'm surprised how under reported this has been. There were basically three scenarios:
1. They knew about the malicious code in 2015 and chose to misrepresent the breach, effectively lie to customers. (note that they didn’t reset passwords in 2015, take that how you will)
2. They didn’t know about the malicious code until somewhat later, and they chose not to inform anyone.
3. They only discovered it recently and they were muddying the water as much as possible to make it look like it was part of the 2015 breach
It turns out it was somewhere between (1) and (2). They've since revealed the did know about the issue soon after that notification, but chose only to disclose it to small number of users who they believed to be effected.
> We initially believed those credentials to be the result of malware or password re-use between services and took immediate action to protect accounts. However, we later concluded the majority of credentials were from accounts that logged in to Slack during the 2015 incident.
But it turns out it effected a much wider pool of people, and they continue to misrepresent the nature of the breach and it's impact. Their communications are very carefully crafted to downplay the situation or muddy the timelines. Even the title of this piece "New Information...". The information is 4 years old, they're only coming clean now!
Slack had an adversary capturing plain text passwords in 2015 and didn't disclose this to potentially effected users. This is a massive trust issue.
And tried to claim in the emails to those affected that it might be the result of malware on their system, performing exfil from a password manager, because they didn't know the real reason.
I spent a couple of days doing nothing but trying to prove to myself that wasn't the case on my machine, and have low-key (and sometimes really seriously) worried about it ever since.
>The attackers also inserted code that allowed them to capture plaintext passwords as they were entered by users at the time.
>Today we are resetting passwords for all accounts that were active at the time of the 2015 incident
Ok so Slack's security is still dependent on the (unrealistic) hope that their authentication-receiving process isn't manipulable and nothing has changed. I.e. "we are very sorry, we are praying this doesn't happen again"
> inserted code that allowed them to capture plaintext passwords as they were entered by users at the time
Huh? I get if you 'lose' a database backup, but hashing on the server side has always been the wrong way to do things. If you hash it in JS before the request is sent, there is no possible way for you, or any attacker to get the raw user password; only if they literally change production code to send a new request to another endpoint.
> users we confirmed to be affected.
So everyone who logged in during the event?
> resetting passwords for all accounts that were active at the time of the 2015 incident
That's what should have happened the first time. Once you have attacker modified code, you cannot trust what you have on file is accurate or not already taken.
Client-side hashing of passwords does not provide any of the benefits of server-side password hashing, as the client-side hash of the password effectively becomes the password, as it's all that's required to authenticate.
Client-side hashing does provide very limited protection against plaintext reuse (on other sites where the user uses the same password) in the case that it's leaked in transit, which is far less of a concern now that HTTPS is cheap and widespread.
What about double hashing? Once on the client and once more on the server? Then, the server never sees the password, but prevents using a leaked password hash from the DB
The problem with client-side hashing is that the client-hashed password then becomes the password. An attacker doesn't care if the password being sent to the server is "password123" or if it's "e2389cbb675c3ef00879482bd1702f76", because either way, all the attacker needs to do to log in as you is send that "e2389cbb..." string to the server.
I suppose double hashing would have the benefit of preventing your server from ever seeing the plaintext password, but I'm don't think I've ever seen that be a major concern.
The point of hashing it server side is that even identical passwords get different hashes in the database. You can't do that just by hashing the password in the js.
A salt being public is usually fine. AFAIK most modern suggested hashing libraries use a random salt and store the salt in plaintext next to the hashed password anyway. A properly salted-and-hashed password is still practically uncrackable even if the attacker knows the exact salt used.
i've always stored the salt plain text, how else do you store a salt? i need to be able to use it with 0 information about the user other than "they are trying to use this password"
I've seen lots of people under the false impression that salts need to be kept secret, and jump through some hoops to do stuff like encrypt salts before storing them. Well intentioned, but misinformed and error prone, and likely indicates that they're not using standard login/password hash libraries and might be doing other things wrong, too.
Hmmm... I don't think so? I honestly never considered that possibility. My only concern would be accidentally leaking whether a login username/email is valid depending on your strategy for returning salt for invalid usernames. That should be able to be worked around though.
An attacker knowing what hash function you used does not reduce your hash security in any material way. The most secure thing you can do is to use just regular out-of-the-box Argon2 or scrypt hashing. And even if an attacker knows the exact code of the Argon2 function you used to hash, they are no closer to cracking it. In fact, customizing it in any way is almost certain to reduce your security.
Hashing password client-side is a terrible idea (not only is it a terrible idea but it's pretty much useless in terms of security), but it's not because of someone being able to "get" your hash function.
Keybase CEO here. Let me tell a quick story. January 2019. I was loading the car to leave for a short family ski vacation when I got a truly horrifying email: that my slack account had been accessed from a distant land (that I hadn't been to).
There goes my weekend!
When we first started Keybase, we used Slack as other teams did, but were gradually moving all Slack-based workflows over to Keybase. As such, we didn't use it for anything beyond communicating when Keybase was down. But I was very worried. I knew I used a good, random, one-time password for Slack, so it couldn't have been that the password was stolen from somewhere else. Had my computer been rooted? Had my side-hustle password manager been compromised (oneshallpass.com). I immediately contacted Slack security and asked them if this issue was on their side, and they neglected to point me to the relevant blog article from 2015 (which didn't detail the extent of the compromise, we now know). They just said they take security very seriously and hinted I was at fault.
In the subsequent few weeks, I reset all of my passwords, threw away all my computers, bought new computers, factory-reset my phone, rotated all of my Keybase devices, and reestablished everything from the ground up. It cost me a lot of time, money and stress. In the end, I was pretty sure but not 100% convinced that if I had been rooted, that the attackers couldn't follow me to my new setup. But with these things, you can really never know for sure.
I got the email today that my account might have been compromised in the attack. I would say for sure that it was compromised, and I can breathe a big sigh of relief, that was the explanation I wanted to hear all along.
It was great to know throughout this ordeal that the product we're building --- Keybase --- solves this problem in a fundamental way, not with adding further workarounds (2FA while better than just password alone, reminds me a bit of the 3-digit verification code on the back of your credit card; and if Slack's credential database is compromised again, 2FA won't help at all). With Keybase, all of your data is E2E encrypted, and your encryption keys never leave your device. A would-be attacker who compromised our database would have no ability to access any important user data.
Summary: estimated cost to me:
Fortunately: Edits: wordsmithing and improvementsAlso: Import your Slack team to Keybase: https://keybase.io/slack-importer
Note that Slack has only sent the email about account access since 2018:
>Additional security features: As of January 2018, we began sending an email every time your account is accessed from a new device; this is a simpler and more immediate way for you to be aware of new logins to your account than periodically reviewing your access logs.
(Quote from the "Slack password reset" email they recently sent out to affected users.)
I think if this happened to me I would just assume the company was hacked due to poor security practices. It seems much more likely than my password being stolen from a device of mine considering that a very significant percentage of companies I have account with seem to have had breaches at some point. But maybe I am just naive.
That was my 90%+ likelihood explanation too, but I figured Slack had its act together, and the risk of being wrong was too high.
I think it’s different depending on who you are and what systems you have access to. The Keybase CEO is much more vulnerable to targeted attacks where 0-days would be potentially worth the cost.
That's true of course, I'm not a valuable target so it is both less likely I would be targeted and less important if I was successfully hit.
Why throw away your computers? Why not remove/disconnect the batteries (if portable) and just store them somewhere in case you eventually no longer suspect a hardware hack, as in this case?
https://www.welivesecurity.com/2018/09/27/lojax-first-uefi-r...
Further: Keybase is a security product and it wasn't deemed worth the risk for the CEO. And while Keybase isn't made of money, the $5k was roughly irrelevant compared to the other costs mentioned here and the _magnitude of the risk_.
If you haven't been through this kind of thing, it's hard to understand how scary it is to have a break-in of unknown origin. If you use strong, unique passwords as Max did, then you're almost certain it's a server break in (and again, this is why Slack is scary for sensitive info)...but being 99% certain isn't enough. Removing that computer permanently from the team gave peace of mind.
tl;dr: UEFI rootkits can survive operating system reinstallation and even a hard disk replacement.
That's why he needed a new physical computer.
Sorry, but you're just being wasteful. You were looking for an excuse and you know it.
I'm surprised how under reported this has been. There were basically three scenarios:
1. They knew about the malicious code in 2015 and chose to misrepresent the breach, effectively lie to customers. (note that they didn’t reset passwords in 2015, take that how you will)
2. They didn’t know about the malicious code until somewhat later, and they chose not to inform anyone.
3. They only discovered it recently and they were muddying the water as much as possible to make it look like it was part of the 2015 breach
It turns out it was somewhere between (1) and (2). They've since revealed the did know about the issue soon after that notification, but chose only to disclose it to small number of users who they believed to be effected.
https://twitter.com/SlackHQ/status/1152005165802614786
> We initially believed those credentials to be the result of malware or password re-use between services and took immediate action to protect accounts. However, we later concluded the majority of credentials were from accounts that logged in to Slack during the 2015 incident.
But it turns out it effected a much wider pool of people, and they continue to misrepresent the nature of the breach and it's impact. Their communications are very carefully crafted to downplay the situation or muddy the timelines. Even the title of this piece "New Information...". The information is 4 years old, they're only coming clean now!
Slack had an adversary capturing plain text passwords in 2015 and didn't disclose this to potentially effected users. This is a massive trust issue.
The blog post doesn't make it clear: In 2015, did Slack let the users know that the plaintext passwords were captured as well?
No
And tried to claim in the emails to those affected that it might be the result of malware on their system, performing exfil from a password manager, because they didn't know the real reason.
I spent a couple of days doing nothing but trying to prove to myself that wasn't the case on my machine, and have low-key (and sometimes really seriously) worried about it ever since.
>The attackers also inserted code that allowed them to capture plaintext passwords as they were entered by users at the time. >Today we are resetting passwords for all accounts that were active at the time of the 2015 incident
Ok so Slack's security is still dependent on the (unrealistic) hope that their authentication-receiving process isn't manipulable and nothing has changed. I.e. "we are very sorry, we are praying this doesn't happen again"
It would be better if their authentication-receiving process wasn't dependent on being not manipulable through a "Password-authenticated key agreement" PAKE: https://en.wikipedia.org/wiki/Password-authenticated_key_agr...
>irreversibly encrypted, or “hashed,” passwords
Please, don't describe hashes inaccurately as enciphering information. "Scrambled" is marketable enough.
Don't forget that Slack is a php shop. Once a system is compromised, it's easy to edit the code.
Strange, the 2015 report says nothing about inserted code. Did they not know about it until 2019?
They knew about it in 2015.
> inserted code that allowed them to capture plaintext passwords as they were entered by users at the time
Huh? I get if you 'lose' a database backup, but hashing on the server side has always been the wrong way to do things. If you hash it in JS before the request is sent, there is no possible way for you, or any attacker to get the raw user password; only if they literally change production code to send a new request to another endpoint.
> users we confirmed to be affected.
So everyone who logged in during the event?
> resetting passwords for all accounts that were active at the time of the 2015 incident
That's what should have happened the first time. Once you have attacker modified code, you cannot trust what you have on file is accurate or not already taken.
Client-side hashing of passwords does not provide any of the benefits of server-side password hashing, as the client-side hash of the password effectively becomes the password, as it's all that's required to authenticate.
Client-side hashing does provide very limited protection against plaintext reuse (on other sites where the user uses the same password) in the case that it's leaked in transit, which is far less of a concern now that HTTPS is cheap and widespread.
What about double hashing? Once on the client and once more on the server? Then, the server never sees the password, but prevents using a leaked password hash from the DB
The problem with client-side hashing is that the client-hashed password then becomes the password. An attacker doesn't care if the password being sent to the server is "password123" or if it's "e2389cbb675c3ef00879482bd1702f76", because either way, all the attacker needs to do to log in as you is send that "e2389cbb..." string to the server.
I suppose double hashing would have the benefit of preventing your server from ever seeing the plaintext password, but I'm don't think I've ever seen that be a major concern.
I don't think that solves the problem. Instead of logging the plaintext passwords they would have just logged the 1-hashed password.
The point of hashing it server side is that even identical passwords get different hashes in the database. You can't do that just by hashing the password in the js.
Is letting the client know the salt a problem?
A salt being public is usually fine. AFAIK most modern suggested hashing libraries use a random salt and store the salt in plaintext next to the hashed password anyway. A properly salted-and-hashed password is still practically uncrackable even if the attacker knows the exact salt used.
i've always stored the salt plain text, how else do you store a salt? i need to be able to use it with 0 information about the user other than "they are trying to use this password"
I've seen lots of people under the false impression that salts need to be kept secret, and jump through some hoops to do stuff like encrypt salts before storing them. Well intentioned, but misinformed and error prone, and likely indicates that they're not using standard login/password hash libraries and might be doing other things wrong, too.
Hmmm... I don't think so? I honestly never considered that possibility. My only concern would be accidentally leaking whether a login username/email is valid depending on your strategy for returning salt for invalid usernames. That should be able to be worked around though.
Anyone can get your JS hash function and reverse it. This has to be on server side
An attacker knowing what hash function you used does not reduce your hash security in any material way. The most secure thing you can do is to use just regular out-of-the-box Argon2 or scrypt hashing. And even if an attacker knows the exact code of the Argon2 function you used to hash, they are no closer to cracking it. In fact, customizing it in any way is almost certain to reduce your security.
Hashing password client-side is a terrible idea (not only is it a terrible idea but it's pretty much useless in terms of security), but it's not because of someone being able to "get" your hash function.
A hash function isn't feasible to reverse, even knowing the algorithm.
https://en.wikipedia.org/wiki/Cryptographic_hash_function