System76 Email Leak Update
It started on March 17th, 2022 and was resolved on May 26th, 2023.
What was potentially exposed? User account information and order details including names, physical addresses, email, phone numbers.
Your system76.com credentials and credit card were not compromised at any point. It was not the contents of our entire database, but a random selection of what happened to be in our web servers' memory at the time. It is not possible to determine which users had their information visible over the course of time when this leak was occurring.
Can any of this data still be accessible? This data was embedded directly into the source code of our website, so it will be included in any archives that were taken during this time period. We have already been in contact with the internet archive, and they have removed their data, but there are other potential places where it could have been stored.
What pages on our site contained this data? Due to the nature of the flaw, the data was being included to some extent on every page on our site.
Why was there a delay before resolving this and making an announcement? We wanted to do our best to clean up any residual customer data that was easily accessible before disclosing the nature of this leak. This involved working with the internet archive to delete their copy of the data before public disclosure.
Is there any evidence that any of this data was misused? To date, the only indications of misuse that we have are reports of spam emails being sent to customers.
Why did it take so long to discover? There were a number of factors that play into this, but they do not change the fact that it should not have been happening for over a year.
You would need to look at just the right location in a very long single-line string. This is a common method for minimizing javascript and most developers are used to seeing such long lines on a website. There's usually not much reason to look into them in more detail. It was not something a dev would ever see in their local environment. It was also not noticeable in our staging environment due to the difference in activity there. It is entirely unexpected to see sensitive data exposed in this fashion.
What are the technical details? One of the technologies that we use on our website is server-side rendering with nuxt. Several years ago a caching layer was implemented so that the server-side rendering wouldn't have to ask our backend API every time it needed the same information. It worked fine, but front-end technologies shifted and the way it was implemented became outdated. Last year our team made an update to a seemingly unrelated piece of code, to fix a deprecation message no less. However, this exposed the initial implementation of this cache to a new style of handling state, which it had not been written for and was not prepared to handle. So what happened is the following: state that shouldn't have been shared was shared between sessions. This is one of the major pitfalls of using server-side rendering in a web application. Unlike an API, the code in an SSR application doesn't always have distinct boundaries between what happens on a single client vs. what happens on the server.
What is being done to prevent similar leaks in the future? For the time being we are running an automated scan to alert us if sensitive data is ever found on our website again. We are also continually performing manual inspections to make sure nothing else is in there that shouldn't be, but we are confident that we have identified and fixed this issue. In the long term, we are moving towards a static model (SSG instead of SSR) for our website, which will eliminate the majority of the risk of shared state.
This is a mistake that we will learn from and we offer our sincerest apologies for leaking information that was entrusted to us. We deeply regret the incident and remain committed to safeguarding your information and trust.
Wow props to System76 for coming clean about this despite the fear of a huge public shaming/backlash from a disclosure like this. Although they took a decision of switching from SSR to SSG as a result, they shouldn’t think that really fixes vulnerabilities in the future. I believe SSG is overall the ‘less secure’ of the two, but no tech stack is perfect and its mostly a matter of use case and performance, not security that should be the decisive factor. At least it wasn’t another Wordpress or PHP breach.
Why? With SSG there's no chance of a state change happening without you knowing, so they may be able to do some QC (see automated scans) before release and be confident it won't change afterwards.
I agree. Just blindly moving from SSR to SSG sounds like it could potentially make things even worse. Especially considering that the breach seems to be due to some interplay between their caching layer and SSR.
I received this email today from System76. Edited slightly (lopped off the intro and sign off, mostly) in order to fit into the 4000 char limit.
Appreciated the clear, thorough, and upfront notification.
Ouch, that's a painful mistake! But good on them for coming clean about it, and explaining fully without a lot of corporate doublespeak.
Its always a good sign when my reaction to a disclosure is "ouch that sucks" instead of "buuuuuullsshhhhiii".
I mean, this was so matter of fact to read I've already forgotten about it.
I also received this email. I'm not familiar with nuxt, although I've used SSR and Universal Rendering with React, Remix and Gatsby.
Does anyone know the specifics of how this flaw occurred and how it can be mitigated against?
It sounds like they "rolled their own" caching layer and the bug occurred somewhere between that and SSR. Which means it was unique to their codebase. I hope so and that it isn't a more generic vulnerability with SSR...
Excellent debrief and thanks to Systen76 for showing how it’s done. The SSR thing may be a JS ecosystem but I feel that’s a weird cop out at the end of an otherwise great AAR. We’ve been rendering results on the server for decades with caching. Can someone provide some insight on the difference here as I feel this is a knowledge gap on my part.
Those caching issues seem quite common of late. I remember Steam and a couple other companies encountering a similar problem a couple years back where they'd exposed cached pages containing their customers' info.