Thanks for the heads-up, the goal was mostly avoiding that typing the author's name in Google brings up this post. I'll have it blurred for the sake of consistency, though.
The author worked in Belarus for Wargaming.net until just before making this commit. Wargaming recently withdrew their operations from Belarus and Russia for obvious reasons, and the author appears to have lost his job with them as a result. Combined with the way he nonchalantly reversed the commit and I’m thinking the theory on r/netsec may not be so far fetched.
Could not find any public use of the package but there's a few interesting things about this repo:
- The author used to work at wargaming.net at the time the repo was created (publisher of world of tanks)
- There are two contributors: the author and one of his ex-colleague from wargaming.net.
- There are a bunch of maintenance commits over the years suggesting actual use and not just a random weekend project.
A bit far-fetched but whoever introduced this backdoor could be attempting a targeted attack against wargaming.net as there's a good chance it's used in there.
Note: it looks like the author of the package removed the malicious commit.
According to https://pypistats.org/packages/fastapi-toolkit, this package had 158 downloads in total in the past month. This would include automated tools (e.g. this GuardDog mentioned in TFA) grabbing every single package version published.
But of course they have to hype it up with "50k stars", "used by Microsoft, Uber, and Netflix" blah blah, otherwise it's a complete non-story.
Hello there! I'm one of the authors of the post. Sorry you feel we "hyped it up", that was definitely not the intent. The malicious package is targeting FastAPI applications. The point is that there are a lot of applications the attacker could attempt to target (through social engineering, malicious pull requests etc.)
Will adapt the wording to make it clearer. Thanks for the feedback!
> The point is that there are a lot of applications the attacker could attempt to target(through social engineering, malicious pull request, etc.)
If you want to discuss the potential damage an attacker can do with a GitHub account, why not hype it up even more unrealistically and talk about how they could have attacked any public repo on GitHub that accepts PRs. The article should either be limited to what actually happened or you should follow the thought through to its logical conclusion. Why do you stop when you've sufficiently scared people enough to start talking about datadog tooling?
More FUD from attention seeking, for-profit organizations. Even a tiny bit of digging shows this is virtually a non-issue. Look at the gh repo, the pypi stats
Likely because they don't really care about the root causes, they are not spending time reviewing a barely used package in order to learn something new or for the betterment of the world as a whole, the article is trying to sell their tooling for detecting malicious packages. Understanding the root cause wouldn't help with that.
One of the authors of the post here. We prefer sticking to the facts rather than speculating the account was compromised without having a solid proof. Someone on /r/netsec also had an interesting theory that this might be an intentional backdoor[1].
For what it's worth, the project referred to in the post is free, open-source, and unrelated to the commercial offering.
You seem to have updated the wording to "has been backdoored by a malicious actor". Isn't that more speculation, with the tentativeness removed? What facts incline you to believe it was a malicious actor, and not the maintainer?
Yes, that's true. And I agree that there is some malicious actor; a bag of Base64-encoded code doesn't get inserted as an innocent accident. But the way you've expressed yourself, more than once, suggests you have reason to believe the malicious actor is other than the maintainer.
Do you have any evidence, one way or the other?
Let's not chop logic. I don't think you've been completely frank about this. The commit was signed by the maintainer, right, using a private key? That means the maintainer "done it", absent evidence to the contrary. And apparently the maintainer is silent.
Apologies; I am not a git user. I thought every git commit was signed, excuse my ignorance. I'm surprised something in PyPi can be shipped without any signatures, but I'm not shocked; I'm not a fan of language-specific repositories. NPM is an example of what I'm not a fan of.
> It is possible the original developer of the package had their account compromised and used by a malicious actor.
> whose maintainer's account was likely compromised by a malicious actor
Seems to still be speculating about the cause without diving deeper into the topic, or is there some cache invalidation of the article that is missing perhaps?
Yes, that would be caching. We kept the first sentence, as it's still possible his account was compromised (we have no strong evidence to prove it, but no strong evidence to refute it either).
This is definitely pretty strange. Account takeovers happen, but just reverting the commit and closing the issue after one gets discovered is not the best way to handle these.
This is the reality of our modern software development process though. Your threat model now must include the GitHub account of every maintainer of every open source project you use.
I’m the guy that opened that issue. To be clear, I’m NOT affiliated with datadog. I am co-founder of a software supply chain company https://phylum.io.
We scan all the open source packages as they’re published, and got a hit for this pretty much right away. The volume of packages that get published that are malware is astonishing…
Kind of unfortunate that these guys just closed the issue, if they aren’t malicious actors. I suspect that this is a fake account, and not an account compromise, though.
In order to be a bit more constructive, what is the ideal process for the author to remove it?
The issue in general of backdoored packages is not new, but that it happened to you can be a new issue if you haven't either thought of it before or not simply encountered before. It would be very helpful if there was a resource out there answering the question "So your package was backdoored, what do you do now?" that people could refer to and get help.
Some thoughts:
1. The blast radius appears to be very minimal, the affected github package has 0 stars, 2 contributers, 1 watcher and 4 issues total.
2. The issue was caught and resolved quickly (within a day?).
3. I haven't seen any explanation by the developer on whether there account was compromised?
Fully agree, worth re-iterating as the title doesn't make it clear.
The vulnerability is not in FastAPI itself, but in a relatively unknown utility package that you probably aren't using.
Still good to raise awareness but a slight bit of scaremongering
Hello! One of the authors of the post here. Just added a sentence in the introduction to make it crystal clear:
> While FastAPI itself is not impacted, this is an interesting occurrence of an attacker attempting to deploy a FastAPI-specific backdoor.
Appreciate your feedback!
FYI: You left a name and email in the github history.
I know they are public via GH, but it feels weird to redact every piece of PII including avatar, then leaving a name and email in there.
Thanks for the heads-up, the goal was mostly avoiding that typing the author's name in Google brings up this post. I'll have it blurred for the sake of consistency, though.
> Still good to raise awareness but a slight bit of scaremongering
It's not scaremongering as much as it is a (thinly veiled) ad for Datadog's own tech:
> using our latest open source tool, GuardDog, which uses heuristics to identify malicious or compromised PyPI packages.
> We recently released GuardDog, a free and open-source tool to identify malicious PyPI packages
> Datadog ASM Vulnerability Monitoring, announced earlier this year at Dash, allows you to identify vulnerable and malicious packages
> Datadog Cloud Workload Security has a number of out-of-the-box rules to detect post exploitation scenarios
The author worked in Belarus for Wargaming.net until just before making this commit. Wargaming recently withdrew their operations from Belarus and Russia for obvious reasons, and the author appears to have lost his job with them as a result. Combined with the way he nonchalantly reversed the commit and I’m thinking the theory on r/netsec may not be so far fetched.
For context: https://old.reddit.com/r/netsec/comments/z30465/investigatin...
Quote:
----------------
Could not find any public use of the package but there's a few interesting things about this repo:
- The author used to work at wargaming.net at the time the repo was created (publisher of world of tanks)
- There are two contributors: the author and one of his ex-colleague from wargaming.net.
- There are a bunch of maintenance commits over the years suggesting actual use and not just a random weekend project.
A bit far-fetched but whoever introduced this backdoor could be attempting a targeted attack against wargaming.net as there's a good chance it's used in there.
Note: it looks like the author of the package removed the malicious commit.
According to https://pypistats.org/packages/fastapi-toolkit, this package had 158 downloads in total in the past month. This would include automated tools (e.g. this GuardDog mentioned in TFA) grabbing every single package version published.
But of course they have to hype it up with "50k stars", "used by Microsoft, Uber, and Netflix" blah blah, otherwise it's a complete non-story.
Hello there! I'm one of the authors of the post. Sorry you feel we "hyped it up", that was definitely not the intent. The malicious package is targeting FastAPI applications. The point is that there are a lot of applications the attacker could attempt to target (through social engineering, malicious pull requests etc.)
Will adapt the wording to make it clearer. Thanks for the feedback!
> The point is that there are a lot of applications the attacker could attempt to target(through social engineering, malicious pull request, etc.)
If you want to discuss the potential damage an attacker can do with a GitHub account, why not hype it up even more unrealistically and talk about how they could have attacked any public repo on GitHub that accepts PRs. The article should either be limited to what actually happened or you should follow the thought through to its logical conclusion. Why do you stop when you've sufficiently scared people enough to start talking about datadog tooling?
More FUD from attention seeking, for-profit organizations. Even a tiny bit of digging shows this is virtually a non-issue. Look at the gh repo, the pypi stats
> a package whose maintainer's account was likely compromised by a malicious actor
They don't say why they think it was an account compromise, rather than a malicious maintainer.
Likely because they don't really care about the root causes, they are not spending time reviewing a barely used package in order to learn something new or for the betterment of the world as a whole, the article is trying to sell their tooling for detecting malicious packages. Understanding the root cause wouldn't help with that.
One of the authors of the post here. We prefer sticking to the facts rather than speculating the account was compromised without having a solid proof. Someone on /r/netsec also had an interesting theory that this might be an intentional backdoor[1].
For what it's worth, the project referred to in the post is free, open-source, and unrelated to the commercial offering.
[1]: https://www.reddit.com/r/netsec/comments/z30465/comment/ixlj...
And yet you go ahead and speculate that the account "was likely compromised", without saying what facts inclined you to that opinion.
We just updated the wording. Thanks for the feedback.
You seem to have updated the wording to "has been backdoored by a malicious actor". Isn't that more speculation, with the tentativeness removed? What facts incline you to believe it was a malicious actor, and not the maintainer?
If the maintainer themselves added the backdoor, can't they be considered a malicious actor?
Yes, that's true. And I agree that there is some malicious actor; a bag of Base64-encoded code doesn't get inserted as an innocent accident. But the way you've expressed yourself, more than once, suggests you have reason to believe the malicious actor is other than the maintainer.
Do you have any evidence, one way or the other?
Let's not chop logic. I don't think you've been completely frank about this. The commit was signed by the maintainer, right, using a private key? That means the maintainer "done it", absent evidence to the contrary. And apparently the maintainer is silent.
The malicious commit (2cd2223dcd90fa9d9c72851427602aa0e179e061) was not signed. Sorry you feel like the writing isn't frank.
Apologies; I am not a git user. I thought every git commit was signed, excuse my ignorance. I'm surprised something in PyPi can be shipped without any signatures, but I'm not shocked; I'm not a fan of language-specific repositories. NPM is an example of what I'm not a fan of.
> It is possible the original developer of the package had their account compromised and used by a malicious actor.
> whose maintainer's account was likely compromised by a malicious actor
Seems to still be speculating about the cause without diving deeper into the topic, or is there some cache invalidation of the article that is missing perhaps?
Yes, that would be caching. We kept the first sentence, as it's still possible his account was compromised (we have no strong evidence to prove it, but no strong evidence to refute it either).
> we have no strong evidence to prove it, but no strong evidence to refute it either
How is that different than "speculation"? That sounds like textbook definition of "speculation".
"Speculation - the activity of guessing possible answers to a question without having enough information to be certain"
Again some esoteric package that likely nobody uses. If you’re worried about such attacks, private registry mirrors can go a long way.
The issue from the researchers appears to be here: https://github.com/timaakulich/fastapi_toolkit/issues/4
This is definitely pretty strange. Account takeovers happen, but just reverting the commit and closing the issue after one gets discovered is not the best way to handle these.
This is the reality of our modern software development process though. Your threat model now must include the GitHub account of every maintainer of every open source project you use.
I’m the guy that opened that issue. To be clear, I’m NOT affiliated with datadog. I am co-founder of a software supply chain company https://phylum.io.
We scan all the open source packages as they’re published, and got a hit for this pretty much right away. The volume of packages that get published that are malware is astonishing…
Kind of unfortunate that these guys just closed the issue, if they aren’t malicious actors. I suspect that this is a fake account, and not an account compromise, though.
In order to be a bit more constructive, what is the ideal process for the author to remove it?
The issue in general of backdoored packages is not new, but that it happened to you can be a new issue if you haven't either thought of it before or not simply encountered before. It would be very helpful if there was a resource out there answering the question "So your package was backdoored, what do you do now?" that people could refer to and get help.
Some kind of post-mortem or statement at all about how the GitHub account got compromised, if that's what happened here.
It could have also been a researcher checking to see if anyone would notice, or something worse.
I wouldn’t bet on an account takeover on this case.
> Your threat model now must include the GitHub account of every maintainer of every open source project you use.
But GitHub is afaik the only site on the Internet that actually does account management correctly, so it least there is that.