During the 2013-2018 boom when tech workers as a class had maximum leverage, used some of the energy organizing broad tech unions and guilds instead of negotiating for fuck-you money and pulling up the ladder by telling other trades to “learn 2 code noobs”
craft broadly populist multi-union legislation to raise taxes for “infra and natsec basic income fund”, for a living stipend to infra workers: railroad, steelworkers, electricians, OSS maintainers
My current take on the #xz situation, not having read the actual source backdoor commits yet (thanks a lot #Github for hiding the evidence at this point...) besides reading what others have written about it (cf. https://boehs.org/node/everything-i-know-about-the-xz-backdoor for a good timeline):
This is going to be an excellent teaching example for advanced supply chain attacks that I will definitely be using in the future - after much more in-depth analysis.
It seems to have been a long game, executed with an impressive sequence of steps and preparation, including e.g. disabling OSSFuzz checks for the particular code path and pressuring the original maintainer into accepting the (malicious) contributions.
Given the luck involved in this case, we need to assume a number of other, currently unknown supply chain backdoors that were successfully deployed with comparable sophistication and are probably active in the field.
Safe(r) languages like #rustlang for such central library dependencies would maybe (really big maybe) have made it a bit harder to push a backdoor like this because - if and only if the safety features are used idiomatically in an open source project - reasonably looking code is (a bit?) more limited in the sneaky behavior it could include. We should still very much use those languages over C/C++ for infrastructure code because the much larger class of unintentional bugs is significantly mitigated, but I believe (without data to back it up) that even such "bugdoor" type changes will be harder to execute. However, given the sophistication in this case, it may not have helped at all. The attacker(s) have shown to be clever enough.
Sandboxing library code may have helped - as the attacker(s) explicitly disabled e.g. landlock, that might already have had some impact. We should create better tooling to make it much easier to link to infrastructure libraries in a sandboxed way (although that will have performance implications in many cases).
Automatic reproducible builds verification would have mitigated this particular vector of backdoor distribution, and the Debian team seems to be using the reproducibility advances of the last decade to verify/rebuild the build servers. We should build library and infrastructure code in a fully reproducible manner and automatically verify it, e.g. with added transparency logs for both source and binary artefacts. In general, it does however not prevent this kind of supply chain attack that directly targets source code at the "leaf" projects in Git commits.
Verifying the real-life identity of contributors to open source projects is hard and a difficult trade-off. Something similar to the #Debian#OpenPGP#web-of-trust would potentially have mitigated this style of attack somewhat, but with a different trade-off. We might have to think much harder about trust in individual accounts, and for some projects requiring a link to a real-world country-issued ID document may be the right balance (for others it wouldn't work). That is neither an easy nor a quick path, though. Also note that sophisticated nation state attackers will probably not have a problem procuring "good" fake IDs. It might still raise the bar, though.
What happened here seems clearly criminal - at least under my IANAL naive understanding of EU criminal law. There was clear intent to cause harm, and that makes the specific method less important. The legal system should also be able to help in mitigating supply chain attacks; not in preventing them, but in making them more costly if attackers can be tracked down (this is difficult in itself, see point 8) and face risk of punishment after the fact.
Regarding the #xz backdoor, I've seen statements like "if you're not running a publicly exposed sshd, you're safe". This is not the case and reflects a pretty outdated security mindset. You're still vulnerable, because you shouldn't assume internal connections are inherently trustworthy.
Yes, it limits exposure, but that's not the same - you still have a high-severit incident on your hands. Anyway, just here stating the obvious, as usual. ✌️
TL;DR: in 5.6.1 there's some code added that looks for specific signatures in files in tests/files, and if found, it grabs some data from these files, deciphers them, and executes them. NO FILES WITH THESE SIGNATURES EXIST YET, so it's like a way to extend the backdooring scripts in the future by just adding new binary test files. Guess things weren't supposed to end here.
The abusive behavior that was being used to manipulate Lasse Collin into bringing on more maintainers for #xz went unnoticed because abusive behavior in Open Source communities is so pervasive. In context, we can clearly see it was part of an orchestrated operation. Out of context, it looks like just another asshole complaining about stuff they have no right to complain about. https://robmensching.com/blog/posts/2024/03/30/a-microcosm-of-the-interactions-in-open-source-projects/
This is someone who knows exactly what they are doing and has been around enough in enough spaces that they know the lingo intimately.
This needs to cause serious reflection among maintainers and users of OSS tools and libraries.
How many of the utilities that we use daily are maintained by one or two burned out people who would be highly susceptible to this kind of social engineering? Who would look at this language and think "this person can help"?
It's important to note how critical it was caught now: all the commercial distributions are making releases over the next 12-18 months: Red Hat with RHEL 10 in May 2025, SUSE with SLE 16 in fall 2025, and Canonical with Ubuntu 24.04 in April. It was key to infect their upstreams (Fedora, openSUSE, Debian) now.
And if you are curious about the #xz#compromise, a little update on the #Debian site:
As already written, the archive processing is currently off (nothing new coming to testing/unstable/experimental, no mirror updates pushed out).
Automated build daemons for the affected architectures have been stopped, and only two of them regenerated with a clean #stable environment. They are building for the security archive only, nothing else, right now. That part is safe.
Members of the Release, FTP, Security, Build-Daemon and Sysadmin team are discussing what the next steps are. There are multiple different ways that can be taken, with different drawbacks and amounts of work involved.
Also, it is not yet fully known what the malicious code all could do, so there might be much more that needs to be done later - or not. Unknown as of now, needs the analysis of it to finish, which is not easy nor fast.
Again the FOSS world has proven to be vigilant and proactive in finding bugs and backdoors, IMHO. The level of transparency is stellar, especially compared to proprietary software companies. What the FOSS world has accomplished in 24 hours after detection of the backdoor code in #xz deserves a moment of humbleness. Instead we have flamewars and armchair experts shouting that we must change everything NOW. Which would introduce even more risks. Progress is made iteratively. Learn , adapt, repeat.
How to quickly check if your #linux system may be affected by the recent XZ utils backdoor.
Update: (thx @scy) I've been advised not to run "xz --version" because the full extend of this backdoor is still being researched. Instead use your package manager to check the version, i.e. for apt that would be:
apt list liblzma5
very bad: versions 5.6.0 or 5.6.1
5.4.6. or earlier - probably ok, no one knows for sure right now, keep an eye out for updates
To everyone losing their shit over the xz/liblzma debacle: This is how Open Source is supposed to work: many eyes looking over work-in-progress to make sure it works as intended. Sometimes it’s reviewing source code commits and other times it’s looking over the behavior of pre-release software, noticing anomalous behavior and chasing down the commit that caused it. This is preciselywhy we have debian-testing and FreeBSD-Current. If anything this is validation that Open Source works #xz#liblzma
Does everyone understand how much luck was involved in this exploit in #xz being discovered so quickly? And, what it tells us about the attacker?
This was a subtle and sophisticated attack implemented over years. The attacker was made a co-maintainer two years ago, and they made numerous innocuous-looking and seemingly unrelated changes over that time, sometimes through a second account, that eventually added up to a backdoor. Along with many innocent commits, too. #Linux
TL;DR #XZ has been backdoored in 5.6.0 and 5.6.1. While Fedora Rawhide and Fedora 41 packages are affected, Red Hat Enterprise Linux is NOT affected. Updates (well, technically downgrades to 5.4.x) for Fedora are being made available through the regular update channels. Our Security Alert explains more details. https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users
Check if your machine is affected: run xz --version and see if it returns 5.6.0 or 5.6.1. If it shows a lower version, you are safe, as far as we can see now.
An incredibly technically complex #backdoor in xz (potentially also in libarchive and elsewhere) was just discovered. This backdoor has been quietly implemented over years, with the assistance of a wide array of subtly interconnected accounts: