Very annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it's "great new features". We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.
He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.
GitHub has suspended @JiaT75's account.
EDIT: Lasse Collin's account @Larhzu has also been suspended.
EDIT: Github has disabled all Tukaani repositories, including downloads from the releases page.
--
EDIT: Just did a bit of poking. xz-embedded was touched by Jia as well and it appears to be used in the linux kernel. I did quick look and it doesn't appear Jia touched anything of interest in there. I also checked the previous mirror at the tukaani project website, and nothing was out of place other than lagging a few commits behind:
https://gist.github.com/Qix-/f1a1b9a933e8847f56103bc14783ab7...
--
Here's a mailing list message from them ca. 2022.
https://listor.tp-sv.se/pipermail/tp-sv_listor.tp-sv.se/2022...
--
MinGW w64 on AUR was last published by Jia on Feb 29: https://aur.archlinux.org/cgit/aur.git/log/?h=mingw-w64-xz (found by searching their public key: 22D465F2B4C173803B20C6DE59FCF207FEA7F445)
--
pacman-static on AUR still lists their public key as a contributor, xz was last updated to 5.4.5 on 17-11-2023: https://aur.archlinux.org/cgit/aur.git/?h=pacman-static
EDIT: I've emailed the maintainer to have the key removed.
--
Alpine was patched as of 6 hours ago.
https://git.alpinelinux.org/aports/commit/?id=982d2c6bcbbb57...
--
OpenSUSE is still listing Jia's public key: https://sources.suse.com/SUSE:SLE-15-SP6:GA/xz/576e550c49a36... (cross-ref with https://web.archive.org/web/20240329235153/https://tukaani.o...)
EDIT: Spoke with some folks in the package channel on libera, seems to be a non-issue. It is not used as attestation nor an ACL.
--
Arch appears to still list Jia as an approved publisher, if I'm understanding this page correctly.
https://gitlab.archlinux.org/archlinux/packaging/packages/xz...
EDIT: Just sent an email to the last committer to bring it to their attention.
EDIT: It's been removed.
--
jiatan's Libera info indicates they registered on Dec 12 13:43:12 2022 with no timezone information.
-NickServ- Information on jiatan (account jiatan):
-NickServ- Registered : Dec 12 13:43:12 2022 +0000 (1y 15w 3d ago)
-NickServ- Last seen : (less than two weeks ago)
-NickServ- User seen : (less than two weeks ago)
-NickServ- Flags : HideMail, Private
-NickServ- jiatan has enabled nick protection
-NickServ- *** End of Info ***
/whowas expired not too long ago, unfortunately. If anyone has it I'd love to know.They are not registered on freenode.
EDIT: Libera has stated they have not received any requests for information from any agencies as of yet (30th Saturday March 2024 00:39:31 UTC).
EDIT: Jia Tan was using a VPN to connect; that's all I'll be sharing here.
I think this has been in the making for almost a year. The whole ifunc infrastructure was added in June 2023 by Hans Jansen and Jia Tan. The initial patch is "authored by" Lasse Collin in the git metadata, but the code actually came from Hans Jansen: https://github.com/tukaani-project/xz/commit/ee44863ae88e377...
> Thanks to Hans Jansen for the original patch.
https://github.com/tukaani-project/xz/pull/53
There were a ton of patches by these two subsequently because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizers. Subsequently the configure script was modified multiple times to detect the use of sanitizers and abort the build unless either the sanitizer was disabled or the use of ifuncs was disabled. That would've masked the payload in many testing and debugging environments.
The hansjans162 Github account was created in 2023 and the only thing it did was add this code to liblzma. The same name later applied to do a NMU at Debian for the vulnerable version. Another "<name><number>" account (which only appears here, once) then pops up and asks for the vulnerable version to be imported: https://www.mail-archive.com/search?l=debian-bugs-dist@lists...
Yesterday sure was fun wasn't it :p Thanks for all your help/working with me on getting this cleaned up in Fedora.
because of it's "great new features"
"great" for whom? I've seen enough of the industry to immediately feel suspicious when someone uses that sort of phrasing in an attempt to persuade me. It's no different from claiming a "better experience" or similar.
Interesting that one of the commits commented on update of the test file that it was for better reproducibility for having been generated by a fixed random seed (although how goes unmentioned). For the future, random test data better be generated as part of the build, rather than being committed as opaque blobs...
Debian have reverted xz-utils (in unstable) to 5.4.5 – actual version string is “5.6.1+really5.4.5-1”. So presumably that version's safe; we shall see…
After reading the original post by Andres Freund, https://www.openwall.com/lists/oss-security/2024/03/29/4, his analysis indicates that the RSA_public_decrypt function is being redirected to the malware code. Since RSA_public_decrypt is only used in the context of RSA public key - private key authentication, can we reasonably conclude that the backdoor does not affect username-password authentication?
I’m surprised there isn’t way more of this stuff. The supply chain is so huge and therefore represents so much surface area.
Github accounts of both xz maintainers have been suspended.
Do you know if it was actually the commit author, of if their commit access was compromised?
Nice. I worked on a Linux disto when I was a wee lad and all we did was compute a new md5 and ship it.
Name and shame this author. They should never be allowed anywhere near any open projects ever again.
Can legal action be taken against the author if it's found he maliciously added the backdoor?
It is not good to take into consideration something with any unreadable text instead of the open text of the programme. It should be excluded.
I wonder who the target was!
his account is active again on github https://github.com/JiaT75
Sleeper.
[dead]
[flagged]
[flagged]
[flagged]
the account was either sold or stolen
Fascinating. Just yesterday the author added a `SECURITY.md` file to the `xz-java` project.
> If you discover a security vulnerability in this project please report it privately. *Do not disclose it as a public issue.* This gives us time to work with you to fix the issue before public exposure, reducing the chance that the exploit will be used before a patch is released.
Reading that in a different light, it says give me time to adjust my exploits and capitalize on any targets. Makes me wonder what other vulns might exist in the author's other projects.
Security Researchers: Is this request-for-private-disclosure + "90-days before public" reasonable?
It's a SEVERE issue, to my mind, and 90 days seems too long to me.
90 day dark window for maintainers is SOP though. Then after 90 days, it’s free game for public disclosure
How many of people like this one exist?
Honestly it seems like a state-based actor hoping to get whatever high value target compromised before it's made public. Reporting privately buys them more time, and allows them to let handlers know when the jig is up.
Looks like one of the backdoor authors even went and disabled the feature the exploit relied on directly on oss-fuzz to prevent accidental discovery: https://social.treehouse.systems/@Aissen/112180302735030319 https://github.com/google/oss-fuzz/pull/10667
But luckily there was some serendipity: "I accidentally found a security issue while benchmarking postgres changes." https://mastodon.social/@AndresFreundTec/112180083704606941
This is getting addressed here: https://github.com/google/oss-fuzz/issues/11760
This in of itself can be legitimate. ifunc has real uses and it indeed does not work when sanitizer is enabled. Similar change in llvm: https://github.com/llvm/llvm-project/commit/1ef3de6b09f6b21a...
and that was in mid 2023. Very funny that Wikipedia on this issue says
> It is unknown whether this backdoor was intentionally placed by a maintainer or whether a maintainer was compromised
Yeah, if you've been compromised for a year your attacker is now your identity. Can't just wave hands, practice infosec hygiene
I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).
Would be interesting to see what's going on here; the person who did the releases has done previous releases too (are they affected?) And has commits going back to 2022 – relatively recent, but not that recent. Many are real commits with real changes, and they have commits on some related projects like libarchive. Seems like a lot of effort just to insert a backdoor.
Edit: anyone with access can add files to existing releases and it won't show that someone else added it (I just tested). However, the timestamp of the file will be to when you uploaded it, not that of the release. On xz all the timestamps of the files match with the timestamp of the release (usually the .tar.gz is a few minutes earlier, which makes sense). So looks like they were done by the same person who did the release. I suspected someone else might have added/altered the files briefly after the release before anyone noticed, but that doesn't seem to be the case.
> I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).
Yeah, I've been banging on that same drum for ages too... for example on this very site a decade ago: https://news.ycombinator.com/item?id=7213563
I'm honestly surprised that this autoconf vector hasn't happened more often... or more often that we know of.
Every single commit this person ever did should immediately be rolled back in all projects.
> they have commits on some related projects like libarchive
Windows started using libarchive to support .rar, .7z, ...
https://arstechnica.com/gadgets/2023/05/cancel-your-winrar-t...
Couldn't the autoconf soup be generated from simpler inputs by the CI/CD system to avoid this kind of problem? Incomprehensible soup as a build artifact (e.g. executables) is perfectly normal, but it seems to me that such things don't belong in the source code.
(This means you too, gradle-wrapper! And your generated wrapper for your generated wrapper. That junk is not source code and doesn't belong in the repo.)
Pure speculation but my guess is a specific state actor ahem is looking for developers innocently working with open source to then strongarm them into doing stuff like this.
I would be curious if their commits could be analyzed for patterns that could then be used to detect commits from their other account
I mean, a backdoor at this scale (particularly if it wasn't noticed for a while and got into stable distros) could be worth millions. Maybe hundreds of millions (think of the insider trading possibilities alone, not to mention espionage). 2 years doesn't seem like that much work relative to the potential pay off.
This is the sort of case where america's over the top hacking laws make sense.
> I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup
If I recall correctly, xz can be built with both autoconf and cmake, are cmake configs similarly affected?
How about wheels in the python ecosystem
Yeah this was my first thought too. Though I think the case against autoconf is already so overwhelming I think anyone still using it is just irredeemable; this isn't going to persuade them.
For those panicking, here are some key things to look for, based on the writeup:
- A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was added in the last month or so. If you're not on a rolling release distro, your version is probably older.
- A debian or RPM based distro of Linux on x86_64. In an apparent attempt to make reverse engineering harder, it does not seem to apply when built outside of deb or rpm packaging. It is also specific to Linux.
- Running OpenSSH sshd from systemd. OpenSSH as patched by some distros only pulls in libsystemd for logging functionality, which pulls in the compromised liblzma5.
Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.
It is possible there are other flaws or backdoors in liblzma5, though.
Focusing on sshd is the wrong approach. The backdoor was in liblzma5. It was discovered to attack sshd, but it very likely had other targets as well. The payload hasn't been analyzed yet, but _almost everything_ links to libzma5. Firefox and Chromium do. Keepassxc does. And it might have made arbitrary changes to your system, so installing the security update might not remove the backdoor.
Ubuntu still ships 5.4.5 on 24.03 (atm).
I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though.
So for 5.4.5 the tagged release and download on github differ.
It does change format strings, e.g.
+#: src/xz/args.c:735
+#, fuzzy
+#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout"
+msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout"
+msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard"
There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs.[Edit] formatting
> Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.
I'm surprised .deb doesn't have a better approach. RPM has epoch for this purpose http://novosial.org/rpm/epoch/index.html
> If you're not on a rolling release distro, your version is probably older.
Ironic considering security is often advertised as a feature of rolling release distros. I suppose in most instances it does provide better security, but there are some advantages to Debian's approach (stable Debian, that is).
The article gives a link to a simple shell script that detects the signature of the compromised function.
> Running OpenSSH sshd from systemd
I think this is irrelevant.
From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.
Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.
I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related?
I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?
My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.
$ dpkg-query -W liblzma5
liblzma5:amd64 5.4.1-0.2
Tumbleweed has a package: liblzma5-5.6.1.revertto5.4-3.2.x86_64 FYI
I hope Lasse Collin is doing OK! Here is a older message from him [1]
"I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.
It's also good to keep in mind that this is an unpaid hobby project. "
Github (Microsoft) are in a unique position to figure out if his account is hacked or not, and find a way to reach him. I hope they reach out and offer him some proper support! Economic support (if that's needed), or just help clearing his name.
This is another tale of how we are building multi trillion dollar industries on the back of unpaid volunteers. It's not github 'job', and many other organisations have benefited even more from Lasses work, but they are in a unique position, and would be literally pocket change for them.
1:https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...
In a movie his mental health issues would likely have been caused intentionally by the attacker, setting the stage for the mole to offer to step in just at the right time. Seems a bit far fetched in this case though for what looks like a tangential attack.
Lasse appears to be active and working on undoing the sabotage. https://git.tukaani.org/?p=xz.git;a=blobdiff;f=CMakeLists.tx...
He came on IRC, he seemed ok. He did some cleanup of access and signed off for easter.
I would like to see more attention given to this. I'm capable of compartmentalization and not over-guilting myself, but holy hell, I really hope he's doing alright. This would kind of destroy me.
I was actually telling my dad about this. I have a project, 500+ users, not quite root access, but enough to cause serious damage. I can think of at least one covert way to backdoor the binary artifacts from it.
About two years ago, someone showed up, started making good commits. In this case, they have some other community rep that goes back a bit further but... man it's an unsettling feeling.
Relevant xkcd:
A couple of years ago I wrote a Go library that wraps the xz C code and allows you to do xz compression in Go: https://github.com/jamespfennell/xz
About a week ago I received the first PR on that repo, to upgrade to 5.6.1. I thought it was odd to get such a random PR...it's not the same GitHub account as upstream though.
As a bit of an aside, I would never accept a PR like this, and would always update $large_vendored_dependency myself. This is unreviewable, and trivial to insert any backdoor (unless you go through the motions of updating it yourself and diffing, at which point the PR becomes superfluous). I'd be wary even from a well-known author unless I knew them personally on some level (real-life or via internet). Not that I wouldn't trust them, but people's machines or accounts can get compromised, people can have psychotic episodes, things like that. At the very least I'd like to have some out-of-band "is this really you?" signal.
This is how I once inserted a joke in one of our (private) repos that would randomly send cryptic messages to our chat channel. This was pretty harmless and just a joke (there's some context that made it funny), but it took them years to find it – and that was only because I told them after I quit.
That said, looking at the GitHub account I'd be surprised if there's anything nefarious going on here. Probably just someone using your repo, seeing it's outdated, and updating it.
Hey all, I’m the author of that PR. Just posted to Github with additional context: https://github.com/jamespfennell/xz/pull/2#issuecomment-2027...
I don't want to read too much into it, but the person (supposedly) submitting the PR seems to work at 1Password since December last year, as per his Linkedin. (And his Linkedin page has a link to the Github profile that made the PR).
> it's not the same GitHub account as upstream
This is valuable information, and a sign that this may be the tip of an iceberg.
There was also a bug report in Debian which requested updating xz-utils to 5.6.1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708
IMO your prior on this should be that it's most likely just someone innocently updating a dependency.
The backdoor (test binary blob and autoconf) is not part of the pull request.
Suddenly anything like that becomes super suspicious.
I wonder how this will affect the OS community in general.
Internet detectives at work in this thread!
I am *not* a security researcher, nor a reverse engineer. There's lots of
stuff I have not analyzed and most of what I observed is purely from
observation rather than exhaustively analyzing the backdoor code.
I love this sort of technical writing from contributors outside the mainstream debugging world who might be averse to sharing. What an excellently summarized report of his findings that should be seen as a template.FWIW, it felt intimidating as hell. And I'm fairly established professionally. Not sure what I'd have done earlier in my career (although I'd probably not have found it in the first place).
For what it's worth the author is a PostgreSQL committer, he's not a security researcher but he's a pretty damn good engineer!
Honestly, you only get this kind of humility when you're working with absolute wizards on a consistent basis. That's how I read that whole analysis. Absolutely fascinating.
Related ongoing threads:
Xz: Disable ifunc to fix Issue 60259 - https://news.ycombinator.com/item?id=39869718
FAQ on the xz-utils backdoor - https://news.ycombinator.com/item?id=39869068
Everything I Know About the XZ Backdoor - https://news.ycombinator.com/item?id=39868673
Out of curiosity I looked at the list of followers of the account who committed the backdoor.
Randomly picked https://github.com/Neustradamus and looked at all their contributions.
Interestingly enough, they got Microsoft to upgrade ([0],[1]) `vcpkg` to liblzma 5.6.0 3 weeks ago.
OMG: look at the other contributions. He is trying to take over projects and pushing some change to sha256 in a hundred projects.
Hey, I remember this guy! Buddy of someone who tried to get a bunch of low quality stuff into ifupdown-ng, including copying code with an incompatible license and removing the notice. He's in every PR, complaining the "project is dead". He even pushes for the account to be made "team member".
https://github.com/ifupdown-ng/ifupdown-ng/pulls/easynetdev
He follows 54k accounts though, so it may indeed just be coincidence.
Dear @0xthr0w4, do you attack me because I have requested the XZ update?
Do not mix, I am not linked to the XZ project.
Imagine a more competent backdoor attempt on xz(1)—one that wouldn't have been noticed this quickly. xz is everywhere. They could pull off a "reflections on trusting trust": an xz which selectively modifies a tiny subset of the files it sees, like .tar.xz software tarballs underlying certain build processes. Not source code tarballs (someone might notice)—tarballs distributing pre-compiled binaries.
edit to add: Arch Linux' entire package system used to run on .tar.xz binaries (they switched to Zstd a few years ago [0]).
[0] https://news.ycombinator.com/item?id=19478171 ("Arch Linux propose changing compression method from xz to zstd (archlinux.org)")
A backdoored xz could also run payloads hidden inside other xz files, allowing targeted attacks.
The same authors have also contributed to Zstd
deb packages are xz compressed...
Unfortunately, this is how good bad actors work: with a very long-term point of view. There is no “harmless” project any more.
And, Joey Hess has counted at least 750 commits to xz from that handle.
https://hachyderm.io/@joeyh/112180715824680521
This does not look trust-inspiring. If the code is complex, there could be many more exploits hiding.
I imagine it might be easier to just compromise a weakly protected account than to actual put in a 2 years long effort with real contributions. If we mandated MFA for all contributors who contribute to these really important projects then we can know with greater certainty if it was really a long con vs. a recently compromised account.
Probably a state actor. You can look far into the future when you’re working for the party.
(I detached this subthread from https://news.ycombinator.com/item?id=39866275, for the sake of pruning the top heavy thread.)
More likely that the account of that dev was breawched, dont you think ?
Warning, drunk brain talking. But a LLM driven email based "collaborator" could play a very long gMw adding basic features to a code made whilst earning trust backed by a generated online presence. My money is on a resurgance in the Web of Trust.
Looks like Lasse Collin has commented on LKML: https://lkml.org/lkml/2024/3/30/188
Also, some info here: https://tukaani.org/xz-backdoor/
Or if you can't stand the lkml.org UI:
https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/
The terrifying part is that this was primarily found because the backdoor was poorly made and causing performance problems.
Makes you wonder what more competent actors can do.
I've analysed the backdoor myself and it's very sophisticated, not poorly made at all. The performance problem is surprising in this context, but I think next time they won't make that mistake.
So many malicious actors have been caught because they accidentally created a mild annoyance for someone that went on to bird-dog the problem.
You must mean, "Makes you wonder what more competent actors are doing"
s/can do/have done/
Funny how Lasse Collin started to ccing himself and Jia Tan from 2024-03-20 (that was a day of tons of xz kernel patches), he never did that before. :)
https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.co...
This is extremely suspicious.
It looks like someone may have noticed a unmaintained or lightly maintained project related to various things, and moved to take control of it.
Otherwhere in the discussion here someone mentions the domain details changed; if you have control of the domain you have control of all emails associated with it.
Also interesting, to me, how the GMail account for the backdoor contributor ONLY appears in the context of "XZ" discussions. Google their email address. Suggests a kind of focus, to me, and a lack of reality / genuineness.
those pipe usages are quite suspicious
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...
pipeing into this shell script which now uses "eval"
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...
i guess this will be revisited and removed soon
"started to cc himself" seems to be simply "contributing to a new project and not having git-send-email fully set up". By default git-send-email Cc the sender, though in practice it's one of the first options one changes.
My favorite part was the analysis of "I'm not really a security researcher or reverse engineer but here's a complete breakdown of exactly how the behavior changes."
You only get this kind of humility when you're working with absolute wizards on a consistent basis.
That's completely crazy, the backdoor is introduced through a very cryptic addition to the configure script. Just looking at the diff, it doesn't look malicious at all, it looks like build script gibberish.
Thanks to autoconf, we're now used to build scripts looking like gibberish. A perfect place to hide a backdoor.
It looks like an earlier commit with a binary blob "test data" contained the bulk of the backdoor, then the configure script enabled it, and then later commits patched up valgrind errors caused by the backdoor. See the commit links in the "Compromised Repository" section.
Also, seems like the same user who made these changes are still submitting changes to various repositories as of a few days ago. Maybe these projects need to temporarily stop accepting commits until further review is done?
The use of "eval" stands out, or at least it should stand out – but there are two more instances of it in the same script, which presumably are not used maliciously.
A while back there was a discussion[0] of an arbitrary code execution vulnerability in exiftool which was also the result of "eval".
Avoiding casual use of this overpowered footgun might make it easier to spot malicious backdoors. Usually there is a better way to do it in almost all cases where people feel the need to reach for "eval", unless the feature you're implementing really is "take a piece of arbitrary code from the user and execute it".
Yeah, now imagine they succeeded and it didn't cause any performance issues...
Can we even be sure no such successful attempt has already been made?
A big part of the problem is all the tooling around git (like the default github UI) which hides diffs for binary files like these pseudo-"test" files. Makes them an ideal place to hide exploit data since comparatively few people would bother opening a hex editor manually.
> "Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the "fixes" mentioned above."
Crazy indeed.
So when are we going to stop pretending that OSS maintainers/projects are reaping what they sow when they "work for free" and give away their source code away using OSS licensed software, while large companies profit off of them? If they were paid more (or in some cases even actually paid), then they could afford to quit their day jobs, reducing burn out, they could actually hire a team of trusted vetted devs instead of relying on the goodwill of strangers who step up "just to help them out" and they could pay security researchers to vet their code.
Turns out burned out maintainers are a great attack vector and if you are willing to play the long game you can ingratiate yourself with the community with your seemingly innocuous contributions.
Paid people get burnt out as well and they are just as likely to accept free help as an unpaid person.
> So when are we going to stop pretending ...
I'm not sure that we are. Doesn't everybody know that developing/maintaining free software is largely thankless work, with little to no direct recompense?
I don't think moving towards unfree software is a good way to make free software more secure. It shouldn't be a surprise that proprietary software is less likely to be exploited in this way simply because they don't accept any patches from outside of the team. What you want is more people that understand and care about free software and low barriers to getting involved.
OSS maintainers aren't reaping anything. Most OSS licenses say the software is provided without warranty.
The discussion to upload it to Debian is interesting on its own https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708
Wow, that's a lot of anonymous accounts adding comments there urging for a fast merge!
And this "Hans Jansen" guy is apparently running around salsa.debian.org pushing for more updates in other projects as well: https://salsa.debian.org/users/hjansen/activity
That name jumped out at me, Hans Jansen is the name Dominic Monaghan used when posing as a German interviewer with Elijah Woods. Not that it can't be a real person
For anyone else feeling some deja vu about ifunc / Valgrind errors, this Red Hat issue [1] was previously linked from HN 12 days ago [2].
I get the feeling that a number of the comments are all the same person / group.
I'd love to be at Microsoft right now and have the power to review this user's connection history to Github, even though VPN exists, many things can be learned from connection habits, links to ISPs, maybe even guess if VPNs were used, roundtrip time on connections can give hints.
I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.
> I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.
Which is why there's probably nothing remotely interesting in them logs.
Nah. I'm sure Microsoft got a call from the alphabet boys and nobody, not even internal employees are allowed to look at the logs right now.
Oh my, another reason not to use github. :D So many reasons poping up just in this comment section alone.
I'm guessing Microsoft just got a call from the Government telling them not to look too deeply into it.
That’d be illegal for an employee to do.
https://github.com/tukaani-project/tukaani-project.github.io...
> Note: GitHub automatically includes two archives Source code (zip) and Source code (tar.gz) in the releases. These archives cannot be disabled and should be ignored.
The author was thinking ahead! Latest commit hash for this repo: 8a3b5f28d00ebc2c1619c87a8c8975718f12e271
Btw, this is not the only project providing a source tarball different from the git repo, for example libusb also does this (and probably others):
- https://github.com/libusb/libusb/issues/1468#issuecomment-19...
For a long time, there was one legitimately annoying disadvantage to the git-generated tarballs though - they lost tagging information. However, since git 2.32 (released June 2021; presumably available on GitHub by August 2021 when they blogged about it) you can use `$Format:%(describe)$` ... limited to once per repository for performance reasons.
Jia Tan "cleaned up" in all their ZSTD branches some hours ago, probably hiding something https://github.com/JiaT75/zstd/branches/all
GitHub/Microsoft likely has a backup. I’d be getting those out about now.
Bad move. Destroying evidence is a felony.
Comment from Andres Freund on how and why he found it [0] and more information on the LWN story about the backdoor. Recommend people read this to see how close we came (and think about what this is going to mean for the future).
That man deserves a Nobel Prize
A mirror of the offending repository created by someone else is available at [1]. GitHub should be keeping the evidence in the open (even if just renamed or archived in a safer format) instead of deleting it/hiding it away.
The offending tarball for v5.6.1 is easier to find, an example being.[2]
m4/.gitignore was updated 2 weeks ago to hide build-to-host.m4 that is only present in the release tarball and is used to inject the backdoor at build time.[3]
[1] https://git.phial.org/d6/xz-analysis-mirror
[2] https://mirrors.xtom.ee/gentoo/distfiles/9f/xz-5.6.1.tar.gz
[3] https://git.phial.org/d6/xz-analysis-mirror/commit/4323bc3e0...
This gist summarizes the current situation very well: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...
Definitely looking like they were most likely some sort of state actor. This is very well done and all in plain sight. It's reassuring that it was discovered but given a simple audit of the release build artifacts would have raised alarms, how prevalent is this behavior in other projects? Terrifying stuff.
A lot of eyes will be dissecting this specific exploit, and investigating this specific account, but how can we find the same kind of attack in a general way if it’s being used in other projects and using other contributor names?
1. Everything must be visible. A diff between the release tarball and tag should be unacceptable. It was hidden from the eyes to begin with.
2. Build systems should be simple and obvious. Potentially not even code. The inclusion was well hidden.
3. This was caught through runtime inspection. It should be possible to halt any Linux system at runtime, load debug symbols and map _everything_ back to the source code. If something can't map back then regard it as a potentially malicious blackbox.
There has been a strong focus and joint effort to make distributions reproducible. What we haven't managed though is prove that the project compromises only of freshly compiled content. Sorta like a build time / runtime "libre" proof.
This should exist for good debugging anyway.
It wouldn't hinder source code based backdoors or malicious vulnerable code. But it would detect a backdoor like this one.
Just an initial thought though, and probably hard to do, but not impossibly hard, especially for a default server environment.
Build-related fixes are only treating the symptoms, not the disease. The real fix would be better sandboxing and capability-based security[1] built into major OSes which make backdoors a lot less useful. Why does a compression library have the ability to "install an audit hook into the dynamic linker" or anything else that isn't compressing data? No amount of SBOMs, reproducible builds, code signing, or banning binaries will change the fact that one mistake anywhere in the stack has a huge blast radius.
[1]: https://en.wikipedia.org/wiki/Capability-based_security
Note that the malicious binary is fairly long and complex.
This attack can be stopped by disallowing any binary testdata or other non-source code to be on the build machines during a build.
You could imagine a simple process which checks out the code, then runs some kind of entropy checker over the code to check it is all unminified and uncompressed source code, before finally kicking off the build process.
autogenerated files would also not be allowed to be in the source repo - they're too long and could easily hide bad stuff. Instead the build process should generate the file during the build.
We should be able to produce a tar and a proof that tar was produced from a specific source code.
Quote from the article:
That line is not in the upstream source of build-to-host, nor is build-to-host used by xz in git.
Zero Knowledge virtual machines, like cartesi.io, might help with this. Idea is to take the source, run a bunch of computational steps (compilation & archiving) and at the same time produce some kind of signature that certain steps were executed.The verifiers can then easily check that the signature and indeed be convinced that the code was executed as it is claimed and source code wasn't tampered with.
The advantage of Zero-Knowledge technology in this case is that one doesn't need to repeat the computational steps themselves nor rely on a trusted party to do it for them (like automated build - that can also be compromised by the state actors). Just having the proof solves this trust problem mathematically: if you have the proof & the tar, you can quickly check source code that produced the tar wasn't modified.
The Guix full source bootstrap is looking less paranoid as time goes on
More reproducible builds, maybe even across distributions? Builds based on specific commits (no tarballs like in this case), possibly signed (just for attribution, not for security per se)? Allow fewer unsafe/runtime modifications The way oss-fuzz ASAN was disabled should've been a warning on its own, if these issues weren't so common.
I'm not aware of any efforts towards it, but libraries should also probably be more confined to only provide intended functionality without being able to hook elsewhere?
NixOS/Pkgs 23.11 unaffected, unstable contains backdoored implementations (5.6.0, 5.6.1) but their OpenSSH sshd does not seem to link against systemd/liblzma, and the backdoor doesn't get configured in (only happens on .deb/.rpm systems).
It may not have really mattered much for NixOS:
> b) argv[0] needs to be /usr/sbin/sshd
For once, the lack of FHS interoperability is a benefit, if only on accident.
Note that NixOS has a unique advantage in that `dlopen` is easier to analyze, but you do have to check for it. A lot of people are looking only at `ldd` and missing that they can be vulnerable at runtime.
Not affected by the latest CVE, but the author had unrestricted access to xz for 2 years, so I would say it is affected until the other contributions are proven safe (never gonna happen) or it reverts to pre-adversarial actor version.
That's one of the advantages of NixOS - viruses and mass hacks have lesser chance to function due to how different this OS is. Until it gets more popular, of course.
I looked at the differences between the GitHub repository and released packages. About 60 files are in a release package that are not in the repo (most are generated files for building) but also some of the .po files have changes.
That's devastating.
If you don't build your release packages from feeding "git ls-files" into tar, you are doing it wrong.
I think this is unfortunately very common practice
Why not `git archive`?
The latest commit from the user who committed those patches is weirdly a simplification of the security reporting process, to not request as much detail:
https://github.com/tukaani-project/xz/commit/af071ef7702debe...
Not sure what to make of this.
I think the reason is pretty obvious. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.
That repository is now disabled. But here's a similar change to the .github repository of tukaani-project from @JiaT75 to the bug report template:
+ or create a private Security Advisory instead.
Under a commit titled "Wrap text on Issue template .yaml files."[1] https://github.com/tukaani-project/.github/commit/44b766adc4...
Potentially the purpose is that if someone goes to the effort to get those details together, they are more likely to send the same report to other trusted individuals. Maybe it was originally there to add legitimacy, then they got a report sent in, and removed it to slow the spread of awareness
> "Docs: Simplify SECURITY.md."
https://github.com/tukaani-project/xz/commit/af071ef7702debe...
Removes instructions about details relevant to security reports. Heh, nice one.
It looks like the person who added the backdoor is in fact the current co-maintainer of the project (and the more active of the two): https://tukaani.org/about.html
In various places they say Lasse Collin is not online right now, but he did make commits a week ago https://git.tukaani.org/?p=xz.git;a=summary
Makes me wonder if he's an owner of the github organization, and what happens with it now?
Why has Github disabled the (apparently official) xz repository, but left the implicated account open to the world? It makes getting caught up on the issue pretty difficult, when GitHub has revoked everyone's access to see the affected source code.
https://github.com/tukaani-project/xz vs https://github.com/JiaT75
The account has been suspended for a while, but for whatever reason that's not displayed on the profile itself (can be seen at https://github.com/Larhzu?tab=following). Repo being disabled is newer, and, while annoying and realistically likely pointless, it's not particularly unreasonable to take down a repository including a real backdoor.
The author (Jia Tan) also changed the xz.tukaani.org (actually the github.io, where the main contributor is, surprise, also them) release description to state all new releases are signed by their OpenPGP key. I'd guess that was one of the first steps to a complete project takeover.
I hope Lasse Collin still has control of his accounts, though the CC on the kernel mailing list looks kind of suspicious to me.
The backdoor is not in the C source directly, but a build script uses data from files in the test dir to only create the backdoor in the release tars. Did I summarize that correctly?
That's how I understand it. A build script that's in the releases tarballs but not the git repo, checks to see if it's being run as part of the debian/build or rpm build processes, and then injects content from one of the "test" files.
"Amazon Linux customers are not affected by this issue, and no action is required. AWS infrastructure and services do not utilize the affected software and are not impacted. Users of Bottlerocket are not affected."
https://aws.amazon.com/security/security-bulletins/AWS-2024-...
The best part is everyone disabling security tests that started failing
I read through the entire report and it gradually got more interesting. Then, I got to the very end, saw Andres Freund's name, and it put a smile on my face. :)
Who else would have run a PostgreSQL performance benchmark and discover a major security issue in the process?
This is another proof that systemd is an anti-pattern for security: with its crawling and ever growing web of dependencies, it extends the surface of vulnerability to orders of magnitude, and once embraced not even large distro communities can defend you from that.
A malware code injection in upstream xz-tools is a vector for remote exploitation of the ssh daemon due to a dependency on systemd for notifications and due to systemd's call to dlopen() liblzma library (CVE-2024-3094). The resulting build interferes with authentication in sshd via systemd.
Please take the systemd trolling to Reddit. They likely targeted xz specifically because it’s so widely used but there are dozens of other libraries which are potential candidates for an attack on sshd, much less everything else which has a direct dependency unrelated to systemd (e.g. dpkg).
Rather than distracting, think about how the open source projects you use would handle an attack like this where someone volunteers to help a beleaguered maintainer and spends time helpfully taking on more responsibilities before trying to weaken something.
Actually you have a point. A collection of shell scripts (like the classical init systems) have obviously a smaller attack surface. In this case the attacker used some integration code with systemd to attack the ssh daemon. So sshd without systemd integration is safe against this specific attack.
In general, I’m not convinced that systemd makes things less secure. I have the suspicion that the attacker would just have used a different vector, if there was no systemd integration. After all it looks like the attacker was also trying to integrate exploits in owner libraries, like zstd.
Still I would appreciate it, if systemd developers would find a better protection against supply chain attacks.
This isn't Twitter you don't have to use hashtags
> systemd's call to dlopen() liblzma library (CVE-2024-3094)
That's technically wrong, but no surprise. Anti-systemd trolls usually don't understand technical details after all.
For bad-3-corrupt_lzma2.xz, the claim was that "the original files were generated with random local to my machine. To better reproduce these files in the future, a constant seed was used to recreate these files." with no indication of what the seed was.
I got curious and decided to run 'ent' https://www.fourmilab.ch/random/ to see how likely the data in the bad stream was to be random. I used some python to split the data into 3 streams, since it's supposed to be the middle one that's "bad":
I used this regex to split in python, and wrote to "tmp":
re.split(b'\xfd7zXZ', x)
I manually used dd and truncate to strip out the remaining header and footer according to the specification, which left 48 bytes: $ ent tmp2 # bad file payload
Entropy = 4.157806 bits per byte.
Optimum compression would reduce the size
of this 48 byte file by 48 percent.
Chi square distribution for 48 samples is 1114.67, and randomly
would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 51.4167 (127.5 = random).
Monte Carlo value for Pi is 4.000000000 (error 27.32 percent).
Serial correlation coefficient is 0.258711 (totally uncorrelated = 0.0).
$ ent tmp3 # urandom
Entropy = 5.376629 bits per byte.
Optimum compression would reduce the size
of this 48 byte file by 32 percent.
Chi square distribution for 48 samples is 261.33, and randomly
would exceed this value 37.92 percent of the times.
Arithmetic mean value of data bytes is 127.8125 (127.5 = random).
Monte Carlo value for Pi is 3.500000000 (error 11.41 percent).
Serial correlation coefficient is -0.067038 (totally uncorrelated = 0.0).
The data does not look random. From https://www.fourmilab.ch/random/ for the Chi-square Test, "We interpret the percentage as the degree to which the sequence tested is suspected of being non-random. If the percentage is greater than 99% or less than 1%, the sequence is almost certainly not random. If the percentage is between 99% and 95% or between 1% and 5%, the sequence is suspect. Percentages between 90% and 95% and 5% and 10% indicate the sequence is “almost suspect”."Now to be fair, such an archive could have been created with a “store” level of compression that doesn’t actually perform any compression.
All these older (4.x, 5.0.x etc) releases that were suddenly uploaded a few months ago should probably also be considered suspect: https://github.com/tukaani-project/tukaani-project.github.io...
Here's a handy bash script I threw together to audit any docker containers you might be running on your machine. It's hacky, but will quickly let you know what version, if any, of xz, is running in your docker containers.
``` #!/bin/bash
# Get list of all running Docker containers containers=$(docker ps --format "{{.Names}}")
# Loop through each container for container in $containers; do # Get container image image=$(docker inspect --format='{{.Config.Image}}' "$container")
# Execute xz --version inside the container
version=$(docker exec "$container" xz --version)
# Write container name, image, and command output to a text file
echo "Container: $container" >> docker_container_versions.txt
echo "Image: $image" >> docker_container_versions.txt
echo "xz Version:" >> docker_container_versions.txt
echo "$version" >> docker_container_versions.txt
echo "" >> docker_container_versions.txt
doneecho "Output written to docker_container_versions.txt" ```
Sadly this is exactly one of the cases where open source is much more vulnerable to a state actor sponsored attack than proprietary software. (it is also easier to find such backdoors in OS software but that's BTW)
Why? Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he. Also they have to be hiring in the right team etc. Your operative has to be in a different country, needs a CV that checks out, passports/ids are checked etc.
But to contribute to an OS project? You just need an email address. Your operative sends good contributions until they build trust, then they start introducing backdoors in the part of the code "no one, but them understands".
The cost of such attack is a lot lower for a state actor so we have to assume every single OS project that has a potential to get back doored had many attempts of doing so. (proprietary software too, but as mentioned, this is much more expensive)
So what is the solution? IDK, but enforcing certain "understandability" requirements can be a part of it.
Is that true? Large companies producing software usually have bespoke infra, which barely anyone monitors. See: the Solarwinds hack. Similarly to the xz compromise they added the a Trojan to the binary artifacts by hijacking the build infrastructure. According to Wikipedia "around 18,000 government and private users downloaded compromised versions", it took almost a year for somebody to detect the trojan.
Thanks to the tiered updates of Linux distros, the backdoor was caught in testing releases, and not in stable versions. So only a very low percentage of people were impacted. Also the whole situation happened because distros used the tarball with a "closed source" generated script, instead of generating it themselves from the git repo. Again proving that it's easier to hide stuff in closed source software that nobody inspects.
Same with getting hired. Don't companies hire cheap contractors from Asia? There it would be easy to sneak in some crooked or even fake person to do some dirty work. Personally I was even emailed by a guy from China who asked me if I was willing to "borrow" him my identity so he could work in western companies, and he would share the money with me. Of course I didn't agree, but I'm not sure if everybody whose email he found on Github did.
https://en.wikipedia.org/wiki/2020_United_States_federal_gov...
> Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he.
Or work for a third-party company that gets access to critical systems without any checks. See for example the incident from 2022 here: https://en.wikipedia.org/wiki/Okta,_Inc.
Or a third-party that rents critical infrastructure to the company (Cloud, SaaS solutions).
It's wild that this could have laid dormant for far longer if the exploit was better written-- if it didn't spike slow down logins or disturb valgrind.
So many security companies publishing daily generic blog posts about "serious supply chain compromises" in various distros on packages with 0 downloads, and yet it takes a developer debugging performance issues to find an actual compromise.
I worked in the software supply chain field and cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.
> cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.
That's the entire point. You did everything you could by getting someone else look at it and saying it's fine.
Thats basically the whole point actually... A company pays for insurance for the business. The insurance company says sure we will insure you, but you need to go through audits A B and C, and you need certifications X and Y to be insured by us. Those audits are often industry dependent, mostly for topics like HIPAA, PCI, SOC, etc.
Insurance company hears about supply chain attacks. Declares that insured must have supply chain validation. Company goes and gets a shiny cert.
Now when things go wrong, the company can point to the cert and go "it wasnt us, see we have the cert you told us to get and its up to date". And the company gets to wash their hands of liability (most of the time).
If you installed xz on macOS using brew, then you have
xz (XZ Utils) 5.6.1
liblzma 5.6.1
which are within the release target for the vuln. As elsewhere in these comments, people say macOS effect is uncertain. If concerned you can revert to 5.4.6 with brew upgrade xz
> the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.
That is actually a major point of a lot of corporate security measures (shifting risk)
That's the entire point of certification, and any certification at all. Certification does not guarantee performance. Actually, I would always cast a suspect glance to anyone who is FOCUSED on getting certification after certification without any side project.
List of pull request requesting the updating to liblzma 5.6.0 [0]
I wonder what amount of scrutiny all the accounts that proposed the upgrade should be put under.
[0] https://github.com/search?q=liblzma+5.6.0&type=pullrequests
When I search for "digital masquerade" on Google, the first result is a book with this title from the author Jia Tan. I assume that is how the attackers got their fake name. Or they think using this author's name is a joke.
A lot of software (including https://gitlab.com/openconnect/openconnect of which I'm a maintainer) uses libxml2, which in turn transitively links to libzma, using it to load and store compressed XML.
I'm not *too* worried about OpenConnect given that we use `libxml2` only to read and parse uncompressed XML…
But I am wondering if there has been any statement from libxml2 devs (they're under the GNOME umbrella) about potential risks to libxml2 and its users.
This doesn't matter, if libxml2 loads .so and the library is malicious, you are already potentially compromised, as it is possible to run code on library load.
> only to read and parse uncompressed XML…
how does libxml2 know to decompress something?
does it require you, as the caller, to explicitly tell it to?
or does it look at the magic bytes or filename or mimetype or something?
Potentially malicious commit by same author on libarchive: https://github.com/libarchive/libarchive/pull/1609
Summary: "The upstream xz repository and the xz tarballs have been backdoored."
It is known to be in version 5.6.0 and 5.6.1, and the obfuscated code is found in the test directory.
Since GitHub disabled the repos.. I uploaded all GitHub Events from the two suspected users and from their shared project repo as easy to consume CSV files:
https://github.com/emirkmo/xz-backdoor-github
For those who want to see the GitHub events (commits, comments, pull_requets, diffs, etc.)
Better make a torrent out of them.
Very strange behavior from the upstream developers. Possible government involvement? I have a feeling LANG is checked to target servers from particular countries
One thing to note is that the person that added the commits only started contributing around late 2022 and appears to have a Chinese name. Might be required by law to plant the backdoor.
That would be quite scary considering they have contributed to a wide variety of projects including C++ https://learn.microsoft.com/en-us/cpp/overview/whats-new-cpp...
LANG only needs to have some value, the concrete value does not seem to matter.
If you have a recently updated NixOS unstable it has the affected version:
$ xz --version
xz (XZ Utils) 5.6.1
liblzma 5.6.1
EDIT: I've been informed on the NixOS matrix that they are 99% sure NixOS isn't affected, based on conversations in #security:nixos.orgPersonally, I use lzip ever since I read https://www.nongnu.org/lzip/xz_inadequate.html Seems like the complexity of XZ has backfired severely, as expected.
> Seems like the complexity of XZ has backfired severely, as expected.
this is a very bad reading of the current situation.
This potentially could be a full automated rootkit type breach right? Great - is any system with 5.6.1 possibly vulnerable?
Also super weird a contributor thought they could slip this in and not have it be noticed at some point. It may point to burning that person (aka, they go to jail) for whatever they achieved with this. (And whoever they are…)
This was only a matter of time. Open source projects are under-staffed, maintainers are overworked and burned out, and everyone relies on the goodwill of all actors.
Obviously a bad actor will make use of these conditions and the assumption of good will.
We need automated tooling to vet for stuff like this. And maybe migrate away from C/C++ while we are at it because they don't make such scanning easy at all.
Wouldn’t be surprised that the ssh auth being made slower was deliberate - that makes it fairly easy to index all open ssh servers on the internet, then to see which ones get slower to fail preauth as they install the backdoor
people are mis-reading the Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708
it wasn't the apparently newly-created identity "Hans Jansen" just asking for a new version to be uploaded, it was "Hans Jansen" providing a new version to be uploaded as a non-maintainer-upload - Debian-speak for "the maintainer is AWOL, someone else is uploading their package". if "Hans Jansen" is another attacker then they did this cleverly, providing the new - compromised - upstream tarballs in an innocent-looking way and avoiding anyone examining the upstream diff.
Looking at how many requests to update to the backdoored version have been made, I wonder if the fact that many people (including developers) have been conditioned to essentially accept updates as "always-good" is a huge contributing factor in how easy it is to spread something like this.
The known unknowns can be better than the unknown unknowns.
Totally agree. With things like Dependabot encouraged by GitHub, people now get automated pull requests for dependency updates, increasing the speed of propagation of such vulnerabilities.
Looks like GitHub has suspended access to the repository, which while it protects against people accidentally compiling and using the code, but certainly complicates forensic analysis for anyone who doesn't have a clone or access to history (which is what I think a lot of people will be doing now to understand their exposure).
It looks like git clone https://git.tukaani.org/xz.git still works for now (note: you will obviously be cloning malware if you do this) - that is, however, trusting the project infrastructure that compromised maintainers could have had access to, so I'm not sure if it is unmodified.
HEAD (git rev-parse HEAD) on my result of doing that is currently 0b99783d63f27606936bb79a16c52d0d70c0b56f, and it does have commits people have referenced as being part of the backdoor in it.
Well that's inconvenient, I was (probably, time permitting) going to propose to some of my friends that we attempt to reverse this for fun tomorrow.
Anyone have a link to the git history? I guess we can use the ubuntu tarball for the evil version.
It seems like based on the (very well written) analysis that this is a way to bypass ssh auth, not something that phones out which would've been even scarier.
My server runs arch w/ a LTS kernel (which sounds dumb on the surface, but was by far the easiest way to do ZFS on Linux that wasn't Ubuntu) and it seems that since I don't have SSH exposed to the outside internet for good reason, and my understanding is Arch never patched shhd to begin with that I and most people who would be in similar situations to me are unaffected.
Still insane that this happened to begin with, and I feel bad for the Archlinux maintainers who are now going to feel more pressure to try to catch things like this.
Being included via libsystemd isn't the only way ssh can load liblzma, it can come as an indirect dependency of Selinux (and its PAM stack) IIUC. Which makes it even a bit more funny (?) since Arch also doesn't officially support any Selinux stuff.
There might be other ways sshd might pull in lzma, but those are the 2 ways I saw commonly mentioned.
On a different note, pacman/makepkg got the ability to checksum source repository checkouts in 6.1.
Interesting commit in January where the actual OpenPGP key was changed: https://github.com/tukaani-project/tukaani-project.github.io...
They just signed each other's keys around that time, and one needs to redistribute the public keys for that; nothing suspicious about it I think. The key fingerprint 22D465F2B4C173803B20C6DE59FCF207FEA7F445 remained the same.
before:
pub rsa4096/0x59FCF207FEA7F445 2022-12-28 [SC] [expires: 2027-12-27]
22D465F2B4C173803B20C6DE59FCF207FEA7F445
uid Jia Tan <jiat0218@gmail.com>
sig 0x59FCF207FEA7F445 2022-12-28 [selfsig]
sub rsa4096/0x63CCE556C94DDA4F 2022-12-28 [E] [expires: 2027-12-27]
sig 0x59FCF207FEA7F445 2022-12-28 [keybind]
after: pub rsa4096/0x59FCF207FEA7F445 2022-12-28 [SC] [expires: 2027-12-27]
22D465F2B4C173803B20C6DE59FCF207FEA7F445
uid Jia Tan <jiat0218@gmail.com>
sig 0x59FCF207FEA7F445 2022-12-28 [selfsig]
sig 0x38EE757D69184620 2024-01-12 Lasse Collin <lasse.collin@tukaani.org>
sub rsa4096/0x63CCE556C94DDA4F 2022-12-28 [E] [expires: 2027-12-27]
sig 0x59FCF207FEA7F445 2022-12-28 [keybind]
Lasse's key for reference: pub rsa4096/0x38EE757D69184620 2010-10-24 [SC] [expires: 2025-02-07]
3690C240CE51B4670D30AD1C38EE757D69184620
uid Lasse Collin <lasse.collin@tukaani.org>
sig 0x38EE757D69184620 2024-01-08 [selfsig]
sig 0x59FCF207FEA7F445 2024-01-12 Jia Tan <jiat0218@gmail.com>
sub rsa4096/0x5923A9D358ADF744 2010-10-24 [E] [expires: 2025-02-07]
sig 0x38EE757D69184620 2024-01-08 [keybind]
GitHub suspended this project
> I am *not* a security researcher, nor a reverse engineer.
Could have fooled me - impressive write-up!
Github making suspect repository private and hiding recent account activity is wrong move and is interfering with citizens investigation efforts.
Going forward this will require more than a citizens investigation. Law enforcement will surely be granted access. Also, tarballs are still available in package managers if you really want to dig into the code.
It's a crime scene. It effectively has the "police" yellow tape around it.
I think the lesson here for packagers is that binary testdata should not be present while doing the build.
It is too easy to hide things in testdata.
Nice idea, but then you just hide the attack in logo.png that gets embedded in the binary. Less useful for libraries, works plenty good for web/desktop/mobile.
Mirror of the report, since the Openwall servers appear to be down.
https://web.archive.org/web/20240329182300/https://www.openw...
Debian is considering that their infrastructure may be compromised[1].
Looks like Arch Linux shipped both compromised versions - and 5.6.1-2 is out to hopefully resolve it.
5.6.1-2 is not an attempted fix, it's just some tweaks to Arch's own build script to improve reproducibility. Arch's build script ultimately delegates to the compromised build script unfortunately, but it also appears the payload itself is specifically targeting deb/RPM based distros, so a narrow miss for Arch here.
(EDIT: as others have pointed out, part of the exploit is in the artifact from libxz, which Arch is now avoiding by switching to building from a git checkout)
I upgraded Arch Linux on my server a few hours ago. Arch Linux does not fetch one of the compromised tarballs but builds from source and sshd does not link against liblzma on Arch.
[root@archlinux ~]# pacman -Qi xz | head -n2
Name : xz
Version : 5.6.1-2
[root@archlinux ~]# pacman -Qi openssh | head -n2
Name : openssh
Version : 9.7p1-1
[root@archlinux ~]# ldd $(which sshd) | grep liblzma
[root@archlinux ~]#
It seems that Arch Linux is not affected.The project has made an official post on the subject
https://archlinux.org/news/the-xz-package-has-been-backdoore...
The writeup indicates that the backdoor only gets applied when building for rpm or deb, so Arch probably would have been okay either way? Same with Nix, Homebrew, etc.
On arch, `ldd $(which sshd)` doesn't list lzma or xz, so I think it's unaffected? Obviously still not great to be shipping malicious code that just happens to not trigger.
Incredible. It's like discovering your colleague for 2 years at the secret nuclear weapon facility is a spy for another country, covering his tracks until the very last minute. Feels like a Hollywood movie is coming up.
Should we start doing background checks on all committers to such critical IT infrastructure?
But how? Let's say you're one of 10 maintainers of an open source project. A new user wants to contribute. What do you do? Do you ask them to send you some form of ID? Assuming this is legal and assuming you could ensure the new user is the actual owner of an actual, non counterfeit ID, what do you do? Do you vet people based on their nationality? If so, what nationality should be blackballed? Maybe 3 maintainers are American, 5 are European and 2 are Chinese. Who gets to decide? Or do you decide based on the company they work for?
Open source is, by definition, open. The PR/merge request process is generally meant to accept or refuse commits based on the content (which is why you have a diff), not on the owner.
Building consensus on which commits are actually valid, even in the face of malicious actors, is a notoriously difficult problem. Byzantine fault tolerance can be achieved with a 2/3 + 1 majority, but if anyone can create new identities and have them join the system (Sybil attack) you're going to have to do things differently.
Not even background check but a foreground check would already help. Like literally, who dis? any identity at all?
Too often maintainers who have no time just blanket approve PRs and see if stuff breaks.
@people who write github scanners for updates and security issues (dependabot and the like)
Can we start including a blacklist of emails and names of contributors (with reasons/links to discussions)?
I can't track them and I don't want them in my projects.
Might not be very helpful as it is easy to create new identities, but I see no reason to make it easier for them. Also, I might approach differently someone with lots of contributions to known projects than a new account, so it still helps.
It takes a minute to create a new email address. And you can change or fake an email address on a git commit trivially. You, too, can writing code as anyone you want by just doing "git commit --author='Joe Biden <icecream@whitehouse.gov>'". On the internet nobody knows you're Joe Biden.
You can write a rather simple GitHub action that would do that: look at a PR and reject / close it if you don't like it for some reason. AFAIK open-source projects have a free quota of actions.
OTOH sticking to the same email for more than one exploit might be not as wise for a malicious agent.
github already suspended the account
Github should probably remove the dopamine hits of green checkmarks etc. like in serious stock broker apps
They should also remove the emojis, there is no need to have people feel good about upvotes. I've long felt uncomfortable with emojis on Slack as well. Responding to a coding or infrastructure issue should not be a social activity, I respond because it's my job and if the issue is worth it, not because a human being should feel appreciated (either them or me).
There's good discussion of the timeline here: https://boehs.org/node/everything-i-know-about-the-xz-backdo...
> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.
It looks to be limited to Linux systems that are running certain patches. macOS and BSD seem unaffected?
FreeBSD is not affected as the payloads in question were stripped out, however we are looking into improvements to our workflow to further improve the import process.
Is the solution against such attacks in the future only to scrutinize more, or are there other reasonable options in terms of hardening?
The lesson here seems to not depend on tools written in languages that have complex, obscure build systems and no one is either able or interested to read. Using tools rewritten in Rust, Go or any other languege which resolves dependencies within project seems the only way to do hardening here.
Am I crazy thinking libraries shouldn't be able to provide _other libraries'_ symbols without the other libraries' "permission"? What am I missing?
> One portion of the backdoor is solely in the distributed tarballs. For easier reference, here's a link to debian's import of the tarball, but it is also present in the tarballs for 5.6.0 and 5.6.1:
Ubuntu 22.04 version:
dpkg -l |grep liblzma ii liblzma5:amd64 5.2.5-2ubuntu1 amd64 XZ-format compression library
Whew!
Is this a crime? Has anyone been prosecuted for adding a backdoor like this?
Has anyone been prosecuted for adding a backdoor
Google up Randal Schwartz. Caution: clickhole.
Kinda relevant, as I saw few comments about how safer languages are the solution.
Here[0] is a very simple example, that shows how easy such supply chain attacks are in Rust; and lets not forget that there was a very large python attack just a few days ago[1].
[0] - https://github.com/c-skills/rust1
[1] - https://checkmarx.com/blog/over-170k-users-affected-by-attac...
I am very concerned about Rust.
Rust’s “decision” to have a very slim standard library has advantages, but it severely amplifies some other issues. In Go, I have to pull in zero dependencies to make an HTTP request. In Rust, pulling reqwest pulls in at least 30 distinct packages (https://lib.rs/crates/reqwest). Date/time, “basic” base64, common hashing or checksums, etc, they all become supply chain vectors.
The Rust ecosystem’s collective refusal to land stable major versions is one of the amplifying issues. “Upgrade fatigue” hits me, at least. “Sure, upgrade ring to 0.17” (which is effectively the 16th major version). And because v0.X versions are usually incompatible, it’s not really possible to opt not to upgrade, because it only takes a short while before some other transitive dependency breaks because you are slow to upgrade. I recently spent a while writing my code to support running multiple versions of the `http` library, for example (which, to be fair, did just land version 1.0). My NATS library (https://lib.rs/crates/async-nats) is at version 34. My transitive base64 dependency is at version 22 (https://lib.rs/crates/base64).
This makes it nearly impossible for me to review these libraries and pin them, because if I pin foo@0.41.7, and bar needs foo@0.42.1, I just get both. bar can’t do =>0.41, because the point of the 0.X series is that it is not backwards compatible. It makes this process so time consuming that I expect people will either just stop (as if they did) reviewing their dependencies, or accept that they might have to reinvent everything from URL parsing to constructing http headers or doing CRC checks.
Combine this with a build- and compile-time system that allows completely arbitrary code execution, which is routinely just a wrapper for stuff like in the zx attack (look at a lot of the low-level libs you inevitably pull in). Sure, the build scripts and the macro system enables stuff like the amazing sqlx library, but said build and macro code is already so hard to read, it really takes proper wizardry to properly understand.
Keeps one wonder how many similar backdoors are there in the wild. What is the best way to execute such a move? This is sophisticated enough, but not good enough to stay unnoticed for a long while. If I were a state actor I'd think about at least 6-12 months.
Both https://github.com/tukaani-project members accounts have been suspended. (to see that, you can list the followers of each account).
Jai Tan's commit history on his github profile suggests he took off for Christmas, new years, and spring break. I smell an American.
Sometimes you smell an American because someone wanted you to smell an American.
Operating on a target region schedule doesn't seem particularly sophisticated, at least compared to the all the efforts put into this exploit.
Interesting. Is there also a pattern in the times of day? (I don't so much mean the times in commits done by the developer because they can be fake. I'd be more interested in authentic times recorded by GitHub, if any such times are publicly accessible.)
Another thing would be to examine everything ever written by the user for linguistic clues. This might point towards particular native languages or a particular variant of English or towards there being several different authors.
Quite ironic: The most recent commit in the git repo is "Simplify SECURITY.md", committed by the same Github account which added the backdoor.
https://github.com/tukaani-project/xz/commit/af071ef7702debe...
It's not ironic, this change is really sinister IMO. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.
This is exactly why I fight the windmills so hard when it comes automatic updates in Linux software.
So much damage is caused just by adding a single maintainer to a project - imagine how much power you would have to wield the remote execution systems put in place by naive developers for "automatic updates".
All it takes is a single malicious maintainer given access to the new version update of some popular user software, and they have a new botnet of thousands of devices at their disposal. Better yet, after the backdoor installation, they can just release the real update and cover their tracks forever.
Automatic updates are like running web applications, but without any sandboxing or protection usually implemented by the browser.
I hope mainstream news cover this so the general population can understand the issue with our software ecoysystems reliance on unpaid open-source maintainers
I worry the mainstream news take would just be "open source bad, microsoft closed source and google cloud good"
> Red Hat assigned this issue CVE-2024-3094.
Does that mean this affects RHEL and Fedora?
RHEL no, Fedora 41 and Rawhide yes.
https://www.redhat.com/en/blog/urgent-security-alert-fedora-...
https://lists.debian.org/debian-security-announce/2024/msg00...
RHEL won't get this bug for 2 years =)
Red Hat helps to do the job of making sure OSS has CVEs so there's common vernacular for the problem.
Given the recent ( not so recent ) attacks/"bugs" I feel there is a need to do more than the already hard task of investigating and detecting attacks but also to bring IRL consequences to these people.
My understanding is that right now it's pretty much a name and shame of people who most of the time aren't even real "people" but hostile agents either working for governments or criminal groups ( or both )
Getting punched in the face is actually a necessary human condition for a healthy civilization.
In the article it says CISA was notified - that sounds like it's going to be a federal investigation if nothing else. If I was this person, I wouldn't be in the USA (or any US friendly nation) ASAP.
> Getting punched in the face is actually a necessary human condition for a healthy civilization.
Aside from signed commits, we need to bring back GPG key parties and web of trust. When using a project you would know how many punches away from the committers you are.
> Getting punched in the face is actually a necessary human condition for a healthy civilization.
This is factually false - in fact, it's literally the direct opposite of the truth. "Getting punched in the face" is base violence that is incompatible with a healthy civilization. A good government with a robust justice system is what is actually needed for a healthy civilization.
> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.
The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded into their address space (even if you don't use systemd as PID 1), increasing the risks of supply chain attacks. Thanks, systemd.
That is all the protocol is. From https://www.freedesktop.org/software/systemd/man/latest/sd_n...:
> These functions send a single datagram with the state string as payload to the socket referenced in the $NOTIFY_SOCKET environment variable.
The simplest implementation (pseudocode, no error handling, not guaranteed to compile), is something like:
const char *addrstr = getenv("NOTIFY_SOCKET");
if (addrstr) {
int fd = socket(AF_UNIX, SOCK_DGRAM, 0);
struct sockaddr_un addr = { .sun_family = AF_UNIX };
strncpy(addr.sun_path, sizeof(addr.sun_path), addrstr);
connect(fd, (struct sockaddr*) &addr);
write(fd, "READY=1");
close(fd);
}
> The systemd notification protocol could have been as simple as just writing a newline to a pipe
It basically is. libsystemd links to liblzma for other features not related to notifications.
(The protocol is that systemd passes the path to a unix socket in the `NOTIFY_SOCKET` env variable, and the daemon writes "READY=1" into it.)
FWIW, I did a quick check on a Devuan system. The sshd in Devuan does link to a libsystemd stub - this is to cut down on their maintenance of upstream packages. However that stub does not link to lzma.
> so now security-critical daemons like openssh have additional dependencies like liblzma
Systemd itself seems security-critical to me. Would removing other dependencies on libsystemd really make a secure system where systemd was compromised through its library?
One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.
Yes, there are things delivered with that complexity. However, as an example, sysvinit is maybe, oh, 20k lines of code including binaries, heck including all core init scripts.
What's systemd? 2M lines? It was >1M lines 4+ years ago.
For an init system, a thing that is to be the core of stability, security, and most importantly glacial, stable change -- that is absurdly complex. It's exceedingly over engineered.
And so you get cases like this. And cases like that, and that over there, and that case over there too. All which could not exist, if systemd didn't try to overengineer, over complicate everything.
Ah well. I'm still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on), and just end up with systemd compatible service files.
But not yet. So... well, oh well.
Also thanks to Debian for modifying openssh.
Uh. systemd documents the protocol at various places and the protocol is trivial: a single text datagram sent to am AF_UNIX socket whose path you get via the NOTIFY_SOCKET. That's trivial to implement for any one with some basic unix programming knowledge. And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise. In particular non-C environments really should do their own native impl and not botjer wrapping libsystemd just for this.
But let me stress two other things:
Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam/pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack.
With that in mind libsystemd git dropped the dep on liblzma actually, all compressors are now dlopen deps and thus only pulled in when needed.
The notify protocol isn't much more complicated than that. From memory you send a string to a unix socket. I have written both systemd notify and listenfd in a few languages for little experiments and it is hard to imagine how the protocols could be simpler.
Looking at most popular projects these days they are a mass of dependencies and I think very few of them can be properly audited and verified by the projects that use them. Rust and Go might be more memory safe than C but look at the number of cargo or go modules in most projects. I have mostly stopped using node/npm on my systems.
Not a programmer, but couldn't the distribution's sshd patches for systemd (and all other distro patches for privileged daemons) use static includes? Wouldn't that have only pulled in the simple client-side communication API? Would that have defeated this vector? Would it be doable?
It's unfortunate that the anti-systemd party lost the war... years ago. But I don't blame systemd, Lennart Pottering or the fanboys (though it would have been so much better if the guy never worked in open source or wasn't such a prolific programmer). I blame Debian and its community for succumbing to this assault on Unix philosophy (again, years ago).
What? I don't get it? Isn't it on Debian if they modified the package to do something like this? Why would you blame systemd for maintainers doing something that upstream has never required or recommended?
xz is so pervasive, I just discovered on my Mac that the (affected?) version 5.6.1 made it into homebrew. The post in the linked article says that only Linux x86-64 systems are affected, but now I'm left scratching my head whether my Mac is also in trouble, just that we don't know it yet.
The two active maintainers seem to be: Lasse Collin <lasse.collin@tukaani.org> and Jia Tan <jiat0218@gmail.com>
Searching DDG for "jiat0218" I came across a blog post which I found weird. Seems to be dated: 2006-05-03
Blog post: "Kuso拍賣.有靈氣的筷子 - 闕小豪" <https://char.tw/blog/post/24397301>
Internet Archive link: <https://web.archive.org/web/20240329182713/https://char.tw/b...>
The contents of the page when translated seems to be about jiat0218 auctioning a pair of spiritual chopsticks as a prank.
The blog entry is basically a QA between jiat0218 and various other people about these chopsticks.
If Jia Tan does turn out to be a compromised maintainer working for a state actor then some of the content on the blog page can be viewed in a more sinister way (i.e. spycraft / hacks for sale etc.).
Example question 38:
Question 38
accounta066 (3): Are these chopsticks really that good? I kind of want to buy
them! But I recently sent money for online shopping but didn’t receive anything.
It’s very risky; currently jiat0218 you don’t have any reviews, you can
interview me. Do you want to hand it over?! … A sincere buyer will keep it.
Reply to
jiat0218 (4): First of all, I would like to express my condolences to you for
your unfortunate experience! What can I say about this kind of thing...My little
sister has always been trustworthy. What’s more, this is a pair of spiritual
chopsticks, so I hope to have a good one. It’s the beginning! As you can see,
my little sister is very careful and takes her time when answering your
questions. Except for the two messages that were accidentally deleted by her,
she always answers your questions. If this still doesn’t reassure you, then I
can only say that I still have room to work hard. You are still welcome
to bid... ^_^
Note however, it could all just be what it purports to be which is a prank auction of spiritual chopsticks.Something about this I found surprising is that Linux distros are pulling and packaging pre-built binaries from upstream projects. I'd have expected them to build from source.
Homebrew is currently shipping 5.6.1 (and was shipping 5.6.0 as well). Hopefully not affected on mac?
Well isn't this an interesting commit. He finished his inject macro to compose the payload at build, so now he can start clearing up the repo so none of that shit gets seen when cruising through it.
https://git.tukaani.org/?p=xz.git;a=commitdiff;h=4323bc3e0c1...
Everybody here In jumping into the pure malice bandwagon, I have a better hypothesis.
Abandonment and inaction, the actual developers of these tools are elsewhere, oblivious to this drama, trying to make living because most of the time you are not compensated nor any corporation cares about making things sustainable at all. This is the default status of everything your fancy cloud depends on underneath.
An attacker took over of the project slowly and stayed dormant until recently.
I'm really curious about if the act of injecting a backdoor into OSS software is legal/illegal ?
Are they somehow in the clear unless we can show they actively exploited it?
Oof, this is on my Sid laptop:
{0}[calvinow@mozart ~] dpkg-query -W liblzma5
liblzma5:amd64 5.6.0-0.2
{0}[calvinow@mozart ~] hexdump -ve '1/1 "%.2x"' /lib/x86_64-linux-gnu/liblzma.so.5 | grep -c f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
1
Glad I stopped running sshd on my laptop a long time ago... still probably going to reinstall :/Anyone have any idea what the code in the malicious liblzma_la-crc64-fast.o is actually doing? It's difficult to follow statically.
The `pack`[0] compression utility that reached the HN front page the other day[1] is setting off my alarm bells right now. (It was at the time too, but now doubly so)
It's written in Pascal, and the only (semi-)documented way to build it yourself is to use a graphical IDE, and pull in pre-compiled library binaries (stored in the git repo of a dependency which afaict Pack is the only dependent of - appears to be maintained by the same pseudonymous author but from a different account).
I've opened an issue[2] outlining my concerns. I'm certainly not accusing them of having backdoored binaries, but if I was setting up a project to be deliberately backdoorable, it'd look a lot like this.
[0] https://pack.ac/
We need to get these complex & bloated build-systems under control.
I'm not trying to troll, but I'm wondering if a distro like Gentoo is less susceptible to such attacks, since the source code feels more transparent with their approach. But then again, it seems that upstream was infected in this case, so I'm not sure if a culture of compiling from source locally would help.
I am not embarrassed to say... is there anything in there that someone who runs a server with ssh needs to know?
I literally can't make heads or tails of the risk here. All I see is the very alarming and scary words "backdoor" and "ssh server" in the same sentence.
If I am keeping stuff up to date, is there anything at all to worry about?
Is it time to deprecate the ability for code to implement linker symbols in other libraries? Shouldn't there be a strict namespace separation between binaries/libraries? liblzma being to implement openssh symbols seems like a symptom of a much larger problem.
Safety through obscurity and weirdness! If you disable ifunc, like any sensible person, this backdoor disables itself.
Why doesn’t GitHub force “releases” to be a simple repo tarball for sources and with binaries from GitHub actions or such…
I find it incredibly ironic that a “version control” site gives no assurance of reproducible builds (nor reproducible source!!)
The real villain is not the perpetrator, it is Microsoft, and it is all of us.
Really disappointed in the number of posters here who are playing down rushing to judgement and suggesting perhaps a legitimate developer was compromised, when it's very clear this is sophisticated and not the work of a single person.
I'm recalling bad memories of the Juniper backdoor years ago.
Whoever did this, was playing the long game. As the top post pointed out, there was an effort to get this into Fedora.... which eventually makes its way into RHEL (read: high value targets). This was not for short term payoffs by some rogue developer trying to mine crypto or other such nonsense. What you are seeing here is the planting of seeds for something months or a year down the road.
It doesn't really relate to this issue other than that both issues share a common source, but I wish we'd never fallen for xz.
I agree with the lzip guy
So, it's been almost 24 hours since I read this yesterday. Is it confirmed that Jia Tan is the perpetrator? do we know who he/she really is? Or are we going to live for the rest of our lives only knowing the pseudo name? just like Satoshi Nakamoto did to us. ;)
https://github.com/tukaani-project/tukaani-project.github.io... Does this mean anything that it changed to a parameter??
So much for a quiet Easter holiday. Fuck
There's a bug in the detection script. The line:
if [ "$path" == "" ]
should be
if [ "$path" = "" ]
Could anyone please tell me if current stable version of Debian has that backdoor or not?
Python for Windows bundles liblzma from this project, but it appears to be version 5.2.5 [0] vendored into the Python project's repo on 2022-04-18 [1], so that should be fine, right?
[0] https://github.com/python/cpython/blob/main/PCbuild/get_exte...
a user offered 5.6.0 and 5.4.5 in an issue to microsoft/vcpkg
5.4.5 can be compromised
Which nation state (if any) is most likely behind this? China based on name, or is this a red herring?
The perpetrator did most GitHub actions between 10 and 18 UTC, which sort of rules out US based, unless the messages were scheduled. Consistent with Europe to Asia.
See clickhouse for data: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...
What a disappointment.
It's something always in the back of our minds as developers using public libraries, but when something like this happens, non-developers that hear about it start to associate it with the rest of the open-source community.
It's essentially a terrorist attack on developer experience. Thankfully, management doesn't follow the same approach as the TSA.
Doesn't this call for criminal charges?
Hello,
Github just disabled the repo : https://github.com/tukaani-project/xz
Do someone have an up to date fork to see the project history ?
Is there any news concerning the payload analysis? Just curious to see if it can be correlated with something I have in my sshd logs (e.g. login attempt with specific RSA keys).
I think we have to assume that all community software is a target. The payoff for bad actors is too great.
For every one of these we spot, assume there are two we have not.
Now consider that your average Linux distribution pulls in tens of thousands of packages, each of which can be similarly compromised. Pretty scary to think about.
Also the attacker included in the 5.6.0 release the support for the long-awaited multi-threading decompression (and - broken - sandbox) making it very attractive to upgrade to...
It was probably a tactic to give a reason to upgrade. It's not always a fault for those who did or tried to do.
Is there a proper reverse engineering of the payload yet?
Anyone keeping current with OpenSUSE Tumbleweed got a update...downgrade. Prior to `zypper dup --no-allow-vendor-change` I had 5.6.0, now I'm at 5.4.6.
It was caught out of luck due to performance degradation. So nobody reads the code - not even once- prior to merging into upstream supply chain?
This is why the less the better... even if it means less comfortable... to a certain point obviously. And that includes SDKs...
On Ubuntu there is a bug report asking to sync the 5.6 version from Debian experimental https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2055...
Saw this on nix, which was using a compromised version in the unstable channel, I hope not too many systems are affected.
State actor or not, let's not ignore that the backdoor has been discovered thanks to the open nature of the projects involved that allowed digging into the code. Just another example like the infamous Borland InterBase backdoor in the early 2K that remained dormant for years and was discovered months after the source code has been released. If the xz malware authors worked for any corp that produced closed source drivers or blobs that can't be properly audited, we would be fucked; I just hope this is not already happening, because the attack surface in all those devices and appliances out there running closed code is huge.
Why are projects like xz and sshd still active? Just freeze it, it works fine. Only changes should be fixes for vulnerabilities. None of this complicated new functionality. If you want something like that make a new project. If it is truly better people will use it.
chmod u+x running detect_sh script just runs with no output on my arch linux box?
Interestingly on of the accounts that the GitHub account who introduced the backdoor follows was suspended very recently [1] who is also part of the org who runs XZ
It seems that to counter this type of supply chain attack, the best practices for managing software dependencies are to pin the version numbers of dependencies instead of using `latest`, and to use static linking instead of dynamic linking.
In the future: automated `diff` or any other A/B check to see whether or not the tarball matches the source repo (if not, auto-flag with a mismatch warning attribute), is that feasible to implement?
For someone who does not understand the packages used, could you please summarize in layman non technical terms. Thanks I did read the main post.
that's... creative. and patient. 11/10 concerning - now I'm wondering how many other projects could have shit like this in them or added right as I'm writing this shudder
Brain fart: would it be possible to attach passwords to a crypto based micro transaction such that every time you attempted a password entry your crypto account was charged a small fee for the login attempt?
This would thwart brute force attacks, but not be a significant cost for users. If you could attach your login to the crypto account it would mean the account would have to be funded to allow the attempt. The token wouldn't store passwords it would just be a gatekeeper to the login attempt.
The fees would be paid to the service providers as mining fees.
E.g. foo@bar.com needs a password and a token provided from a designated crypto address to gain access to the service.
Damn. I'm on macOS and use homebrew. To my surprise I had "xz" version 6.5.1 installed on my computer!
I ran "brew upgrade" and that downgraded to version 5.4.6.
xz is just a horribly designed format, and always has been. If you use it, please switch to Lzip. Same compression level, but designed by someone competent.
Looks like Jonathan Blow was right about open source.
Why is the Long Range Zip lrzip compression format not used? It gives better compression than xz when using the correct switches.
Why isn't he identified personally? Very likely he is 'contributing' to other projects under different accounts.
Maybe @JiaT75 got forced to do it. Maybe someone has more personal contact with him and can check how he is doing.
Is there already a list of distributions that included the affected versions in non-prereelase channels?
Surely the real target of this was Tor (which links liblzma) not random SSH servers.
Has this affected OpenBSD at all?
I wonder which browsers link liblzma and can this lead to https eavesdropping?
Fairly deep bugs for a Bazaar.
we should take this diagram and change "random person in nebraska" to "possibly a state-level attacker"
nice
Candidly how would someone protect against a vulnerability like this?
Which OS are affected by this compromise?? Is Ubuntu affected?
How that backdoor is triggered and what exactly it does?
Maybe it's finally time to start sunsetting LZMA and xz all together in favor of newer algorithms like Zstandard that also offer better performance but compression rates on par with LZMA.
Was Debian 12/stable unaffected? Only sid?
- * _ring ring_ * - "Hello?" - "It's Lasse Collin." - "Why are you collin me? Why not just use the backdoor?"
Please note: the changes have been made after GitHub has enforced 2FA (certainly not for "better security", but for promotion of FIDO2 and Windows Hello biometric impl of FIDO2, see https://codeberg.org/KOLANICH/Fuck-GuanTEEnomo for more info. Until recent times (for now access via git protocol is blocked for my acc, I guess based on lack of 2FA set up) it was even possible to push into all repos one has access by just using single-factor SSH key even without enabling 2FA in the account). As I have warned, nothing will protect when a backdoor is introduced by a malicious maintainer, or a "smart entrepreneur" who sold his project to a ad-company, or a loyal "patriot" living and earning money within reach of some state, or just a powerless man who got an offer he can't refuse. In general supply chain attacks by "legitimate" maintainers cannot be prevented. "Jia Tan" is just a sockpuppet to mitigate consequences to maintainers to make it look like they are not involved into it. They surely are. At least according to the current info it were they who have given the malicious account the permission to publish releases on behalf of the project and access to the repo.
IMHO all maintainers of the backdooored projects anyhow related to accepting the malicious changes should be considered as accomplices and boycotted. We don't need evidence of their liability, it is they who need to maintain their reputation. We are just free to take our decisions based on their reputation. Even if they were hacked themselves, it is not our problem, it is their problem. Our problem is to keep ourselves safe. It may feel "unjust" to ruin reputation of a person based on the fact he may be cheated or hacked… But if a person can be cheated or hacked, why should he/she have such a good reputation as everyone else?! So, it makes a lot of sense to just exclude and replace everyone, for whome there exists evidence of comprometation, no matter due to unconcern or malice. But FOSS is a doocracy serving products at dumpling prices ($0, free of charge), and for majority backdoored software is completely acceptable given that they get them free of charge. And powerful actors who can afford to pay for software will just hire devs to develop their private versions, while allowing the public to pay $0 for their free versions and use the backdoors placed into them themselves. In other words a complete market failure.
I think that 1. xz project must be shut down completely. I mean projects should stop using it as a dependency, exclude from distros, boycott it. LZMA algo was developed by Igor Pavlov in 7z project, but somehow it has happenned that liblzma was developed and maintained by unrelated folks. liblzma should be developed as a part of 7z project taking no code other than the trivial one for API compatibility adapter from xz. 2. Projects created by compromised authkrs should be boycotted. 3. Other projects touched by the compromised devs/maintainers should be audited. 4. All the projects using autotools should be audited and must replace autotools with cmake/meson. Autotools is a piece of shit, completely uncomprehensible. There is no surprise it was used to hude a backdoor - according to my experience in FOSS noone likes to touch its scripts anyhow. 5. No project should be built from releases. Project should be built from git directly. Implementing full support of SHA256 in git and git forges (GitHub, GitLab, Codeberg, sr.ht) should be accelerated to mitigate attacks using collisions to replace approved commits (I guess the randomness can be concealed from reviewer's eye in binary resource files, like pictures).
TLDR: Some people have been throwing around “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone.
These are my notes on time stamps/zones. There are a few interesting bits that I haven't fully fleshed out.
The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.
Observation 1: Time zone basic analysis
Here is the data on Jia’s time zone and the number of times he was recorded in that time zone:
3: + 0200 (in winter: February and November)
6: +0300 (in summer: in Jun, Jul, early October)
440: +0800
1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert.
2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March).
Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…
Observation 2: Time zone inconsistencies
Let’s analyze the few times where Jia was recorded in a non +800 time zone. Here, we notice that there are some situations where Jia switches between +800 and +300/+200 in a seemingly implausible time. Indicating that perhaps he is not actually in +800 CST time, as his profile would like us to believe.
Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but flight from China to anywhere in Eastern Europe is at a min 10 hours
Jia Tan Thu, 5 May 2022 20:53:42 +0800
Jia Tan Sat, 19 Nov 2022 23:18:04 +0800
Jia Tan Mon, 7 Nov 2022 16:24:14 +0200
Jia Tan Sun, 23 Oct 2022 21:01:08 +0800
Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1)
Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1)
Jia Tan Wed, 5 Oct 2022 23:54:12 +0800
Jia Tan Wed, 5 Oct 2022 20:57:16 +0800
—> again, given the flight time, this is even more impossible
Jia Tan Fri, 2 Sep 2022 20:18:55 +0800
Jia Tan Thu, 8 Sep 2022 15:07:00 +0300
Jia Tan Mon, 25 Jul 2022 18:30:05 +0300
Jia Tan Mon, 25 Jul 2022 18:20:01 +0300
Jia Tan Fri, 1 Jul 2022 21:19:26 +0800
Jia Tan Thu, 16 Jun 2022 17:32:19 +0300
Jia Tan Mon, 13 Jun 2022 20:27:03 +0800
—> the ordering of these time stamps, and the switching back and forth looks strange.
Jia Tan Thu, 15 Feb 2024 22:26:43 +0800
Jia Tan Thu, 15 Feb 2024 01:53:40 +0800
Jia Tan Mon, 12 Feb 2024 17:09:10 +0200
Jia Tan Mon, 12 Feb 2024 17:09:10 +0200
Jia Tan Tue, 13 Feb 2024 22:38:58 +0800
—> this travel time is possible, but the duration of stay is unlikely
Observation 3: Strange record of time stamps It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.
Observation 4: Bank holiday inconsistencies
We notice that Jia’s work schedule and holidays seem to align much better with an Eastern European than a Chinese person.
Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bak holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)
Chinese bank holidays (just looking at 2023):
- Working on 2023, 29 September: Mid Autumn Festival
- Working on 2023, 05 April: Tomb Sweeping Day
- Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year
Eastern European holidays:
- Never working on Dec 25: Christmas (for many EET countries)
- Never working Dec 31 or Jan 1: New Years
Observation 5: No weekend work —> salary job?
The most common working days for Jia was Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to be EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night.
These times also line up well with Hans Jansen and Lasse Collin.
I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.
Thank the gods I didn't plan on having a life this weekend
Is this sev0?
Hello everybody.
I am taking the initiative to gather more information regarding the possible precursors and perpetrators of the backdoor.
The purpose of this commentary is focused on open source information (OSINT).
I am not a judge of anyone or any action that may occur, the objective of this comment is to help through accurate and quick information to help the core developers of the affected packages and consequently the Linux kernel (which may have been indirectly or directly affected) take action necessary in relation to the fact that occurred.
NOTE: This comment will always have "edit" so always review it for information.
Information I have so far.
Summary: 1. GitHub Account Suspension: - The accounts of @JiaT75 and @Larhzu were suspended by GitHub. - All Tukaani repositories, including downloads, were disabled. - Investigate the cause of the account suspensions and whether there is any correlation with suspicious activities.
2. Possible Backdoor in xz/liblzma: - There are concerns about the presence of a backdoor in xz/liblzma. - Investigate whether there is evidence of compromise in the source code and recent updates. - Examine potential impacts, especially if the software is used in critical systems.
3. Updates and Patches in Packages: - Note recent updates in packages such as MinGW w64, pacman-static, Alpine, and OpenSUSE. - Review changelogs to understand if these updates are related to security fixes.
4. Jia's Activities on Platforms and Projects: - Investigate Jia's contributions to different projects and platforms, such as Arch Linux, Alpine Linux, and OpenSUSE. - Check for correlations between Jia's activities and reported security issues.
5. Libera Registration Information: - Analyze Jia's registration details on Libera to determine the timeline of their online activities. - Consider correlating this information with other online activities of Jia.
6. VPN Usage: - Confirm Jia's use of VPN and assess its impact on security investigations. - Explore possible reasons for using a VPN and how it may affect the identification and tracking of online activities.
Links related to user JiaT75 [xz] Remove JiaT75 as a contact, determine correct contacts #11760 - Google/oss-fuzz https://github.com/google/oss-fuzz/issues/11760
Tuktest index hash #7 - tukaani-project/xz/pull/7 https://web.archive.org/web/20240329230522/https://github.co...
Time for another OS wipe. Glad I keep bleeding edge versions VMd
is this sev0?
Jesus! Does anyone know if Debian stable is affected?
now I wonder which browsers link liblzma?
Notes on time stamps and time zones.
A few interesting bits that I haven't fully fleshed out. TLDR: Some people have been throwing around that Jia is from “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone.
The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.
Observation 1: Time zone basic analysis
Here is the data on Jia’s time zone and the number of times he was recorded in that time zone: 3: + 0200 (in winter: February and November) 6: +0300 (in summer: in Jun, Jul, early October) 440: +0800
1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert. 2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March). 1. Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…
Observation 2: Time zone inconsistencies
Let’s analyze the few times where Jia was recorded in a non +800 time zone. Here, we notice that there are some situations where Jia switches between +800 and +300/+200 in a seemingly implausible time. Indicating that perhaps he is not actually in +800 CST time, as his profile would like us to believe.
Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but a flight from China to anywhere in Eastern Europe is at a min 10 hours
Jia Tan Thu, 5 May 2022 20:53:42 +0800 Jia Tan Sat, 19 Nov 2022 23:18:04 +0800 Jia Tan Mon, 7 Nov 2022 16:24:14 +0200 Jia Tan Sun, 23 Oct 2022 21:01:08 +0800 Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1) Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1) Jia Tan Wed, 5 Oct 2022 23:54:12 +0800 Jia Tan Wed, 5 Oct 2022 20:57:16 +0800 —> again, given the flight time, this is even more impossible
Jia Tan Fri, 2 Sep 2022 20:18:55 +0800 Jia Tan Thu, 8 Sep 2022 15:07:00 +0300 Jia Tan Mon, 25 Jul 2022 18:30:05 +0300 Jia Tan Mon, 25 Jul 2022 18:20:01 +0300 Jia Tan Fri, 1 Jul 2022 21:19:26 +0800 Jia Tan Thu, 16 Jun 2022 17:32:19 +0300 Jia Tan Mon, 13 Jun 2022 20:27:03 +0800 —> the ordering of these time stamps and the switching back and forth between time zones looks strange.
Jia Tan Thu, 15 Feb 2024 22:26:43 +0800 Jia Tan Thu, 15 Feb 2024 01:53:40 +0800 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Tue, 13 Feb 2024 22:38:58 +0800 —> this travel time is possible, but the duration of stay is unlikely
Observation 3: Strange record of time stamps
It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.
Observation 4: Bank holiday inconsistencies
We notice that Jia’s work schedule and holidays seems to align much better with an Eastern European than a Chinese person.
Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bank holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)
Chinese bank holidays (just looking at 2023): - Working on 2023, 29 September: Mid Autumn Festival - Working on 2023, 05 April: Tomb Sweeping Day - Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year
Eastern European holidays: - Never working on Dec 25: Christmas (for many EET countries) - Never working Dec 31 or Jan 1: New Years
Observation 5: Little weekend work —> salary job?
The most common working days for Jia were Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night.
These times also line up well with Hans Jansen and Lasse Collin.
I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.
Another interesting data point: about 2 years ago there was a clear pressure campaign to name a new maintainer: https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h...
At the time I thought it was just rude, but maybe this is when it all started.
Wait, I'm on mobile. Did this partially slip by because of the ABSURD PRACTICE of publishing release.tarballs that do not 1:1 correspond with source?
Let me guess, autotools? I want to rage shit post but I guess I'll wait for confirmation first.
EDIT: YUP, AT LEAST PARTIALLY. Fucking god damn autotools.
I think its much more likely this was not a bad actor, given their long history of commits.
It's a known fact that China will "recruit" people to operate them. A quote:
> They talk to them, say my friend, I see you like our special menu. Are you from China? Are you here on a VISA? Do you have family back there? Would you like your family to stay alive? Is your loyalty to this temporary employer or is your loyalty to your motherland? You know, a whole bunch of stuff like that. That’s how Chinese intelligence operations acts...
This just gives feelings of less "compromised account" and more "Your account is now our account"
Yikes! Do you have any info on the individual's background or possible motivations?
Tukaani website states "jiatan" as the nickname of the malicious code committer on Libera Chat.
WHOWAS jiatan provided me the following information:
jiatan ~jiatan 185.128.24.163 * :Jia Tan jiatan 185.128.24.163 :actually using host jiatan jiatan :was logged in as jiatan tungsten.libera.chat :Fri Mar 14:47:40 2024
WHOIS yields nothing, the user is not present on the network at the moment.
Given that 185.128.24.163 is covered with a range-block on the English Wikipedia, it appears this is a proxy.
[dead]
[dead]
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
can someone ELI5 ?
pRoBaBlY a StaTe AcToR
zero definition of what that means...
egos of people who just like to say cool words they don't understand
lol
this comment will probably get deleted, but let the action of this comment being deleted stand that in 2024 we're all allowed to use big words with no definition of what they mean -> bad
state actor? who? what motive? what country? all comments involving "state actor" are very broad and strange... i would like people to stop using words that have no meaning, as it really takes away from the overall conversation of what is going on.
i mean you're seriously going to say "state actor playing the long game" to what end? the issue was resolved in 2 hours... this is stupid
It's always Debian, like last time when they removed RNG randomness from ssh because of a warning.
This is why we never upgrade software versions. I’ve been asked by our customers why we use such an old AMI version. This is why.
Waiting for the new YouTube videos on this. "Woah! Linux has a back door dudes!". My distribution, Ubuntu (now Kubuntu) 2022 isn't affected.
I guess that rewriting liblzma in Rust would not have prevented this backdoor. But would have likely increased the confidence in its safety.
Using the build system (and potentially the compiler) to insert malicious backdoors is far from a new idea, and I don't see why this example would the only case.
Pretty much proof that OSS != automatically more secure. And proof that OSS projects can get backdoored. See this for more ideas on this issue: https://seirdy.one/posts/2022/02/02/floss-security/
"Lasse Collin," as other posters here have found, does not seem to exist as an experienced coder. Oddly, there is a Swedish jazz musician named Lasse Collin, which would otherwise be one of those names, especially the last name, that would stick out. Instead it is buried under a lot of mentions of a musician.