GitHub has suspended @JiaT75's account.

EDIT: Lasse Collin's account @Larhzu has also been suspended.

EDIT: Github has disabled all Tukaani repositories, including downloads from the releases page.

--

EDIT: Just did a bit of poking. xz-embedded was touched by Jia as well and it appears to be used in the linux kernel. I did quick look and it doesn't appear Jia touched anything of interest in there. I also checked the previous mirror at the tukaani project website, and nothing was out of place other than lagging a few commits behind:

https://gist.github.com/Qix-/f1a1b9a933e8847f56103bc14783ab7...

--

Here's a mailing list message from them ca. 2022.

https://listor.tp-sv.se/pipermail/tp-sv_listor.tp-sv.se/2022...

--

MinGW w64 on AUR was last published by Jia on Feb 29: https://aur.archlinux.org/cgit/aur.git/log/?h=mingw-w64-xz (found by searching their public key: 22D465F2B4C173803B20C6DE59FCF207FEA7F445)

--

pacman-static on AUR still lists their public key as a contributor, xz was last updated to 5.4.5 on 17-11-2023: https://aur.archlinux.org/cgit/aur.git/?h=pacman-static

EDIT: I've emailed the maintainer to have the key removed.

--

Alpine was patched as of 6 hours ago.

https://git.alpinelinux.org/aports/commit/?id=982d2c6bcbbb57...

--

OpenSUSE is still listing Jia's public key: https://sources.suse.com/SUSE:SLE-15-SP6:GA/xz/576e550c49a36... (cross-ref with https://web.archive.org/web/20240329235153/https://tukaani.o...)

EDIT: Spoke with some folks in the package channel on libera, seems to be a non-issue. It is not used as attestation nor an ACL.

--

Arch appears to still list Jia as an approved publisher, if I'm understanding this page correctly.

https://gitlab.archlinux.org/archlinux/packaging/packages/xz...

EDIT: Just sent an email to the last committer to bring it to their attention.

EDIT: It's been removed.

--

jiatan's Libera info indicates they registered on Dec 12 13:43:12 2022 with no timezone information.

    -NickServ- Information on jiatan (account jiatan):
    -NickServ- Registered : Dec 12 13:43:12 2022 +0000 (1y 15w 3d ago)
    -NickServ- Last seen : (less than two weeks ago)
    -NickServ- User seen : (less than two weeks ago)
    -NickServ- Flags : HideMail, Private
    -NickServ- jiatan has enabled nick protection
    -NickServ- *** End of Info ***

They are not registered on freenode.

EDIT: Libera has stated they have not received any requests for information from any agencies as of yet (30th Saturday March 2024 00:39:31 UTC).

EDIT: Jia Tan was using a VPN to connect; that's all I'll be sharing here.

I think this has been in the making for almost a year. The whole ifunc infrastructure was added in June 2023 by Hans Jansen and Jia Tan. The initial patch is "authored by" Lasse Collin in the git metadata, but the code actually came from Hans Jansen: https://github.com/tukaani-project/xz/commit/ee44863ae88e377...

> Thanks to Hans Jansen for the original patch.

https://github.com/tukaani-project/xz/pull/53

There were a ton of patches by these two subsequently because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizers. Subsequently the configure script was modified multiple times to detect the use of sanitizers and abort the build unless either the sanitizer was disabled or the use of ifuncs was disabled. That would've masked the payload in many testing and debugging environments.

The hansjans162 Github account was created in 2023 and the only thing it did was add this code to liblzma. The same name later applied to do a NMU at Debian for the vulnerable version. Another "<name><number>" account (which only appears here, once) then pops up and asks for the vulnerable version to be imported: https://www.mail-archive.com/search?l=debian-bugs-dist@lists...

Yesterday sure was fun wasn't it :p Thanks for all your help/working with me on getting this cleaned up in Fedora.

because of it's "great new features"

"great" for whom? I've seen enough of the industry to immediately feel suspicious when someone uses that sort of phrasing in an attempt to persuade me. It's no different from claiming a "better experience" or similar.

Interesting that one of the commits commented on update of the test file that it was for better reproducibility for having been generated by a fixed random seed (although how goes unmentioned). For the future, random test data better be generated as part of the build, rather than being committed as opaque blobs...

Debian have reverted xz-utils (in unstable) to 5.4.5 – actual version string is “5.6.1+really5.4.5-1”. So presumably that version's safe; we shall see…

After reading the original post by Andres Freund, https://www.openwall.com/lists/oss-security/2024/03/29/4, his analysis indicates that the RSA_public_decrypt function is being redirected to the malware code. Since RSA_public_decrypt is only used in the context of RSA public key - private key authentication, can we reasonably conclude that the backdoor does not affect username-password authentication?

I’m surprised there isn’t way more of this stuff. The supply chain is so huge and therefore represents so much surface area.

Github accounts of both xz maintainers have been suspended.

Do you know if it was actually the commit author, of if their commit access was compromised?

Nice. I worked on a Linux disto when I was a wee lad and all we did was compute a new md5 and ship it.

Name and shame this author. They should never be allowed anywhere near any open projects ever again.

Can legal action be taken against the author if it's found he maliciously added the backdoor?

It is not good to take into consideration something with any unreadable text instead of the open text of the programme. It should be excluded.

I wonder who the target was!

his account is active again on github https://github.com/JiaT75

Sleeper.

[dead]

[flagged]

[flagged]

[flagged]

the account was either sold or stolen

Security Researchers: Is this request-for-private-disclosure + "90-days before public" reasonable?

It's a SEVERE issue, to my mind, and 90 days seems too long to me.

90 day dark window for maintainers is SOP though. Then after 90 days, it’s free game for public disclosure

How many of people like this one exist?

Honestly it seems like a state-based actor hoping to get whatever high value target compromised before it's made public. Reporting privately buys them more time, and allows them to let handlers know when the jig is up.

This is getting addressed here: https://github.com/google/oss-fuzz/issues/11760

This in of itself can be legitimate. ifunc has real uses and it indeed does not work when sanitizer is enabled. Similar change in llvm: https://github.com/llvm/llvm-project/commit/1ef3de6b09f6b21a...

and that was in mid 2023. Very funny that Wikipedia on this issue says

> It is unknown whether this backdoor was intentionally placed by a maintainer or whether a maintainer was compromised

Yeah, if you've been compromised for a year your attacker is now your identity. Can't just wave hands, practice infosec hygiene

> I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).

Yeah, I've been banging on that same drum for ages too... for example on this very site a decade ago: https://news.ycombinator.com/item?id=7213563

I'm honestly surprised that this autoconf vector hasn't happened more often... or more often that we know of.

Every single commit this person ever did should immediately be rolled back in all projects.

> they have commits on some related projects like libarchive

Windows started using libarchive to support .rar, .7z, ...

https://arstechnica.com/gadgets/2023/05/cancel-your-winrar-t...

Couldn't the autoconf soup be generated from simpler inputs by the CI/CD system to avoid this kind of problem? Incomprehensible soup as a build artifact (e.g. executables) is perfectly normal, but it seems to me that such things don't belong in the source code.

(This means you too, gradle-wrapper! And your generated wrapper for your generated wrapper. That junk is not source code and doesn't belong in the repo.)

Pure speculation but my guess is a specific state actor ahem is looking for developers innocently working with open source to then strongarm them into doing stuff like this.

I would be curious if their commits could be analyzed for patterns that could then be used to detect commits from their other account

I mean, a backdoor at this scale (particularly if it wasn't noticed for a while and got into stable distros) could be worth millions. Maybe hundreds of millions (think of the insider trading possibilities alone, not to mention espionage). 2 years doesn't seem like that much work relative to the potential pay off.

This is the sort of case where america's over the top hacking laws make sense.

> I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup

If I recall correctly, xz can be built with both autoconf and cmake, are cmake configs similarly affected?

How about wheels in the python ecosystem

Yeah this was my first thought too. Though I think the case against autoconf is already so overwhelming I think anyone still using it is just irredeemable; this isn't going to persuade them.

Focusing on sshd is the wrong approach. The backdoor was in liblzma5. It was discovered to attack sshd, but it very likely had other targets as well. The payload hasn't been analyzed yet, but _almost everything_ links to libzma5. Firefox and Chromium do. Keepassxc does. And it might have made arbitrary changes to your system, so installing the security update might not remove the backdoor.

Ubuntu still ships 5.4.5 on 24.03 (atm).

I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though.

So for 5.4.5 the tagged release and download on github differ.

It does change format strings, e.g.

   +#: src/xz/args.c:735
   +#, fuzzy
   +#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout"
   +msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout"
   +msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard"

[Edit] formatting

> Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.

I'm surprised .deb doesn't have a better approach. RPM has epoch for this purpose http://novosial.org/rpm/epoch/index.html

> If you're not on a rolling release distro, your version is probably older.

Ironic considering security is often advertised as a feature of rolling release distros. I suppose in most instances it does provide better security, but there are some advantages to Debian's approach (stable Debian, that is).

The article gives a link to a simple shell script that detects the signature of the compromised function.

> Running OpenSSH sshd from systemd

I think this is irrelevant.

From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.

Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related?

I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?

My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.

$ dpkg-query -W liblzma5

liblzma5:amd64 5.4.1-0.2

Tumbleweed has a package: liblzma5-5.6.1.revertto5.4-3.2.x86_64 FYI

In a movie his mental health issues would likely have been caused intentionally by the attacker, setting the stage for the mole to offer to step in just at the right time. Seems a bit far fetched in this case though for what looks like a tangential attack.

Lasse appears to be active and working on undoing the sabotage. https://git.tukaani.org/?p=xz.git;a=blobdiff;f=CMakeLists.tx...

He came on IRC, he seemed ok. He did some cleanup of access and signed off for easter.

I would like to see more attention given to this. I'm capable of compartmentalization and not over-guilting myself, but holy hell, I really hope he's doing alright. This would kind of destroy me.

I was actually telling my dad about this. I have a project, 500+ users, not quite root access, but enough to cause serious damage. I can think of at least one covert way to backdoor the binary artifacts from it.

About two years ago, someone showed up, started making good commits. In this case, they have some other community rep that goes back a bit further but... man it's an unsettling feeling.

Relevant xkcd:

https://xkcd.com/2347/

As a bit of an aside, I would never accept a PR like this, and would always update $large_vendored_dependency myself. This is unreviewable, and trivial to insert any backdoor (unless you go through the motions of updating it yourself and diffing, at which point the PR becomes superfluous). I'd be wary even from a well-known author unless I knew them personally on some level (real-life or via internet). Not that I wouldn't trust them, but people's machines or accounts can get compromised, people can have psychotic episodes, things like that. At the very least I'd like to have some out-of-band "is this really you?" signal.

This is how I once inserted a joke in one of our (private) repos that would randomly send cryptic messages to our chat channel. This was pretty harmless and just a joke (there's some context that made it funny), but it took them years to find it – and that was only because I told them after I quit.

That said, looking at the GitHub account I'd be surprised if there's anything nefarious going on here. Probably just someone using your repo, seeing it's outdated, and updating it.

Hey all, I’m the author of that PR. Just posted to Github with additional context: https://github.com/jamespfennell/xz/pull/2#issuecomment-2027...

I don't want to read too much into it, but the person (supposedly) submitting the PR seems to work at 1Password since December last year, as per his Linkedin. (And his Linkedin page has a link to the Github profile that made the PR).

> it's not the same GitHub account as upstream

This is valuable information, and a sign that this may be the tip of an iceberg.

There was also a bug report in Debian which requested updating xz-utils to 5.6.1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

IMO your prior on this should be that it's most likely just someone innocently updating a dependency.

The backdoor (test binary blob and autoconf) is not part of the pull request.

Suddenly anything like that becomes super suspicious.

I wonder how this will affect the OS community in general.

Internet detectives at work in this thread!

FWIW, it felt intimidating as hell. And I'm fairly established professionally. Not sure what I'd have done earlier in my career (although I'd probably not have found it in the first place).

For what it's worth the author is a PostgreSQL committer, he's not a security researcher but he's a pretty damn good engineer!

Honestly, you only get this kind of humility when you're working with absolute wizards on a consistent basis. That's how I read that whole analysis. Absolutely fascinating.

OMG: look at the other contributions. He is trying to take over projects and pushing some change to sha256 in a hundred projects.

Example: https://github.com/duesee/imap-flow/issues/96

Hey, I remember this guy! Buddy of someone who tried to get a bunch of low quality stuff into ifupdown-ng, including copying code with an incompatible license and removing the notice. He's in every PR, complaining the "project is dead". He even pushes for the account to be made "team member".

https://github.com/ifupdown-ng/ifupdown-ng/pulls/easynetdev

He follows 54k accounts though, so it may indeed just be coincidence.

Dear @0xthr0w4, do you attack me because I have requested the XZ update?

Do not mix, I am not linked to the XZ project.

A backdoored xz could also run payloads hidden inside other xz files, allowing targeted attacks.

The same authors have also contributed to Zstd

deb packages are xz compressed...

And, Joey Hess has counted at least 750 commits to xz from that handle.

https://hachyderm.io/@joeyh/112180715824680521

This does not look trust-inspiring. If the code is complex, there could be many more exploits hiding.

I imagine it might be easier to just compromise a weakly protected account than to actual put in a 2 years long effort with real contributions. If we mandated MFA for all contributors who contribute to these really important projects then we can know with greater certainty if it was really a long con vs. a recently compromised account.

Probably a state actor. You can look far into the future when you’re working for the party.

(I detached this subthread from https://news.ycombinator.com/item?id=39866275, for the sake of pruning the top heavy thread.)

More likely that the account of that dev was breawched, dont you think ?

Warning, drunk brain talking. But a LLM driven email based "collaborator" could play a very long gMw adding basic features to a code made whilst earning trust backed by a generated online presence. My money is on a resurgance in the Web of Trust.

Or if you can't stand the lkml.org UI:

https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/

I've analysed the backdoor myself and it's very sophisticated, not poorly made at all. The performance problem is surprising in this context, but I think next time they won't make that mistake.

So many malicious actors have been caught because they accidentally created a mild annoyance for someone that went on to bird-dog the problem.

You must mean, "Makes you wonder what more competent actors are doing"

s/can do/have done/

This is extremely suspicious.

It looks like someone may have noticed a unmaintained or lightly maintained project related to various things, and moved to take control of it.

Otherwhere in the discussion here someone mentions the domain details changed; if you have control of the domain you have control of all emails associated with it.

Also interesting, to me, how the GMail account for the backdoor contributor ONLY appears in the context of "XZ" discussions. Google their email address. Suggests a kind of focus, to me, and a lack of reality / genuineness.

those pipe usages are quite suspicious

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...

pipeing into this shell script which now uses "eval"

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...

i guess this will be revisited and removed soon

"started to cc himself" seems to be simply "contributing to a new project and not having git-send-email fully set up". By default git-send-email Cc the sender, though in practice it's one of the first options one changes.

Thanks to autoconf, we're now used to build scripts looking like gibberish. A perfect place to hide a backdoor.

It looks like an earlier commit with a binary blob "test data" contained the bulk of the backdoor, then the configure script enabled it, and then later commits patched up valgrind errors caused by the backdoor. See the commit links in the "Compromised Repository" section.

Also, seems like the same user who made these changes are still submitting changes to various repositories as of a few days ago. Maybe these projects need to temporarily stop accepting commits until further review is done?

The use of "eval" stands out, or at least it should stand out – but there are two more instances of it in the same script, which presumably are not used maliciously.

A while back there was a discussion[0] of an arbitrary code execution vulnerability in exiftool which was also the result of "eval".

Avoiding casual use of this overpowered footgun might make it easier to spot malicious backdoors. Usually there is a better way to do it in almost all cases where people feel the need to reach for "eval", unless the feature you're implementing really is "take a piece of arbitrary code from the user and execute it".

[0] https://news.ycombinator.com/item?id=39154825

Yeah, now imagine they succeeded and it didn't cause any performance issues...

Can we even be sure no such successful attempt has already been made?

A big part of the problem is all the tooling around git (like the default github UI) which hides diffs for binary files like these pseudo-"test" files. Makes them an ideal place to hide exploit data since comparatively few people would bother opening a hex editor manually.

> "Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the "fixes" mentioned above."

Crazy indeed.

Paid people get burnt out as well and they are just as likely to accept free help as an unpaid person.

> So when are we going to stop pretending ...

I'm not sure that we are. Doesn't everybody know that developing/maintaining free software is largely thankless work, with little to no direct recompense?

I don't think moving towards unfree software is a good way to make free software more secure. It shouldn't be a surprise that proprietary software is less likely to be exploited in this way simply because they don't accept any patches from outside of the team. What you want is more people that understand and care about free software and low barriers to getting involved.

OSS maintainers aren't reaping anything. Most OSS licenses say the software is provided without warranty.

Wow, that's a lot of anonymous accounts adding comments there urging for a fast merge!

And this "Hans Jansen" guy is apparently running around salsa.debian.org pushing for more updates in other projects as well: https://salsa.debian.org/users/hjansen/activity

That name jumped out at me, Hans Jansen is the name Dominic Monaghan used when posing as a German interviewer with Elijah Woods. Not that it can't be a real person

https://youtu.be/IfhMILe8C84

For anyone else feeling some deja vu about ifunc / Valgrind errors, this Red Hat issue [1] was previously linked from HN 12 days ago [2].

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2267598

[2]: https://news.ycombinator.com/item?id=39733185

I get the feeling that a number of the comments are all the same person / group.

> I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.

Which is why there's probably nothing remotely interesting in them logs.

Nah. I'm sure Microsoft got a call from the alphabet boys and nobody, not even internal employees are allowed to look at the logs right now.

Oh my, another reason not to use github. :D So many reasons poping up just in this comment section alone.

I'm guessing Microsoft just got a call from the Government telling them not to look too deeply into it.

That’d be illegal for an employee to do.

Btw, this is not the only project providing a source tarball different from the git repo, for example libusb also does this (and probably others):

- https://github.com/libusb/libusb/issues/1468#issuecomment-19...

- https://github.com/orgs/community/discussions/6003

For a long time, there was one legitimately annoying disadvantage to the git-generated tarballs though - they lost tagging information. However, since git 2.32 (released June 2021; presumably available on GitHub by August 2021 when they blogged about it) you can use `$Format:%(describe)$` ... limited to once per repository for performance reasons.

GitHub/Microsoft likely has a backup. I’d be getting those out about now.

Bad move. Destroying evidence is a felony.

That man deserves a Nobel Prize

1. Everything must be visible. A diff between the release tarball and tag should be unacceptable. It was hidden from the eyes to begin with.

2. Build systems should be simple and obvious. Potentially not even code. The inclusion was well hidden.

3. This was caught through runtime inspection. It should be possible to halt any Linux system at runtime, load debug symbols and map _everything_ back to the source code. If something can't map back then regard it as a potentially malicious blackbox.

There has been a strong focus and joint effort to make distributions reproducible. What we haven't managed though is prove that the project compromises only of freshly compiled content. Sorta like a build time / runtime "libre" proof.

This should exist for good debugging anyway.

It wouldn't hinder source code based backdoors or malicious vulnerable code. But it would detect a backdoor like this one.

Just an initial thought though, and probably hard to do, but not impossibly hard, especially for a default server environment.

Build-related fixes are only treating the symptoms, not the disease. The real fix would be better sandboxing and capability-based security[1] built into major OSes which make backdoors a lot less useful. Why does a compression library have the ability to "install an audit hook into the dynamic linker" or anything else that isn't compressing data? No amount of SBOMs, reproducible builds, code signing, or banning binaries will change the fact that one mistake anywhere in the stack has a huge blast radius.

[1]: https://en.wikipedia.org/wiki/Capability-based_security

Note that the malicious binary is fairly long and complex.

This attack can be stopped by disallowing any binary testdata or other non-source code to be on the build machines during a build.

You could imagine a simple process which checks out the code, then runs some kind of entropy checker over the code to check it is all unminified and uncompressed source code, before finally kicking off the build process.

autogenerated files would also not be allowed to be in the source repo - they're too long and could easily hide bad stuff. Instead the build process should generate the file during the build.

We should be able to produce a tar and a proof that tar was produced from a specific source code.

Quote from the article:

    That line is not in the upstream source of build-to-host, nor is build-to-host used by xz in git.

The verifiers can then easily check that the signature and indeed be convinced that the code was executed as it is claimed and source code wasn't tampered with.

The advantage of Zero-Knowledge technology in this case is that one doesn't need to repeat the computational steps themselves nor rely on a trusted party to do it for them (like automated build - that can also be compromised by the state actors). Just having the proof solves this trust problem mathematically: if you have the proof & the tar, you can quickly check source code that produced the tar wasn't modified.

The Guix full source bootstrap is looking less paranoid as time goes on

More reproducible builds, maybe even across distributions? Builds based on specific commits (no tarballs like in this case), possibly signed (just for attribution, not for security per se)? Allow fewer unsafe/runtime modifications The way oss-fuzz ASAN was disabled should've been a warning on its own, if these issues weren't so common.

I'm not aware of any efforts towards it, but libraries should also probably be more confined to only provide intended functionality without being able to hook elsewhere?

It may not have really mattered much for NixOS:

> b) argv[0] needs to be /usr/sbin/sshd

For once, the lack of FHS interoperability is a benefit, if only on accident.

Note that NixOS has a unique advantage in that `dlopen` is easier to analyze, but you do have to check for it. A lot of people are looking only at `ldd` and missing that they can be vulnerable at runtime.

Not affected by the latest CVE, but the author had unrestricted access to xz for 2 years, so I would say it is affected until the other contributions are proven safe (never gonna happen) or it reverts to pre-adversarial actor version.

That's one of the advantages of NixOS - viruses and mass hacks have lesser chance to function due to how different this OS is. Until it gets more popular, of course.

I think this is unfortunately very common practice

Why not `git archive`?

https://news.ycombinator.com/item?id=39872062

I think the reason is pretty obvious. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.

That repository is now disabled. But here's a similar change to the .github repository of tukaani-project from @JiaT75 to the bug report template:

    + or create a private Security Advisory instead.

[1] https://github.com/tukaani-project/.github/commit/44b766adc4...

Potentially the purpose is that if someone goes to the effort to get those details together, they are more likely to send the same report to other trusted individuals. Maybe it was originally there to add legitimacy, then they got a report sent in, and removed it to slow the spread of awareness

In various places they say Lasse Collin is not online right now, but he did make commits a week ago https://git.tukaani.org/?p=xz.git;a=summary

Makes me wonder if he's an owner of the github organization, and what happens with it now?

The account has been suspended for a while, but for whatever reason that's not displayed on the profile itself (can be seen at https://github.com/Larhzu?tab=following). Repo being disabled is newer, and, while annoying and realistically likely pointless, it's not particularly unreasonable to take down a repository including a real backdoor.

That's how I understand it. A build script that's in the releases tarballs but not the git repo, checks to see if it's being run as part of the debian/build or rpm build processes, and then injects content from one of the "test" files.

Please take the systemd trolling to Reddit. They likely targeted xz specifically because it’s so widely used but there are dozens of other libraries which are potential candidates for an attack on sshd, much less everything else which has a direct dependency unrelated to systemd (e.g. dpkg).

Rather than distracting, think about how the open source projects you use would handle an attack like this where someone volunteers to help a beleaguered maintainer and spends time helpfully taking on more responsibilities before trying to weaken something.

Actually you have a point. A collection of shell scripts (like the classical init systems) have obviously a smaller attack surface. In this case the attacker used some integration code with systemd to attack the ssh daemon. So sshd without systemd integration is safe against this specific attack.

In general, I’m not convinced that systemd makes things less secure. I have the suspicion that the attacker would just have used a different vector, if there was no systemd integration. After all it looks like the attacker was also trying to integrate exploits in owner libraries, like zstd.

Still I would appreciate it, if systemd developers would find a better protection against supply chain attacks.

This isn't Twitter you don't have to use hashtags

> systemd's call to dlopen() liblzma library (CVE-2024-3094)

That's technically wrong, but no surprise. Anti-systemd trolls usually don't understand technical details after all.

Now to be fair, such an archive could have been created with a “store” level of compression that doesn’t actually perform any compression.

Is that true? Large companies producing software usually have bespoke infra, which barely anyone monitors. See: the Solarwinds hack. Similarly to the xz compromise they added the a Trojan to the binary artifacts by hijacking the build infrastructure. According to Wikipedia "around 18,000 government and private users downloaded compromised versions", it took almost a year for somebody to detect the trojan.

Thanks to the tiered updates of Linux distros, the backdoor was caught in testing releases, and not in stable versions. So only a very low percentage of people were impacted. Also the whole situation happened because distros used the tarball with a "closed source" generated script, instead of generating it themselves from the git repo. Again proving that it's easier to hide stuff in closed source software that nobody inspects.

Same with getting hired. Don't companies hire cheap contractors from Asia? There it would be easy to sneak in some crooked or even fake person to do some dirty work. Personally I was even emailed by a guy from China who asked me if I was willing to "borrow" him my identity so he could work in western companies, and he would share the money with me. Of course I didn't agree, but I'm not sure if everybody whose email he found on Github did.

https://en.wikipedia.org/wiki/2020_United_States_federal_gov...

> Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he.

Or work for a third-party company that gets access to critical systems without any checks. See for example the incident from 2022 here: https://en.wikipedia.org/wiki/Okta,_Inc.

Or a third-party that rents critical infrastructure to the company (Cloud, SaaS solutions).

> cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

That's the entire point. You did everything you could by getting someone else look at it and saying it's fine.

Thats basically the whole point actually... A company pays for insurance for the business. The insurance company says sure we will insure you, but you need to go through audits A B and C, and you need certifications X and Y to be insured by us. Those audits are often industry dependent, mostly for topics like HIPAA, PCI, SOC, etc.

Insurance company hears about supply chain attacks. Declares that insured must have supply chain validation. Company goes and gets a shiny cert.

Now when things go wrong, the company can point to the cert and go "it wasnt us, see we have the cert you told us to get and its up to date". And the company gets to wash their hands of liability (most of the time).

If you installed xz on macOS using brew, then you have

  xz (XZ Utils) 5.6.1
  liblzma 5.6.1

  brew upgrade xz

> the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

That is actually a major point of a lot of corporate security measures (shifting risk)

That's the entire point of certification, and any certification at all. Certification does not guarantee performance. Actually, I would always cast a suspect glance to anyone who is FOCUSED on getting certification after certification without any side project.

This doesn't matter, if libxml2 loads .so and the library is malicious, you are already potentially compromised, as it is possible to run code on library load.

> only to read and parse uncompressed XML…

how does libxml2 know to decompress something?

does it require you, as the caller, to explicitly tell it to?

or does it look at the magic bytes or filename or mimetype or something?

Better make a torrent out of them.

One thing to note is that the person that added the commits only started contributing around late 2022 and appears to have a Chinese name. Might be required by law to plant the backdoor.

That would be quite scary considering they have contributed to a wide variety of projects including C++ https://learn.microsoft.com/en-us/cpp/overview/whats-new-cpp...

LANG only needs to have some value, the concrete value does not seem to matter.

> Seems like the complexity of XZ has backfired severely, as expected.

this is a very bad reading of the current situation.

Totally agree. With things like Dependabot encouraged by GitHub, people now get automated pull requests for dependency updates, increasing the speed of propagation of such vulnerabilities.

It looks like git clone https://git.tukaani.org/xz.git still works for now (note: you will obviously be cloning malware if you do this) - that is, however, trusting the project infrastructure that compromised maintainers could have had access to, so I'm not sure if it is unmodified.

HEAD (git rev-parse HEAD) on my result of doing that is currently 0b99783d63f27606936bb79a16c52d0d70c0b56f, and it does have commits people have referenced as being part of the backdoor in it.

Well that's inconvenient, I was (probably, time permitting) going to propose to some of my friends that we attempt to reverse this for fun tomorrow.

Anyone have a link to the git history? I guess we can use the ubuntu tarball for the evil version.

Being included via libsystemd isn't the only way ssh can load liblzma, it can come as an indirect dependency of Selinux (and its PAM stack) IIUC. Which makes it even a bit more funny (?) since Arch also doesn't officially support any Selinux stuff.

There might be other ways sshd might pull in lzma, but those are the 2 ways I saw commonly mentioned.

On a different note, pacman/makepkg got the ability to checksum source repository checkouts in 6.1.

They just signed each other's keys around that time, and one needs to redistribute the public keys for that; nothing suspicious about it I think. The key fingerprint 22D465F2B4C173803B20C6DE59FCF207FEA7F445 remained the same.

before:

    pub   rsa4096/0x59FCF207FEA7F445 2022-12-28 [SC] [expires: 2027-12-27]
        22D465F2B4C173803B20C6DE59FCF207FEA7F445
    uid                             Jia Tan <jiat0218@gmail.com>
    sig        0x59FCF207FEA7F445 2022-12-28   [selfsig]
    sub   rsa4096/0x63CCE556C94DDA4F 2022-12-28 [E] [expires: 2027-12-27]
    sig        0x59FCF207FEA7F445 2022-12-28   [keybind]

    pub   rsa4096/0x59FCF207FEA7F445 2022-12-28 [SC] [expires: 2027-12-27]
        22D465F2B4C173803B20C6DE59FCF207FEA7F445
    uid                             Jia Tan <jiat0218@gmail.com>
    sig        0x59FCF207FEA7F445 2022-12-28   [selfsig]
    sig        0x38EE757D69184620 2024-01-12   Lasse Collin <lasse.collin@tukaani.org>
    sub   rsa4096/0x63CCE556C94DDA4F 2022-12-28 [E] [expires: 2027-12-27]
    sig        0x59FCF207FEA7F445 2022-12-28   [keybind]

    pub   rsa4096/0x38EE757D69184620 2010-10-24 [SC] [expires: 2025-02-07]
        3690C240CE51B4670D30AD1C38EE757D69184620
    uid                             Lasse Collin <lasse.collin@tukaani.org>
    sig        0x38EE757D69184620 2024-01-08   [selfsig]
    sig        0x59FCF207FEA7F445 2024-01-12   Jia Tan <jiat0218@gmail.com>
    sub   rsa4096/0x5923A9D358ADF744 2010-10-24 [E] [expires: 2025-02-07]
    sig        0x38EE757D69184620 2024-01-08   [keybind]

GitHub suspended this project

Going forward this will require more than a citizens investigation. Law enforcement will surely be granted access. Also, tarballs are still available in package managers if you really want to dig into the code.

It's a crime scene. It effectively has the "police" yellow tape around it.

Nice idea, but then you just hide the attack in logo.png that gets embedded in the binary. Less useful for libraries, works plenty good for web/desktop/mobile.

5.6.1-2 is not an attempted fix, it's just some tweaks to Arch's own build script to improve reproducibility. Arch's build script ultimately delegates to the compromised build script unfortunately, but it also appears the payload itself is specifically targeting deb/RPM based distros, so a narrow miss for Arch here.

(EDIT: as others have pointed out, part of the exploit is in the artifact from libxz, which Arch is now avoiding by switching to building from a git checkout)

I upgraded Arch Linux on my server a few hours ago. Arch Linux does not fetch one of the compromised tarballs but builds from source and sshd does not link against liblzma on Arch.

  [root@archlinux ~]# pacman -Qi xz | head -n2  
  Name            : xz  
  Version         : 5.6.1-2  
  [root@archlinux ~]# pacman -Qi openssh | head -n2
  Name            : openssh
  Version         : 9.7p1-1
  [root@archlinux ~]# ldd $(which sshd) | grep liblzma
  [root@archlinux ~]#

The project has made an official post on the subject

https://archlinux.org/news/the-xz-package-has-been-backdoore...

The writeup indicates that the backdoor only gets applied when building for rpm or deb, so Arch probably would have been okay either way? Same with Nix, Homebrew, etc.

On arch, `ldd $(which sshd)` doesn't list lzma or xz, so I think it's unaffected? Obviously still not great to be shipping malicious code that just happens to not trigger.

But how? Let's say you're one of 10 maintainers of an open source project. A new user wants to contribute. What do you do? Do you ask them to send you some form of ID? Assuming this is legal and assuming you could ensure the new user is the actual owner of an actual, non counterfeit ID, what do you do? Do you vet people based on their nationality? If so, what nationality should be blackballed? Maybe 3 maintainers are American, 5 are European and 2 are Chinese. Who gets to decide? Or do you decide based on the company they work for?

Open source is, by definition, open. The PR/merge request process is generally meant to accept or refuse commits based on the content (which is why you have a diff), not on the owner.

Building consensus on which commits are actually valid, even in the face of malicious actors, is a notoriously difficult problem. Byzantine fault tolerance can be achieved with a 2/3 + 1 majority, but if anyone can create new identities and have them join the system (Sybil attack) you're going to have to do things differently.

Not even background check but a foreground check would already help. Like literally, who dis? any identity at all?

Too often maintainers who have no time just blanket approve PRs and see if stuff breaks.

It takes a minute to create a new email address. And you can change or fake an email address on a git commit trivially. You, too, can writing code as anyone you want by just doing "git commit --author='Joe Biden <icecream@whitehouse.gov>'". On the internet nobody knows you're Joe Biden.

You can write a rather simple GitHub action that would do that: look at a PR and reject / close it if you don't like it for some reason. AFAIK open-source projects have a free quota of actions.

OTOH sticking to the same email for more than one exploit might be not as wise for a malicious agent.

github already suspended the account

They should also remove the emojis, there is no need to have people feel good about upvotes. I've long felt uncomfortable with emojis on Slack as well. Responding to a coding or infrastructure issue should not be a social activity, I respond because it's my job and if the issue is worth it, not because a human being should feel appreciated (either them or me).

FreeBSD is not affected as the payloads in question were stripped out, however we are looking into improvements to our workflow to further improve the import process.

The lesson here seems to not depend on tools written in languages that have complex, obscure build systems and no one is either able or interested to read. Using tools rewritten in Rust, Go or any other languege which resolves dependencies within project seems the only way to do hardening here.

Am I crazy thinking libraries shouldn't be able to provide _other libraries'_ symbols without the other libraries' "permission"? What am I missing?

Has anyone been prosecuted for adding a backdoor

Google up Randal Schwartz. Caution: clickhole.

I am very concerned about Rust.

Rust’s “decision” to have a very slim standard library has advantages, but it severely amplifies some other issues. In Go, I have to pull in zero dependencies to make an HTTP request. In Rust, pulling reqwest pulls in at least 30 distinct packages (https://lib.rs/crates/reqwest). Date/time, “basic” base64, common hashing or checksums, etc, they all become supply chain vectors.

The Rust ecosystem’s collective refusal to land stable major versions is one of the amplifying issues. “Upgrade fatigue” hits me, at least. “Sure, upgrade ring to 0.17” (which is effectively the 16th major version). And because v0.X versions are usually incompatible, it’s not really possible to opt not to upgrade, because it only takes a short while before some other transitive dependency breaks because you are slow to upgrade. I recently spent a while writing my code to support running multiple versions of the `http` library, for example (which, to be fair, did just land version 1.0). My NATS library (https://lib.rs/crates/async-nats) is at version 34. My transitive base64 dependency is at version 22 (https://lib.rs/crates/base64).

This makes it nearly impossible for me to review these libraries and pin them, because if I pin foo@0.41.7, and bar needs foo@0.42.1, I just get both. bar can’t do =>0.41, because the point of the 0.X series is that it is not backwards compatible. It makes this process so time consuming that I expect people will either just stop (as if they did) reviewing their dependencies, or accept that they might have to reinvent everything from URL parsing to constructing http headers or doing CRC checks.

Combine this with a build- and compile-time system that allows completely arbitrary code execution, which is routinely just a wrapper for stuff like in the zx attack (look at a lot of the low-level libs you inevitably pull in). Sure, the build scripts and the macro system enables stuff like the amazing sqlx library, but said build and macro code is already so hard to read, it really takes proper wizardry to properly understand.

Sometimes you smell an American because someone wanted you to smell an American.

Operating on a target region schedule doesn't seem particularly sophisticated, at least compared to the all the efforts put into this exploit.

Interesting. Is there also a pattern in the times of day? (I don't so much mean the times in commits done by the developer because they can be fake. I'd be more interested in authentic times recorded by GitHub, if any such times are publicly accessible.)

Another thing would be to examine everything ever written by the user for linguistic clues. This might point towards particular native languages or a particular variant of English or towards there being several different authors.

It's not ironic, this change is really sinister IMO. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.

I worry the mainstream news take would just be "open source bad, microsoft closed source and google cloud good"

RHEL no, Fedora 41 and Rawhide yes.

https://www.redhat.com/en/blog/urgent-security-alert-fedora-...

https://lists.debian.org/debian-security-announce/2024/msg00...

RHEL won't get this bug for 2 years =)

Red Hat helps to do the job of making sure OSS has CVEs so there's common vernacular for the problem.

In the article it says CISA was notified - that sounds like it's going to be a federal investigation if nothing else. If I was this person, I wouldn't be in the USA (or any US friendly nation) ASAP.

> Getting punched in the face is actually a necessary human condition for a healthy civilization.

Aside from signed commits, we need to bring back GPG key parties and web of trust. When using a project you would know how many punches away from the committers you are.

> Getting punched in the face is actually a necessary human condition for a healthy civilization.

This is factually false - in fact, it's literally the direct opposite of the truth. "Getting punched in the face" is base violence that is incompatible with a healthy civilization. A good government with a robust justice system is what is actually needed for a healthy civilization.

That is all the protocol is. From https://www.freedesktop.org/software/systemd/man/latest/sd_n...:

> These functions send a single datagram with the state string as payload to the socket referenced in the $NOTIFY_SOCKET environment variable.

The simplest implementation (pseudocode, no error handling, not guaranteed to compile), is something like:

    const char *addrstr = getenv("NOTIFY_SOCKET");
    if (addrstr) {
        int fd = socket(AF_UNIX, SOCK_DGRAM, 0);
        struct sockaddr_un addr = { .sun_family = AF_UNIX };
        strncpy(addr.sun_path, sizeof(addr.sun_path), addrstr);
        connect(fd, (struct sockaddr*) &addr);
        write(fd, "READY=1");
        close(fd);
    }

> The systemd notification protocol could have been as simple as just writing a newline to a pipe

It basically is. libsystemd links to liblzma for other features not related to notifications.

(The protocol is that systemd passes the path to a unix socket in the `NOTIFY_SOCKET` env variable, and the daemon writes "READY=1" into it.)

FWIW, I did a quick check on a Devuan system. The sshd in Devuan does link to a libsystemd stub - this is to cut down on their maintenance of upstream packages. However that stub does not link to lzma.

> so now security-critical daemons like openssh have additional dependencies like liblzma

Systemd itself seems security-critical to me. Would removing other dependencies on libsystemd really make a secure system where systemd was compromised through its library?

One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.

Yes, there are things delivered with that complexity. However, as an example, sysvinit is maybe, oh, 20k lines of code including binaries, heck including all core init scripts.

What's systemd? 2M lines? It was >1M lines 4+ years ago.

For an init system, a thing that is to be the core of stability, security, and most importantly glacial, stable change -- that is absurdly complex. It's exceedingly over engineered.

And so you get cases like this. And cases like that, and that over there, and that case over there too. All which could not exist, if systemd didn't try to overengineer, over complicate everything.

Ah well. I'm still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on), and just end up with systemd compatible service files.

But not yet. So... well, oh well.

Also thanks to Debian for modifying openssh.

Uh. systemd documents the protocol at various places and the protocol is trivial: a single text datagram sent to am AF_UNIX socket whose path you get via the NOTIFY_SOCKET. That's trivial to implement for any one with some basic unix programming knowledge. And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise. In particular non-C environments really should do their own native impl and not botjer wrapping libsystemd just for this.

But let me stress two other things:

Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam/pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack.

With that in mind libsystemd git dropped the dep on liblzma actually, all compressors are now dlopen deps and thus only pulled in when needed.

The notify protocol isn't much more complicated than that. From memory you send a string to a unix socket. I have written both systemd notify and listenfd in a few languages for little experiments and it is hard to imagine how the protocols could be simpler.

Looking at most popular projects these days they are a mass of dependencies and I think very few of them can be properly audited and verified by the projects that use them. Rust and Go might be more memory safe than C but look at the number of cargo or go modules in most projects. I have mostly stopped using node/npm on my systems.

Not a programmer, but couldn't the distribution's sshd patches for systemd (and all other distro patches for privileged daemons) use static includes? Wouldn't that have only pulled in the simple client-side communication API? Would that have defeated this vector? Would it be doable?

It's unfortunate that the anti-systemd party lost the war... years ago. But I don't blame systemd, Lennart Pottering or the fanboys (though it would have been so much better if the guy never worked in open source or wasn't such a prolific programmer). I blame Debian and its community for succumbing to this assault on Unix philosophy (again, years ago).

What? I don't get it? Isn't it on Debian if they modified the package to do something like this? Why would you blame systemd for maintainers doing something that upstream has never required or recommended?

This is likely just a coincidence. 0218 looks like a birthday and jiat is probably the name + initial. 18 years is also too long of a time horizon for this.

Chopsticks could also be a codeword for something. Maybe some sort a backdoor into a system somewhere.

Crazy to think that the time horizon for these kinds of attacks span decades. This absolutely does not read like a coincidence. Chopsticks, little sister, "room to work hard", all sound like codewords.

They were pulling a tarball from upstream and building it - the tarball was compromised.

Not in this case as the other commenter pointed out but for example Vivaldi on Arch Linux is just a repackaged upstream build.

https://gitlab.archlinux.org/archlinux/packaging/packages/vi...