Just for posterity since I can no longer edit: Libera staff has been firm and unrelenting in their position not to disclose anything whatsoever about the account. I obtained the last point on my own. Libera has made it clear they will not budge on this topic, which I applaud and respect. They were not involved whatsoever in ascertaining a VPN was used, and since that fact makes anything else about the connection information moot, there's nothing else to say about it.

> EDIT: Github has disabled all Tukaani repositories, including downloads from the releases page.

Why? Isn't it better to freeze them and let as many people as possible analyze the code?

The alpine patch includes gettext-dev which is likely also exploited as the same authors have been pushing gettext to projects where their changes have been questioned

FWIW, that's mingw-w64-xz (cross-compiled xz utils) in AUR, not ming-w64 (which would normally refer to the compiler toolchain itself).

It appears to be an RCE, not a public key bypass: https://news.ycombinator.com/item?id=39877312

I've posted an earlier WHOWAS of jiatan here: https://news.ycombinator.com/item?id=39868773

Asking this here too: why isn't there an automated A/B or diff match for the tarball contents to match the repo, auto-flag with a warning if that happens? Am I missing something here?

account is back online https://github.com/JiaT75

1 week ago "Hans Jansen" user "hjansen" was created in debian and opened 8 PRs including the upgrade to 5.6.1 to xz-utils

From https://salsa.debian.org/users/hjansen/activity

Author: Hans Jansen <hansjansen162@outlook.com>

- [Debian Games / empire](https://salsa.debian.org/games-team/empire): opened merge request "!2 New upstream version 1.17" - March 17, 2024

- [Debian Games / empire](https://salsa.debian.org/games-team/empire): opened merge request "!1 Update to upstream 1.17" - March 17, 2024

- [Debian Games / libretro / libretro-core-info](https://salsa.debian.org/games-team/libretro/libretro-core-i...): opened merge request "!2 New upstream version 1.17.0" - March 17, 2024

- [Debian Games / libretro / libretro-core-info](https://salsa.debian.org/games-team/libretro/libretro-core-i...): opened merge request "!1 Update to upstream 1.17.0" - March 17, 2024

- [Debian Games / endless-sky](https://salsa.debian.org/games-team/endless-sky): opened merge request "!6 Update upstream branch to 0.10.6" - March 17, 2024

- [Debian Games / endless-sky](https://salsa.debian.org/games-team/endless-sky): opened merge request "!5 Update to upstream 0.10.6" - March 17, 2024

- [Debian / Xz Utils](https://salsa.debian.org/debian/xz-utils): opened merge request "!1 Update to upstream 5.6.1" - March 17, 2024

Make it two years.

Jia Tan getting maintainer access looks like it is almost certainly to be part of the operation. Lasse Colling mentioned multiple times how Jia has helped off-list and to me it seems like Jia befriended Lasse as well (see how Lasse talks about them in 2023).

Also the pattern of astroturfing dates back to 2022. See for example this thread where Jia, who has helped at this point for a few weeks, posts a patch, and a <name><number>@protonmail (jigarkumar17) user pops up and then bumps the thread three times(!) lamenting the slowness of the project and pushing for Jia to get commit access: https://www.mail-archive.com/xz-devel@tukaani.org/msg00553.h...

Naturally, like in the other instances of this happening, this user only appears once on the internet.

Also I saw this hans jansen user pushing for merging the 5.6.1 update in debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

Wow, what a big pile of infrastructure for a non-optimization.

An internal call via ifunc is not magic — it’s just a call via the GOT or PLT, which boils down to function pointers. An internal call through a hidden visibility function pointer (the right way to do this) is also a function pointer.

The even better solution is a plain old if statement, which implements the very very fancy “devirtualization” optimization, and the result will be effectively predicted on most CPUs and is not subject to the whole pile of issue that retpolines are needed to work around.

> because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizers

for example, https://github.com/google/oss-fuzz/pull/10667

>Hans Jansen and Jia Tan

Are they really two people conspiring?

Unless proven otherwise, it is safe to assume one is just a pseudonym alias of the other.

Also I see this PR: https://github.com/tukaani-project/xz/pull/64

Does anybody know anything about Jia Tan? Is it likely just a made up persona? Or is this a well-known person.

It seems like Hans Jansen has also an account on proton.me (hansjansen162@proton.me) with the Outlook address configured as recovery-email.

PSA: I just noticed homebrew installed the compromised version on my Mac as a dependency of some other package. You may want to check this to see what version you get:

   xz --version

Is it normal that when I try to uninstall xz it is trying to install lzma?

I made a library where version 2 is really really much faster than version 1. I'd want everyone to just move to version 2.

You can find more examples of that kind of puffer if you go to a website's cookie consent pop-up and find the clause after "we use cookies to...".

I agree on principle, but sometimes programmatic generating test data is not so easy.

E.g.: I have a specific JPEG committed into a repository because it triggers a specific issue when reading its metadata. It's not just _random_ data, but specific bogus data.

But yeah, if the test blob is purely random, then you can just commit a seed and generate in during tests.

Is that version truly vetted? "Jia Tan" has been the official maintainer since 5.4.3, could have pushed code under any other pseudonym, and controls the signing keys. I would have felt better about reverting farther back, xz hasn't had any breaking changes for a long time.

TIL that +really is a canonical string. [0]

[0]: https://www.debian.org/doc/debian-policy/ch-controlfields.ht...

There are suggestions to roll back further

Isn't it rather that the attacker can log in to the compromised server by exploiting the RSA code path?

There probably is. Way more than anyone knows. I bet every major project on github is riddled with state actors.

Imagine if sshd was distributed by PyPI or cargo or npm instead of by a distro.

Not true, the original author wasn't suspended: https://github.com/Larhzu

https://github.com/JiaT75 was suspended for a moment, but isn't anymore?

These shouldn't be suspended, and neither should their repositories. People might want to dig through the source code. It's okay if they add a warning on the repository, but suspending _everything_ is a stupid thing to do.

If it was a compromise it also included the signing keys as the release tarball was modified vs the source available on GitHub.

Please don't?

1. You don't actually know what has been done by whom or why. You don't know if the author intended all of this, or if their account was compromised. You don't know if someone is pretending to be someone else. You don't know if this person was being blackmailed, forced against their will, etc. You don't really know much of anything, except a backdoor was introduced by somebody.

2. Assuming the author did do something maliciously, relying on personal reputation is bad security practice. The majority of successful security attacks come from insiders. You have to trust insiders, because someone has to get work done, and you don't know who's an insider attacker until they are found out. It's therefore a best security practice to limit access, provide audit logs, sign artifacts, etc, so you can trace back where an incursion happened, identify poisoned artifacts, remove them, etc. Just saying "let's ostracize Phil and hope this never happens again" doesn't work.

3. A lot of today's famous and important security researchers were, at one time or another, absolute dirtbags who did bad things. Human beings are fallible. But human beings can also grow and change. Nobody wants to listen to reason or compassion when their blood is up, so nobody wants to hear this right now. But that's why it needs to be said now. If someone is found guilty beyond a reasonable doubt (that's really the important part...), then name and shame, sure, shame can work wonders. But at some point people need to be given another chance.

They might have burnt the reputation built for this particular pseudonym but what is stopping them from doing it again? They were clearly in it for the long run.

Good luck with that. We don't even know what country is he from. Probably from China but even if so. Good luck finding him among 1.5 Billions.

Every Linux box inside AWS, Azure, and GCP and other cloud providers that retains the default admin sudo-able user (e.g., “ec2”) and is running ssh on port 22.

I bet they intended for their back door to eventually be merged into the base Amazon Linux image.

Distro build hosts and distro package maintainers might not be a bad guess. Depends on whether getting this shipped was the final goal. It might have been just the beginning, part of some bootstrapping.

Probably less of an individual and more of an exploit to sell.

Not sure why are people downvoting you... it's pretty unlikely that various Chinese IoT companies would just decide it's cool to add a backdoor, which clearly implies that no matter how good their intentions are, they simply might have no other choice.

Don't blame the guy. Could have happened to anyone. Even you.

That's pure speculation and there are plenty of hints to the contrary.

In this particular case, there is a strong reason to expect exploitation in the wild to already be occurring (because it's an intentional backdoor) and this would change the risk calculus around disclosure timelines.

But in the general case, it's normal for 90 days to be given for the coordinated patching of even very severe vulnerabilities -- you are giving time not just to the project maintainers, but to the users of the software to finish updating their systems to a new fixed release, before enough detail to easily weaponize the vulnerability is shared. Google Project Zero is an example of a team with many critical impact findings using a 90-day timeline.

Whether its reasonable is debatable, but that type of time frame is pretty normal for things that aren't being actively exploited.

This situation is perhaps a little different as its not an accidental bug waiting to be discovered but an intentionally placed exploit. We know that a malicious person already knows about it.

I think you have to take the credibility of the maintainer into account.

If it's a large company, made of people with names and faces, with a lot to lose by hacking its users, they're unlikely to abuse private disclosure. If it's some tiny library, the maintainers might be in on it.

Also, if there's evidence of exploitation in the wild, the embargo is a gift to the attacker. The existence of a vulnerability in that case should be announced, even if the specifics have to be kept under embargo.

imho it depends on the vuln. I've given a vendor over a year, because it was a very low risk vuln. This isn't a vuln though - this is an attack.

The fraudulent author must have enjoyed the 'in joke' -- He's the one create vulnerabilities..

I've always laughed my ass off at the idea of a disclosure window. It takes less than a day to find RCE that grants root privileges on devices that I've bothered to look at. Why on earth would I bother spending months of my time trying to convince someone to fix something?

If this question had a reliable (and public) answer then the world would be a very different place!

That said, this is an important question. We, particularly those us who work on critical infrastructure or software, should be asking ourselves this regularly to help prevent this type of thing.

Note that it's also easy (and similarly catastrophic) to swing too far the other way and approach all unknowns with automatic paranoia. We live in a world where we have to trust strangers every day, and if we lose that option completely then our civilization grinds to a halt.

But-- vigilance is warranted. I applaud these engineers who followed their instincts and dug into this. They all did us a huge service!

EDIT: wording, spelling

> How many of people like this one exist?

I guess every 3 letter agency has at least one. You can do the math. They havent't learned anything after Solar Winds.

Because of the exploit, so, why should we use configurations in production that were not covered by these tests?

Could that commit also be made by a bad actor?

Given that this was discovered by sheer luck, I'd expect way more such exploits in the wild.

It's weird and disturbing that this isn't the default perspective.

Rolling back two years worth of commits made by a major contributor is going to be hell. I'm looking forward to see how they'll do this.

Hoe will you do that practically though? That’s probably thousands of commits upon which tens or hundred thousand commits from others were built. You can’t just rollback everything two years and expect it not to break or bring back older vulnerabilities that were patched in those commits.

I don’t thinks that’s necessary: there are enough eyes on this person’s work now.

Yes, it's usually regenerated already. However even the source is often pretty gnarly.

And in general, the build system of a large project is doing a lot of work and is considered pretty uninteresting and obscure. Random CMake macros or shell scripts would be just as likely to host bad code.

This is also why I like meson, because it's much more constrained than the others and the build system tends to be more modular and the complex parts split across multiple smaller, mostly independent scripts (written in Python or bash, 20-30 lines max). It's still complex, but I find it easier to organize.

The other thing besides the autoconf soup is the XZ project contains incomprehensible binaries as "test data"; the "bad-3-corrupt_lzma2.xz" part of the backdoor that they even put in the repo.

It's entirely possible they could have got that injection through review, even if they had that framwork and instead put it in source files used to generate autoconf soup.

gradle-wrapper is just a convenience, you can always just build the project with an installed version of gradle. Although I get your point, it’s a great place to hide nefarious code.

Or hiring them to do it for years without telling them why until they need a favor.

many people are patriots of their countries. if state agency would approach them proposing to have paid OSS work and help their country to fight terrorism/dictatorships/capitalists/whatever-they-believe, they will feel like killing two birds with one job

One thing that is annoying is that many open source projects have been getting "garbage commits" apparently from people looking to "build cred" for resumes or such.

Easier and easier to hide this junk in amongst them.

There was a DARPA program on this topic called Social Cyber. [1]

1. https://www.darpa.mil/program/hybrid-ai-to-protect-integrity...

And what law would you use to target someone who wrote some code and posted it for free on the internet that was willingly consumed?

Yes, there is evidence of sabotage on the CMake configs too.

https://git.tukaani.org/?p=xz.git;a=commit;h=f9cf4c05edd14de...

From what I'm understanding it's trying to patch itself into the symbol resolution step of ld.so specifically for libcrypto under systemd on x86_64. Am I misreading the report?

That's a strong indication it's targeting sshd specifically.

It checks for argv[0] == "sshd"

FYI, your formatting is broken. Hacker News doesn't support backtick code blocks, you have to indent code.

Anyway, so... the xz project has been compromised for a long time, at least since 5.4.5. I see that this JiaT75 guy has been the primary guy in charge of at least the GitHub releases for years. Should we view all releases after he got involved as probably compromised?

"#, fuzzy" means the translation is out-of-date and it will be discarded at compile time.

RHEL9 is shipping 5.2.5; RHEL8 is on 5.2.4.

Thanks for the heads up.

Debian has epochs, but it's a bad idea to use them for this purpose.

Two reasons:

1. Once you bump the epoch, you have to use it forever. 2. The deb filename often doesn't contain the epoch (we use a colon which isn't valid on many filesystems), so an epoch-revert will give the same file name as pre-epoch, which breaks your repository.

So, the current best practice is the +really+ thing.

.deb has epochs too, but I think Debian developers avoid it where possible because 1:5.4.5 is interpreted as newer than anything without a colon, so it would break eg. packages that depend on liblzma >= 5.0, < 6. There may be more common cases that aren't coming to mind now.

I really like the XBPS way of the reverts keyword in the package template that forces a downgrade from said software version. It's simple but works without any of the troubles RPM epochs have with resolving dependencies as it's just literally a way to tell xbps-install that "yeah, this is a lower version number in the repository but you should update anyway".

Debian packages can have epochs too. I’m not sure why the maintainers haven’t just bumped the epoch here.

Maybe they’re expecting a 5.6.x release shortly that fixes all these issues & don’t want to add an epoch for a very short term packaging issue?

>Ironic considering security is often advertised as a feature of rolling release distros.

Security is a feature of rolling release. But supply-chain attacks like this are the exception to the rule.

i mean, rolling implies rolling 0-days, too.

> From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.

It looks like the backdoor "deactivates" itself when it detects being started interactively, as a security researcher might. I was eventually able to circumvent that, but unless you do so, it'll not be active when started interactively.

However, the backdoor would also be active if you started it with an shell script (as the traditional sys-v rc scripts did) outside the context of an interactive shell, as TERM wouldn't be set either in that context.

> Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

There's no xz code in openssh.

> Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

OpenSSH is developed by the OpenBSD project, and systemd is not compatible with OpenBSD. The upstream project has no systemd or liblzma code to strip. If your sshd binary links to liblzma, it's because the package maintainers for your distro have gone out of their way to add systemd's patch to your sshd binary.

> From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.

From what I understand, the backdoor detects if it's in any of a handful of different debug environments. If it's in a debug environment or not launched by systemd, it won't hook itself up. ("nothing to see here folks...") But if sshd isn't linked to liblzma to begin with, none of the backdoor's code even exists in the processes' page maps.

I'm still downgrading to an unaffected version, of course, but it's nice to know I was never vulnerable just by typing 'ldd `which sshd`' and not seeing liblzma.so.

I think the distributions that do use systemd are the ones that add the libsystemd code, which in turn brings in the liblzma5 code. So, it may not be entirely relevant how it is run, but it needs to be a version of OpenSSH patched.

> I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related?

Possible but unlikely.

> I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?

It injects code that runs early during sshd connection establishment. Likely allowing remote code execution if you know the right magic to send to the server.

Are you on stable/testing/unstable?

With our current knowledge, stable shouldn’t be affected by this.

revertto probably just means "revert to" but it does sound quite italian lol.

In a movie, he was killed by foreign state actors, and his identity assumed by the foreign state hacker. Actually, someone should check on him.

or > Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.

Is actually Jia Tan has him tied up in a basement and is posing as him. State actors can do that kind of thing.

> what looks like a tangential attack

Does it? I expect that finding someone vulnerable was the more likely approach rather than messing with the life of a stable maintainer, but it does seem very much like the attacker was acting with malicious intent from the start of his interaction with the xz project.

I would start restoring trust by reverting all this guys commits. It's the best way to be sure.

I mean, he was right at least. Jia Tan did have a bigger role.

which IRC channel ?

> I'm capable of compartmentalization

teach me how. help me learn how, please. any resources with practical utility you can share? or any class of therapists that are good at teaching this with right frameworks offered? thank you

The (most?) popular SQLite driver for Go often gets PRs to update the SQLite C amalgamation, which the owner politely declines (and I appreciate him for that stance, and for taking on the maintenance burden it brings).

e.g., https://github.com/mattn/go-sqlite3/pull/1042#issuecomment-1...

In this case, the project is using Git submodules for its vendored dependencies, so you can trivially cryptographically verify that they have vendored the correct dependency just by checking the commit hash. It looks really crazy on Github but in most git clients it will just display the commit hash change.

The dopamine hits from updating stuff should come to an end, it should be thought of as adding potentially new bugs or exploits, unless the update fixes a CVE. Also Github needs to remove the green colors and checkmarks in PR's to prevent these dopamine traps from overriding any critical thinking

That sucks to have people write mails to your employer...

I appreciated your detailed update!

They're definitely a real person. I know cause that "1Password employee since December" is a person I know IRL and worked with for years at their prior employer. They're not a no-name person or a fake identity just FYI. Please don't be witch hunting; this genuinely looks like an unfortunate case where Jared was merely proactively doing their job by trying to get an externally maintained golang bindings of XZ to the latest version of XZ. Jared's pretty fantastic to work with and is definitely the type of person to be filing PRs on external tools to get them to update dependencies. I think the timing is comically bad, but I can vouch for Jared.

https://github.com/jamespfennell/xz/pull/2

If I were trying to compromise supply chains, getting into someplace like 1Password would be high up on the list.

Poor guy, he's probably going to get the third degree now.

As a 1Password user, I just got rather nervous.

Yeah the GitHub account looks really really legitimate. Maybe it was compromised though?

That's the same Hans Jansen mentioned here: https://boehs.org/node/everything-i-know-about-the-xz-backdo...

More caution might not be a bad thing.

> Not sure what I'd have done earlier in my career

To anybody in this sorta situation, you should absolutely share whatever you have. It doesn’t need to be perfect, good, or 100% accurate, but if there’s a risk you could help a lot of people

This story is an incredible testament to how open-source software can self-regulate against threats, and more broadly, it reminds us that we all stand on the shoulders of contributors like you. Thank you!

I hope you've hired a PR person for all the interviews :)

This guy's interactions seem weird but it might just be because of the non-native english or a strange attitude, or he's very good at covering his track e.g. found a cpython issue where he got reprimanded for serially opening issues: https://github.com/python/cpython/issues/115195#issuecomment...

But clicking around he seems to mostly be interacting with interest around these bits e.g. https://github.com/python/cpython/issues/95341#issuecomment-... or pinging the entire python team to link to the PR... of a core python developer: https://github.com/python/cpython/issues/95341#issuecomment-...

If I saw that on a $dayjob project I'd pit him as an innocuous pain in the ass (overly excited, noisy, dickriding).

Here's a PR from 2020 where he recommends / requests the addition of SCRAM to an SMTP client: https://github.com/marlam/msmtp/issues/36 which is basically the same thing as the PR you found. The linked documents seem genuine, and SCRAM is an actual challenge/response authentication method for a variety of protocols (in this case mostly SMTP, IMAP, and XMPP): https://en.wikipedia.org/wiki/Salted_Challenge_Response_Auth...

Although, and that's a bit creepy, he shows up in the edition history for the SCRAM page, the edit mostly seem innocent though he does plug his "state of play" github repository.

What? They're just asking for some features there?

Ya'll need to calm down; this is getting silly. Half the GitHub accounts look "suspicious" if you start scrutinizing everything down the the microscopic detail.

I appreciate the way that duesee handled that whole issue.

reported the account to github, just in case.

The PR + angry user pushing for the PR author to gain commit access spiel is definitely suspiciously similar to what happened with xz-utils. Possible coincidence but worth investigating further.

I wouldn't be surprised if that is just a bot.

He even follows me, though I have never published any open-source project on my own.

The parent comment doesn't read like an attack to me. Just an observation. Would be curious why you wanted the update though.

details please? I do not see any such contributions to https://github.com/facebook/zstd

my freaking kernels/initrd are xz or zstd compressed!

... and Debian is very serious about it: https://fulda.social/@Ganneff/112184975950858403

clickhouse has pretty good github_events dataset on playground that folks can use to do some research - some info on the dataset https://ghe.clickhouse.tech/

Example of what this user JiaT75 did so far:

https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

pull requests mentioning xz, 5.6 without downgrade, cve being mentioned in the last 60 days:

https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0Z...

> If the code is complex, there could be many more exploits hiding.

Then the code should not be complex. Low-level hacks and tricks (like pointer juggling) should be not allowed and simplicity and readability should be preferred.

If this is a conspiracy or a state-sponsored attack, they might have gone specifically for embedded devices and the linux kernel. Here archived from tukaani.org:

https://web.archive.org/web/20110831134700/http://tukaani.or...

> XZ Embedded is a relatively small decompressor for the XZ format. It was developed with the Linux kernel in mind, but is easily usable in other projects too.

> *Features*

> * Compiled code 8-20 KiB

> [...]

> * All the required memory is allocated at initialization time.

This is targeted at embedded and real-time stuff. Could even be part of boot loaders in things like buildroot or RTEMS. And this means potentially millions of devices, from smart toasters or toothbrushes to satellites and missiles which most can't be updated with security fixes.

I have some questions.

1) Are there no legit code reviews from contributors like this? How did this get accepted into main repos while flying under the radar? When I do a code review, I try to understand the actual code I'm reviewing. Call me crazy I guess!

2) Is there no legal recourse to this? We're talking about someone who managed to root any linux server that stays up-to-date.

750 commits... is xz able to send e-mails yet?

Anyone have any level of confidence that for example EL7/8 would not be at risk even if more potential exploits at play?

For some random server, sure. For a state sponsored attack? Having an embedded exploit you can use when convenient, or better yet an unknown exploit affecting every linux-based system connected to the internet that you can use when war breaks out - that's invaluable.

I find it funny how MFA is treated as if it would make account takeover suddenly impossible. It's just a bit more work, isn't it? And a big loss in convenience.

I'd much rather see passwords entirely replaced by key-based authentication. That would improve security. Adding 2FA to my password is just patching a fundamentally broken system.

This PR from July 8 2023 is suspicious, so it was very likely a long con: https://github.com/google/oss-fuzz/pull/10667

This is a state sponsored event. Pretty poorly executed though as they were tweaking and modifying things in their and other tools after the fact though.

As a state sponsored project. What makes you think this is their only project and that this is a big setback? I am paranoid myself to think yesterdays meeting went like : "team #25 has failed/been found out. Reallocate resources to the other 49 teams."

As I said recently in a talk I gave, 2FA as implemented by pypy or github is meaningless, when in fact all actions are performed via tokens that never expire, that are saved inside a .txt file on the disk.

they might not have been playing the long con. maybe approached by actors willing to pay them a lot of money to try and slip in a back door. I'm sure a deep dive into code contributions would clear that up for anyone familiar with the code base and some free time.

github already mandates MFA for members of important projects

Not MFA but git commit signing. I don't get why such core low-level projects don't mandate it. MFA doesn0t help if a github access token is stolen and I bet most of use such a token for pushing from an IDE.

Even if an access token to github is stolen, the sudden lack of signed commit should raise red flags. github should allow projects to force commit signing (if not already possible).

Then the access token plus the singing key would need to be stolen.

But of course all that doesn't help in the here more likley scenario of a long con by a state-sponsored hacker or in case of duress (which in certain countries seems pretty likley to happen)

This seems like a great way to invest in supporting open source projects in meantime if these projects are being used by these actors. Just have to maintain an internal fork without the backdoors

Maybe someone can disrupt the open source funding problem by brokering exploit bounties /s

Which like, also wouldn't be totally weird if I found out that the xz or whatever library maintainer worked for the DoE as a researcher? I kind of expect governments to be funding this stuff.

And that long term perspective could be used constructively instead!

The web of trust is a really nice idea, but it works badly against that kind of attacks. Just consider that in the real world, most living people (all eight billions) are linked by only six degrees of separation. It really works, for code and for trusted social relations (like "I lend you 100 bucks and you pay me them back when you get your salary") mostly when you know the code author in person.

This is also not a new insight. In the beginning of the naughties, there was a web site named kuro5hin.org, which experiemented with user ratings and trust networks. It turned out impossible to prevent take-overs.

Clearly a human is even better at it.

State actors have equaly long horizons to compromise

State level actor? China?

I guess it seems like the operational parts are a bit poorly done. Valgrind issues, adding a new version with symbols removed, the aforementioned performance issues. Like i would assume the type of person who would do this sort of thing, over a 2 year period no less, would test extensively and be sure all their i's are dotted. Its all kind of surprising given how audacious the attack is.

Do you have a writeup or any details as to what it does? The logical thing based on this post is that it hooks the SSH key verification mechanism to silently allow some attacker-controlled keys but I wonder if there's more to it?

Which is why a really good backdoor is a one line logic bug somewhere which is fiendishly difficult to trigger.

Unrelated: as a dog/pointer lover i really like the term "to bird-dog the problem". Never heard of it (iam from germany though)

Case in point: https://news.ycombinator.com/item?id=39843930

This also means that Google might know who they are, unless they were careful to hide behind VPN or other such means.

> pipeing into this shell script which now uses "eval"

I don’t actually see an issue with that `eval`. Why would one consider running `xz` followed by `eval`-ing its output more insecure than just running `xz`? If `xz` wants to do shenanigans with the privileges it already has, then it wouldn’t need `eval`’s help for that.

This is my main take-away from this. We must stop using upstream configure and other "binary" scripts. Delete them all and run "autoreconf -fi" to recreate them. (Debian already does something like this I think.)

Maybe it's time to dramatically simplify autoconf?

How long do we need to (pretend to) keep compatibility with pre-ANSI C compilers, broken shells on exotic retro-unixes, and running scripts that check how many bits are in a byte?

Not just autoconf. Build systems in general are a bad abstraction, which leads to lots and lots of code to try to make them do what you want. It's a sad reality of the mismatch between a prodecural task (compile files X, Y, and Z into binary A) and what we want (compile some random subset of files X, Y, and Z, doing an arbitrary number of other tasks first, into binary B).

For fun, you can read the responses to my musing that maybe build systems aren't needed: https://news.ycombinator.com/item?id=35474996 (People can't imagine programming without a build system - it's sad)

Autoconf is m4 macros and Bourne shell. Most mainstream programming languages have a packaging system that lets you invoke a shell script. This attack is a reminder to keep your shell scripts clean. Don't treat them as an afterthought.

I'm wondering is there i.e. no way to add an automated flagging system that A/B / `diff` checks the tarball contents against the repo's files and warns if there's a mismatch? This would be on i.e. GitHub's end so that there'd be this sort of automated integrity test and subsequent warning? Just a thought, since tainted tarballs like these might be altogether be (and become) a threat vector, regardless of the repo.

Maybe the US Government needs to put its line in the sand and mandate the end of autotools. :D

Unfortunately eval in a shell script has an effect on the semantics but is not necessary to do some kind of parsing of the contents of a variable, unlike Python or Perl or JavaScript. A

    $goo

    eval "$goo"

eval in autoconf macros is nothing unusual.

In (pre-backdoor) xz 5.4.5:

  $ grep -wl eval m4/*
  m4/gettext.m4
  m4/lib-link.m4
  m4/lib-prefix.m4
  m4/libtool.m4

> Usually there is a better way to do it in almost all cases where people feel the need to reach for "eval"

unfortunately thats just standard in configure scripts, for example from python:

``` grep eval Python-3.12.2/configure | wc -l 165 ```

and its 32,958 lines of code, plenty of binary fixtures as well in the tarball to hide stuff.

who knows, but I have feeling us finding the backdoor in this case was more of a happy accident.

You can be certain it has happened, many times. Now think of all the software we mindlessly consume via docker, language package managers, and the like.

Remember, there is no such thing as computer security. Make your decisions accordingly :)

No, we can't.

How many people read autoconf scripts, though? I think those filters are symptom of the larger problem that many popular C/C++ codebases have these gigantic build files which even experts try to avoid dealing with. I know why we have them but it does seem like something which might be worth reconsidering now that the tool chain is considerably more stable than it was in the 80s and 90s.

I don't see how showing the binary diffs would help. 99.99999% of people would just scroll right past them anyways.

00011900: 0000 4883 f804 7416 b85f 5f5f 5f33 010f ..H...t..____3.. │ 00011910: b651 0483 f25a 09c2 0f84 5903 0000 488d .Q...Z....Y...H. │ 00011920: 7c24 40e8 5875 0000 488b 4c24 4848 3b4c |$@.Xu..H.L$HH;L │ 00011930: 2440 7516 4885 c074 114d 85ff 0f84 3202 $@u.H..t.M....2. │ 00011940: 0000 498b 0ee9 2c02 0000 b9fe ffff ff45 ..I...,........E │ 00011950: 31f6 4885 db74 0289 0b48 8bbc 2470 1300 1.H..t...H..$p.. │ 00011960: 0048 85ff 0f85 c200 0000 0f57 c00f 2945 .H.........W..)E │ 00011970: 0048 89ac 2470 1300 0048 8bbc 2410 0300 .H..$p...H..$... │ 00011980: 0048 8d84 2428 0300 0048 39c7 7405 e8ad .H..$(...H9.t... │ 00011990: e6ff ff48 8bbc 24d8 0200 0048 8d84 24f0 ...H..$....H..$. │ 000119a0: 0200 0048 39c7 7405 e893 e6ff ff48 8bbc ...H9.t......H.. │ 000119b0: 2480 0200 0048 8d84 2498 0200 0048 39c7 $....H..$....H9. │ 000119c0: 7405 e879 e6ff ff48 8bbc 2468 0100 004c t..y...H..$h...L │ Please tell me what this code does, Sheldon

testdata should not be on the same machine as the build is done. testdata (and tests generally) aren't as well audited, and therefore shouldn't be allowed to leak into the finished product.

Sure - you want to test stuff, but that can be done with a special "test build" in it's own VM.

in this case the backdoor was hidden in a nesting doll of compressed data manipulated with head/tail and tr, even replacing byte ranges inbetween. Would've been impossible to find if you were just looking at the test fixtures.

That's true, but many of these maintainers work a day job on top of doing the open source work precisely because the open source work doesn't pay the bills. If they could get back 40 hours of their time I think many would appreciate it

> Doesn't everybody know that developing/maintaining free software is largely thankless work, with little to no direct recompense?

No I don't think that is a universally acknowledged feeling. Numerous maintainers have detailed recieving entitled demands from users, as if they were paying customers to the open source software projects. Georges Stavracas' interview on the Tech over Tea podcast^1 describes many such experiences. Similarly, when Aseprite transitioned its license^2 to secure financial stability, it faced backlash from users accusing the developer of betraying and oppressing the community.

On the flipside, if everyone truly does know this is the case, then it's a shame that so many people know, and yet are unwilling to financially support developers to change that. See all of the developers for large open source projects who have day jobs, or take huge pay cuts to work on open source projects. I get that not everyone can support a project financially, but I've personally tried to break that habit of expecting everything I use to be free, and go out of my way to look for donation buttons for project maintainers, and raise awareness during fundraisers. Now if only I could donate directly to Emacs development... I'd encourage other people to do the same.

> What you want is more people that understand and care about free software and low barriers to getting involved.

This is tough. For example, the intention behind initiatives like DigitalOcean's Hacktoberfest, are designed to do just this. It is a good idea in theory, submit 4 pull requests and win a tshirt, but not in practice. The event has been criticized for inadvertently encouraging superficial contributions, such as minor text edits or trivial commits, which burden maintainers^3, causing many maintainers to just archive their repos for the month of October.

So, while there's a recognition of the need for more people who understand and value free software, along with lower barriers to entry, the current state of affairs often falls short. The path forward should involve not just increasing awareness and participation but also providing meaningful support and compensation to maintainers. By doing so, we can foster a more sustainable, secure, and vibrant open source community. Or at least that is how I feel...

1. https://www.youtube.com/watch?v=kO0V7BE1bEo 2. https://github.com/aseprite/aseprite/issues/1242 3. https://twitter.com/shitoberfest?lang=en

>that's a lot of anonymous accounts

Just FYI, krygorin4545@proton.me (the latest message before the upload) was created Tue Mar 26 18:30:02 UTC 2024, about an hour earlier than the message was posted.

Proton generates PGP key upon creating the account, with the real datetime of the key (but the key does not include the timezone).

> running around salsa.debian.org pushing for more updates in other projects as well

This is quite common in most (all?) distributions. People are going through lists of outdated packages, updating them, testing them, and pushing them.

And now we see why I don't trust anons, aliases, or anime characters to make contributions.

My GitHub says exactly who I am!

Hans Gruber would have Been a much more stylish choice…

See comments about "Hans Janson" upthread, he appeared to collaborate on the exploit in other ways as well.

Intelligence agencies get caught red handed all the time so I wouldn't be too sure.

If it was an organised group I'm sure they were careful, of course, but it only takes one fuckup.

Why?

It's very common in autoconf codebases because the idea is that you untar and then run `./configure ...` rather than `autoreconf -fi && ./configure ...`. But to do that either you have to commit `./configure` or you have to make a separate tarball (typically with `make dist`). I know because two projects I co-maintain do this.

Except this change was made in 2023, it is just scary how good this threat actor was.

I believe they also do not include sub modules, which is a big disadvantage for some projects

If only you could prosecute people in adversarial countries for a felony, lol.

If you are this deep into it, it doesn't matter.

Not everywhere, and only if you can prove that were evidences :)

That's why I always raise concerns about JEP 411 - removal of SecurityManager from Java without any replacement.

Just ban autotools

This requires a more comprehensive redesign of the build process. Most Linux distributions also run the tests of the project they're building as part of the build process.

I don’t think zero knowledge systems are practical at the moment. It will take over around 8 orders of magnitude more compute and memory to produce a ZKP proof of generic computation like compilation. Even 2 orders of magnitude is barely acceptable.

I haven't looked at Guix but in the discussions around this exploit for NixOS they mentioned that regenerating autoshit for xz-utils would not be something they can/want to do because that would add a lot more dependencies to the bootstrap before other packages can be build. Kind of funny how a requirement for bootstrapped builds can add a requirement for trusting not-quite-binaries-but-also-not-really-source blobs.

Right, but in this case it's not even compiled it, which is arguably better than compiled in but assumed dormant :) (at least until someone actually does a full analysis of the payload).

It's actually not an advantage. The reason why the exploit wasn't included is because the attacker specifically decided to only inject x86_64 Debian and RHEL to reduce the chances of this getting detected.

That's just security by obscurity, not something I'd consider a good security measure.

Because I didn't know about it.

Although if I look at its documentation, it's already a somewhat complicate invocation with unclear effects (lots of commandline options). Git seems to not be able to do KISS.

git ls-files and tar is a simple thing everybody understands and can do without much issues.

> Affected versions of XZ Utils

Most people, to find the affected versions, would either have to bisect or delve deep enough to find the offending commit. Either of which would reveal the attacker.

By not asking for the version, there is a good chance you just report "It's acting oddly, plz investigate".

Taking down the repo prevents more people inadvertendly pulling and building the backdoor so that makes sense. They should have immediately rehosted and archived the state at a different URL which makes it clear to not use it.

I could imagine another similar attack done against an image processing library, include some "test data" of corrupted images that should "clean up" (and have it actually work!) but the corruption data itself is code to be run elsewhere.

Those other libraries dependend on by sshd are hopefully more closely monitored. The upstream sshd developers probably did not even consider that liblzma could end up being loaded in the process.

Make excuses for systemd all you want but loading multiple additional libraries into crytical system deamons just to write a few bytes into a socket is inexcusable and directly enabled this attack vector.

You are distracting from facts with speculations and trolling FUD. I refer to what is known and has happened, you are speculating on what is not known.

It’s also tricky to reason about risk: for example, ShellShock caused a bunch of vulnerabilities in things which used shell scripts and the classic SysV Init system was a factor in a ton of vulnerabilities over the years because not having a standard way to do things like drop privileges or namespace things, manage processes, difficulties around managing chroot, etc. meant that you had a bunch of people implementing code which ran with elevated privileges because it needed to do things like bind to a low network port and they either had vulnerabilities in the privileged part or messed up some detail. I think in general it’s been much better in the systemd era where so much of that is builtin but I have been happy to see them starting to trim some of the things like the compression format bindings and I expect this will spur more.

I really appreciate your tone and dialectic reasoning, thanks for your reply. And yes, as simple as it sounds, I believe that shell scripts help a lot to maintain mission critical tools. One hands-on example is https://dyne.org/software/tomb where I took this approach to replace whole disk encryption which is nowadays also dependent on systemd-cryptsetup.

This isn't Xitter, you don't have to tell people how to write.

It is 10 and more years that I experience such ad-hominem attacks.

You are so quickly labeling an identifiable professional as troll, while hiding behind your throwaway identity, that I am confident readers will be able to discern.

Meanwhile let us be precise and add more facts https://github.com/systemd/systemd/pull/31550

Our community is swamped by people like you, so I will refrain from answering further provocations, believing I have provided enough details to back my assertion.

My reading of the commit message is they're claiming the "data" should look random.

Or exactly this kind of backdoor in open source but target proprietary software. I don't know of any survey but I'd be surprised if less than half of proprietary software used open source software one way or another and not surprised if it was quite a bit more than that.

This needs a Rust joke. You know, the problem with the whole certification charade is it slows down jobs and prevents __actual_problems getting evaluated. But is it safe?

What you describe is a normal process in order to minimise damage from attacks. The damage of hacking is ultimately property damage. The procedures you've described allow you to minimise it.

And that's a good thing.

> And the company gets to wash their hands of liability (most of the time).

Certification theater.

It's completely performative.

Similarly if you're using MacPorts, make sure to sync and upgrade xz if you have it installed.

5.6.1 was available for a few days and just rolled back ~20 minutes ago: https://github.com/macports/macports-ports/commit/a1388aee09...

Thank you for this tip. `brew upgrade xz` worked.

I was going to uninstall but it's used by so many things…

     brew uninstall xz
    Error: Refusing to uninstall /opt/homebrew/Cellar/xz/5.6.1
    because it is required by aom, composer, curl, ffmpeg, gcc, gd, ghostscript, glib, google-cloud-sdk, grc, harfbuzz, httpie, img2pdf, jbig2enc, jpeg-xl, leptonica, libarchive, libavif, libheif, libraw, libtiff, libzip, little-cms2, numpy, ocrmypdf, openblas, openjpeg, openvino, php, pillow, pipx, pngquant, poppler, python@3.11, python@3.12, rsync, tesseract, tesseract-lang, unpaper, webp, wp-cli, yt-dlp and zstd, which are currently installed.

It's been reverted now: https://github.com/Homebrew/homebrew-core/blob/9a0603b474804...

And that's a good thing.

> how does libxml2 know to decompress something? > > does it require you, as the caller, to explicitly tell it to?

In the entry point/function that we use, `xmlReadMemory` (https://gnome.pages.gitlab.gnome.org/libxml2/devhelp/libxml2...), it doesn't handle compressed XML at all.

But there are indeed others where it attempts to auto-detect compression, although as I understand it from the docs only ZLib compression is autodetected… though I suspect these may be out-of-date and it may autodetect any/all compiled -in compression algorithms.

Regardless, the fact that it links with liblzma is cause for concern, given the mechanism of operation of the liblzma/xz backdoor.

I don't think you need to worry about the C++ contribution: https://github.com/MicrosoftDocs/cpp-docs/commit/9a96311122a...

> appears to have a Chinese name

Given the complexity of the attack, I'd assume the name is fake.

The contribution to C++ is just a simple markdown change: https://github.com/MicrosoftDocs/cpp-docs/pull/4716 C++ is fine.

I would think a Chinese state hacker would not assume a Chinese name, just in case he was discovered like now.

[flagged]

No one is being “required by law” to add vulnerabilities, it’s more likely they are foreign agents to begin with.

This kind of shallow dismissal is really unhelpful to those of us trying to follow the argument. You take a tone of authoritative expert, without giving any illuminating information to help outsiders judge the merit of your assertion. Why is it a very bad reading of the current situation? What is a better reading?

Apparently there's a wayback machine for git repos and it "just coincidentally" archived this repo the day before the news broke:

https://archive.softwareheritage.org/browse/origin/visits/?o...

> https://git.tukaani.org/xz.git

it's throwing 403 now.

This entire thread is above my pay grade, but isn’t minimizing the attack surface always a good thing?

Are you sure about that? The diff moves away from using the compromised tarballs to the not-compromised (by this) git source. The comment message says it's about reproducibility, but especially combined with the timing it looks to me like that was just to avoid breaking an embargo.

5.6.1-1 was built from what I understand to be one of the affected tarballs. This was patched in 5.6.1-2: https://gitlab.archlinux.org/archlinux/packaging/packages/xz...

I agree on the sshd linking part.

Deleted per below

My Arch setup is the same, they must not patch openssh.

The emojis reduce (but not eliminate) the number of "me too!"s PRs will get, which IMO is a good thing.

Many people write code for fun and slack is a social communications platform.

If you can't imagine people using these tools for other reasons than pure unemotional business value then you don't understand their market.

Your suggestions would lose those platforms users and revenue.

I agree there's safer languages than C, but nobody reads the 50,000 lines changed when you update the vendoring in a random golang project. It would be easy to introduce something there that nobody notices too.

You don't need a complex obscure build system for most C code. There's a lot of historical baggage here, but many projects (including xz, I suspect) can get away with a fairly straight-forward Makefile. Double so when using some GNU make extensions.

Rust is the worst in terms of build system transparency. Ever heard of build.rs? You can hide backdoors in any crate, or in any crate's build.rs, or the same recursively.

Wouldn't a supply chain attack like this be much worse with Rust and Cargo because of the fact it's not just a single dynamic library that needs to be reinstalled system-wise, but, instead, every binary would require a new release?

People are going to be upset with this perspective but I completely agree. The whole autoconf set of tools is a complete disaster.

Once somebody actually does this people are gonna complain the same as always: "The sole purpose of your project is to rewrite perfectly fine stuff in Rust for the sake of it" or something along these lines.

Is this really the lesson here? We are talking about a maintainer here, who had access to signing keys and a full access to the repository. Deb packages which were distributed are also different than the source code. Do you honestly believe that the (arguably awful) autotools syntax is the single root cause of this mess, Rust will save us from everything, and this is what we should take away from this situation?

I call bullshit.

The fundamental problem here was a violation of chain of trust. Open source is only about the source being open. But if users are just downloading blobs with prebuilt binaries or even _pre-generated scripts_ that aren't in the original source, there is nothing a less-obscure build system will save you from as you are putting your entire security on the chain of trust being maintained.

As far as I remember, he added no backdoors.

He was a consultant/sysadmin for Intel, and he did 3 things which he thought his employer would support, and was astonished to find that not only did his employer not support, but actively had him prosecuted for doing it. Ouch.

1. He ran a reverse-proxy on two machines so he could check in on them from home.

2. He used the crack program to find weak passwords.

3. He found a weak password, and used it to log into a system, which he copied the /etc/shadow file from to look for additional weak passwords.

https://www.giac.org/paper/gsec/4039/intel-v-randal-l-schwar...

https://web.archive.org/web/20160216204357/http://www.lightl...

He didn't try and hide his activities, and didn't do anything else untoward, it was literally just these things which most people wouldn't bat an eyelid at. These days, it is completely normal for a company to provide VPNs for their employees, and completely normal to continually scan for unexpected user accounts or weak passwords. But... because he didn't explain this to higher-ups and get their buy-in, they prosecuted him instead of thanking him.

Seems a little different. Based on a quick read, he gained unauthorized access to systems.

In this case, backdoor code was offered to and accepted by xz maintainers.