Post Mortem: axios NPM supply chain compromise

(github.com)

64 points | by Kyro38 5 hours ago

8 comments

  • Zopieux 3 hours ago
    Not much we didn't know (you're basically SOL since an owner was compromised), however we now have a small peek into the actual meat of the social engineering, which is the only interesting news imho: https://github.com/axios/axios/issues/10636#issuecomment-418...
    • hatmanstack 3 hours ago
      jasonsaayman and voxpelli had useful write ups from the "head on a swivel" perspective of what to watch out for. Jason mentioned "the meeting said something on my system was out of date." they were using Microsoft meeting and that's how they got RCE. Would love more color on that.
    • lrvick 1 hour ago
      An owner being compromised is absolutely survivable on a responsibly run FOSS project with proper commit/review/push signing.

      This and every other recent supply chain attack was completely preventable.

      So much so I am very comfortable victim blaming at this point.

      This is absolutely on the Axios team.

      Go setup some smartcards for signing git push/commit and publish those keys widely, and mandate signed merge commits so nothing lands on main without two maintainer sigs, and no more single points of failure.

      • fortuitous-frog 1 hour ago
        Did you investigate the maintainer compromise and publication path? The malicious version was never committed or pushed via git. The maintainer signs his commits, and v1 releases were using OIDC and provenance attestations. The malicious package versions were published locally using the npm cli after the maintainer's machine was compromised via a RAT; there's no way for package maintainers to disable/forbid local publication on npmjs.

        It seems the Axios team was largely practicing what you're preaching. To the extent they aren't: it still wouldn't have prevented this compromise.

        • lrvick 35 minutes ago
          I can not find a single signed recent commit on the axios repo. It is totally yolo mode. Those "signed by github" signatures are meaningless. I stand by my comment in full.

          One must sign commits -universally- and -also- sign reviews/merges (multi-party) and then -also- do multi party signing on releases. Doing only one step of basic supply chain security unfortunately buys you about as much defense as locking only a single door.

          I do however certainly assign significant blame to the NPM team though for repeatedly refusing optional package signing support so packages with signing enabled can be refused at the server and client if unsigned by a quorum of pinned keys, but even aside from that if packages were signed manually then canary tools could have detected this immediately.

      • TheTaytay 1 hour ago
        It wasn’t done through git. It was a direct npm publish from the compromised machine. If you read further down in the comments (https://github.com/axios/axios/issues/10636#issuecomment-418...), it seems difficult to pick the right npm settings to prevent this attack.

        If I understand it correctly, your suggestions wouldn’t have prevented it, which is evidence that this is not as trivially fixable as you believe it is.

        • lrvick 20 minutes ago
          To prevent supply chain attacks you need multi party cryptographic attestation at every layer, which is pretty straight forward, but you are correct, NPM and GitHub controls absolutely will not save you. Microsoft insists their centralized approach can work, but we have plenty of evidence it does not.

          Operate under the assumption all accounts will be taken over because centralized corporate auth systems are fundamentally vulnerable.

          This is how you actually fix it:

          1. Every commit must be signed by a maintainer key listed in the MAINTAINERS file or similar

          2. Every review/merge must be signed by a -second- maintainer key

          3. Every artifact must be build deterministically and be signed by multiple maintainers.

          4. Have only one online npm publish key maintained in a deterministic and remotely attestable enclave that validates multiple valid maintainer signatures

          5. Automatically sound the alarm if an NPM release is pushed any other way, and automatically revoke it.

  • robshippr 2 hours ago
    The interesting detail from this thread is that every legitimate v1 release had OIDC provenance attestations and the malicious one didn't, but nobody checks. Even simpler, if you're diffing your lockfile between deploys, a brand new dependency appearing in a patch release is a pretty obvious red flag.
    • GCUMstlyHarmls 46 minutes ago
      To be honest, I would have assumed the tooling would do attestation verification for me. The diffing the lockfile would be on me though.
    • clawfund 2 hours ago
      npm could solve half of this by letting packages opt into OIDC-only publishing at the registry level. v1 already had provenance attestations but the registry happily accepted the malicious publish without them.
  • anematode 1 hour ago
    Looks like a very sophisticated operation, and I feel for the maintainer who had his machine compromised.

    The next incarnation of this, I worry, is that the malware hibernates somehow (e.g., if (Date.now() < 1776188434046) { exit(); }) to maximize the damage.

  • akersten 3 hours ago
    Any good payload analysis been published yet? Really curious if this was just a one and done info stealer or if it potentially could have clawed its way deeper into affected systems.
  • fraywing 3 hours ago
    Incredible uptick in supply chain attacks over the last few weeks.

    I feel like npm specifically needs to up their game on SA of malicious code embedded in public projects.

    • simulator5g 3 hours ago
      That's the reality of modern war. Many countries are likely planting malware on a wide scale. You can't even really prove where an attack originated from, so uninvolved countries would also be smart to take advantage of the current conflict. Like if you primarily wrote German, you would translate your malware to Chinese, Farsi, English, or Hebrew, and take other steps to make it appear to come from one of those warring countries. Any country who was making a long term plan involving malware would likely do it around this time.
    • ipnon 3 hours ago
      NPM is designed to let you run untrusted code on your machine. It will never work. There is no game to step up. It's like asking an ostrich to start flying.
      • dcrazy 3 hours ago
        It’s far from a complete solution, but to mitigate this specific avenue of supply chain compromise, couldn’t Github/npm issue single-purpose physical hardware tokens and allow projects (or even mandate, for the most popular ones) maintainers use these hardware tokens as a form of 2FA?
        • yjftsjthsd-h 2 hours ago
          What would a physical token give you that totp doesn't?

          Edit: wait, did the attacker intercept the totp code as it was entered? Trying to make sense of the thread

          • dcrazy 2 hours ago
            The attacker installed a RAT on the contributor’s machine, so if they had configured TOTP or saved the recovery codes anywhere on that machine, the attacker could defeat 2FA.
            • yjftsjthsd-h 30 minutes ago
              Oh, yes, I missed that the TOTP machine was compromised:\ Would that then imply that it would have been okay if codes came from a separate device, eg. a TOTP app on a Palm OS device with zero network connectivity? (Or maybe these days the easiest airgapped option is an old android phone that stays in airplane mode...)
              • dcrazy 7 minutes ago
                The easiest approach is a provider-issued hardware dongle like a SecurID or Yubikey. Lack of end-user programmability is a feature, not a bug.
                • yjftsjthsd-h 4 minutes ago
                  > Lack of end-user programmability is a feature, not a bug.

                  I would argue that the problem is network accessibility, not programmability.

              • nurettin 8 minutes ago
                Yes, unfortunately authenticator apps just generate TOTP codes based on a binary key sitting in plain sight without any encryption. Not that it would help if the encrypting/decrypting machine is pwned.
      • lrvick 1 hour ago
        All maintainers need to do is code signing. This is a solved problem but the NPM team has been actively rejecting optional signing support for over a decade now. Even so maintainers could sign their commits anyway, but most are too lazy to spend a few minutes to prevent themselves from being impersonated.
        • yawaramin 18 minutes ago
          If the solution is 'maintainers just need to do xyz', then it's not a solution, sorry. It's not scalable and which projects become 'successful' and which maintainers accidentally become critical parts of worldwide codebases, is almost pure chance. You will never be able to get all the maintainers you need to 'just' do xyz. Just like you will never be able to get humans to 'just' stop making mistakes. So you had better start looking for a solution that doesn't rely on humans not making mistakes.
          • jiggawatts 4 minutes ago
            "Discipline doesn't scale" as become one of my favourite quotes for a reason.
  • lrvick 1 hour ago
    I ask this on every supply chain security fail: Can we please mandate signing packages? Or at least commits?

    NPM rejected PRs to support optional signing multiple times more than a decade ago now, and this choice has not aged well.

    Anyone that cannot take 5 minutes to set up commit signing with a $40 usb smartcard to prevent impersonation has absolutely no business writing widely depended upon FOSS software.

    Normalized negligence is still negligence.

    • 4ndrewl 1 hour ago
      Is the onus really on people who write code here? It really should be on those who choose to use this unsigned code, surely?
      • lorenzohess 1 hour ago
        Perhaps, but if it's gotten to the point where millions of people download the unsigned code, signing should probably become required. Even reproducible builds.
        • 4ndrewl 1 hour ago
          Required by who though? If your business etc depends upon some code, it's up to you to ensure its quality, surely? You copy some code onto your machine then it's your codebase, right?
          • lrvick 1 hour ago
            While I think anyone unwilling to sign their code is negligent, I also feel anyone unwilling to ensure credible review of code has been done before pushing it to production is equally negligent.
      • lrvick 1 hour ago
        Anyone that maintains code for others to consume has a basic obligation to do the bare minimum to make sure their reputations are not hijacked by bad actors.

        Just sign commits and reviews. It is so easy to stop these attacks that not doing so is like a doctor that refuses to wash their hands between patients.

        If you are not going to wash your hands do not be a doctor.

        If you are not going to sign your code do not be a FOSS maintainer.

  • uticus 4 hours ago
    > March 31, around 01:00 UTC: community members file issues reporting the compromise. The attacker deletes them using the compromised account.

    Interesting it got caught when it did.

  • charcircuit 3 hours ago
    Does OIDC flow block this same issue of being able to use a RAT to publish a malicious package?
    • fortuitous-frog 2 hours ago
      No. axios (v1 at least; not v0) were setup to publish via OIDC, but there's no option on npmjs for package maintainers to restrict their package to *only* using OIDC. The maintainer says his machine was infected via RAT, so if he was using software-based 2FA, nothing could have prevented this.
    • hsbauauvhabzb 3 hours ago
      No, once the computer is compromised nothing really helps assuming the attacker is patient enough.