A supply chain breach of security scanner Trivy, which is built into many CI/CD pipelines, has escalated into a broad set of attacks on the npm ecosystem and the compromise of downstream Kubernetes clusters.

Trivy, which has been starred over 33,000 times on GitHub, is maintained by Tel Aviv-headquartered Aqua Security. The initial attack targeted aquasecurity/trivy-action, a GitHub Action used in CI/CD workflows. 

Today Aqua Security admitted that the attackers had reestablished access to its repositories in an “ongoing and evolving attack” – which has also seen its Docker images compromised, researchers at security firm Socket said. 

The attackers used their access to poison 76 of Trivy’s 77 release tags (mutable commands) via so-called “git tag repointing” – adding 105 lines of attack code to the project that executes “a multi-stage credential theft operation”, then runs Trivy itself so the workflow looks normal. 

In affected environments, the payload was designed to collect sensitive information, including API tokens, cloud credentials (AWS, GCP, Azure), SSH keys, Kubernetes tokens, Docker configuration files, Git credentials, and other secrets available within CI/CD systems... compromised workflows appeared to complete normally while silently exfiltrating data to attacker-controlled infrastructure." - Aqua Security

The attackers exploited the way in which so many organizations implicitly trust third-party CI/CD workflows, often added from the likes of the GitHub Actions community marketplace; workflows that typically grant powerful and direct access to sensitive build environments, secrets, and tokens. 

Notably, security researchers at Upwind pointed out, "the attacker may have deliberately published GitHub Immutable Releases when poisoning the tags. GitHub’s “Immutable” badge was intended as a trust signal – this attack demonstrates it cannot be relied upon in isolation to verify tag integrity."

The threat group responsible, “TeamPCP” appears to have also created the self-propagating "CanisterWorm" that has compromised multiple npm packages with a persistent Python backdoor – and pledged on Telegram to go after “many of your favourite security tools and open-source projects.”

Socket is tracking affected artifacts here.

The Trivy breach 

The incident started, Aqua Security said in a March 22 security advisory, in late February when the attackers “exploited a misconfiguration in Trivy’s GitHub Actions environment, extracting a privileged access token and establishing a foothold into repository automation and release processes…”

It added that the attackers “injected malicious code into workflows that organizations were already running. Because many CI/CD pipelines rely on version tags rather than pinned commits, these pipelines continued to execute without any indication that the underlying code had changed.”

Aqua Security admitted the Trivy attack after Crowdstrike, Socket and cybersecurity researcher Paul McCarty on March 20 all published blogs on the incident. (Crowdstrike has a technical write-up and IOCs here; Wiz too. (Notably, exfiltration of stolen credentials was initially seen via HTTPS POSTs to the typosquatted scan.aquasecurtiy[.]org domain, Crowdstrike said.) 

On March 20, Aikido researcher Charlie Eriksen published a report showing that credentials stolen in the initial compromise were used to infect npm packages with a worm the team has dubbed CanisterWorm. 

Initially manually spread using tokens stolen in the Trivy attacks, this systemd backdoor disguises itself as PostgreSQL tooling. The attackers later updated it, to scrape every npm token it can find and spawn the worm with them – i.e. by accessing packages maintained by those tokens’ users. 

Eriksen told The Stack, “We've seen TeamPCP compromise Aqua Security, and retain access over a period of time. And publicly demonstrating that they have retained access, even after Aqua believed the situation was contained. TeamPCP has stated they plan to partner with other teams, to perpetuate more chaos against security tools and open source.” 

“This underlines how critical it is, that we continue to harden our software supply chains across the board. These attacks are consistently showing how fragile the supply chain is, and how much impact a single breach can have.”

Aqua Security initially said it had  “identified and contained the attack, removing malicious artifacts from distribution channels” on March 19.  

On Sunday, March 22, TeamPCP defaced 44 repos on Aqua Security’s internal GitHub profile and exposed them to the public, researchers at OpenSourceMalware reported. The descriptions on every repo on  aquasec-com, which is separate to Aqua Security’s public GitHub profile hosting the OSS Trivy tool, read “TeamPCP Owns Aqua Security.” 

"New image tags 0.69.5 and 0.69.6 were pushed on March 22 without corresponding GitHub releases or tags. Both images contain indicators of compromise associated with the same TeamPCP infostealer observed in earlier stages of this campaign," Socket’s Philipp Burckhardt wrote.

Aikido also updated its reports of the CanisterWorm linked to the Trivy compromise and TeamPCP on March 22. According to Aikido, CanisterWorm features a “geopolitically targeted destructive payload” that has targeted Iranian Kubernetes clusters and wiped them if they were identified. 

Kubernetes clusters that aren’t set in an Iranian timezone or include Farsi were also hit – with the backdoor disguised as PostgreSQL tooling.

On Monday, Aqua Security said it had onboarded security firm Sygnia to help with the investigation. While in the process of onboarding the team on Sunday, they became aware of the ongoing repo tampering and that the threat actor had reestablished access to Aqua Security’s system. “This development suggests that the incident is part of an ongoing and evolving attack, with the threat actor reestablishing access,” the team said. 

“Our investigation is actively focused on validating that all access paths have been identified and fully closed. We have no indication that the versions of Trivy used within Aqua’s commercial products are impacted at this time.” 

Eriksen, the Aikido security researcher who discovered the CanisterWorm attack and Iranian-focussed payload said, “The most concerning part of this is the speed with which TeamPCP has been able to compromise significant targets at scale, retain, and expand access, even after discovery. 

“They appear to make great use of LLMs to improve their workflows, add capabilities, and discover targets at a scale that we are not used to…”

Eriksen clarified, “Luckily, we're not seeing… any real signs of consistent spread, based on what data can gather publicly. (...) But that could change very suddenly, as we have reason to believe the threat actors are sitting on a lot of credentials they have yet to put to use.” 

The link has been copied!