MSP Process, a software company that serves over 1,000 Managed Service Providers (MSPs) has been hacked – with the attackers using their initial access to pivot to client tenants in what runs the risk of being a troubling and cascading supply chain incident.
Cybersecurity company Huntress is a managed security partner for MSPs and was scrambling to react over the weekend. One Huntress staffer said on r/MSP that it had seen “successful malicious token grants in 47 tenants during the attack window” after the incident, with one tenant showing signs of “significant post-compromise activity.”
MSP Process software typically has User.ReadWrite.All and authentication method permissions in client tenants; any bad actor holding an MSP Process application’s client secret can “use those permissions directly in your client tenants, with no user sign-in or MFA” it admitted in an alert pushed to customers on Friday November 7.
The data reportedly stolen from MSP Process includes Microsoft Entra app client IDs and secrets, admin refresh tokens, and GDAP (Granular Delegated Admin Privileges) relationship lists. MSP Process had not commented publicly as The Stack published, but its LinkedIn page had commented below a post by Blackpoint Cyber on the incident, saying simply “We will keep everyone posted as things progress…”
Join peers following The Stack on LinkedIn
The company’s CEO Patrick Leonard was seemingly active on the /r/MSP sub-Reddit however as discussion of the breach mounted – telling customers that he would share a post-mortem. He added: “This was a very complex attack that we are still reviewing… We caught this early enough to not have had it affect all of our partners and have since reached out via email and SMS broadcast to all affected partners…”
Leonard had not responded to a request for comment from The Stack as we published. MSP Process boasts a footprint across “1000+ MSPs and IT Teams across 22 countries… We connect with your PSA (professional services automation) and remote monitoring and management tools, including ConnectWise PSA, Datto Autotask, HaloPSA, Kaseya BMS, SuperOps, and Syncro,” according to its homepage.
Huntress urged MSP Process customers to rotate secrets and check your Entra audit logs for the patterns we’ve dropped into the collapsible widget below. (h/t Huntress.)
Huntress says to check for...
Timeline (Nov 6 to 7)
* Nov 6 from about 03:50 (possibly related; none of these IPs ever got a token): password spray against MSP staff accounts from residential proxies in PL, RU, KW, TH, RO, BR, ES, CZ and US. UAs `Python-urllib/3.11` and `curl/7.88.1`. Lots of smart lockouts, no successes we saw.
* Nov 6 13:00 to 22:05: failed client secret sign ins against MSP Process modules and other apps, same proxy pool and UAs. At 22:04 a DigitalOcean host (below) tried a third party password reset app's secret.
* Nov 7 09:38 to 09:51: more failed secret tries, then a replay of a long list of MSP Process app IDs against a single tenant.
* Nov 7 10:01: first real token. Test passes ran across many partner tenants at 10:27 to 10:31 and 11:30 to 11:31 (two tokens per app).
* Nov 7 10:30 to 10:38: refresh token attempts against partner admin accounts. All failed for us.
* Nov 7 10:22 to 14:48: hands on activity in one MSP tenant (details below). Last attacker sign in we saw was 14:47:56.
What hands on looked like
Over the course of several hours, the threat actor granted the MSP Process app additional roles, planted additional secrets across dozens of apps, created additional service accounts, exfiltrated data, and re-enabled accounts disabled by Huntress we responded to the compromise.
IOCs
Infrastructure. Most of it is Microsoft Azure address space that other tenants share.
```
Azure (most activity, UA "node" / "Wget" / "Python-urllib/3.6" / bare "Mozilla/5.0"):
172[.]169[.]63[.]30 172[.]169[.]207[.]227 172[.]169[.]187[.]35 172[.]169[.]63[.]29
172[.]169[.]63[.]35 172[.]169[.]29[.]166 172[.]169[.]58[.]181 172[.]169[.]206[.]55
172[.]168[.]41[.]166 172[.]168[.]117[.]170 172[.]168[.]253[.]63
64[.]236[.]112[.]134 64[.]236[.]112[.]237 64[.]236[.]112[.]251 64[.]236[.]112[.]221
64[.]236[.]112[.]120 64[.]236[.]112[.]111 64[.]236[.]113[.]15 64[.]236[.]113[.]23
64[.]236[.]113[.]32
52[.]230[.]237[.]27 52[.]230[.]237[.]16 52[.]230[.]237[.]18 52[.]230[.]237[.]21
52[.]230[.]237[.]13 52[.]230[.]237[.]9 52[.]228[.]167[.]149 52[.]228[.]167[.]121
52[.]154[.]255[.]0 40[.]89[.]240[.]243 40[.]89[.]240[.]113 40[.]83[.]16[.]22
13[.]89[.]111[.]197 20[.]84[.]171[.]92 130[.]131[.]144[.]244 135[.]233[.]107[.]143
172[.]215[.]180[.]155
DigitalOcean (UA Python-urllib/3.6, curl/7.58.0): 143[.]198[.]68[.]206
NordVPN exit (UA curl/7.88.1, first token): 187[.]15[.]92[.]123
Residential proxy pool (Nov 6 to 7 spray / secret tries, Python-urllib/3.11, curl/7.88.1):
178[.]61[.]224[.]154 46[.]148[.]159[.]8 109[.]171[.]38[.]144 213[.]5[.]16[.]40 91[.]210[.]236[.]74
202[.]44[.]193[.]79 24[.]220[.]57[.]81 188[.]137[.]78[.]112 188[.]25[.]140[.]193 193[.]188[.]198[.]233
46[.]151[.]219[.]60 178[.]205[.]170[.]18 188[.]137[.]76[.]53 213[.]226[.]214[.]89 177[.]52[.]113[.]45
46[.]253[.]235[.]237 138[.]118[.]86[.]78
```
What to hunt for (Entra audit log, Nov 6 onward)
* `Add app role assignment to service principal` where the initiator is an MSP Process app (or any vendor app) granting roles **to itself**.
* `Update application – Certificates and secrets management` initiated by a service principal, especially in bursts across many apps.
* `Enable account` by an app shortly after a disable.
* Unified Audit Log: `MailItemsAccessed` and `FileDownloaded` where ClientAppId or ApplicationId is your MSP Process app.
* Service principal sign ins for your MSP Process apps from anything other than the vendor's usual egress, and new secret key IDs you don't recognize.
* Non interactive sign ins from the IPs above. That shows refresh token replay, even when it fails.
If you use MSP Process
- Delete every client secret on the partner apps you connected (and on any app that MSP Process could write to). Then check for secrets **you** didn't add.
- Revoke sessions for the admin who originally connected MSP Process. The vendor stored that account's refresh token. In the tenants we reviewed, those tokens had already expired and every attempt to use them failed, and deleting the app secret also blocks them. Revoking sessions closes the gap for any token we couldn't see.
- Audit the patterns above in your partner tenant, not only client tenants.
- If you keep documentation exports or password sheets in SharePoint or OneDrive, assume they were the target. Rotate what's in them.
- Don't reconnect until the vendor confirms their platform is clean. A new secret handed back to a breached platform just restarts the clock.
MSPs remain a prominent target for cybercriminals and other bad actors.
The most well-remembered incident for many will be a breach of MSP software provider Kaseya in 2021, which saw thousands of its customers hit by ransomware.
MSP Process is a 40-strong team. Many other MSPs themselves are significantly smaller. The Stack has encountered various firms over the years who are working with multiple (uncoordinated) MSPs all with remote admin access into various elements of their environment; doing patching, etc. Not all of them are wildly mature from a security perspective. In Europe, MSPs and MSSPs alike are now are explicitly categorized as a high-priority sector (Annex 1, B2B ICT service management) under the EU NIS2 Directive with associated security compliance implications.
