All Articles

All You Need to Know About the October 2026 Cloud Breaches: TiDB, eMudhra, LiteLLM

All You Need to Know About the October 2026 Cloud Breaches

The TiDB Cloud Breach

Introduction

On 19 October 2026, PingCAP published a security update confirming that someone had accessed its internal systems between 9 September and 7 October, copied data, and took source code. It says that data stored in TiDB is not impacted.

A cyber-security research group (FulcrumSec) published a security breach report on its own account. It says it took 10.6 TB, including the control plane of TiDB Cloud, the SQL that customers typed into the product, four years of connection logs, the CRM, the billing system, and 413 private repositories. It has published about 200 GB as a sample and says more is coming. It also says it did this for nearly two months without a single alert.

Key Takeaways

  • TiDB is a MySQL-compatible distributed database. TiDB Cloud is PingCAP running it for customers. The breach hit the layer that manages customers. PingCAP says it found no evidence that data stored inside TiDB was impacted.
  • The control plane is the crown jewels of any multi-tenant platform. It holds the credentials, so a breach there reaches every tenant at once.
  • A 2022 internal ticket already documented that every role in a customer organisation, including billing admin, could reset the root password on every cluster.
  • FulcrumSec’s totals and PingCAP’s account disagree on scale and on whether the stolen keys were usable. Both agree that customer data and source code left the building.

What TiDB actually is

TiDB is an open-source database that speaks MySQL. If your application already talks to MySQL, you can point it at TiDB instead and mostly not change your code. The difference is that TiDB splits storage and query processing across many machines, so you can add capacity by adding servers rather than buying a bigger one.

PingCAP runs TiDB cloud on infrastructure it operates. You get a connection string instead of a rack of servers.

Control Plane : It decides who you are and what you can touch. It holds accounts, organisations, projects, cluster records, API keys, billing details, and support history. Since TiDB cloud is multi-tenant, all these tables contains all the tenants data.

Data Plane : It has your tables, your queries, your backups.

A useful analogy is an apartment building. The data plane is the apartments. The control plane is the front desk, the tenant list, and the cabinet of spare keys.

What FulcrumSec says it took

The group’s own summary, with the counts rounded:

10.6 TB taken, 4.9 million files, 88 exports of the production control plane, one every six hours, none detected, about 50 GCP storage buckets, a 534-million-row BigQuery warehouse, 180K tenant organisations, 403K unique email addresses, 115K API keys stored in plain text, 43,000 still active, 20.6 million customer SQL statements, 102 million gateway connections, 5 million customer server IPs, 377K Jira issues, 434K CRM contacts, 92 GB of per-tenant AWS cost data, 413 private source repositories

Everything in this section comes from that writeup. PingCAP disputes parts of it saying none of the API keys are stored in plaintext, they are encrypted. PingCAP confirmed that user names and email addresses, organization and project associations data is leaked.

Custome SQLs

TiDB Cloud has a SQL editor and a natural-language query feature called Chat2Query. Both run statements on PingCAP’s side, which means PingCAP’s systems see the text.

The group says it captured the raw statements, keyed by organisation, project, and cluster. The important part is not the count. It is that customers typed their own schema, their own table names, and sometimes their own data into a box that logged it.

Four years of who connected from where

TiDB Cloud runs a gateway in front of customer connections. Its access logs record who connected, from which IP address, to which cluster, using which client program.

The group says the logs cover 846K clusters, with client fingerprints that include Rails, ProxySQL, MySQL Workbench, and Dify. That is not data in the usual sense. It is a map of the installed base: which companies use TiDB, where their application servers are, and what software they run.

The business behind the database

AWS publishes a Cost and Usage Report: a line-by-line bill of everything a company spends. PingCAP tagged each line with the customer it belonged to. The group says the dataset covers 57 months and about $107M of AWS spend, with enterprise discounts of roughly 47% off list and $16.5M in cross-AZ transfer costs alone.

Join that to the finance snapshots and the annual-recurring-revenue logs, and you can compute gross margin per named customer. The result is the entire commercial engine of a private company, exposed as a spreadsheet.

The CRM is similarly specific. HubSpot allegedly holds 434K contacts, 191K companies, and deals with a nominal value of $552M:

413 repositories, and two design choices worth reading twice

The group says 413 repositories with source code is leaked. The group describes a billing engine that trusts an X-ACCOUNT-USER-ID header without verifying it. A header is just text the caller sends with a request. If the server believes it, the caller chooses which account it is. That is not authentication. It is a name tag.

It describes an account service where “admin” is a tenant ID and impersonation skips organisation checks while granting the owner role. In plain terms, administrative identity is a number you can type.

All roles of the organization can reset root password🫣

“All roles of the organization (including: owner, audit admin, billing admin, member) can unconditionally reset the root password of all clusters in the organization. This will cause all users in the organization to reset the cluster root password, and then users can connect to the cluster as root, modify the cluster state, and access cluster data.”

Read that again. Not “an owner can reset a root password”. Every role. Owner, audit admin, billing admin, member. Every cluster in the organisation. Root can read every table, change every row, and drop everything.

PingCAP’s security team wrote this down in August 2022. The breach happened in 2026.

What PingCAP says, and where the stories differ

PingCAP’s report is worth reading in full, because it is more specific than the headlines suggest. It confirms unauthorised access between 9 September and 7 October using stolen service account credentials, confirms that data was copied, and confirms that source code was taken. It says four vulnerabilities were reported by an external firm on 12 and 13 October and fixed by 14 October.

It also pushes back on several of the group’s claims:

Question FulcrumSec PingCAP
Volume 10.6 TB across 4.9 million files No total published
Window Exports from 8 to 30 September Access from 9 September to 7 October
API keys 115,000 keypairs in plain text, 43,000 active Exported table held encrypted key values; the private portion was not exported, and a test value could not be used as a key
Customer database contents 20.6 million SQL statements; external endpoints with root credentials No evidence that data stored in TiDB was impacted
Credentials in support tickets Live AWS, GitHub, and AI platform keys Claims about credentials in support materials have not been verified
Entry point Credentials committed in PingCAP repositories Stolen service account credentials; four vulnerabilities fixed, earlier exploitation still under review

Learnings for a multi-tenant product platform

The pattern here is not exotic. Every item on this list is a known failure mode that appears in postmortems every year:

  1. Treat the control plane as production data. It holds the credentials for every tenant. Encrypt per tenant, not with one global secret, and assume a single config leak should not unlock everything.
  2. Remove global impersonation. If support needs to act as a customer, make it time-boxed, approved, logged, and scoped to the ticket. “Admin is a tenant ID” is not a design.
  3. Test your role model at the database boundary. Log in as a billing admin and try to reset a cluster password. If it works, you have the 2022 PingCAP ticket.
  4. Scan repositories for cloud credentials continuously. Deleting the commit does not revoke the key. Assume any committed key is already in use.
  5. Alert on exports, not just logins. A new service account reading every table is the signal. Login alerts will not catch it.
  6. Put Jira, the CRM, and the analytics warehouse in scope. They hold credentials, personal data, and revenue figures. Most threat models stop at the production VPC.
  7. Ask vendors the awkward questions. Who can impersonate a tenant? Where are our credentials encrypted, and with what key? Can you show export alerting? What happens when a support engineer pastes a connection string into a ticket?

If you are a TiDB Cloud customer

  • Rotate TiDB Cloud API keys. PingCAP says the private portion was not exported, and it is still the cheapest precaution available.
  • Review access logs from 9 September to at least 14 October for new keys, role changes, unfamiliar IPs, and cluster configuration changes.
  • Rotate any database credential that ever appeared in a support ticket or was shared with PingCAP support.
  • Expect convincing phishing. Your account name, contract terms, and billing contact may be in the leak, which is enough to write an email that looks internal.

The eMudhra CA Key Compromise

Key Takeaways

  • eMudhra, an Indian certificate authority, generated eight intermediate CA keys with values below 256. Anyone can recover them by trying all 256 possibilities.
  • Browsers blocked the intermediates within days, but CI runners, container images, JVMs, Go binaries, and CA bundle packages often still trust them.
  • Forged certificates for major domains have appeared in Certificate Transparency logs since 19 October. No confirmed attack yet.
  • The risk is concentrated in machine-to-machine traffic: build pipelines, API clients, and installers that hold cloud keys and do not check revocation.

What a certificate authority actually does

When your browser connects to https://bank.example, the server presents a certificate. The browser does not take it at face value. It checks the certificate’s integrity in three main ways:

  • The issuer. The certificate names the CA that signed it and carries that CA’s signature. The browser only accepts a signature from a CA it already trusts.
  • The expiry date. The certificate has a validity window with a start and end date. An expired certificate, or one that is not valid yet, fails.
  • The certificate chain. The server usually sends the site certificate plus one or more intermediate certificates. The browser follows the signatures upward until it reaches a root certificate in its trust store. If any link is missing or invalid, the browser rejects the certificate.

The browser also checks that the certificate covers the hostname it asked for, and it may check revocation lists to see whether the certificate was cancelled before its expiry date.

The chain is the part worth understanding. A root CA is usually kept offline and is used only to sign intermediates. An intermediate CA is a certificate that is allowed to sign other certificates. It sits between the root and the site certificate: the root signs the intermediate, the intermediate signs the site certificate, and the browser walks that path back to the root.

The split exists so the root key is not used for everyday signing. If an intermediate key leaks, the CA can revoke that intermediate and keep the root. If the root key leaks, every certificate under it is suspect.

A CA private key, whether root or intermediate, is therefore a skeleton key for the domains it can sign. If you have it, you can create a certificate for any domain, and clients that trust that CA will accept it. You do not need to break encryption. You just need to be in the path of the traffic.

eMudhra is an Indian certificate authority. It issues TLS certificates and code-signing certificates, and its roots are trusted by browsers, operating systems, Java runtimes, and many software packages.

The keys were a number below 256

On 30 April 2025, eMudhra created eight intermediate CAs. These are the CA certificates that sign the site certificates users see.

The private keys were supposed to be 384-bit numbers. They were not. Each one was a number below 256. That means there are only 256 possible keys, and a laptop can try all of them in under a second.

eMudhra says a bug in its key generation script produced a second set of weak keys, and staff picked the wrong set because the labels looked similar. One researcher later published a recovered key whose value was simply 4.

Eight intermediates came out of that mistake. Six issue TLS certificates, and two issue code-signing certificates. Each one was also cross-signed by Sectigo, which gave it a second path to a trusted root. Sixteen certificates, eight keys.

The discovery and the browser response

A researcher credited as Wayne found the weakness on 17 October 2026 while scanning public certificate data. Chrome and Firefox blocked the intermediates the same day. Apple followed on 19 October, Microsoft on 20 October, and Mozilla removed the emSign ECC Root CA - G3 root from NSS 3.131.1.

eMudhra revoked the eight certificates on 18 and 19 October, and Sectigo revoked its cross-signed copies. Go released 1.27.3 and 1.26.10. Node.js, Red Hat, and the major Linux distributions followed with updates. Chrome will remove the root on 27 October.

That is the fast part. Browser vendors can push a blocklist to every browser without waiting for a release. The rest of the world cannot.

The part that is still exposed

A TLS client is affected if three things are true:

  1. It trusts emSign ECC Root CA - G3, emSign Root TLS CA - G3, or AAA Certificate Services.
  2. It does not block the 16 intermediate certificates.
  3. It does not check revocation.

Browsers are not affected because their vendors pushed a block. Many other clients are. Container images, JVMs, Go binaries compiled before the fix, Python’s certifi, Rust’s webpki-roots, PHP’s composer/ca-bundle, Electron apps, and the command-line tools that ship with macOS all read a CA bundle from disk and often skip revocation checks. A local machine or a cloud CI runner still running an older version of one of those bundles is exposed, even if the browser on the same machine is patched.

What to do

  • Check every server certificate you operate. Replace any that chain to emSign ECC Root CA - G3 now; Firefox already rejects them, and Chrome will on 27 October.
  • Test your clients where they run: CI jobs, production images, developer laptops, and every TLS inspection proxy. The tracker at emudhra.we-have-a-problem.com has probes for curl, Python, Node.js, Go, Java, PHP, and Rust.
  • Rebuild container images with an updated ca-certificates package. A host upgrade does not change an image that is already built.
  • Rebuild Go binaries with 1.27.3 or 1.26.10 or later. The fix is compiled into the binary; a new Go version on the build host does not help a deployed binary.
  • Upgrade JDKs, or add the 16 intermediates to blocked.certs if your vendor has no fix. Updating the OS trust store does not update a JDK’s own cacerts.
  • Upgrade certifi, webpki-roots, rustls-webpki, composer/ca-bundle, and Electron.
  • Search proxy and TLS inspection logs for the 16 intermediates. If a forged certificate was used by a client that sent secrets, rotate those credentials and start an incident.

Language and runtime updates

The lesson

A CA key is not supposed to be guessable. That is the whole point of a CA. The eMudhra incident is a reminder that trust is transitive: your browser trusts a root, the root trusts an intermediate, the intermediate signs a certificate, and every client that skipped the blocklist inherits the weakness.

The other lesson is about time. Browsers can block a bad CA in hours. An image built last year and never rebuilt, a Go binary on a shelf, or a JDK that ships its own trust store can keep trusting it for years. The patch is not the hard part. Finding every copy of the trust decision is.

To learn more about how TLS and mTLS work, see SSL/TLS and mTLS.

The LiteLLM Supply Chain Breach

What LiteLLM actually is

LiteLLM is an open-source AI gateway. It gives an application one OpenAI-compatible API for more than 100 model providers, so a team can switch between OpenAI, Anthropic, Bedrock, Azure, Gemini, and self-hosted models without rewriting the code that calls them. It also tracks spend, applies rate limits, retries failed calls, and logs traffic.

The useful part is also the dangerous part. A LiteLLM proxy sits in the middle of every model call, which means it holds the provider API keys for everything behind it. One gateway, one database, every key. That is a more valuable target than any single application.

Key Takeaways

  • LiteLLM had two supply-chain incidents: malicious PyPI releases in March 2026, and a compromised 1.107.0 release in October 2026.
  • Both payloads were credential stealers. The March one came through Trivy, a security scanner that LiteLLM ran as a GitHub Actions step in its release pipeline. Later reporting tied it to roughly 2,500 organisations.
  • The October attacker got in through GitHub CLI tokens that LiteLLM’s own gateway had logged, then used them to publish through LiteLLM’s official release pipeline.
  • The lesson is the same as TiDB’s: the layer that holds everyone’s keys is the layer worth attacking.

March 2026: two packages and a .pth file

On 24 March 2026, two LiteLLM releases, 1.82.7 and 1.82.8, appeared on PyPI with malicious code. LiteLLM says they were live for about 40 minutes before PyPI quarantined them.

1.82.7 carried a payload in proxy_server.py. 1.82.8 added litellm_init.pth. The payload was primarily a credential stealer: it collected environment variables, SSH keys, AWS, GCP, and Azure credentials, Kubernetes tokens, and database passwords. It encrypted the bundle and sent it as an outbound HTTPS request to models.litellm[.]cloud. That domain looks like LiteLLM, but it is not LiteLLM’s; the real product lives at litellm.ai. The request looks like ordinary traffic to any firewall that does not block the domain by name.

The chain that led there is worth following in order:

  1. The attacker poisoned Trivy’s GitHub Action tags. Trivy is Aqua Security’s open-source vulnerability scanner, and many projects run it inside CI. In late February, the attacker exploited a misconfiguration in Trivy’s GitHub Actions environment, took a privileged access token, and kept a foothold. Aqua rotated credentials on 1 March, but the rotation was incomplete, so the attacker stayed in. On 19 March, the attackers force-pushed 76 of the 77 trivy-action version tags to malicious commits and published a malicious Trivy binary, v0.69.4. The tags still looked normal. A workflow that referenced a version tag instead of a full commit SHA ran the attacker’s code.

  2. LiteLLM’s pipeline ran the poisoned action. LiteLLM used trivy-action in its CI/CD security scanning workflow. When the action ran, it read the runner’s environment and stole the secrets available to that job, including the PyPI credentials of a LiteLLM developer.

  3. The attacker published directly to PyPI. With that token, they uploaded 1.82.7 and 1.82.8 themselves. They did not need to merge code, pass review, or break into a developer’s laptop. They needed credentials that the publishing pipeline would accept, and the scanner inside the pipeline was the easiest place to get them.

  4. The packages ran without anyone calling LiteLLM. 1.82.7 hid the payload in proxy_server.py, so it ran when the module was imported. 1.82.8 added litellm_init.pth. A .pth file is meant to add a path to Python’s import search, but it can also contain an import line, and Python executes it when the interpreter starts. The stealer therefore ran in every Python process in that environment, including a CI job that installed LiteLLM and never called it.

The “Trivy dependency” was not a Python library that LiteLLM shipped to users. It was a security tool in LiteLLM’s build pipeline. That is why the damage cascaded: the scanner had access to the same secrets as the release job, so compromising the scanner was enough to compromise the release.

Later reporting put the fallout at around 2,500 organisations and a 153 GB archive of stolen credentials. LiteLLM has not published a victim count, so treat those numbers as reporting rather than a vendor figure.

October 2026: the release pipeline

On 21 October 2026, LiteLLM published 1.107.0. According to LiteLLM’s incident report, the PyPI release was live for about two hours and the Docker and GHCR images for about two and a half hours. The package and images contained a credential stealer and a backdoor. The investigation is still open, so treat the timeline as LiteLLM’s current account rather than a final one.

The malware came in three files, as SafeDep’s analysis lays out:

  • cook.pth runs cook.py on every Python start. It collects SSH keys, cloud credentials, CI/CD secrets, browser passwords, AI coding tool tokens, and the provider API keys stored in a LiteLLM proxy database. It also reads AWS instance metadata for role credentials.
  • sitecustomize.py starts a background process that keeps running after the main program exits. Every hour it reads a server address from a GitHub file, reruns the stealer, and executes whatever code that server returns.
  • The exfiltration endpoint was www.litellm[.]dev, another lookalike domain.

The backdoor is the more serious part. A credential stealer takes what is on the machine and leaves. The backdoor stays running in the background, checks a text file every hour for a new address, and runs whatever code that address returns. The attacker can point it somewhere else by editing one file.

The compromised Docker tags included latest, main-latest, and main-stable. A deployment that tracked latest pulled the malware without anyone changing a version number.

How the release pipeline was taken

This is the part worth reading twice.

LiteLLM had already hardened its pipeline after March. It moved releases to a private repository, used Trusted GitHub Publishers, made Docker tags immutable, and signed its images. The October attacker defeated all of it with developer tokens.

LiteLLM says the attacker had GitHub CLI credentials for three developers, and one of those developers was an organisation owner. LiteLLM believes the tokens came from the logs of an internal LiteLLM deployment that was exposed to the internet. The way they leaked is worth spelling out: LiteLLM says it observed Claude Code embedding a user’s GitHub CLI OAuth token in a response body, and its deployment logged response bodies.

With the organisation owner’s token, the attacker disabled branch protection and force-pushed a change to the release workflow. The workflow was pointed at a lookalike repository, Berri-AI/litellm instead of BerriAI/litellm. A second developer’s credential satisfied the production approval gate. A third deleted the production and staging workflow records, which limited the logs available for the investigation.

The official v1.107.0 source tag was clean. The artifacts were built from the attacker’s repository and signed by LiteLLM’s own pipeline. That is the detail to remember: a clean source tag says nothing about the binary you installed.

If you run LiteLLM

  • Check whether 1.82.7, 1.82.8, or 1.107.0 was installed anywhere, including CI images and transitive dependencies.
  • For the October incident, pin 1.106.2 or a later verified release. For March, 1.82.6 or earlier is safe.
  • Look for cook.pth, cook.py, sitecustomize.py, /tmp/remote_code_loader.lock, and PRIVCOLLECT_ACTIVE=1.
  • Do not start with pip install. Any Python start can rerun the payload. Delete the files, kill the background process it started, then rebuild the environment.
  • Rotate everything the host could reach: cloud keys, SSH keys, CI/CD secrets, GitHub tokens, LiteLLM master and virtual keys, the provider keys in the proxy database, and the DATABASE_URL password.
  • Check CloudTrail for GetCallerIdentity from the instance role around the install time, then for use of that role from outside your network.
  • Block models.litellm[.]cloud, www.litellm[.]dev, and checkmarx[.]zone at the network edge.

References

Published Oct 22, 2026

Lokesh Sanapalli is a software engineer who loves to solve real world problems using software engineering principles.