Databricks just bought… a baby Postgres provider for $1 billion
An emerging AWS Aurora Postgres rival is getting a heavyweight new owner
Databases
Databricks has agreed to buy database firm Neon, just a year after the company took its serverless, managed Postgres offering GA.
The acquisition brings an 800-pound data gorilla directly into the managed database space for the first time in a major strategic move.
Neon, founded by Heikki Linnakangas and Stas Kelvich in 2021, describes itself as a “serverless open-source alternative to AWS Aurora Postgres.”
The Apache 2.0 licensed service provider decouples compute and storage and prices them separately for independent scaling. Databricks has paid $1 billion for the company, a spokesperson confirmed to The Stack.
Databricks is targeting the agentic AI database space, it said. (Neon, founded in 2021, claims that 80% of the databases provisioned on its service were created automatically by AI agents rather than by humans.)
“The era of AI-native, agent-driven applications is reshaping what a database must do,” said Databricks’ CEO Ali Ghodsi in a canned statement.
“By bringing Neon into Databricks, we’re giving developers a serverless Postgres that can keep up with agentic speed, pay-as-you-go economics and the openness of the Postgres community,” he added.
Neon seems to actually tick the actual “serverless” box of usage-based pricing with no minimum and nigh-instant availability with an API call.
As one reviewer of Neon, Allen Helton, put it on GA last year: “One of the really cool features I liked about Neon was data branching.
"They treat database data similar to code in a GitHub repository. You can branch your data and use it in a sandbox environment, like a CI/CD pipeline or a trial run for an ETL job. When you’re done with the branch, you can either discard it or merge it back into its parent,” he added in a blog post.
“This is a huge capability for any managed database. It simplifies many workflows and provides an easy and instant way to get a snapshot of data.
"It follows a copy-on-write principle, meaning the data is copied whenever it’s modified in your branch. If you’re just doing reads it uses a pointer to look at the parent branch data. This helps reduce storage costs and increase availability of data when you branch,” Helton wrote last year.
"A fully isolated Postgres instance in 500 milliseconds or less..."
Databricks, which recently raised a colossal $10 billion at a $62 billion valuation, started as a managed Apache Spark provider and now serves a range of data wrangling capabilities via its “Databricks Intelligence Platform.” That’s a one-stop hub for data engineering, governance, science, and AI/ML: The latter building on its 2023 Mosaic AI buyout.
“Agents operate at machine speed and traditional database provisioning often becomes a bottleneck — Neon can spin up a fully isolated Postgres instance in 500 milliseconds or less and supports instant branching and forking of not only database schema but also data, so experiments never disturb production [and its] full separation of compute and storage keeps the total cost of ownership for thousands of ephemeral databases proportional to the queries they actually run.”
“The integration of Neon’s serverless Postgres architecture with the Databricks Data Intelligence Platform will help developers and enterprise teams efficiently build and deploy AI agent systems,” it added.
Neon's architecture
The Neon Control Plane manages cloud resources across both storage and compute. Storage is a multi-tenant key-value store for Postgres pages that is custom-built for the cloud, it says, with storage split into "Safekeepers, Pageservers, and cloud object storage."
"Safekeepers are responsible for durability of recent updates. Postgres streams Write-Ahead Log (WAL) to the Safekeepers, and the Safekeepers store the WAL durably until it has been processed by the Pageservers and uploaded to a cloud object store.
"Pageservers are responsible for serving read requests. To do that, Pageservers process the incoming WAL stream into a custom storage format that makes all page versions easily accessible.
"Pageservers also upload data to cloud object storage, and download the data on demand. Safekeepers can be thought of as an ultra-reliable write buffer that holds the latest data until it is processed and uploaded to cloud storage. Safekeepers implement the Paxos protocol for reliability. Pageservers also function as a read cache for cloud storage, providing fast random access to data pages" - Neon.
Incoming WAL data is "initially stored across multiple availability zones in a Paxos cluster before being uploaded to a cloud object store, such as Amazon S3 (99.999999999% durability), both in raw WAL and materialized form. Additional copies are maintained across Pageservers to enhance the read performance of frequently accessed data," says Neon.
Databricks and Neon will now work together to help customers "eliminate data silos, simplify architecture and build AI agents that are more responsive, reliable and secure" Databricks boasted. How that plays out as part of the latter's GTM will be interesting to watch once the deal close. Watch this space.
Views on what this means for the DBaaS space? Get in touch.