Of Airflow, DAGs… and hedge fund data bottlenecks

Astronomer, fresh from landing a $93 million Series D round in May, is expanding fast – including in the UK where it has opened a new office to serve as its EMEA HQ. The company provides a managed Apache Airflow service dubbed “Astro” among other offerings, helping customers like Adobe, Marriott, Sonos handle some of the gnarly “DataOps” that underpins not just AI efforts but broader work to improve data pipelines. 

Sitting down with The Stack, the company’s new R&D boss Carter Page – who recently joined from famously tech-centric hedge fund Two Sigma, where he was Head of Data Engineering – said he’d seen first-hand there “what it looks like when you really do AI, on steroids; what the data demands are.”

“When I showed up [at the fund], the data was often the bottleneck that was impacting the bottom line. What I learned over my time there was that the ways to accelerate the data, to control the data, to govern it, understand it all came back down to the orchestration layer… which was frankly, a surprise to me; was not what I was expecting” Page said.

He’s no data naïf: Prior to his time at Two Sigma, Page was at Google where he led the development of its Bigtable service from inception to one of the largest revenue products at Google Cloud. Why the surprise?