Can something as low level as your build tool really be strategic rather than an obvious technical choice driven by whatever language your developers code in? 

Building code is something almost every developer deals with, that’s such a notorious time sink there’s an xkcd cartoon about it. At best it’s an interruption, at worst it’s an integration nightmare. 

Google, struggling to stay on top of large amounts of code from tens of thousands of coders in half a dozen languages from C++ to Haskell and LISP, spent years creating a build system that promised to be both fast and correct by scaling out reproducible, deterministic builds that use strict dependency management to rebuild only what’s needed. 

That’s good for developer productivity, especially in large, fast-moving codebases. It’s not just about getting back to work more quickly, but staying in flow: you’re more likely to remember what you were doing if the build only takes a couple of minutes. If you’re hosting your build system in the cloud, faster builds mean a lower compute bill too.

Platform but for developers 

As the open source version of Google’s internal tool Blaze, Bazel promises the same economies of scale as any other platform engineering: “Unifying your developer infrastructure gives you more bang for the buck and overall gives you better tools for less investment,” explains EngFlow CTO Ulf Adams, who spent 11 years running the Bazel team at Google and helped open source the project in 2015. 

It's a bigger problem for large organisations but any business with multiple developer teams (or who acquires another company) is likely to use multiple languages, with a different build system for each. Take BMW, an early adopter of Bazel. 

Adams tells The Stack, “They have [the software for] the cars, of course, but they also have the web site, they have mobile apps for their customers… Companies don’t want to spend a lot of money on developer tools and the question is how do you invest that money effectively?”

Alex Eagle, another ex-Bazel team member and founder of Aspect Build says, “Companies know it's not to their advantage to have every team inventing their own stack and deploying to a different cloud and having different monitoring systems,” adding, the same applies to your build system.

Bazel may not be the best build system for any particular language or framework, “but it's the only build system for all languages and frameworks,” Eagle argues.

“A few years ago, we observed a growing trend among top tech companies toward adopting Bazel as their primary build tool,” says Justin Kaeser, head of the Bazel team at JetBrains. 

“Traditional build tools are not designed to handle projects at arbitrary scale,” Kaeser tells The Stack. Rather than focussing on a single language like most build tools, “Bazel provides a flexible framework for running build rules across multiple languages in a consistent way,” he adds.

Powerfully correct

Bazel is so good because of Google’s focus on the fundamentals of developer infrastructure, says Max Kanat-Alexander, an ex-technical lead for Google's Code Health initiative. 

The Bazel engine uses declarative build rules for each language with the details of your code and dependencies to create a graph of inputs, outputs, tools, environment variables, and dependencies. Bazel transforms that graph into your build, making sure the same inputs give you the same outputs every time, so it can be built once and resused instead of rebuilt. 

Think of the rules (written in a cutdown version of Python called Starlark) as a recipe for building things and the engine as the system that follows the recipes.

That works so well because it’s hermetic, Kanat-Alexander explains: “a self-contained build environment that couldn’t possibly depend on anything on the machine that isn’t in source control.”

“The most incredible thing was the developers realized hermeticity and reproducibility were the two most important factors for a build tool — factors almost no other build tool guarantees even today. Statically knowing the dependency graph was incredibly powerful for the tool, but also for us as developers,” he says.

The former Google lead also calls out the importance of Starlark, (which he views as “the best build configuration language of all time”), because the rules can support almost any language developers want to work with

“It was so brilliant to realize the  build configuration could be declarative in a way that worked for every programming language and everything you’d ever want to build.”

Reuse don’t rebuild

“It's about trying to build things in the most optimal way, so you can reuse as much as possible,” explains Luis Pino, co-CTO at EngFlow and another former engineer from the Bazel team.  Within the Bazel engine, code that’s already been built can be reused so you hardly ever have to do a clean build. 

“On traditional build systems, if you run clean builds you have to redo everything from scratch and that can be very expensive and take a long time,” Adams adds.

Being hermetic avoids the ‘works on my machine’ problem when asking a colleague for help. You don’t have to worry which version of a library you have installed – or whether you’re running something demanding in the background. It also lets you use the cloud instead of building locally, hoping your laptop is powerful enough to do it quickly, Pino says. 

“You can build it in the cloud and test it in the cloud, in a hermetic environment, which means you and I would be building the same and testing the same code. (...) We have the same version, same files.” 

As well as making code reusable, that makes builds – and bugs – reproducible. “If you have a problem, it's easier to debug,” Pino continues. Instead of sharing a bug report, you can share a command that lets someone else see the same failure you’re getting, which Pino says is a multiplier in larger organisations. 

Bazel has a plethora of logs, profiles and multiple query tools for investigating build performance to look for patterns and discover whether builds are consistently slow or unpredictable. In fact, you need to be careful not to gather too much data because it can get overwhelming, but telemetry helps you understand where your bottlenecks are and whether your developer and build environments are consistent (a common cause of failures).

You can take it with you

Despite staying in beta for two years, Bazel quickly became popular at companies with large amounts of code and multiple teams coding in different languages. 

“Google did zero marketing for Bazel open sourcing,” recalls EngFlow CEO Helen Altshuler (who joined the Bazel team soon after): the company just launched the website, dropped the code on GitHub and left it for Hacker News to discover, she tells The Stack. 

“Any time an engineer leaves Google for another software company, the first thing they miss is their tools. They look for the open source equivalent – and Bazel is the open source equivalent,” Altshuler says. Before it was open sourced, people tried to recreate it:  Pants at Twitter and Buck at Facebook were inspired by Blaze.

“Even in its beta state [Bazel] was miles better, so SpaceX and Robinhood and Wix and Stripe and Lyft and Uber started adopting it and replacing CMake and Maven, the hardcore build systems they relied on,” Altshuler says. 

“Bazel was developed to support the breadth of Google and the breadth of Google is very, very broad,” Adams points out, adding that even for non-Googlers “there’s a good chance some chunks of Bazel works for you, even if you don’t use everything.”

Solve for scale

Typical Bazel users have large-scale software with problems they can’t solve by tweaking existing systems, Altshuler says. “A million lines of code or more. Polyglot, so multiple programming languages which means you're going to need different build systems in a traditional setup.” She also says Bazel is often used by companies with a strong engineering base: 100 or more developers with “software engineering best practices and good engineering culture.”

“You won't see Bazel adoption in organizations that ship every quarter and have nightly builds,” according to Altshuler, “but you will see organizations adopt Bazel when time to market is important, and they are compensating their developers by time to ship. If a company has a developer productivity team, Bazel is likely one of their tools of choice.”

“That’s important because you can only get optimizations if you shape your build graph nicely,” Pino warns. “If you have spaghetti where everything depends on everything, there's really not much you can do in terms of how to build it fast, how to get it reproducible, how to get it hermetic.” 

Some of the pain of migrating to Bazel is down to the way it expects (and rewards) good development habits.

You don't have to be Google

Big names like Adobe, DoorDash, Snapchat, Nvidia, Snowflake, Salesforce, Asana, Canva, Braintree, Dropbox, Volkswagen, Toyota, Etsy, Peloton – even LinkedIn picked Bazel.

 So far, it’s less commonly used in open source, but a handful of projects including Envoy and Selenium have adopted it.

Bazel has been mainly adopted by large companies, not because they’re the only ones with these problems, “it's just they were the only ones willing to invest the effort in something so complicated,” Eagle points out.

“In practice, Bazel is very complex and complicated, it has a lot of knobs, it has a lot of concepts, it has a lot of abstractions,” Pino agrees. 

There’s on-going work to make Bazel what Eagle calls “good enough that more companies can figure it out,” which often means dealing with the legacy of Google’s choices about development practices that remain in the code. (The upcoming Bazel 9 release completes a multiyear migration to a better way of handling transitive dependencies when you have a repo and an artefact registry, which was historically complicated on Bazel as Google hadn’t needed to build the feature for itself.) 

Continuous and productive

Bazel may be complicated, but “it solves a problem everyone has,” Eagle says.

Microservices encourage engineering teams to work in isolation, even when you put multiple projects into the same repo, so it’s easier to track changes, run tests, reuse code and manage dependencies. 

That’s rarely a true “monorepo” as Google uses the term, where every project, dependency and line of code in the company is in one single code repository with no external artefacts or package managers (although Databricks uses Bazel with a monorepo containing 100 million lines of code from 3,000 developers). It’s unlikely to catch on, but Eagle prefers the term “omnirepo”: a repo with all kinds of code, in multiple languages.

Pinterest has a repo like that for its iOS builds and finds Bazel a natural fit. Snowflake picked Bazel rather than keeping two build systems – one for business logic written in Java, the other for its C++ query engine, both of which were slow and unreliable (as were the homegrown scripts that integrated them). Redfin was an early adopter (when Bazel was harder to work with than it is now), moving from Maven to get Java and JavaScript builds from the same repo.

The goal is to make continuous integration truly continuous, with everything always using the latest version of any code or library; again, that’s about business value rather than fashions in software engineering. 

“Everybody wants to have better DORA metrics, including the ability to continuously deliver. The most agile companies are successful partly because they don't have to spend weeks in a deploy cycle trying to get everything integrated,” Eagle points out. Shift left is really about saving time and money by “finding out the software doesn’t work sooner.”

Most companies can adopt Bazel without completely changing how they do software engineering and it will still bring advantages to their environments, Eagle suggests. (New and rigorous software engineering processes work best with Bazel.) 

If you have full stack developers, working on a Java or Go backend but also doing JavaScript frontend work, they can use the same commands for both. If you handle builds that call Node and Maven and cope with multiple languages like Python and Scala for data engineers as well Rust, Lua or whatever other else you use to manage infrastructure, Bazel can be an improvement on accidentally Frankenstein-ing the scripts for your CI system into your own build system.

Eagle tells The Stack, “Sometimes, my current build system is bad enough I'm willing to switch to Bazel, even if I don't have a large team or multiple languages and the most common example is CMake.”

Get some help

What Kaeser calls a “critical mass of early adopters” has created a community large enough to spawn a thriving Bazel ecosystem of open source and commercial vendors. 

Bazel gets much of its speed from caching – if one build produces an artefact, other builds can reuse it – and using remote execution to scale out builds. 

You can do that yourself with the open source Buildbarn project or choose vendors like Aspect Build, BuildBuddy and EngFlow, many of them run by former Blaze and Bazel engineers who still contribute to the project. 

“At Google, Bazel was a foundation for developer tooling,” Altshuler points out. “On top of Bazel, developers expected parallelization, distributed execution, remote execution platform, observability tools, insights into their builds.” 

As Blaze/Bazel only represents a portion of Google’s developer platform, EngFlow and other Bazel companies offer elements that wrap around and extend Bazel, such as building, testing and deploying software, but for the more complex, less homogenous infrastructure common outside Google.

 For example, Aspect Build adds a CLI with an extension language for workflow automation using Starlark and a multi-language linter.

Bazel doesn’t replace standard developer tools: there are plugins and integrations for a range of common IDEs and JetBrains recently shipped a new Bazel plugin for IntelliJ IDEA, GoLand and PyCharm. The limitations of XCode make it popular for macOS build tasks. Support for Windows, where MSBuild remains dominant, is weaker in Bazel, but improving.

Eagle calls the Bazel ecosystem “incredibly ambitious”: “there's so many things we can build on top of a multi-language build system that understands where dependencies came from and can do provenance attestation.” (That’s what Google already uses to deliver secure boot.) 

A project of two halves

Bazel is open source, but the Bazel engine is still built by Google and the weekly release on GitHub is essentially the same code as the internal Blaze codebase, Eagle says. 

“External contributors can submit changes, but ultimately Google will accept or reject those changes,” Altshuler notes. “Google created it for itself, and Google critically relies on it. Google can't afford external changes happening organically and breaking Google internally.”

That can lead to frustrations where open source contributions to the engine aren’t adopted if they break an internal workflow. “Bazel does still have something of a wall that the code gets tossed over,” Eagle agrees, although he says the Bazel team does “a good job compensating for the wall”.

Over time, the rules for each language have gradually migrated out of the engine and into Starlark.  Those community rules are managed by the open source community through the Linux Foundation. The Bazel special interest group, hosted by the foundation, has grown into a Technical Steering Committee  and helps run the annual BazelCon event. 

Even at Google the rules for languages were usually owned by different teams, making this a natural split, Pino says; some Google teams even use community rules and contribute to them. 

When the core of an open source project comes from one vendor, there can be concerns about what could happen to the project if that vendor ever decides to move on (Facebook replaced Buck with Buck2 and the original project is now archived). 

There’s no sign Google is likely to stop investing in Blaze or Bazel any time soon. Although some of the biggest Google projects like Android, Chrome and Google Earth use their own build systems, it’s recently migrated some projects like Quantum AI and Google Distributed Cloud to Bazel and has projects that rely on code portability between Bazel and Blaze. 

But as a fallback, members of the Bazel community, which include Altshuler and Eagle, set about creating the BUILD Foundation as part of the Linux Foundation, to offer governance for the community rules ecosystem and Bazel components not managed by Google. 

According to Altshuler, the foundation has received funding from Canva, Spotify and Uber which will allow the new foundation to pay for engineering time to fix issues that Google isn’t working on or improvements to the Bazel documentation (making life easier for new adopters).

The new foundation will also be “the voice of the customer for the prioritization internally at Google,” says Altshuler, which is increasingly important as “there’s now more people using Bazel outside of Google than inside.”

The link has been copied!