Our open supply construct system

• Buck2, our new open source, large-scale build system, is now accessible on GitHub.
• Buck2 is an extensible and performant construct system written in Rust and designed to make your construct expertise quicker and extra environment friendly. 
• In our inside assessments at Meta, we noticed that Buck2 accomplished builds 2x as quick as Buck1.

Buck2, Meta’s open supply large-scale construct system, is now publicly accessible by way of the Buck2 website and the Buck2 GitHub repository. Whereas it shares some commonalities with different construct techniques (like Buck1 and Bazel), Buck2 is a from-scratch rewrite. Buck2 incorporates a full separation of the core and language-specific guidelines, with elevated parallelism, integration with distant execution and digital file techniques, and a redesigned console output. All of those modifications are geared toward serving to engineers and builders spend much less time ready, and extra time iterating on their code.

1000’s of builders at Meta are already utilizing Buck2 and performing thousands and thousands of builds per day, with builds finishing twice as quick as with Buck1. Our personal inside evaluation has proven that engineers have been capable of produce meaningfully extra code when their builds have been executed by Buck2, and we hope the broader business may even see advantages.

Why rebuild Buck?

Construct techniques stand between a programmer and working their code, so something we are able to do to make the expertise faster or extra productive instantly impacts how efficient a developer might be. The objective of Buck2 was to maintain what we preferred about Buck1 (the core ideas and workflows), draw inspiration from improvements after Buck1 (together with Bazel, Adapton, and Shake), and concentrate on pace and enabling new experiences.

Buck2’s design relies on the next rules:

• The core construct system has no information of any language-specific guidelines. Having the foundations separated from the core signifies that the foundations are simpler to vary and perceive. The core of Buck2 is written in Rust, and its language guidelines (like methods to construct C++) are written in Starlark. This separation is in distinction to Buck1 (the place all guidelines are written within the core) and Bazel (the place C++/Java are written within the core).
• The construct system is powered by a single incremental dependency graph, avoiding any phases (in distinction to Buck1 or Bazel). This determination eliminates many sorts of bugs and will increase parallelism.
• The principles API is designed to include superior options for efficiency, together with dynamic (or monadic) dependency options for expressibility. On the similar time, these options are rigorously restricted to make sure different properties (for instance, quick queries or hermeticity) will not be harmed.
• The open supply launch is nearly an identical to our inside model. The one items swapped out are the toolchains (which level on the inside copies of our compilers) and distant execution (which factors at our inside servers) — each have open supply alternate options equipped. We’re additionally releasing all the foundations precisely as they’re used internally. Moreover, we have now separated among the logical parts into separate crates (e.g. Starlark, Superconsole, Allocative, Gazebo) in order that they can be utilized outdoors Buck2.
• Buck2 is written to combine with distant execution, with the power to run actions on distant machines. We use the identical API as Bazel, and have been testing distant execution with Buildbarn and EngFlow. Whereas not required (and not likely anticipated for folks beginning out with the open supply model), we’re capable of effectively compute recursive digests and ship them to distant execution effectively.
• Buck2 is written to combine with digital file techniques, the place the complete repository isn’t all checked out, however fetched on demand because the recordsdata are accessed. Particularly, we assist Sapling-based file systems. To combine effectively, we look ahead to file notifications (with Watchman) and request each recordsdata and file-digests with out direct file operations. The profit is that we are able to make digital file techniques as quick as a full checkout, however with the advantages of a lot quicker checkout and far decrease disk utilization.

The important thing takeaway from all these enhancements is that we have now designed Buck2 to be quick. In actual world utilization, relying on the construct, Buck2 is considerably quicker than Buck1. If there are not any supply code modifications, Buck2 is nearly prompt on subsequent builds. If there’s a number of work to do, Buck2 begins executing quicker and has higher parallelism. This improve in pace is each a consequence of lots of the elements above, but in addition care and a focus.

The consumer view

For finish customers, Buck2 works largely the identical as Buck1 (which, to a primary approximation, is pretty just like Bazel). A consumer defines targets in a BUCK file:

rust_binary(
    identify = “my_binary”,
    srcs = [“main.rs”],
    deps = [“:my_library”],
)

A consumer can then construct with buck2 construct //:my_binary. The worth foremost.rs is a supply file, and :my_library is a dependency outlined in the identical BUCK file. It’s price noting that Buck2 is usually appropriate with the BUCK recordsdata of Buck1. 

In addition to the rise in pace, there are two main further user-visible variations in comparison with Buck1.

First, the console output has been redesigned on high of the Superconsole library, which we particularly developed for Buck2. The console exhibits a couple of extra particulars and feels loads nicer to make use of:

Second, there’s a persistent daemon that maintains a single dependency graph. Once you change a BUCK file, a dependency, or a supply file, we invalidate the suitable issues on the dependency graph, then request the output artifacts per the command line. In Buck1 there are a number of distinct dependency graphs, which end in phases like goal graph development, motion graph development, after which motion graph execution. There are additionally some operations that aren’t carried out on the graph. If sure issues change in Buck1, then complete graphs are thrown away, fairly than the minimal items being invalidated. With a single dependency graph, Buck2 is easier, avoids extra redundant work, and avoids express phases. All the pieces on the dependency graph has a key (how it’s recognized) and a worth, together with a perform to compute the worth from the important thing and different associated keys (following the mannequin within the paper, “Build Systems a la Carte”).

The rule writer view

Whereas the consumer mannequin follows Buck1 very intently, the strategy for guidelines is totally completely different. There are many guidelines in Buck, for instance rust_binary used above. Whereas a rule in Buck1 was a Java class, baked into Buck1, a rule in Buck2 is totally decoupled. Buck2 additionally ships with a “prelude” of guidelines that implement a lot of the Buck1 guidelines. 

Buck1 guidelines have been tuned over time, had a lot of efficiency optimizations and highly effective options like graph traversal, however these guidelines have been additionally anticipated to obey a number of advanced invariants—generally breaking these guidelines. For Buck2, the rule API is totally in Starlark, which pressured us to summary these options as generically reusable APIs, aiming to make them secure, expressive, and performant—a tough stability. We’ll contact on two such examples.

OCaml dependencies

The dependency construction of the OCaml library is difficult to specific in Buck1. An OCaml library consists of a variety of OCaml recordsdata. These have to be compiled in dependency order—so if A.ml makes use of B.ml, you could compile B.ml first. Bazel requires the dependency of A.ml on B.ml to be written explicitly within the BUCK file. Buck1 and Buck2 each depart that inside dependency implicit and run the software ocamldep to deduce it, which requires much less upkeep because the construction modifications.  What Buck1 did is run ocamldep simply after parsing the BUCK file, which wasn’t actually allowed, and it didn’t monitor dependencies, so should you modified the imports an excessive amount of Buck1 gave spurious compilation failures. With Buck2, we are able to use the new primitive dynamic_output, which helps you to run a command, learn the output of the file, then wire up the remainder of the graph—placing within the appropriate dependencies between the .ml recordsdata mechanically.

C++ hyperlink dependencies

Think about the C++ linking mannequin: To provide a library, you often have to hyperlink collectively its construct output, together with the transitive closure of the construct output of its dependencies. Should you merely duplicate the set of dependencies at every layer as you progress up the graph, you find yourself with O(n²) reminiscence utilization. In Buck1, there was customized code in lots of guidelines to seize this sample, counting on the power to share Java values in reminiscence and for the dependencies to be represented in place throughout the rule construction (as there was no reified dependency graph). In Buck2, there are a lot stronger abstraction boundaries, so such reuse needs to be made extra express. Due to this fact, we introduced transitive-sets (tsets) to seize this sample of units representing a transitive closure. By making tsets extra summary, we have been additionally capable of wire the tset instantly into the underlying dependency graph, which means this illustration is environment friendly in each reminiscence and computation time.

Strive Buck2 now

We’re eager for folks to provide Buck2 a strive, and we might be completely happy to listen to any suggestions (GitHub issues are one of the best ways). We anticipate Buck2 will likely be most attention-grabbing to reasonably sized multi-language initiatives. Go to the Buck2 getting started page for extra info.