/ Compiler says no!

The second best build system

Your Makefile is likely a glorified shell script - create a much simpler build.sh instead!

I have used my share of general purpose build-systems and package managers over the years, including make, CMake, SCons, waf, ninja, webpack, gulp and others. The BabelJS setup page alone already has me scratching my head when I look at its 16 (!) supported build systems, which I assume excludes plenty others not deemed important enough for a seat at that particular table.

Unfortunately none of the tools are satisfying to use; compared to native single-language tools like cargo and elm make they are either clumsy or complicated to configure. It is hard to fault them for it though, as they lack the unfair advantage of insight into the code itself that native tools have built in.

Most developers seem to reach the particular fork in the road of being dissatisfied with their build tools at some point on their journey and face a decision: Accept fate or doing something about it? At least this time I was able to resist taking the wrong turn, one which invariably involves writing a replacement and dooming oneself to xkcd #927 in the process.

That still leaves me without a build system though, so we will attempt a building something usable out of tools available right now.

Goals and non-goals

Let us state the goals beforehand to avoid us losing track of them later:

There are some things we specifically do not want to see in our system:

We will consider some of these later in a different article.

The scenario

For the moment, we will assume we have a complicated but small application: lots of moving parts from all over but no need to squeeze every last bit of performance out of the build system yet.

Our application will use multiple tools to

While the project is not large, it involves tools from at least three different languages (SASS, Elm, Rust), with more (Javascript, HTML, CSS) being part of the end result.

Our goal is to create an easy-to-understand but maintainable way of handling all these complex tools to create the end-product.

Choosing make

Our first attempt will use make, which at the time of this writing is 44 years old. That’s older than Linux (1991), TeX (1978), or the x86 instruction set architecture (1978). It is also in widespread use and there really is no way around it - if you are working with software, knowing make is as essential as being able to write a shell-script.

Its syntax is typically concise, even though its built-in rules are not always up to the task. The model it follows, output files depending on input files, is also flexible and language agnostic.

We can count on make always being available and it handles parallel builds very well, provided it is given enough information to do so.

The big danger of make is that it lacks of some features, inviting users to write complicated Makefiles to overcome these perceived shortcomings. We will keep this in mind when writing ours, aiming for simplicity over flexibility. We are also using GNU make if not otherwise noted.

Starting out with the client app

We can start out with a simple, almost no-frills Makefile and just the Elm application:

DEBUG=0

ifeq ($(DEBUG), 1)
	ELM_FLAGS=--debug
else
	ELM_FLAGS=--optimize
endif

output/app.js: client/elm.json $(wildcard client/**/*.elm)
	cd client && elm make --output=../$@ $(ELM_FLAGS) src/Main.elm

clean:
	rm -rf output/ client/elm-stuff

.PHONY: clean

Our single configuration option is DEBUG, which we will need to distinguish between debug and release builds. The difference is huge and not just optimizations, so we cannot do away with this complexity.

We opt for an output directoy. Traditional make use will put the object files next to the source files, but by putting everything into a build directory, cleaning up is a lot simpler.

Instead of defining placeholders and customization options for everything, we just assume all our tools are in the $PATH. In the same vein, we do not define a lot of placeholders (e.g. $BUILD_DIR) but use hardcoded relative paths instead. If necessary, we can introduces these to allow overriding at a later stage.

Adding static files

With one wildcard rule, we can overlay files from a static directory, in case we need to copy them over, while the added all target, which needs to be at the top, describes how to assemble our application:

all: output/index.html output/app.js

# ...

output/%: static/%
	@mkdir -p $(dir $@)
	cp -af $< output/

# ...

.PHONY: clean all

Intermediate codegen

The intermediate codegen step through humblegen is a bit tricky; not only does its Elm backend not integrate with elm make, but it also expects to be able to write to an empty folder. Still, the resulting Makefile fragment is soothingly concise:

client/src/Api: api.humble
	@rm -rf $@ && mkdir -p $@
	humblegen --language elm --output $@ $^

# ..

clean:
	rm -rf output/ client/elm-stuff client/src/Api/

Just adding client/src/Api as dependency to our app.js and to the clean targets finishes up the Elm side of the integration.

Adding a rust application

Our Rust application lives in server and has its own target directory. Luckily humblegen supports build.rs, so we can set it up without extra effort and even integrate clean & friends properly:

ifeq ($(DEBUG), 1)
	ELM_FLAGS=--debug
	CARGO_FLAGS=
	CARGO_TARGET=debug/
else
	ELM_FLAGS=--optimize
	CARGO_FLAGS=--release
	CARGO_TARGET=release/
endif

# ..

output/server: server/Cargo.toml server/Cargo.lock api.humble $(wildcard server/**/*.rs)
	cd server && cargo build $(CARGO_FLAGS)
	cp -af server/target/$(CARGO_TARGET)/server output/server

clean:
	rm -rf output/ client/src/Api/ client/elm-stuff
	cd server && cargo clean

We are seeing a downside of make here: Smarter tools like cargo and elm make would rebuild if their equivalent of DEBUG changes, but since make does not keep track of these settings we need to run make clean each time.

We have now multiple options to “clean”:

Optimizing

We can go the extra mile and properly model the dependency tree of Rust’s debug and release builds, at the cost of an extra target:

ifeq ($(DEBUG), 1)
	ELM_FLAGS=--debug
	SERVER_BINARY=server/target/debug/server
else
	ELM_FLAGS=--optimize
	CARGO_FLAGS=--release
	SERVER_BINARY=server/target/release/server
endif

# ..

output/server: $(SERVER_BINARY)
	@mkdir -p $(dir $@)
	cp -af $^ $@

$(SERVER_BINARY): server/Cargo.toml server/Cargo.lock api.humble $(wildcard server/**/*.rs)
	cd server && cargo build $(CARGO_FLAGS)

Not used: depfiles

rustc has a nice feature where it automatically outputs .d files that can be included into the Makefile, similarly to gcc’s -M flags. Ideally these would be supported by all compilers (e.g. elm make as well and even style generators like sassc) but they are not at the time of this writing. They also are tricky to get to work right, we simply avoid them for being a bit too magical for too little gain — they only relieve us from having to specify dependencies for the $(SERVER_BINARY) target.

Styles

Our application uses a vendored copy of Bootstrap that we customize, which means that our entrypoint is a single SASS file called sass/app.scss. Make rules are straightforward. We do not bother with a generic rule because we expect this to be our only top-level stylesheet:

output/app.css: scss/app.scss $(wildcard scss/*.scss)
	@mkdir -p $(dir $@)
	sassc -t compressed $< $@

Copying over Boostrap assets

For the final product, we will need to copy over the bootstrap.min.js file, which we keep separate from our app JS code. Here we can take advantage of the fact that make will try multiple rules to create a file if it cannot find a prerequisite:

output/%: vendor/bootstrap-4.5.0/dist/js/%
	@mkdir -p $(dir $@)
	cp -af $< output/

This allows us to add output/bootstrap.min.js to the prerequisites for the all target and we are done.

make final result

Our resulting Makefile looks like this:

DEBUG=0

ifeq ($(DEBUG), 1)
	ELM_FLAGS=--debug
	SERVER_BINARY=server/target/debug/server
else
	ELM_FLAGS=--optimize
	CARGO_FLAGS=--release
	SERVER_BINARY=server/target/release/server
endif

all: output/index.html output/app.js output/app.css output/bootstrap.min.js output/server

output/app.js: client/elm.json client/src/Api $(wildcard client/**/*.elm)
	cd client && elm make --output=../$@ $(ELM_FLAGS) src/Main.elm

output/%: static/%
	@mkdir -p $(dir $@)
	cp -af $< output/

output/%: vendor/bootstrap-4.5.0/dist/js/%
	@mkdir -p $(dir $@)
	cp -af $< output/

output/app.css: scss/app.scss $(wildcard scss/*.scss)
	@mkdir -p $(dir $@)
	sassc -t compressed $< $@

client/src/Api: api.humble
	@rm -rf $@ && mkdir -p $@
	humblegen --language elm --output $@ $^

output/server: $(SERVER_BINARY)
	@mkdir -p $(dir $@)
	cp -af $^ $@

$(SERVER_BINARY): server/Cargo.toml server/Cargo.lock api.humble $(wildcard server/**/*.rs)
	cd server && cargo build $(CARGO_FLAGS)

clean:
	rm -rf output/ client/src/Api/ client/elm-stuff
	cd server && cargo clean

.PHONY: all clean

That is 33 lines of fairly readable code compiling a server application, a web frontend, custom styles, copying over static assets, all executable in parallel. A build with no changes takes less than 10 ms to run, which makes it pleasant to include in other tools.

Of course, this is not perfect: When changing build type from DEBUG to RELEASE, the frontend might not get rebuilt and thus will be stuck to DEBUG.

Even simpler: build.sh

Looking at Makefiles like these we realize that they are barely more than shell scripts — almost all of the important smarts that save work are inside the language specific tooling. We can translate the Makefile above into a small shell script:

#!/bin/sh -e

if [ "$1" = "debug" ]; then
  ELM_FLAGS=--debug
  SERVER_BINARY=server/target/debug/server
else
  CARGO_FLAGS=--release
  SERVER_BINARY=server/target/release/server
fi;

# Build frontend app
rm -rf client/src/Api
mkdir -p client/src/Api
humblegen --language elm --output client/src/Api api.humble
cd client && elm make --output=../output/app.js ${ELM_FLAGS} src/Main.elm && cd ..

# Compile styles
sassc -t compressed scss/app.scss output/app.css

# Build backend
cd server && cargo build ${CARGO_FLAGS} && cd ..

# Assemble application
cp ${SERVER_BINARY} static/index.html vendor/bootstrap-4.5.0/dist/js/bootstrap.min.js output/

The resulting shell script is a bit slower, but still runs in well under 50 ms. We also do not have to worry about build configuration issues anymore!

Conclusion

We set out to find a simple, easy to understand (and thus maintaineable) way of writing build systems that fits even fringe projects and as usual the pattern of calling a binary to produce an output file from input files is the lowest common denominator.

A good approach is to start out with build.sh and gradually move to a Makefile once increasing build times on the “non-smart” components dictate this. Ideally all of our compoonents would be either smart or fast enough for this never to be an issue, which is the reality for many modern programming languages.

For now, I am doing away with the top-level Makefile, delegating make back to the job it has always had: Building software written in languages like C, which do not ship with an actual build tool themselves — and it will not moonlight as a package manager or software configurator anymore.