Thursday 22 October 2020

Proposal to add build graph output to GNU Make

Background

In 2015 I worked as a consultant at a large company in Lund. My position was with the build team and one of our responsibilities was managing and maintaining the build system for their Android based phones.

The problem I was tasked with solving was the fact that running 'make' for a product after a successful build resulted in a lot of stuff being rebuilt unnecessarily.

A stock Android build tree behaved nicely: a second run of 'make' only produced a line about everything being up-to-date. But the company products were taking a good 15 minutes for a rebuild even if nothing had been changed.

The Android build system works by including all recipes to be built (programs / libraries / etc) using the GNU Make include directive, so that you end up with one giant Makefile that holds all rules for building the platform. Possibly to avoid the problems laid out in the paper Recursive make considered harmful.

As you can imagine this results in quite a large Makefile that is near impossible to debug. To help us out GNU Make has an option that helps you figure out what is going on with every decision it makes:

-p, --print-data-base       Print make's internal database.

This is very powerful and lets you investigate a lot about what Make is doing. It may be a bit too powerful though. It is a lot of information to digest. You can see example output from when I run it against the project from my last blog  post, riscv-asm-hello-morse in a gist here.

The bugs we found and the fixes we implemented were mostly about targets depending on non-existing files or depending on phony targets. The fixes were almost never the hard part, it was finding the bugs that was hard.

Lately I have been working a bit with the Yocto build system which uses Bitbake to build recipes. And Bitbake has a -g option to generate a dependency graph that lets you figure out why something was built.

I wondered if something similar could be made useful for GNU Make. So I attempted to implement it as:

  -g, --output-graph          Output (dot) graph of modified targets for each goal.

Here I might need to pause to inject that I am not the first person to have this idea:


And I am sure I am missing more submissions.

In my defense my approach is a bit different. I only want to include targets that have been updated from Make goals that have changed, which will trim the graph quite a bit. It is not an attempt to show the information from --print-database in a new way.

Example 1

If I use my proposal to generate a graph from the riscv project mentioned above:

$ make -g
riscv64-unknown-elf-as -march=rv32imac -mabi=ilp32 -g -o0  -c -o hello-morse.o hello-morse.S
riscv64-unknown-elf-as -march=rv32imac -mabi=ilp32 -g -o0  -c -o wait.o wait.S
riscv64-unknown-elf-as -march=rv32imac -mabi=ilp32 -g -o0  -c -o led.o led.S
riscv64-unknown-elf-as -march=rv32imac -mabi=ilp32 -g -o0  -c -o morse.o morse.S
riscv64-unknown-elf-ld hello-morse.o wait.o led.o morse.o -m elf32lriscv -nostartfiles -nostdlib -Thello-morse.lds -o hello-morse.elf
riscv64-unknown-elf-objcopy -O ihex hello-morse.elf hello-morse.hex
make: Writing dependency graph to '/home/jonas/sandbox/riscv/riscv-asm-hello-morse/all.dot'
The last line here (my bold) is new and added by my patch. If we now look at the graph generated by this build.
$ cat all.dot
strict digraph "all" {
  "hello-morse.elf" -> "hello-morse.o" 
  "hello-morse.elf" -> "wait.o" 
  "hello-morse.elf" -> "led.o" 
  "hello-morse.elf" -> "morse.o" 
  "hello-morse.hex" -> "hello-morse.elf" 
  "all" -> "hello-morse.hex" 
  "all"
}

And we can generate a PNG from it:

$ dot -Tpng all.dot -o all.png


And if I force a rebuild  and re-generate the PNG:

$ touch morse.S
$ make -g 



Example 2

In my last blog post I mentioned the freedom-e-sdk used to build software for the hifive1-revb board. I noticed that it had the same problem as the Android based build system above, it always rebuilt on a second 'make' run.

The graph for the second run looks like:



Which helps us see that the build is forced because Make cannot find the libmetal-pico.a file. And looking at the Makefile we find that, yes, it does not check if the bsp support picolibc before depending on libmetal-pico.a. Fixing that will make rebuilding on the second 'make' run go away.

Example 3

If I want to track what happens on rebuild of the Make project itself I run into an issue.
$ touch src/file.c
$ make -g
The resulting all.dot file hardly contains any information about the rebuild:
strict digraph "all" {
  "all" -> "all-recursive" [label="forced: PHONY prerequisite"]
  "all"
}
The problem is that the make process is using make to build sub-targets, this comes from the all-recursive target. In order to solve this we need to tell, in this case automake, that all calls to make should use the -g option.
$ touch src/file.c
$ AM_MAKEFLAGS="-g" make -g

This gives us an additional all-am.dot file from the all-am make goal, which holds the missing information:

strict digraph "all-am" {
  "make" -> "src/file.o" 
  "all-am" -> "make" 
  "all-am"
}

Status and feedback

I have submitted this proposal to the GNU Make project. The maintainers will decide if this is something that is worthwhile and something they could consider maintaining.

You can also find the implementation in my Make fork at GitHub, here.

I would love to hear your thoughts on this. Is this something you would find useful? Please comment here or drop me a line on Twitter (@jonasdn) with your opinions!

2 comments:

  1. great idea, but as a lot of projects use make via autotools, which has a recursive approach, doesn't that limit the usefulness of the feature?

    ReplyDelete
  2. Thanks!

    Recursive invocation limit the feature a bit. At least in the same regard as recursive make is limiting in general, you lose the complete picture of the dependencies.

    Autotools and recursive make is why I chose to use a flag "-g" and that the graph files are named after the make goals. And not provide a filename, it allows us to support the recursive make case pretty well. The trick, for automake, is to use AM_MAKEFLAGS.

    We can take strace as an example.


    $ ./configure
    [...]
    $ AM_MAKEFLAGS="-g" make -g
    [...]
    $ find . -name \*.dot
    ./tests-m32/all.dot
    ./tests/all.dot
    ./all-recursive.dot
    ./all.dot
    ./tests-mx32/all.dot
    ./all-am.dot

    This gives us graph files for each of the different invocations / recursions. The meat of building strace is in the all-am dot file. We lose the overview because of the use of recursive make, but we get a bit of insight into the automake machinery by which dot files are produced.

    ReplyDelete