Proposal to add build graph output to GNU Make

Background

In 2015 I worked as a consultant at a large company in Lund. My position was with the build team and one of our responsibilities was managing and maintaining the build system for their Android based phones.

The problem I was tasked with solving was the fact that running 'make' for a product after a successful build resulted in a lot of stuff being rebuilt unnecessarily.

A stock Android build tree behaved nicely: a second run of 'make' only produced a line about everything being up-to-date. But the company products were taking a good 15 minutes for a rebuild even if nothing had been changed.

The Android build system works by including all recipes to be built (programs / libraries / etc) using the GNU Make include directive, so that you end up with one giant Makefile that holds all rules for building the platform. Possibly to avoid the problems laid out in the paper Recursive make considered harmful.

As you can imagine this results in quite a large Makefile that is near impossible to debug. To help us out GNU Make has an option that helps you figure out what is going on with every decision it makes:

-p, --print-data-base       Print make's internal database.

This is very powerful and lets you investigate a lot about what Make is doing. It may be a bit too powerful though. It is a lot of information to digest. You can see example output from when I run it against the project from my last blog  post, riscv-asm-hello-morse in a gist here.

The bugs we found and the fixes we implemented were mostly about targets depending on non-existing files or depending on phony targets. The fixes were almost never the hard part, it was finding the bugs that was hard.

Lately I have been working a bit with the Yocto build system which uses Bitbake to build recipes. And Bitbake has a -g option to generate a dependency graph that lets you figure out why something was built.

I wondered if something similar could be made useful for GNU Make. So I attempted to implement it as:

  -g, --output-graph          Output (dot) graph of modified targets for each goal.

Here I might need to pause to inject that I am not the first person to have this idea:

• In 2009 an anonymous user suggested XML output for Make's dependency graph: https://savannah.gnu.org/bugs/index.php?25751
  
  
• In 2016 user Yuri proposed something similar: https://savannah.gnu.org/bugs/index.php?47308
  
  
• In 2019 user Shlomi Fish brought it up again: https://savannah.gnu.org/bugs/index.php?56204

And I am sure I am missing more submissions.

In my defense my approach is a bit different. I only want to include targets that have been updated from Make goals that have changed, which will trim the graph quite a bit. It is not an attempt to show the information from --print-database in a new way.

Example 1

If I use my proposal to generate a graph from the riscv project mentioned above:

$ make -g

riscv64-unknown-elf-as -march=rv32imac -mabi=ilp32 -g -o0  -c -o hello-morse.o hello-morse.S

riscv64-unknown-elf-as -march=rv32imac -mabi=ilp32 -g -o0  -c -o wait.o wait.S

riscv64-unknown-elf-as -march=rv32imac -mabi=ilp32 -g -o0  -c -o led.o led.S

riscv64-unknown-elf-as -march=rv32imac -mabi=ilp32 -g -o0  -c -o morse.o morse.S

riscv64-unknown-elf-ld hello-morse.o wait.o led.o morse.o -m elf32lriscv -nostartfiles -nostdlib -Thello-morse.lds -o hello-morse.elf

riscv64-unknown-elf-objcopy -O ihex hello-morse.elf hello-morse.hex

make: Writing dependency graph to '/home/jonas/sandbox/riscv/riscv-asm-hello-morse/all.dot'

The last line here (my bold) is new and added by my patch. If we now look at the graph generated by this build.

$ cat all.dot

strict digraph "all" {

  "hello-morse.elf" -> "hello-morse.o" 

  "hello-morse.elf" -> "wait.o" 

  "hello-morse.elf" -> "led.o" 

  "hello-morse.elf" -> "morse.o" 

  "hello-morse.hex" -> "hello-morse.elf" 

  "all" -> "hello-morse.hex" 

  "all"

}

And we can generate a PNG from it:

$ dot -Tpng all.dot -o all.png

And if I force a rebuild  and re-generate the PNG:

$ touch morse.S

$ make -g 

Example 2

In my last blog post I mentioned the freedom-e-sdk used to build software for the hifive1-revb board. I noticed that it had the same problem as the Android based build system above, it always rebuilt on a second 'make' run.

The graph for the second run looks like:

Which helps us see that the build is forced because Make cannot find the libmetal-pico.a file. And looking at the Makefile we find that, yes, it does not check if the bsp support picolibc before depending on libmetal-pico.a. Fixing that will make rebuilding on the second 'make' run go away.

Example 3

If I want to track what happens on rebuild of the Make project itself I run into an issue.

$ touch src/file.c

$ make -g

The resulting all.dot file hardly contains any information about the rebuild:

strict digraph "all" {

  "all" -> "all-recursive" [label="forced: PHONY prerequisite"]

  "all"

}

The problem is that the make process is using make to build sub-targets, this comes from the all-recursive target. In order to solve this we need to tell, in this case automake, that all calls to make should use the -g option.

$ touch src/file.c

$ AM_MAKEFLAGS="-g" make -g

This gives us an additional all-am.dot file from the all-am make goal, which holds the missing information:

strict digraph "all-am" {

  "make" -> "src/file.o" 

  "all-am" -> "make" 

  "all-am"

}

Status and feedback

I have submitted this proposal to the GNU Make project. The maintainers will decide if this is something that is worthwhile and something they could consider maintaining.

You can also find the implementation in my Make fork at GitHub, here.

I would love to hear your thoughts on this. Is this something you would find useful? Please comment here or drop me a line on Twitter (@jonasdn) with your opinions!