Build system

Problems solved

• Forgotten dependencies: build in isolated environment
• Reactive builds (auto-rebuild): mek watch
• Access to the build environment: eval $(mek)
• What changed since last good build(s) without doing git commit all the time: mek source --good (also: cd toto.err.source/)
• Track downloads of third-party components: build in isolated environment without network access
• Create an archive of the source and all dependencies, including downloaded stuff: mek archive
• No global state: everything is specified by the mekfile. No global state, just a global cache of the output of pure computations.

• Prerequisites for building:

  • Using an archive: ./archive.run
  • Using the source: ./build.sh or make

  both do the same, the source should always embed a self-bootstrapping copy of mek (it's tiny), and the only external dependency is third-party components that are automatically downloaded.

  • Using the source + downloaded.run: ./build.sh or make

Basic usage

$(mek daemon)
ls
editor some-file

Extended usage

mek build toto           # builds toto or toto.err in case of an error, then builds everything else
                         # Cached errors are displayed early, but the build is retried in case it was a cosmic ray.

mek clean                # only necessary if a hardware or system failure (e.g. out of memory) made a build appear as SUCCESSFUL when it was not
                         # (e.g. a poorly-written test that expected an error, caught an OOM but the real execution would not have produced an error)

mek watch                # rebuilds everything incrementally as soon as sources are modified (threads 1 and 3 or 2 and 3, see below).
mek daemon               # same but with '&' after initialization
mek watch toto           # rebuilds toto incrementally as soon as one of its transitive dependencies is modified

ls                       # show progress next to (future) targets and a symlink to the backup of their source
ls _build.err            # stderr of the last build (if non-empty)
ls _build/err            # stderr of the last build
ls _build/source/        # source of the last build
ls _build/good.source/   # source of the last successful build
ls _build/err.source/    # source of the last failed build
ls toto.err              # stderr when building toto (if non-empty)
ls toto.source           # source of the last build of toto
ls toto.good.source      # source of the last successful build of toto
ls toto.err.source       # source of the last failed build of toto
ls toto.log              # log of all the builds of toto

mek source               # dumps the source of the last build
mek source --good        # dumps the source of the last successful build
mek source --err         # dumps the source of the last failed build
mek source toto          # dumps the source of the last successful build of toto
mek source toto.err      # dumps the source of the last failed build of toto

mek log                  # log of all the builds (from these we can get the source and the built output)

eval $(mek)              # builds and adds the output bin directory to $PATH etc.
eval "$(mek)"            # same
$(mek)                   # same
. mekfile.sh             # same
mek; copy-paste output   # same
$(mek daemon)            # same but detaches right away and builds incrementally in the background

eval $(mek build toto)   # adds an output bin directory containing only toto (can be a collection of outputs) to $PATH etc.
mek shell toto           # subshell in the directory and with the env of the recipe that builds toto
mek shell toto.err       # same as above
eval $(mek shell toto)   # cd to build directory for toto and add to $PATH etc.

mek archive toto         # toto.run now contains the source of toto and of all its dependencies
mek archive --downloaded # downloaded.run now contains the source of all network dependencies

Example session

$ $(mek daemon)
$ ls
(toto 60%)  toto.source  toto.c  toto.h
$ ls
toto  toto.source  toto.c  toto.h
$ echo "bad stuff" >> toto.h
$ ls
toto (34%)  toto.c  toto.h
$ ls
toto  toto.source  toto.err  toto.err.source  toto.c  toto.h
$ meld toto.source/ toto.err.source/
$ echo "fix bug" >> toto.h
$ ls
toto  toto.source  toto.c  toto.h

In a mekfile

toto: toto.h
# automatic dependency on the gcc executable and on toto.c
# "," is the unquote from scheme, it escapes from the implicitly-quoted shell command.
# TODO: Maybe gcc (a variable pointing to a third-party tool) should be distinguished from toto.c (a local file)
> ,gcc ,toto.c -o ,output

• Uses hashes, not timestamps
• Builds are done in isolated environments (cd, proot, Nix, chroot, container, VM, whatever is available), which only contain the dependencies.
• Transitions are atomic (mv of a symlink), so that the bin folder in the $PATH contains executables from the same version of the source, not toto from one version and tata from another.
• Two builds can run in parallel without interfering with each other, yet if work can be shared it will be.

Note about build threads

• Invariant when threads 2 and 3 are enabled: if the user constantly modifies a file, e.g. while sleep 1; do date > somesource; done which produces an infinite stream of changes, and one of the versions causes the compiler to deadlock / go in an infinite loop, mek will still eventually produce an infinite stream of output binaries, where the latest produced binary is not based on a "very old" change (i.e. it is not a queue of jobs that grows indefinitely). It tries to build the latest changes, but it is resistant to the compiler hanging forever on some inputs, and it is resistant to a rapid stream of changes that could cause a naive algorithm to always restart without ever finishing any build.
• Thread 1 builds, and then checks for new changes. (If the build gets stuck in an infinite loop or deadlocks, the build never finishes and new changes are never taken into account.)
• Thread 2 builds, but aborts and restarts as soon as there's any change. Upon completion it kills threads 1 and 3 because it got a successful build of a more recent version. (If you're constantly modifying and the build takes a while, it never gets a chance to finish.)
• Thread 3 works like thread 1 but aborts if there are new changes and the build took longer than a timeout, e.g. twice the time of the last successful build by any thread and increasing geometrically

TODO: other requirements

• meta-rules: a rule which returns rules. Can be memoized easily, and be part of the normal reactive flow.
• Easy changes of config: dev vs. build, -O3, -Odebug etc.
• Easily modify the dependencies / have several projects that are developped in parallel, without the need to publish, increment version number, etc. all the time.