# The GAP build system

This file is meant to give an overview of how the GAP build system works. It
is targeted at people who need to work on the build system itself (to extend
it, fixes bugs in it, etc.). It should not be necessary to read this if all
you want to do is compile GAP and work on the GAP library and kernel.

Note that this really is just an overview; for details, please refer to the
comments inside the various parts of the build system.


## Prerequisites

In order to work on the build system, you need at least the following:

* GNU autoconf (we recommend 2.69 or later)
* GNU make

Note that we extensively use features provided by GNU make, so in general
another version of make, such as BSD make, is not suitable.


## Quick start: building GAP with no frills

If you are working with a fresh clone of the GAP repository, you need to
run the `autogen.sh` script first, which will generate the `configure`
script. Afterwards, or if you are using a release version of GAP, you
can follow the standard procedure:

    ./configure
    make


== Overview of the files constituting the GAP build system

* `autogen.sh`: sets up the build system; typically the first thing to run in
  a fresh clone of the GAP repository. It runs `autoreconf` which in turn runs
  several tools include `aclocal`, `autoheader`, `autoconf`, `libtoolize`

* `configure`: generated by `autogen.sh` from `configure.ac`.

* `configure.ac`: the GNU autoconf source of our configure script.

* `GNUmakefile`, `GNUmakefile.in`: The former file is generated from the
  latter by `configure`. It is the primary Makefile (GNU make prefers it
  over `Makefile`). It only contains variables and vpath settings, and
  includes `Makefile.rules` for the actual build rules.

* `Makefile`: This is a placeholder file, and serves two purposes:
   1. If the user runs `make` before `configure`, it prints a warning.
   2. If `configure` did run, but `make` is not GNU make, it produces
      a corresponding error message.

* `Makefile.rules`: This is the core of the build system. If you want
  to add or remove a kernel C source file, you need to add or remove
  its name here and only here.

* `bin/`: This directory is created for compatibility mode (see below).

* `cnf/`: All files in this directory are part of the build system.

* `extern/`: External libraries we bundle with GAP (such as GMP) are
  put in here.

* `build/`: Generated code (such as `config.h` and `version.c`) is put
  into this directory.

* `build/obj/`: All `*.o` resp. `*.lo` files are placed into this directory.

- `build/deps/` directories contain `*.d` files generated by the build system,
  and which are used to track dependencies, e.g. of C source files on header
  files.

- `.libs/` directories are created by libtool. Please refer to the libtool
  documentation for details.


## Out-of-tree builds

The old GAP build system had a concept of "configs" and "CONFIGNAME", which
allowed you to build GAP in different configurations from a single set of
sources. This is gone in the current build system. However, a similar goal can
be achieved by using so-called "out-of-tree builds".

In the following and also in the files that make up the build system, "srcdir"
refers to the directory containing the GAP files, i.e. it contains this
README, the src and lib directories and more.

To create a new out-of-tree build, create a new directory anywhere
in your filesystem. A typical setup places the out-of-tree dirs into
subdirectories of a "build" directory inside the srcdir. So you might
have directories

    srcdir/build/default
    srcdir/build/default32
    srcdir/build/hpcgap
    srcdir/build/hpcgap32
    ...

We will refer to this directory from now on as the "builddir".

To initialize the out-of-tree build, change into the builddir and
execute the configure script from the srcdir, like this:

    cd $builddir
    $srcdir/configure

You can pass any additional options you like to configure, e.g. `ABI=32`
or `--enable-hpcgap`.

Once the configure script has completed, you can run `make` as usual,
and all the object files and the gap executable will be placed inside
builddir. Your srcdir will remain untouched.


## Compatibility mode

Compatibility mode emulates the build environment of the old GAP build system
in order to allow packages with kernel extensions to be used with the new
build system unmodified. As such, it mainly exists to ease the transition
between new and old build system, and the plan is to remove it once all
packages have been adapted to the new build system. However, that is still
far off.

Compatibility mode does the following things:

* create a symlink `sysinfo.gap-default$ABI` pointing at `sysinfo.gap`
* create a `bin/$GAPARCH/config.h` symlink pointing at `build/config.h`
* create a `bin/$GAPARCH/gac` symlink pointing at `gac`
* create a `bin/$GAPARCH/src` symlink pointing at `$srcdir/src`
* for out-of-tree builds, it creates a `${builddir}/src/compiled.h` file
* ...

For now, using compatibility mode is required if one wants to build the
kernel extension for most packages which have one. In the future, we will
provide an alternative way to do this, and also will extend `gac` to
cleanly supported building kernel extensions.


## Dependency tracking

The build system tracks depdencies between files, such as between C source and
header files, via `*.d` files stored in `build/deps/`.
These files are mostly generated by the compiler; for this, the compiler needs
to support the relevant flags (gcc, clang, icc all do so).

For a detailed explanation of a very similar scheme, see here:
<https://make.mad-scientist.net/papers/advanced-auto-dependency-generation/>


## HPC-GAP integration

One of the main features of the new build system is that it optionally allows
to build HPC-GAP instead of plain GAP. HPC-GAP is an experimental fork of GAP
which implements concurrent programming, multi-threading, etc..

The HPC-GAP kernel and library were forked from the GAP kernel and library and
developed semi-independently for several years, with occasional merges between
the two. In order to recombine the two, we merged the HPC-GAP fork into a
subdirectory `hpcgap` of the GAP repository.  Then, all files inside `hpcgap`
which were identical to their counterparts in the GAP repository were deleted
(e.g. `hpcgap/src/ariths.c` was deleted as it was identical to `src/ariths.c`).
At this point, `hpcgap/src` has been fully merged, but there are still files
in `hpcgap/lib/` which differ from their counterparts in `lib/`

The new build system can optionally be instructed to build HPC-GAP, by
passing the `--enable-hpcgap` flag to the `configure` script. For the
resulting HPC-GAP binary to work, a trick is used:  HPC-GAP mode uses multiple
GAP root paths. Specifically, the GAP kernel function `SySetGapRootPath` was
modified so that for every root directory `FOO` that gets added, we first add
`FOO/hpcgap` to the list of root directories. This way, `GAPROOT/hpcgap/lib`
is searched first for files, and only if no matching file is found there does
GAP also search in `GAPROOT/lib`.


## Cross compilation

The GAP build system supports cross compilation by leveraging the GNU autoconf
support for this (please consult the GNU autoconf manual for details).
When building GAP from a release archive, everything should just work (if
it doesn't, please report it to us as a bug).

However, for development versions of GAP built directly from its git sources,
there is a complication: GAP uses a few C source files which are the output of
the GAP-to-C compiler `gac`, which in turn needs a working `gap` executable.
This requires a bootstrapping process, which works because GAP can actually be
built and used without those files, they are merely a performance
optimization. So what our build system does is to first compile a version of
GAP without those files, then run that to generate those files (if they are
missing or not up-to-date). Then finally the actual GAP executable is compiled.

Unfortunately, this poses a problem for cross compilation: when compiling GAP
on one computer (the "build" architecture) for a different computer with a
different OS / CPU architecture / whatnot (the "host" architecture), then the
first GAP executable needs to be built for the "build" architecture, so that
it can run during the build process and generated code; while the second GAP
executable needs to be compiled for the "host" architecture.

A second cross compilation obstacle is that there files `build/ffdata.c` and
`build/ffdata.h` are generated by a C program `ffgen` that thus needs to be
built for the "build" architecture.

To overcome these obstacles, you may proceed as follows:

1. First build GAP for the "build" architecture as usual.
2. This produces files `build/c_*.c` and `build/ffdata.*` which you should
   copy into the `src` directory.
3. Clean all build artifacts (alternatively, perform steps 1 & 2 in an
   out-of-tree build directory)
3. Build GAP a second time, for the "host" architecture. The build system
   will detect the files you copied into `src` in step 2, and will then not
   attempt to generate them.

Here is a minimal example to illustrate this:

    ./configure && make                 # build native GAP
    cp build/c_*.c build/ffdata.* src   # copy the generated code
    make clean
    ./configure --host=HOST_ARCH        # build for the "host" architecture
    make

Note that this is really a minimalistic example; for actual cross compilation,
you may need to provide further flags to the `configure` script and/or set
environment variables, e.g. to ensure it finds dependencies like GMP or GNU
readline, or to specify an installation prefix, and so on.

One final remark: some of the generated files differ for HPC-GAP. If you want
to cross compile HPC-GAP, you need to generate them with HPC-GAP, and place
them into `src/hpc` instead of `src`, like in this example:


    ./configure --enable-hpcgap
    make
    cp build/c_*.c src/hpc/
    cp build/ffdata.* src/
    make clean
    ./configure --host=HOST_ARCH --enable-hpcgap
    make


## Open tasks

There are many things that still need to be done in the new build system. For
an overview, see
<https://github.com/gap-system/gap/issues?q=is%3Aopen+is%3Aissue+label%3A%22topic%3A+build+system%22>

The main open task is to add support for `make install`. There are some
pitfalls to that, esp. when it comes to handling packages with kernel
extensions.


## FAQ

### Q: Why don't you just use `automake`? Then you get `make install` for free!

A: This is simply wrong. Using `automake` does not give us `make install`
support for free. While it does indeed provide various parts of the puzzle,
from our perspective those are mostly the easy ones (they may be tedious, but
are not conceptually difficult). The real obstacles for `make install` support
are unfortunately not resolved by using `automake`. Among these are:
- making `gac` installable; in particular installing a version of GNU
  libtool needed by `gac`
- The probably hardest problem of them all is coming up with a sane way for
  dealing with the installation of GAP packages with kernel modules in such a
  way that existing workflows are not unduly broken. This has the added
  difficulty that it requires convincing the authors of ~20 GAP packages to
  apply changes needed to support this, in a way that the packages can still
  be used with the at that time latest stable release of GAP; then getting
  them to make releases of their packages with those changes.
- There are probably more bits and pieces that also need to be resolved, which
  I just can't remember right now.

Of course none of this is rocket science, but it is a lot of tedious and
finicky work, and care must be taken not to break stuff. Help with this is
welcome, but it is strongly suggested that you first talk to us, to avoid
disappointments (e.g. due to rejected pull requests) later on.

In any case: GNU `automake` does not help with this at all.
