Building images
This part of hackable:1 now has a name: "strap:1".
About strap1
Requirements
strap1 should run as-is on any GNU/Linux distribution, and even on other Unix systems (like BSD). It depends on these specific packages in particular:
- binutils (for ar and strip)
- fakeroot (optional)
- grep
- gzip
- mtd-utils (depending on the target hardware), found there
- sed
- sh
- sudo (optional)
- tar
- wget, found http://ftp.gnu.org/pub/gnu/wget/
The core component of strap1 consists of a simple POSIX shell script, currently containing less than a thousand lines of code.
On a Debian-based GNU/Linux system, it is possible to install the required software this way:
# aptitude install bash binutils fakeroot grep gzip mtd-utils sed sudo tar wget
Obtaining strap1
strap1 is found inside hackable:1's source code management system. It uses SVN, and can be obtained this way:
$ svn co svn://trac.hackable1.org/hackable1/trunk/build
It is then found inside the trunk/build directory inside hackable:1's SVN tree.
There is no official release of strap1 yet.
Project structure
strap1 consists of the following components:
- build.sh, the script invoked to create images
- profiles, defining the target hardware images for their respective purposes
- packages, containing the packages post-install configuration scripts
- depends, a staging directory where dependencies are resolved
- destdir, another staging directory where the filesystem is prepared
build.sh
build.sh is a shell script, joining the different pieces together. It accepts the following syntax:
$ ./build.sh Usage: build.sh [option=value...] target... Targets: archive Create an archive of an image contents clean Remove temporary data config Configure installed packages help Display this help screen image Create a native image file list List the available profiles Options: VENDOR (Openmoko) MODEL (Freerunner) PURPOSE (user)
A profile consists of a VENDOR-MODEL-PURPOSE combination. Any internal variable from build.sh, or within the given profile can also be overriden as an option.
Some typical examples:
To build a JFFS2 flash image for the Openmoko Freerunner:
$ ./build.sh VENDOR=Openmoko MODEL=Freerunner image
To build an image archive for the Openmoko Freerunner, with a developer environment:
$ ./build.sh VENDOR=Openmoko MODEL=Freerunner PURPOSE=developer archive
Useful options include:
| Variable | Description |
| DESTDIR | The staging directory to use while building the filesystem |
| MODEL | The model name of the target hardware platform |
| PURPOSE | The intended purpose of the image to generate |
| USERNAME | The name of the user account on the image |
| VENDOR | The vendor name of the target hardware platform |
| VERSION | The version number or string to use for this image |
More advanced options and variables are also detailed below.
Profiles
The profiles directory contains one file per supported VENDOR-MODEL-PURPOSE combination (with the ".profile" extention), which in turn may include common files (with the ".include" extention).
They typically only define a number of variables, of which the most relevant are:
| Variable | Description |
| CLEAN_APT | "yes" removes the packages database |
| CLEAN_DEVEL | "yes" removes include files and development libraries |
| CLEAN_DOC | "yes" removes any documentation |
| CLEAN_KERNEL | "yes" removes the kernel image |
| CLEAN_LOCALES | "yes" removes the locales information |
| CLEAN_STRIP | "yes" strips binaries and libraries |
| DEBIAN_ARCH | the Debian port to use |
| DISTRIBUTIONS | the Debian repositories to use |
| PACKAGES_BLACKLIST | a list of packages not to install |
| PACKAGES_PRIORITY | packages with either one of these priorities are installed |
| PACKAGES | an additional list of packages to install |
These can also be overriden on the command line with build.sh.
Packages configuration
As packages are simply extracted into the staging directory before building the actual image, they are configured with their default settings, or sometimes not at all. Each package can be configured inside the packages directory, within a shell script of the same name.
These shell scripts rely on a number of variables. The options defined above can be used to alter the behaviour of the scripts, and hence the final configuration of the images to be generated (eg the VENDOR-MODEL-PURPOSE profile).
Other essential commands are available, and can be overriden as well:
| Variable | Description |
| AR | Create, analyze and extract archives |
| CHMOD | Change the permissions of files and directories |
| CHOWN | Change the ownership of files and directories |
| CP | Copy files and directories |
| CUT | Process text data on the command-line |
| DU | Query the disk space used by files and directories |
| GREP | Look for given text patterns in files or on the command-line |
| GUNZIP | Decompress files |
| LN | Create and modify hard and symbolic links |
| MKDIR | Create directories |
| MKNOD | Create device nodes |
| MV | Rename and move files and directories |
| RMDIR | Delete directories |
| RM | Delete files and directories |
| SED | Replace content in files or on the command-line |
| STRIP | Remove debugging |
| TAR | Create, analyze and extract archives |
| TOUCH | Create files and alter timestamps |
| WGET | Download files over HTTP or FTP |
Some examples follow.
Within packages/base-passwd:
#/etc/passwd
$SUDO$MKDIR "$DESTDIR/home/$USERNAME" || exit 2
$SUDO$CHMOD 0750 "$DESTDIR/home/$USERNAME" || exit 2
$SUDO$CHOWN 1000:1000 "$DESTDIR/home/$USERNAME" || exit 2
${SUDO}sh -c "cat > \"$DESTDIR/etc/passwd\"" << EOF
root:x:0:0:root:/root:/bin/sh
$USERNAME:x:1000:1000:Hackable1 user:/home/hackable1:/bin/sh
EOF
Within packages/bluez-utils:
#/etc/rc?.d/S25bluetooth
for i in 2 3 4 5; do
$SUDO$MKDIR "$DESTDIR/etc/rc$i.d" || exit 2
$SUDO$LN -s "../init.d/bluetooth"
"$DESTDIR/etc/rc$i.d/S25bluetooth" \
|| exit 2
done
Within packages/fakeroot:
#/usr/bin/fakeroot
$SUDO$MKDIR "$DESTDIR/etc/alternatives" "$DESTDIR/usr/bin" || exit 2
$SUDO$LN -s "/usr/bin/fakeroot-tcp" "$DESTDIR/etc/alternatives/fakeroot" \
|| exit 2
$SUDO$LN -s "/etc/alternatives/fakeroot" "$DESTDIR/usr/bin/fakeroot" \
|| exit 2
Generating images
The operational mode of strap1 is fairly simple, and can be decomposed this way:
- Resolve and cache dependencies
- Extract the required packages
- Create the device nodes
- Configure the packages installed
- Clean up the filesystem as required
- Generate the final image
Packages dependencies
Debian packages repositories contain a Packages file, gathering the meta-data from the different packages in one convenient place. This is particularly welcome when resolving dependencies, and therefore listing the packages to be installed. There are a few relevant parameters there:
- priorities: the essential packages have a particular "Priority" header, like "required", "important" or "standard"; if it matches PACKAGES_PRIORITY, the packages gets selected;
- dependencies: the "Depends", "Pre-Depends" and "Recommends" headers are also used to select packages.
Two files are created during this process:
- depends/VENDOR-MODEL-PURPOSE.Depends
- depends/VENDOR-MODEL-PURPOSE.Packages
The "Depends" file contains the list of the packages selected for inclusion, while the "Packages" file gathers all of the meta-data in one file. The latter resembles dpkg's "available" file.
Packages extraction
Debian packages consist of an ar archive, with a meta-data archive member, and an optional archive to decompress. The extraction is performed without the dpkg tool, using the ar, gzip, bzip2 and tar utilities as required.
The packages database is completed at the same time, always appending information to the "available" database, and appending information to the "status" database for the packages extracted. The list of files extracted, and the package meta-data is also placed in "/var/lib/dpkg/info". Minor modifications to ar's output are also necessary to format the "package.list" file properly.
Device nodes creation
At this stage, device nodes are created. The regular device node creation script from Debian is used in this process, with the appropriate architecture specified on the command line.
This phase is only performed if the sbin/MAKEDEV script was installed on the target platform. Otherwise, it has to be performed from within the packages configuration scripts.
This phase is the only one currently known not to be portable. It is automatically skipped by Debian's device node creation script when it detects a non-Linux system.
Configure the packages installed
As packages get extracted, they also need to be configured in the context of the image generated. Many packages also actually require some scripts and executables to be ran right before or after extraction.
Unfortunately, in the case of hackable:1, the staging filesystem is often compiled for a different architecture than the native host. It is therefore impossible to run the configuration scripts. Instead, the necessary steps are reproduced within the "packages" directory.
For each configuration script found in the "packages" directory, it is checked if the package of the same name was actually installed on the filesystem (by testing the presence of the "/var/lib/dpkg/info/package.list" file). If it is the case, the configuration script is sourced from within the build.sh script.
It is possible to reproduce this phase at will, using the "config" target of the build.sh script. Beware that this can only work if the packages database information was kept (eg without the CLEAN_APT option set to "yes").
Clean up the filesystem as required
Some operations can then be conducted in order to gain space. They were mentioned in the 2.2 section:
| Variable | Description |
| CLEAN_APT | Deletes everything in /var/lib/dpkg |
| CLEAN_DEVEL | Deletes /usr/include, /usr/lib/pkgconfig, /usr/lib/*.a... |
| CLEAN_DOC | Deletes /usr/share/doc, /usr/share/man... |
| CLEAN_KERNEL | Deletes /boot |
| CLEAN_LOCALES | Deletes /usr/share/locales |
| CLEAN_STRIP | Strips all the binaries found in /usr/bin, /usr/lib... |
Generate the final image
Lastly, the filesystem is either saved as a compressed tar archive (the "archive" target), or packed as a ready-to-flash software image (the "image" target). Unfortunately, this phase is currently hard-coded within the build.sh script.
Comparison with debootstrap
In fact, strap1 implements parts of deboostrap, and extends it in some regards.
Why debootstrap is not enough
deboostrap is also useful to create Debian systems from scratch, without requiring dpkg or apt either. It is actually used by the Debian installer to initialize the system being installed. debootstrap consists of several thousands of lines of perl, of which cdebootstrap is an alternative implementation in C.
However, both assume that the installer will resume the installation by chrooting to the target system, and then install and configure packages using the native tools. In many cases, this is not likely during embedded development.
Limitations of debootstrap
First, in Debian packages the configuration process is relying on the possibility to run native binary programs from within the target system. This is used to either automatically detect, or ask the user specifically the settings to apply.
Moreover, debootstrap can not conveniently be used to generate systems from multiple source repositories. It is also not able to initialize the packages database as well as apt does itself.
Finally, debootstrap stops at the packages extraction phase. It does not provide filesystem images ready to flash, nor a way to conveniently automatically reproduce images from updated packages.
Advantages of strap1
strap1 addresses some of these issues already:
- it can use multiple package repositories at once
- it can generate valid packages databases, almost as complete as what apt does itself
- it can configure packages cross-platform
- it can generate final images.
It is also very portable, and easy to extend: it consists only of a simple core script, along with individual profiles and configuration files.
Remaining issues
Incomplete dependencies
Dependencies are not resolved recursively at the moment (this problem is also found in debootstrap). The images generated may therefore be incomplete in some cases.
Device nodes creation
Device nodes creation is not portable across Unix systems. This can sometimes be work-arounded with a tar archive of the necessary nodes, however it cannot be conveniently forced during the process at the moment.
Not always embedded
The stock Debian packages may not always fit in an embedded environment. More cooperation with the Emdebian project is certainly desirable.
Performance
The overall performance could be improved.
