Changes between Version 12 and Version 13 of BuildingImages

Timestamp:

06/23/09 01:41:46 (15 months ago)

Author:

ppronchery (IP: 83.236.217.161)

Comment:

Completed merge of strap:1 paper

BuildingImages

@@ -132 +132 @@
 === Packages configuration ===
 
-The configuration of packages is found [http://trac.hackable1.org/trac/browser/trunk/build/packages here]. There is one file per package, and they are sourced after the extraction process if they are installed. They can have a different logic depending on the target platform and actual purpose of the image.
-
-== Known problems ==
-
-=== Dependency resolving is not perfect ===
-
-Dependencies are not resolved recursively at the moment. They rely on proper information from the `Packages.gz` files.
-
-=== Installation hook scripts are not executed ===
-
-There's no simple way around this one. It needs to be duplicated in the configuration files as described above.
+As packages are simply extracted into the staging directory before
+building the actual image, they are configured with their default
+settings, or sometimes not at all. Each package can be configured inside
+the packages directory, within a shell script of the same name.
+
+These shell scripts rely on a number of variables. The options defined
+above can be used to alter the behaviour of the scripts, and hence the
+final configuration of the images to be generated (eg the
+VENDOR-MODEL-PURPOSE profile).
+
+Other essential commands are available, and can be overriden as well:
+
+||'''Variable'''||'''Description'''||
+||AR||Create, analyze and extract archives||
+||CHMOD||Change the permissions of files and directories||
+||CHOWN||Change the ownership of files and directories||
+||CP||Copy files and directories||
+||CUT||Process text data on the command-line||
+||DU||Query the disk space used by files and directories||
+||GREP||Look for given text patterns in files or on the command-line||
+||GUNZIP||Decompress files||
+||LN||Create and modify hard and symbolic links||
+||MKDIR||Create directories||
+||MKNOD||Create device nodes||
+||MV||Rename and move files and directories||
+||RMDIR||Delete directories||
+||RM||Delete files and directories||
+||SED||Replace content in files or on the command-line||
+||STRIP||Remove debugging||
+||TAR||Create, analyze and extract archives||
+||TOUCH||Create files and alter timestamps||
+||WGET||Download files over HTTP or FTP||
+
+Some examples follow.
+
+Within packages/base-passwd:
+
+{{{
+#/etc/passwd
+$SUDO$MKDIR "$DESTDIR/home/$USERNAME"                           || exit 2
+$SUDO$CHMOD 0750 "$DESTDIR/home/$USERNAME"                      || exit 2
+$SUDO$CHOWN 1000:1000 "$DESTDIR/home/$USERNAME"                 || exit 2
+${SUDO}sh -c "cat > \"$DESTDIR/etc/passwd\"" << EOF
+root:x:0:0:root:/root:/bin/sh
+$USERNAME:x:1000:1000:Hackable1 user:/home/hackable1:/bin/sh
+EOF
+}}}
+
+Within packages/bluez-utils:
+
+{{{
+#/etc/rc?.d/S25bluetooth
+for i in 2 3 4 5; do
+        $SUDO$MKDIR "$DESTDIR/etc/rc$i.d"                       || exit 2
+        $SUDO$LN -s "../init.d/bluetooth"
+"$DESTDIR/etc/rc$i.d/S25bluetooth" \
+                                                                || exit 2
+done
+}}}
+
+Within packages/fakeroot:
+
+{{{
+#/usr/bin/fakeroot
+$SUDO$MKDIR "$DESTDIR/etc/alternatives" "$DESTDIR/usr/bin"      || exit 2
+$SUDO$LN -s "/usr/bin/fakeroot-tcp" "$DESTDIR/etc/alternatives/fakeroot" \
+                                                                || exit 2
+$SUDO$LN -s "/etc/alternatives/fakeroot" "$DESTDIR/usr/bin/fakeroot" \
+                                                                || exit 2
+}}}
+
+
+== Generating images ==
+
+The operational mode of strap1 is fairly simple, and can be decomposed
+this way:
+
+ 1. Resolve and cache dependencies
+ 1. Extract the required packages
+ 1. Create the device nodes
+ 1. Configure the packages installed
+ 1. Clean up the filesystem as required
+ 1. Generate the final image
+
+
+=== Packages dependencies ===
+
+Debian packages repositories contain a Packages file, gathering the
+meta-data from the different packages in one convenient place. This is
+particularly welcome when resolving dependencies, and therefore listing
+the packages to be installed. There are a few relevant parameters there:
+
+ * priorities: the essential packages have a particular "Priority" header, like "required", "important" or "standard"; if it matches PACKAGES_PRIORITY, the packages gets selected;
+ * dependencies: the "Depends", "Pre-Depends" and "Recommends" headers are also used to select packages.
+
+Two files are created during this process:
+
+ * depends/VENDOR-MODEL-PURPOSE.Depends
+ * depends/VENDOR-MODEL-PURPOSE.Packages
+
+The "Depends" file contains the list of the packages selected for
+inclusion, while the "Packages" file gathers all of the meta-data in one
+file. The latter resembles dpkg's "available" file.
+
+
+=== Packages extraction ===
+
+Debian packages consist of an ar archive, with a meta-data archive
+member, and an optional archive to decompress. The extraction is
+performed without the dpkg tool, using the ar, gzip, bzip2 and tar
+utilities as required.
+
+The packages database is completed at the same time, always appending
+information to the "available" database, and appending information to
+the "status" database for the packages extracted. The list of files
+extracted, and the package meta-data is also placed in
+"/var/lib/dpkg/info". Minor modifications to ar's output are also
+necessary to format the "package.list" file properly.
+
+
+=== Device nodes creation ===
+
+At this stage, device nodes are created. The regular device node
+creation script from Debian is used in this process, with the
+appropriate architecture specified on the command line.
+
+This phase is only performed if the sbin/MAKEDEV script was installed on
+the target platform. Otherwise, it has to be performed from within the
+packages configuration scripts.
+
+This phase is the only one currently known not to be portable. It is
+automatically skipped by Debian's device node creation script when it
+detects a non-Linux system.
+
+
+=== Configure the packages installed ===
+
+As packages get extracted, they also need to be configured in the
+context of the image generated. Many packages also actually require some
+scripts and executables to be ran right before or after extraction.
+
+Unfortunately, in the case of hackable:1, the staging filesystem is
+often compiled for a different architecture than the native host. It is
+therefore impossible to run the configuration scripts. Instead, the
+necessary steps are reproduced within the "packages" directory.
+
+For each configuration script found in the "packages" directory, it is
+checked if the package of the same name was actually installed on the
+filesystem (by testing the presence of the
+"/var/lib/dpkg/info/package.list" file). If it is the case, the
+configuration script is sourced from within the build.sh script.
+
+It is possible to reproduce this phase at will, using the "config"
+target of the build.sh script. Beware that this can only work if the
+packages database information was kept (eg without the CLEAN_APT option
+set to "yes").
+
+=== Clean up the filesystem as required ===
+
+Some operations can then be conducted in order to gain space. They were
+mentioned in the 2.2 section:
+
+||'''Variable'''||'''Description'''||
+||CLEAN_APT||Deletes everything in /var/lib/dpkg||
+||CLEAN_DEVEL||Deletes /usr/include, /usr/lib/pkgconfig, /usr/lib/*.a...||
+||CLEAN_DOC||Deletes /usr/share/doc, /usr/share/man...||
+||CLEAN_KERNEL||Deletes /boot||
+||CLEAN_LOCALES||Deletes /usr/share/locales||
+||CLEAN_STRIP||Strips all the binaries found in /usr/bin, /usr/lib...||
+
+
+=== Generate the final image ===
+
+Lastly, the filesystem is either saved as a compressed tar archive (the
+"archive" target), or packed as a ready-to-flash software image (the
+"image" target). Unfortunately, this phase is currently hard-coded
+within the build.sh script.
+
+
+== Comparison with debootstrap ==
+
+In fact, strap1 implements parts of deboostrap, and extends it in some
+regards.
+
+=== Why debootstrap is not enough ===
+
+deboostrap is also useful to create Debian systems from scratch, without
+requiring dpkg or apt either. It is actually used by the Debian
+installer to initialize the system being installed. debootstrap consists
+of several thousands of lines of perl, of which cdebootstrap is an
+alternative implementation in C.
+
+However, both assume that the installer will resume the installation by
+chrooting to the target system, and then install and configure packages
+using the native tools. In many cases, this is not likely during
+embedded development.
+
+
+=== Limitations of debootstrap ===
+
+First, in Debian packages the configuration process is relying on the
+possibility to run native binary programs from within the target system.
+This is used to either automatically detect, or ask the user
+specifically the settings to apply.
+
+Moreover, debootstrap can not conveniently be used to generate systems
+from multiple source repositories. It is also not able to initialize the
+packages database as well as apt does itself.
+
+Finally, debootstrap stops at the packages extraction phase. It does not
+provide filesystem images ready to flash, nor a way to conveniently
+automatically reproduce images from updated packages.
+
+
+=== Advantages of strap1 ===
+
+strap1 addresses some of these issues already:
+
+ * it can use multiple package repositories at once
+ * it can generate valid packages databases, almost as complete as what apt does itself
+ * it can configure packages cross-platform
+ * it can generate final images.
+
+It is also very portable, and easy to extend: it consists only of a
+simple core script, along with individual profiles and configuration files.
+
+
+== Remaining issues ==
+
+=== Incomplete dependencies ===
+
+Dependencies are not resolved recursively at the moment (this problem is
+also found in debootstrap). The images generated may therefore be
+incomplete in some cases.
+
+=== Device nodes creation ===
+
+Device nodes creation is not portable across Unix systems. This can
+sometimes be work-arounded with a tar archive of the necessary nodes,
+however it cannot be conveniently forced during the process at the moment.
+
+=== Not always embedded ===
+
+The stock Debian packages may not always fit in an embedded environment.
+More cooperation with the Emdebian project is certainly desirable.
+
+=== Performance ===
+
+The overall performance could be improved.