It has been quite a few years since I first got acquainted with Linux. When I was still in primary school I remember getting an old desktop PC which definitely would not run Windows XP, so I got my hands on a Slackware Linux ISO and managed to get it up and running. I didn’t understand much of what I was doing back then, but I did succeed in hosting an IRC bot for my Counter-Strike team. In more recent years I have been using Debian-based distributions such as Linux Mint and Ubuntu as my primary OS on both my laptop and desktop PC and have become fairly proficient on the command line. While studying electronics engineering I started tinkering with a Raspberry Pi with the Raspbian operating system (now Raspberry Pi OS) and also took an Embedded Linux course where I got a better understanding of the operating system and the Linux philosophy.
Since finishing my studies I have mainly been working on microcontrollers and only briefly on embedded Linux systems. I have been wanting to get more into Embedded Linux and to learn more about writing kernel modules and device drivers, building custom distributions with Buildroot and Yocto, configuring and managing the operating system and get a deeper understanding of how the kernel works.
So, as a first step I decided to order an Embedded Linux development kit in the form a BeagleBone Green Wireless, which features a TI AM335x ARM Cortex-A8 processor, 4 GB on-board eMMC, WiFi + Bluetooth 4.1 capability and peripherals like ADCs and PWM units (which you don’t get on a Raspberry Pi).
To start off somewhere, I figured I might as well start by figuring out what happens after you power up the thing. So in this article I will be exploring the Linux boot process both on a regular desktop PC and on an embedded system, in this case the BeagleBone Green (BBG).
Booting Linux on a desktop PC
On a regular PC the boot process depends on whether you are using the old-school Basic Input/Output System (BIOS) mode or the newer Unified Extensible Firmware Interface (UEFI) mode. The figure below outlines the boot sequence using the two different modes:
BIOS vs UEFI boot
When the system starts in BIOS mode, the CPU starts executing the BIOS firmware which is located on a memory chip on the motherboard. In the early days of the PC this would be a ROM chip, but since updating the firmware would require replacing the ROM chip altogether, we started using flash memory instead which can simply be reprogrammed. As the name implies, the BIOS initializes only the most basic hardware and then performs a power-on self test (POST). In the beginning, the BIOS memory was not very large (on the first IBM PC it was only 8 kB) so you could only fit very primitive code on the chip – not anything fancy like filesystem drivers. Instead, it searches the connected storage devices for bootstrap code which is located at the very beginning of the device in what is called the master boot record (MBR). The order in which the storage devices are searched (called the “boot order” or “boot sequence”) can be configured in the BIOS setup and is stored in non-volatile memory. Apart from bootstrap code, the MBR also contains things like the disk’s partition table and a boot signature to ensure that the boot sector is valid. Since the entire MBR is limited to 512 bytes, the bootstrap code is typically only a few hundred bytes. Since this is not enough space to store filesystem drivers, it contains adresses of sections to load from the storage device where the filesystem drivers are located. Then it reads the partition table, locates the boot partition and executes the bootloader. We will get back to that in a second.
The UEFI firmware is also stored on a flash chip on the motherboard. Unlike BIOS, UEFI firmware is filesystem aware and thus there is no need for bootstrap code. It is capable of reading disks formatted with a traditional MBR partitioning scheme, but the recommended method is to format the disk using the GUID Partition Table (GPT) scheme instead. Compared to the MBR scheme, which only supports disks up to 2 TB and a maximum of four primary partitions, the GPT scheme supports disks up to 9.4 ZB (zettabytes, 1021 bytes) and 128 primary partitions. That ought to keep us going for some time into the future. When searching through the storage devices, the firmware looks for an EFI System Partition (ESP) where the bootloader is located.
The most common Linux bootloader for PCs in use today is the GRand Unified Bootloader 2 (GRUB2). I also remember seeing the LInux LOader (LILO) on quite a few old distros but it doesn’t seem to be used anymore. In BIOS mode, GRUB2 consists of both the bootstrap code (/boot/grub/boot.img) located in the MBR and the core image (/boot/grub/core.img) located on the boot partition. These are also referred to as the first-stage and second-stage bootloader, respectively. The core image is what contains the filesystem drivers and other programs required to load the Linux kernel. When the bootloader is executed, it searches all the connected storage devices for operating systems and kernels. It then lists the available options in a boot menu, which can be configured in the /boot/grub/grub.cfg configuration file. In UEFI mode, the entire bootloader is stored directly on the ESP and consists of a main executable (e.g. grubx64.efi) and a core image (core.efi). On my Ubuntu 22.04 system these are located under /boot/efi and /boot/grub, respectively.
After selecting an operating system kernel in the boot menu, the bootloader proceeds to load and transfer control to the kernel.
Loading the Linux kernel and initializing the operating system
The kernel image is located on the boot partition (/boot) and is usually in a self-extracting, compressed format and named vmlinuz or zImage. If uncompressed, the corresponding names would be vmlinux or Image.
After the bootloader loads the image, it uncompresses itself and performs some hardware initialization and then loads the initial RAM disk (initrd) or initial RAM filesystem (initramfs) into memory. The initramfs is used as a temporary root filesystem and contains various utilities and drivers required to mount the actual root filesystem from the disk.
After mounting the root filesystem, the init program (/sbin/init) is executed. On most modern systems the init program is systemd, but sysvinit, runit and openrc are other popular choices. The init program is the first process that is started and thus has PID 1. It starts by mounting all configured filesystems and then starts all processes and daemons that are required to reach a given target (in systemd) or runlevel (in sysvinit, runit and openrc). In the older SysV init style, the filesystems to be mounted and the processes to be started are configured in the filesystem table /etc/fstab and init table /etc/inittab. Systemd instead uses the concept of units (such as mount units and target units).
The most common targets are multi-user.target (runlevel 3), which boots to a command-line, and graphical.target (runlevel 5) which boots to a desktop environment. After the target or runlevel is reached, the user is presented with a login prompt and the system is ready to use.
Booting Linux on an embedded system
The boot process on an embedded Linux system, such as the BBG, is actually fairly similar to that of a PC. However, because of resource constraints and different processor architecture, there are a few differences – and it will also vary between different SoC’s. The figure below outlines the boot process on a stock BBG:
When the board is powered on, the first thing that is executed is the primary program loader (PPL) which is stored on ROM inside the SoC. Its only purpose is to look for a secondary program loader (SPL) on the on-board eMMC or microSD card. On the BeagleBone the specific SPL used is the TI x-loader, but U-Boot SPL is a popular alternative. It is located in the first sector of the eMMC in a file name MLO (for Memory LOader). At this point, since the dynamic RAM (DRAM) controller is not initialized, the only memory available is in the form of static RAM (SRAM), which does not require a memory controller. The SPL is only necessary because there is not enough SRAM available to hold the entire bootloader. Instead, the SPL is loaded into SRAM and is then responsible for initializing the DRAM controller, locating the tertiary program loader (TPL or bootloader) on the boot partition of the eMMC or micro SD card, and loading it into said DRAM.
The GRUB2 bootloader that we know from the previous section only works on x86 and x86_64 architectures. On embedded systems, the most popular bootloader is Das U-Boot (or just U-Boot) which works on several different architectures, including ARM. Another popular choice is BareBox (not to be confused with BusyBox). After the bootloader is loaded into DRAM and executed, it finds the Linux kernel image on the boot partition. This would normally be named zImage for a compressed kernel image, but U-Boot requires an image with some additional information which is then named uImage.
When the bootloader hands over control to the kernel, it also loads both the initramfs and the device tree binary into memory and passes it to the kernel. The device tree describes the hardware configuration of the system, and some systems also feature a device tree overlay, which makes it possible to make changes to the hardware configuration during run-time without recompiling the device tree source. After the kernel is loaded and executed, the system initialization is very similar to that of a PC. You might still see systemd and sysvinit being used as init programs, but for resource-constrained systems it is common to choose a more lightweight init program like BusyBox init or MUSL init. After the init process is finished, it might not make sense to show a login prompt, since an embedded system will often be headless. Instead, you might start running user mode applications to perform whatever tasks the system was designed for.
Final thoughts
This was just a brief overview of the Linux boot process and there is still much left to explore. I am currently working my way through Mastering Embedded Linux Programming by Chris Simmonds which (so far) seems to give a very nice overview of everything from cross toolchains, bootloaders, kernels and root filesystem to build systems such as Buildroot and Yocto, developing device drivers and system management. Definitely worth a read if you are looking to learn more about Embedded Linux.