[PATCH 0/5] test: arm: boot test Debian image in CI

[PATCH 0/5] test: arm: boot test Debian image in CI

[Ahmad Fatoum]

So far, we had no automated tests that actually booted a kernel
on every push (or PR). This series addresses this by 
a Debian ARM64 image and trying to boot it three times:

  - via GRUB
  - directly via kernel EFI stub
  - directly without EFI

Ahmad Fatoum (5):
  test: arm: multi_v8_efiloader_defconfig: use qcow2 image
  test: py: strategy: add helpers for booting kernel and bootm
  test: py: efiloader: prepare get_dmesg for more general usage
  test: py: add more kernel boot tests
  ci: add Debian ARM64 EFI loader boot test

 .github/workflows/test-labgrid-pytest.yml  | 20 ++++++
 .gitignore                                 |  4 ++
 scripts/fetch-os.sh                        | 35 ++++++++++
 test/arm/multi_v8_efiloader_defconfig.yaml |  7 +-
 test/py/test_fit.py                        |  2 +-
 test/py/test_linux_efiloader.py            | 75 +++++++++++++++++++++-
 test/strategy.py                           | 36 ++++++++++-
 7 files changed, 171 insertions(+), 8 deletions(-)
 create mode 100755 scripts/fetch-os.sh

-- 
2.47.3

[PATCH 1/5] test: arm: multi_v8_efiloader_defconfig: use qcow2 image

[Ahmad Fatoum]

The qcow2 image is 1/4th of the raw image size, so use that for shorter
downloads and easier storage.

Signed-off-by: Ahmad Fatoum <a.fatoum@barebox.org>
---
 test/arm/multi_v8_efiloader_defconfig.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/arm/multi_v8_efiloader_defconfig.yaml b/test/arm/multi_v8_efiloader_defconfig.yaml
index 2aba23f3c0c1..42bb418af6f8 100644
--- a/test/arm/multi_v8_efiloader_defconfig.yaml
+++ b/test/arm/multi_v8_efiloader_defconfig.yaml
@@ -9,7 +9,7 @@ targets:
         kernel: barebox-dt-2nd.img
         display: qemu-default
         extra_args: >
-          -drive if=virtio,format=raw,snapshot=on,file=debian-13-nocloud-arm64.raw
+          -drive if=virtio,snapshot=on,file=debian-13-nocloud-arm64.qcow2
           -device virtio-rng
       BareboxDriver:
         prompt: 'barebox@[^:]+:[^ ]+ '
-- 
2.47.3

[PATCH 2/5] test: py: strategy: add helpers for booting kernel and bootm

[Ahmad Fatoum]

The current boot helper activates the barebox strategy with the
assumption that barebox is booting itself, which is indeed the
case in the FIT boot test.

In preparation for adding customized boot tests of actual kernels
(and not only barebox masquerading as one), rename it to boot_barebox
and add boot_kernel, which behaves similarly, but activates the shell
driver.

Additionally, add support for using bootm instead of boot, which is more
convenient if we have no script, no ESP and no bootloader spec file.

Signed-off-by: Ahmad Fatoum <a.fatoum@barebox.org>
---
 test/py/test_fit.py |  2 +-
 test/strategy.py    | 36 +++++++++++++++++++++++++++++++++---
 2 files changed, 34 insertions(+), 4 deletions(-)

diff --git a/test/py/test_fit.py b/test/py/test_fit.py
index 97fdac0c79d9..e0999a01b0cb 100644
--- a/test/py/test_fit.py
+++ b/test/py/test_fit.py
@@ -78,7 +78,7 @@ def test_fit(barebox, strategy, testfs, fit_testdata):
 
     boottarget = generate_bootscript(barebox, fit_name('gzipped'))
 
-    with strategy.boot(boottarget):
+    with strategy.boot_barebox(boottarget) as barebox:
         assert of_get_property(barebox, "/chosen/barebox-version") == f'"{ver}"', \
                "/chosen/barebox-version suggests we did not chainload"
 
diff --git a/test/strategy.py b/test/strategy.py
index 0ca08b0cdd15..c04d38c180a7 100644
--- a/test/strategy.py
+++ b/test/strategy.py
@@ -86,20 +86,50 @@ class BareboxTestStrategy(Strategy):
             )
         self.status = status
 
+    def barebox_bootm(self, image=None):
+        self.barebox._run(f"global.loglevel={self.barebox.saved_log_level}",
+                          adjust_log_level=False)
+        if image:
+            self.console.sendline(f"bootm -v {image}")
+        else:
+            self.console.sendline("bootm -v")
+
     @contextmanager
-    def boot(self, boottarget=None):
+    def boot_barebox(self, boottarget=None, bootm=False):
         self.transition(Status.barebox)
 
         try:
-            self.barebox.boot(boottarget)
+            if bootm:
+                self.barebox_bootm(boottarget)
+            else:
+                self.barebox.boot(boottarget)
+
             self.target.deactivate(self.barebox)
             self.target.activate(self.barebox)
-            yield
+            yield self.barebox
         finally:
             self.target.deactivate(self.barebox)
             self.power.cycle()
             self.target.activate(self.barebox)
 
+    @contextmanager
+    def boot_kernel(self, boottarget=None, bootm=False):
+        self.transition(Status.barebox)
+
+        try:
+            if bootm:
+                self.barebox_bootm(boottarget)
+            else:
+                self.barebox.boot(boottarget)
+
+            self.barebox.await_boot()
+            self.target.activate(self.shell)
+            yield self.shell
+        finally:
+            self.target.deactivate(self.shell)
+            self.power.cycle()
+            self.target.activate(self.barebox)
+
     def force(self, state):
         self.transition(Status.off)  # pylint: disable=missing-kwoa
 
-- 
2.47.3

[PATCH 3/5] test: py: efiloader: prepare get_dmesg for more general usage

[Ahmad Fatoum]

Despite the generic sounding name, get_dmesg actually filter for the 'efit'
string, which is not always what's intended. Rename it to get_journalctl
to make the dependency clearer and make it more generic.

Signed-off-by: Ahmad Fatoum <a.fatoum@barebox.org>
---
 test/py/test_linux_efiloader.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/test/py/test_linux_efiloader.py b/test/py/test_linux_efiloader.py
index 22531bff81f4..8530747a7a6b 100644
--- a/test/py/test_linux_efiloader.py
+++ b/test/py/test_linux_efiloader.py
@@ -4,8 +4,13 @@ import re
 import pytest
 
 
-def get_dmesg(shell):
-    stdout, _, ret = shell.run("journalctl -k --no-pager --grep efi -o cat")
+def get_journalctl(shell, kernel=True, grep=None):
+    opts = ''
+    if grep is not None:
+        opts += f" --grep={grep}"
+    if kernel:
+        opts += " -k"
+    stdout, _, ret = shell.run(f"journalctl --no-pager {opts} -o cat")
     assert ret == 0
     return stdout
 
@@ -19,7 +24,7 @@ def test_efi_kernel_no_warn(shell):
 
 @pytest.mark.lg_feature(['bootable', 'efi'])
 def test_expected_efi_messages(shell, env):
-    dmesg = get_dmesg(shell)
+    dmesg = get_journalctl(shell, 'efi')
 
     expected_patterns = [
         r"efi:\s+EFI v2\.8 by barebox",
-- 
2.47.3

[PATCH 4/5] test: py: add more kernel boot tests

[Ahmad Fatoum]

We have one boot test for a Debian kernel via GRUB. The kernel in this
same image can be EFI booted directly thanks to its EFI stub and it can
be even booted without EFI at all via the normal boot protocol.

Add a test exercising both.

Signed-off-by: Ahmad Fatoum <a.fatoum@barebox.org>
---
 test/arm/multi_v8_efiloader_defconfig.yaml |  5 ++
 test/py/test_linux_efiloader.py            | 64 ++++++++++++++++++++++
 2 files changed, 69 insertions(+)

diff --git a/test/arm/multi_v8_efiloader_defconfig.yaml b/test/arm/multi_v8_efiloader_defconfig.yaml
index 42bb418af6f8..b24058da8e99 100644
--- a/test/arm/multi_v8_efiloader_defconfig.yaml
+++ b/test/arm/multi_v8_efiloader_defconfig.yaml
@@ -27,6 +27,11 @@ targets:
       - smbios
       - bootable
       - efi
+    options:
+      root_dev: virtioblk0.0
+      bootm.image: 'boot/vmlinuz-*'
+      bootm.initrd: 'boot/initrd.img-*'
+
 images:
   barebox-dt-2nd.img: !template "$LG_BUILDDIR/images/barebox-dt-2nd.img"
 imports:
diff --git a/test/py/test_linux_efiloader.py b/test/py/test_linux_efiloader.py
index 8530747a7a6b..b9ea82d27c0d 100644
--- a/test/py/test_linux_efiloader.py
+++ b/test/py/test_linux_efiloader.py
@@ -15,6 +15,70 @@ def get_journalctl(shell, kernel=True, grep=None):
     return stdout
 
 
+@pytest.mark.lg_feature(['bootable', 'efi'])
+@pytest.mark.parametrize('efiloader', [False, True])
+def test_boot_manual_with_initrd(strategy, barebox, env, efiloader):
+    """Test booting Debian kernel directly without GRUB"""
+
+    barebox.run_check(f"global.bootm.efi={'required' if efiloader else 'disabled'}")
+
+    def get_option(strategy, opt):
+        config = strategy.target.env.config
+        return config.get_target_option(strategy.target.name, opt)
+
+    try:
+        root_dev = get_option(strategy, "root_dev")
+        kernel_path = get_option(strategy, "bootm.image")
+    except KeyError:
+        pytest.fail("Feature bootable enabled, but root_dev/bootm.image option missing.")  # noqa
+
+    # Detect block devices
+    barebox.run_check("detect -a")
+    barebox.run_check(f"ls /mnt/{root_dev}/")
+
+    [kernel_path] = barebox.run_check(f"ls /mnt/{root_dev}/{kernel_path}")
+
+    try:
+        initrd_path = get_option(strategy, "bootm.initrd")
+        [initrd_path] = barebox.run_check(f"ls /mnt/{root_dev}/{initrd_path}")
+        barebox.run_check(f"global.bootm.initrd={initrd_path}")
+    except KeyError:
+        pass
+
+    barebox.run_check(f"global.bootm.image={kernel_path}")
+    barebox.run_check(f"global.bootm.root_dev=/dev/{root_dev}")
+    barebox.run_check("global.bootm.appendroot=1")
+    # Speed up subsequent runs a bit
+    barebox.run_check("global linux.bootargs.noapparmor=apparmor=0")
+
+    # Boot the kernel - it should use EFI stub by default
+    with strategy.boot_kernel(bootm=True) as shell:
+        shell.run_check("grep -q apparmor=0 /proc/cmdline")
+
+        initrd_freed = any("Freeing initrd memory"
+                           in line for line in get_journalctl(shell, 'initrd'))
+        assert initrd_freed, "initrd was not loaded or freed"
+
+        # Verify we booted to shell
+        dmesg = get_journalctl(shell, 'efi')
+
+        uefi_not_found = re.search("efi: UEFI not found.",
+                                   "\n".join(dmesg)) is not None
+
+        if efiloader:
+            test_efi_kernel_no_warn(shell)
+            test_expected_efi_messages(shell, env)
+            test_efi_systab(shell, env)
+            test_efivars_filesystem_not_empty(shell)
+
+            assert not uefi_not_found, \
+                   "EFI stub was not used despite global.bootm.efi=required"
+        else:
+            # Verify that EFI was NOT used
+            assert uefi_not_found, \
+                   "EFI stub was used despite global.bootm.efi=disabled"
+
+
 @pytest.mark.lg_feature(['bootable', 'efi'])
 def test_efi_kernel_no_warn(shell):
     stdout, stderr, ret = shell.run("journalctl -k --no-pager --grep efi -o cat -p warning")
-- 
2.47.3

[PATCH 5/5] ci: add Debian ARM64 EFI loader boot test

[Ahmad Fatoum]

Download the Debian nocloud image, boot with barebox into Linux and do
some rudimentary tests on the shell and then cache it for next time.

To make it easier for manual use, the downloading is done by a separate
script. The download info message is intentionally a warning, so we have
a chance of noticing if the Github Actions caching doesn't work as we
expect, so we can take measures instead of needlessly hitting Debian
infra too often.

I tried using runs-on: ubuntu-latest-arm, but there are less ARM workers
and the job takes longer to schedule it seems, so sticking with the normal
x86 runners.

Signed-off-by: Ahmad Fatoum <a.fatoum@barebox.org>
---
 .github/workflows/test-labgrid-pytest.yml | 20 +++++++++++++
 .gitignore                                |  4 +++
 scripts/fetch-os.sh                       | 35 +++++++++++++++++++++++
 3 files changed, 59 insertions(+)
 create mode 100755 scripts/fetch-os.sh

diff --git a/.github/workflows/test-labgrid-pytest.yml b/.github/workflows/test-labgrid-pytest.yml
index d9822fc206a5..de06fab3e136 100644
--- a/.github/workflows/test-labgrid-pytest.yml
+++ b/.github/workflows/test-labgrid-pytest.yml
@@ -28,6 +28,11 @@ jobs:
             defconfig: multi_v8_defconfig
             lgargs: --runxfail
 
+          - ARCH: arm
+            lgenv: test/arm/multi_v8_efiloader_defconfig.yaml
+            defconfig: multi_v8_efiloader_defconfig
+            osimg: debian-13-nocloud-arm64.qcow2
+
           - ARCH: mips
             lgenv: test/mips/qemu-malta_defconfig.yaml
             defconfig: qemu-malta_defconfig
@@ -64,6 +69,21 @@ jobs:
     - name: Checkout code
       uses: actions/checkout@v4
 
+    - name: Cache OS image
+      id: cache-osimg
+      if: ${{ matrix.osimg != '' }}
+      uses: actions/cache@v4
+      with:
+        path: ${{ matrix.osimg }}
+        key: osimg-${{ matrix.osimg }}
+        restore-keys: osimg-${{ matrix.osimg }}-
+
+    - name: Download OS image
+      if: ${{ steps.cache-osimg.outputs.cache-hit != 'true' && matrix.osimg != '' }}
+      run: |
+        echo "::warning::Cache miss - downloading fresh OS image from Debian infra."
+        scripts/fetch-os.sh
+
     - name: Determine used features
       id: used-features
       run: |
diff --git a/.gitignore b/.gitignore
index 74cb7b092834..1b335c330091 100644
--- a/.gitignore
+++ b/.gitignore
@@ -112,3 +112,7 @@ default.profdata
 default.profraw
 coverage.info
 coverage_html/
+
+# Disk images (used for testing)
+/*.raw
+/*.qcow2
diff --git a/scripts/fetch-os.sh b/scripts/fetch-os.sh
new file mode 100755
index 000000000000..649e4efe9a05
--- /dev/null
+++ b/scripts/fetch-os.sh
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+wgetopts=""
+if [ -n "$GITHUB_ACTIONS" ]; then
+	wgetopts="-nv"
+fi
+
+set -euo pipefail
+
+declare -A images=(
+	["debian-13-nocloud-arm64.qcow2"]="https://cloud.debian.org/images/cloud/trixie/latest/debian-13-nocloud-arm64.qcow2"
+)
+
+found=0
+dl=0
+
+for image in "${!images[@]}"; do
+	if [ -e "$image" ]; then
+		((++found))
+	else
+		wget -c $wgetopts -O "$image" "${images[$image]}"
+		((++dl))
+	fi
+done
+
+if [ "$found" -eq 0 ]; then
+	echo -n "No images found. ";
+else
+	echo -n "Found $found images(s). ";
+fi
+if [ "$dl" -eq 0 ]; then
+	echo "Nothing needed to be downloaded.";
+else
+	echo "$dl missing images downloaded.";
+fi
-- 
2.47.3