Re: [PATCH 3/7] test: move dm-verity testdata generation to fixture

[Tobias Waldekranz <tobias@waldekranz.com>]

On fre, sep 26, 2025 at 16:25, Jonas Rebmann <jre@pengutronix.de> wrote:
> Hi all,
>
> On 2025-09-26 12:14, Jonas Rebmann wrote:
>> Simplify barebox integration test setup by moving logic away from
>> scripts/ and .github/.
>> 
>> Instead of generating testdata in separate scripts, they should be
>> implemented as testfs fixtures which are automatically ran as part of
>> the test suite.
>> 
>> Includes error handling and cleanup.
>> 
>> Signed-off-by: Jonas Rebmann <jre@pengutronix.de>
>> ---
>>   scripts/generate_testfs.sh | 44 ---------------------------------------
>>   test/py/test_dm.py         | 51 ++++++++++++++++++++++++++++++++++++++++++++--
>>   2 files changed, 49 insertions(+), 46 deletions(-)
>> 
>> diff --git a/scripts/generate_testfs.sh b/scripts/generate_testfs.sh
>> index 1c358ff846..3c200bd401 100755
>> --- a/scripts/generate_testfs.sh
>> +++ b/scripts/generate_testfs.sh
>> @@ -28,47 +28,3 @@ generate_fit()
>>   	       ${KBUILD_OUTPUT}/testfs/barebox-gzipped.fit
>>   }
>>   [ -f .github/testfs/${KBUILD_DEFCONFIG}-gzipped.its ] && generate_fit
>
> It took me some time to figure out why this patch broke CI.
>
> With the set -e in this script, this last line of fit generation logic
> should have been
>
>    [ -f .github/testfs/${KBUILD_DEFCONFIG}-gzipped.its ] && generate_fit || true
>
> given the intent here is instead of erroring out, to just skip fit
> generation if the its does not exist.

Quite right. Sorry for leaving that foot gun there :)

> The only reason CI didn't break when Tobias added mipsel and riscv, for
> which no .github/testfs/${KBUILD_DEFCONFIG}-gzipped.its exist, in commit
> 739a1a7855f5 ("ci: pytest: Open up testfs to more consumers than the FIT
> test"), is that the github workflow (incorrectly) invokes the script
> with exec. This is why only when this check became the last line of the
> script, the error return code propagated into github ci.
>
> I would like to drop scripts/generate_testfs.sh entirely in v2,
> migrating the generate_fit logic to a pytest fixture too.
>
> The reason I haven't yet is mainly that there we cannot rely on
> KBUILD_DEFCONFIG to select the test data.
>
> I thought about selecting the its from something in barebox_config or
> something in strategy.qemu instead. Any suggestions?
>
> And I'd like to also move those testfs files out of .github. I would
> like to consider the pytest tests something more than just artifacts of
> the github workflow.
>
> As this excursion into cleanup around the integration tests is getting a
> bit more involved than I hoped, I'm happy about feedback already while
> I'm working on v2.

I'll leave some thoughts that went through my mind as I was doing the dm
work. This is from the perspective of someone with almost no experience
of neither pytest in general, nor of labgrid.

It would be great if much of the logic from the workflow file could be
moved to a regular Make target or something, so that it is easier to
answer questions like "Will the CI job run with testfs for this config
now?"

As an example, in Infix, our GitHub workflow for launching the test
suite is just 'make test'. This will then launch a docker container,
which in turn runs a bunch of QEMU instances and the tests. One very
nice property with this approach is that it is very easy to launch the
exact same test environment on your local machine.

>> -
>> -alias pad128k="dd if=/dev/zero bs=128k count=1 status=none"
>> -
>> -generate_dm_verity()
>> -{
>> -    work=$(mktemp -d)
>> -    cd ${work}
>> -
>> -    # Create two dummy files; use lots of padding to make sure that
>> -    # when we alter the contents of 'english' in root-bad, 'latin' is
>> -    # still be readable, as their contents wont (a) share the same
>> -    # hash block and (b) the block cache layer won't accedentally read
>> -    # the invalid block.
>> -
>> -    pad128k  >latin
>> -    echo -n "veritas vos liberabit" >>latin
>> -    pad128k >>latin
>> -
>> -    pad128k  >english
>> -    echo -n "truth will set you free" >>english
>> -    pad128k >>english
>> -
>> -    truncate -s 1M good.fat
>> -    mkfs.vfat good.fat
>> -    mcopy -i good.fat latin english ::
>> -
>> -    veritysetup format \
>> -		--root-hash-file=good.hash \
>> -		good.fat good.verity
>> -
>> -    sed 's/truth will set you free/LIAR LIAR PANTS ON FIRE/' \
>> -	<good.fat >bad.fat
>> -
>> -    cd -
>> -    cp \
>> -	${work}/good.fat \
>> -	${work}/good.verity \
>> -	${work}/good.hash \
>> -	${work}/bad.fat \
>> -	${KBUILD_OUTPUT}/testfs
>> -
>> -    rm -rf ${work}
>> -}
>> -generate_dm_verity
>> diff --git a/test/py/test_dm.py b/test/py/test_dm.py
>> index 8bdacf7b4e..837881f4af 100644
>> --- a/test/py/test_dm.py
>> +++ b/test/py/test_dm.py
>> @@ -4,10 +4,57 @@ import re
>>   import pytest
>>   from .helper import of_get_property
>>   
>> +import os, subprocess, shutil
>>   
>> +def pad128k(f):
>> +    f.write(b"\0" * 128 * 1024)
>>   
>> -def test_dm_verity(barebox, testfs):
>> -    barebox.run_check("cd /mnt/9p/testfs")
>> +@pytest.fixture(scope="module")
>> +def dm_testdata(testfs):
>> +    path = os.path.join(testfs, "dm")
>> +    os.makedirs(path, exist_ok=True)
>> +    cwd = os.getcwd()
>> +    os.chdir(path)
>> +
>> +    with open("latin", "wb") as f:
>> +        pad128k(f)
>> +        f.write(b"veritas vos liberabit")
>> +        pad128k(f)
>> +
>> +    with open("english", "wb") as f:
>> +        pad128k(f)
>> +        f.write(b"truth will set you free")
>> +        pad128k(f)
>> +
>> +    try:
>> +        subprocess.run(["truncate", "-s", "1M", "good.fat"], check=True)
>> +        subprocess.run(["mkfs.vfat", "good.fat"], check=True)
>> +        subprocess.run(["mcopy", "-i", "good.fat", "latin", "english", "::"], check=True)
>> +
>> +        subprocess.run([
>> +            "veritysetup", "format",
>> +            "--root-hash-file=good.hash",
>> +            "good.fat", "good.verity"
>> +        ], check=True)
>> +    except FileNotFoundError as e:
>> +        pytest.skip(f"Skip dm tests due to missing dependency: {e}")
>> +
>> +    with open("good.fat", "rb") as f: data = f.read()
>> +    with open("bad.fat", "wb") as f:
>> +        f.write(data.replace(
>> +            b"truth will set you free",
>> +            b"LIAR LIAR PANTS ON FIRE"
>> +        ))
>> +
>> +    os.chdir(cwd)
>> +
>> +    yield path
>> +
>> +    shutil.rmtree(path)
>> +
>> +
>> +def test_dm_verity(barebox, dm_testdata):
>> +    barebox.run_check("cd /mnt/9p/testfs/dm")
>>   
>>       # Since commands run in a subshell, export the root hash in a
>>       # global, so that we can access it from subsequent commands
>> 
>
> Regards,
> Jonas
>
> -- 
> Pengutronix e.K.                           | Jonas Rebmann               |
> Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |