[PATCH 1/5] test: conftest.py: move top-level

[PATCH 1/5] test: conftest.py: move top-level

[Ahmad Fatoum]

We have a top level pytest.ini to support running pytest --lg-env=$some_env
without extra arguments. This works for running tests nornally, but not if
extra options like --qemu should be used, because the conftest.py won't
be loaded early enough.

The easy fix is moving the conftest.py a level higher as described in:

  https://github.com/pytest-dev/pytest/issues/5822#issuecomment-697331920

Signed-off-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
---
 test/conftest.py => conftest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
 rename test/conftest.py => conftest.py (99%)

diff --git a/test/conftest.py b/conftest.py
similarity index 99%
rename from test/conftest.py
rename to conftest.py
index 12daf3a058ff..56d4b512a912 100644
--- a/test/conftest.py
+++ b/conftest.py
@@ -1,7 +1,7 @@
 import pytest
 import os
 import argparse
-from .py import helper
+from test.py import helper
 
 
 @pytest.fixture(scope='function')
-- 
2.39.2

[PATCH 2/5] MAKEALL: return error code from do_build_target

[Ahmad Fatoum]

The script currently records only if any config build failed to control
the exit code, but doesn't return an error code from each single
do_build_target. We need this though, when we start running the test
suite for successful builds, so let's add a return value and rename the
global ret variable to a more descriptive exitcode.

Signed-off-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
---
 MAKEALL | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/MAKEALL b/MAKEALL
index 861d397814d9..df318c6d123c 100755
--- a/MAKEALL
+++ b/MAKEALL
@@ -8,7 +8,7 @@ warnings_list=""
 nb_errors=0
 errors_list=""
 nb_defconfigs=0
-ret=0
+exitcode=0
 
 time_start=$(date +%s)
 
@@ -72,7 +72,7 @@ stats() {
 	fi
 	echo "----------------------------------------------------------"
 
-	exit ${ret}
+	exit ${exitcode}
 }
 
 check_pipe_status() {
@@ -89,6 +89,7 @@ do_build_target() {
 	local target_time_start=$(date +%s)
 	local log_report="${LOGDIR}/${target}/report.log"
 	local log_err="${LOGDIR}/${target}/errors.log"
+	local err=0
 
 	[ "$INCREMENTAL" != "1" ] && rm -rf "${BUILDDIR}"
 	mkdir -p "${LOGDIR}/${target}"
@@ -148,12 +149,14 @@ do_build_target() {
 			printf "FAILED \n" | tee -a "${log_report}"
 			nb_errors=$((nb_errors + 1))
 			errors_list="${errors_list} ${target}"
-			ret=1
+			err=1
+			exitcode=1
 		fi
 	else
 		printf "FAILED \n" | tee -a "${log_report}"
 		printf "Compile: ------ \n" | tee -a "${log_report}"
-		ret=1
+		err=1
+		exitcode=1
 	fi
 
 	if [ -s "${log_err}" ] ; then
@@ -168,6 +171,8 @@ do_build_target() {
 	target_time_stop=$(date +%s)
 	target_time_diff=$((${target_time_stop} - ${target_time_start}))
 	printf "Compiled in %4is\n" ${target_time_diff} | tee -a "${log_report}"
+
+	return $err
 }
 
 do_build() {
-- 
2.39.2

[PATCH 3/5] MAKEALL: don't build all configs wen defconfigs given

[Ahmad Fatoum]

We currently build all defconfigs when no architecture was specified,
which leads to surprising behavior, e.g.

  ./MAKEALL some_defconfig

will start building for all architectures and ignore the argument
completely. Improve upon this by calling do_build_all only if
there are no positional arguments.

Signed-off-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
---
 MAKEALL | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/MAKEALL b/MAKEALL
index df318c6d123c..68ebe6ef98a3 100755
--- a/MAKEALL
+++ b/MAKEALL
@@ -281,20 +281,20 @@ then
 	REGEX="*"
 fi
 
-if [ ! "${ARCH}" ] || [ ! -d arch/${ARCH} ]
-then
-	do_build_all
-	if [ $? -eq 0 ]
-	then
-		echo "You need to specify the ARCH or CROSS_COMPILE_<arch> or CROSS_COMPILE_<target> in your config file"
-		usage
-		exit 1
-	fi
-	exit 0
-fi
-
 if [ $# -eq 0 ]
 then
+	if [ ! "${ARCH}" ] || [ ! -d arch/${ARCH} ]
+	then
+		do_build_all
+		if [ $? -eq 0 ]
+		then
+			echo "You need to specify the ARCH or CROSS_COMPILE_<arch> or CROSS_COMPILE_<target> in your config file"
+			usage
+			exit 1
+		fi
+		exit 0
+	fi
+
 	do_build ${ARCH} "${REGEX}"
 else
 	for i in $*; do
-- 
2.39.2

[PATCH 4/5] MAKEALL: add support for running pytest after build

[Ahmad Fatoum]

As a first step towards removal of emulate.pl, let's teach MAKEALL to
accept labgrid environment YAML files instead of defconfigs, in which
case the will be built.

A bit of extra care is needed, because some YAMLs are symlinks and we
don't want to needlessly build/test more than once.

The benefit of this is that we can have a single command to get QEMU
running on a target, e.g.:

  scripts/container.sh ./MAKEALL test/arm/multi_v8_defconfig.yaml

Signed-off-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
---
 MAKEALL | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 96 insertions(+), 6 deletions(-)

diff --git a/MAKEALL b/MAKEALL
index 68ebe6ef98a3..3b93bfe5660b 100755
--- a/MAKEALL
+++ b/MAKEALL
@@ -7,7 +7,10 @@ nb_warnings=0
 warnings_list=""
 nb_errors=0
 errors_list=""
+test_errors_list=""
 nb_defconfigs=0
+nb_tests=0
+nb_tests_failed=0
 exitcode=0
 
 time_start=$(date +%s)
@@ -61,15 +64,21 @@ stats() {
 	echo ""
 	echo "--------------------- SUMMARY ----------------------------"
 	echo "defconfigs compiled: ${nb_defconfigs}"
-	time_stop=$(date +%s)
-	time_diff=$((${time_stop} - ${time_start}))
-	printf "compiled in %4is\n" ${time_diff}
 	if [ ${nb_errors} -gt 0 ] ; then
-		echo "defconfigs with errors: ${nb_errors} (${errors_list} )"
+		echo -e "\tdefconfigs with errors:   ${nb_errors} (${errors_list} )"
 	fi
 	if [ ${nb_warnings} -gt 0 ] ; then
-		echo "defconfigs with warnings: ${nb_warnings} (${warnings_list} )"
+		echo -e "\tdefconfigs with warnings: ${nb_warnings} (${warnings_list} )"
 	fi
+	if [ ${nb_tests} -gt 0 ]; then
+		echo "defconfigs tested:   ${nb_tests}"
+		if [ "${nb_tests_failed}" -gt 0 ]; then
+			echo -e "\tdefconfigs with errors:   ${nb_tests_failed} (${test_errors_list} )"
+		fi
+	fi
+	time_stop=$(date +%s)
+	time_diff=$((${time_stop} - ${time_start}))
+	printf "Total time spent: %4is\n" ${time_diff}
 	echo "----------------------------------------------------------"
 
 	exit ${exitcode}
@@ -175,6 +184,44 @@ do_build_target() {
 	return $err
 }
 
+if command -v labgrid-pytest >/dev/null; then
+	alias pytest=labgrid-pytest
+fi
+
+do_test_target() {
+	local yaml=$1
+	local target=$2
+	shift 2
+	local target_time_start=$(date +%s)
+	local log_report="${LOGDIR}/${target}/report.log"
+	local err=0
+
+	LG_BUILDDIR=$BUILDDIR pytest --lg-env $yaml "$@" 2>&1 >> "${log_report}"
+
+	check_pipe_status
+	compile_result="$?"
+
+	printf "Test: " ${yaml} | tee -a "${log_report}"
+
+	if [ "$compile_result" = "0" ]; then
+		printf "OK     \n" | tee -a "${log_report}"
+	else
+		printf "FAILED \n" | tee -a "${log_report}"
+		nb_tests_failed=$((nb_tests_failed + 1))
+		test_errors_list="${test_errors_list} ${yaml}"
+		exitcode=1
+		err=1
+	fi
+
+	nb_tests=$((nb_tests + 1))
+
+	target_time_stop=$(date +%s)
+	target_time_diff=$((${target_time_stop} - ${target_time_start}))
+	printf "Tested in %4is\n" ${target_time_diff} | tee -a "${log_report}"
+
+	return $err
+}
+
 do_build() {
 	local arch=$1
 	local regex=$2
@@ -297,8 +344,51 @@ then
 
 	do_build ${ARCH} "${REGEX}"
 else
+	declare -a configs=()
+	declare -a pytest_opts=()
 	for i in $*; do
-		do_build_target ${ARCH} $i
+		if [[ "$i" = "-"* ]] || [ ${#pytest_opts[@]} -gt 0 ]; then
+			pytest_opts+=($i)
+			continue;
+		fi
+
+		skip=0
+
+		# test/*/ may contain local symlinks, so resolve them
+		for j in "${configs[@]}"; do
+			if [ "$i" = "$j" ]; then
+				skip=1
+				break
+			fi
+
+			# drop duplicates, e.g. via globbing in directory with symlinks
+			if [[ $i =~ / && $j =~ / &&
+			   "$(readlink -f $i)" == "$(readlink -f $j)" ]]; then
+				skip=1
+				break
+			fi
+		done
+
+		if [ $skip = 1 ]; then
+			continue;
+		fi
+
+		configs+=($i)
+	done
+	for i in "${configs[@]}"; do
+		config=$i
+		if [[ $i =~ ^.*/([^/]+)/([^@]*@|)([^.]+).yaml$ ]]; then
+			arch=${BASH_REMATCH[1]}
+			defconfig=${BASH_REMATCH[3]}
+			do_build_target $arch $defconfig
+			if [ $? -eq 0 ]; then
+				do_test_target $config $defconfig "${pytest_opts[@]}"
+			else
+				echo "Skipping test due to failed build"
+			fi
+		else
+			do_build_target ${ARCH} $config
+		fi
 	done
 fi
 
-- 
2.39.2

[PATCH 5/5] Documentation: board: emulated: rewrite to use MAKEALL/pytest directly

[Ahmad Fatoum]

I added test/emulate.pl for my own use, but it didn't catch on. The CI
uses MAKEALL and lots of functionality has been migrated into pytest
proper, e.g. test/emulate.pl --emulate is now pytest --interactive.

Everything that's described in the docs is now supported though by
MAKEALL/pytest, so let's rewrite the documentation to use them.

There's still some features remaining that only emulate.pl supports,
once these are present in MAKEALL/pytest, we can drop it too.

Signed-off-by: Ahmad Fatoum <a.fatoum@pengutronix.de>
---
 Documentation/boards/emulated.rst | 85 +++++++++++++++++--------------
 1 file changed, 47 insertions(+), 38 deletions(-)

diff --git a/Documentation/boards/emulated.rst b/Documentation/boards/emulated.rst
index a67533613ec5..99c55d2051c2 100644
--- a/Documentation/boards/emulated.rst
+++ b/Documentation/boards/emulated.rst
@@ -20,56 +20,65 @@ fix up the virtio mmio regions into the device tree and barebox will
 discover the devices automatically, analogously to what it does with
 VirtIO over PCI.
 
-test/emulate.pl
----------------
+labgrid
+-------
 
-The ``emulate.pl`` script shipped with barebox can be used to easily
-start VMs. It reads a number of YAML files in ``test/$ARCH``, which
-describe some virtualized targets that barebox is known to run on.
-
-Controlled by command line options, these targets are built with
-tuxmake if available and loaded into the emulator for either interactive
-use or for automated testing with Labgrid ``QEMUDriver``.
-
-.. _tuxmake: https://pypi.org/project/tuxmake/
-.. _Labgrid: https://labgrid.org
-
-Install dependencies for interactive use::
-
-  cpan YAML::XS # or use e.g. libyaml-libyaml-perl on Debian
-  pip3 install tuxmake # optional
+Labgrid is used to run the barebox test suite, both on real and emulated
+hardware. A number of YAML files located in ``test/$ARCH`` describe some
+of the virtualized targets that barebox is known to run on.
 
 Example usage::
 
-  # Switch to barebox source directory
-  cd barebox
+  # Run x86 VM runnig the EFI payload from efi_defconfig
+  pytest --lg-env test/x86/efi_defconfig.yaml --interactive
 
-  # emulate x86 VM runnig the EFI payload from efi_defconfig
-  ARCH=x86 ./test/emulate.pl efi_defconfig
+  # Run the test suite against the same
+  pytest --lg-env test/x86/efi_defconfig.yaml
 
-  # build all MIPS targets known to emulate.pl and exit
-  ARCH=mips ./test/emulate.pl --no-emulate
+The above assumes that barebox has already been built for the
+configuration and that labgrid is available. If barebox has been
+built out-of-tree, the build directory must be pointed at by
+``LG_BUILDDIR``, ``KBUILD_OUTPUT`` or a ``build`` symlink.
 
-The script can also be used with a precompiled barebox tree::
+Additional QEMU command-line options can be added by specifying
+them after the ``--qemu`` option::
 
-  # Switch to build directory
-  export KBUILD_OUTPUT=build
+  # appends -device ? to the command line. Add --dry-run to see the final result
+  pytest --lg-env test/riscv/rv64i_defconfig.yaml --interactive --qemu -device '?'
 
-  # run a barebox image built outside tuxmake on an ARM virt machine
-  ARCH=arm ./test/emulate.pl virt@multi_v7_defconfig --no-tuxmake
+Some of the QEMU options that are used more often also have explicit
+support in the test runner, so paravirtualized devices can be added
+more easily::
 
-  # run tests instead of starting emulator interactively
-  ARCH=arm ./test/emulate.pl virt@multi_v7_defconfig --no-tuxmake --test
+  # Run tests and pass a block device (here /dev/virtioblk0)
+  pytest --lg-env test/arm/virt@multi_v8_defconfig.yaml --blk=rootfs.ext4
 
-``emulate.pl`` also has some knowledge on paravirtualized devices::
+For a complete listing of possible options run ``pytest --help``.
 
-  # Run target and pass a block device (here /dev/virtioblk0)
-  ARCH=riscv ./test/emulate.pl --blk=rootfs.ext4 rv64i_defconfig
+MAKEALL
+-------
 
-Needed command line options can be passed directly to the
-emulator/``pytest`` as well by placing them behind ``--``::
+The ``MAKEALL`` script is a wrapper around ``make`` to more easily build
+multiple configurations. It also accepts YAML Labgrid environment files
+as arguments, which will cause it to build and then run the tests::
 
-  # appends -device ? to the command line. Add -n to see the final result
-  ARCH=riscv ./test/emulate.pl rv64i_defconfig -- -device ?
+  ./MAKEALL test/mips/qemu-maltael_defconfig.yaml
 
-For a complete listing of options run ``./test/emulate.pl -h``.
+This expects ``CROSS_COMPILE`` (or ``CROSS_COMPILE_mips``) to have been
+set beforehand to point at an appropriate toolchain prefix.
+
+The barebox-ci container provides an easy way to run ``MAKEALL`` against
+all configurations supported by barebox, even if the host system
+lacks the appropriate toolchains::
+
+  # Run MAKEALL and possibly pytest in the container
+  alias MAKEALL="scripts/container.sh ./MAKEALL"
+
+  # Build a single configuration
+  MAKEALL test/mips/qemu-maltael_defconfig.yaml
+
+  # Build all configurations for an architecture, no test
+  MAKEALL -a riscv
+
+  # Build all mips platforms that can be tested
+  MAKEALL test/mips/*.yaml
-- 
2.39.2

Re: [PATCH 1/5] test: conftest.py: move top-level

[Sascha Hauer]

On Thu, 22 Aug 2024 13:57:26 +0200, Ahmad Fatoum wrote:
> We have a top level pytest.ini to support running pytest --lg-env=$some_env
> without extra arguments. This works for running tests nornally, but not if
> extra options like --qemu should be used, because the conftest.py won't
> be loaded early enough.
> 
> The easy fix is moving the conftest.py a level higher as described in:
> 
> [...]

Applied, thanks!

[1/5] test: conftest.py: move top-level
      https://git.pengutronix.de/cgit/barebox/commit/?id=884fd2ec8497 (link may not be stable)
[2/5] MAKEALL: return error code from do_build_target
      https://git.pengutronix.de/cgit/barebox/commit/?id=4a89f37d0c2b (link may not be stable)
[3/5] MAKEALL: don't build all configs wen defconfigs given
      https://git.pengutronix.de/cgit/barebox/commit/?id=3c0c3ebaa523 (link may not be stable)
[4/5] MAKEALL: add support for running pytest after build
      https://git.pengutronix.de/cgit/barebox/commit/?id=d0ef5a0c8e3a (link may not be stable)
[5/5] Documentation: board: emulated: rewrite to use MAKEALL/pytest directly
      https://git.pengutronix.de/cgit/barebox/commit/?id=af54b28a8c1d (link may not be stable)

Best regards,
-- 
Sascha Hauer <s.hauer@pengutronix.de>