From e71eb22bddaff1087b0e8ea9eec5c8504fd21d16 Mon Sep 17 00:00:00 2001 From: Josef Schlehofer Date: Tue, 30 Jun 2026 08:15:38 +0200 Subject: [PATCH] contributing: update CI section with current test infrastructure Update the Continuous Integration section to reflect the current state: - Update the list of runtime-tested architectures (add i386_pentium-mmx and mips_24kc, remove outdated i386_pentium4) - Document the generic test suite (executable, version, hardcoded path, strip, linked library, and SONAME checks) - Add documentation for test-version.sh (version check override) and pre-test.sh (pre-test setup) scripts alongside the existing test.sh - Document available environment variables (PKG_NAME, PKG_VERSION, CI_HELPERS) for test scripts Signed-off-by: Josef Schlehofer --- CONTRIBUTING.md | 102 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 90 insertions(+), 12 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c8b523beae..2faf3c327d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -195,22 +195,56 @@ To simplify review and require less human resources, a CI tests all packages. Passing CI tests are not a hard requirement but a good indicator what the Buildbots will think about the proposed patch. -The CI builds modified packages for multiple architectures using the latest -snapshot SDK. For supported architectures (`aarch64_generic`, -`arm_cortex-a15_neon-vfpv4`, `i386_pentium4` and `x86_64`) an additional -runtime test is executed. A running OpenWrt is simulated which tries to install -created packages and runs a script called `test.sh` located next to the package -Makefile. The script is executed with the two arguments `PKG_NAME` and -`PKG_VERSION`. The `PKG_NAME` can be used to distinguish package variants, e.g. -`foobar` vs. `foobar-full`. The `PKG_VERSION` can be used for a trivial test -checking if `foobar --version` prints the correct version. +The CI builds modified packages for multiple +architectures using the latest snapshot SDK. For supported architectures +(`aarch64_generic`, `arm_cortex-a15_neon-vfpv4`, `i386_pentium-mmx`, +`mips_24kc` and `x86_64`) an additional runtime test is executed inside a +Docker container with QEMU user-mode emulation, simulating a running OpenWrt +system. + +### Generic tests + +The CI automatically runs a set of **generic tests** on every installed +package: + +- **Executable check** — verifies that files installed in standard executable + paths (`/usr/bin/`, `/usr/sbin/`, etc.) are actually executable. +- **Version check** — attempts to detect the package version by running each + executable with common flags (`--version`, `-V`, `--help`, etc.) and checking + the output for `PKG_VERSION`. +- **Hardcoded path check** — scans ELF binaries for leftover build directory + paths (`/build_dir/`). +- **Strip check** — warns if ELF binaries are not stripped. +- **Linked library check** — verifies that all shared library dependencies are + present on the system. +- **SONAME check** — for libraries in standard library paths, verifies that the + SONAME is set correctly and the corresponding symlink exists. + +### Test scripts + +In addition to the generic tests, package maintainers can provide up to three +optional shell scripts placed next to the package Makefile: + +#### `test.sh` — Functional test + +A package-specific functional test script that runs **in addition** to the +generic tests. This is useful for verifying functionality that the generic +checks cannot cover (e.g. importing a Python module, encrypting/decrypting +data, or testing a specific command-line workflow). The following environment +variables are available: + +| Variable | Description | +| ------------- | ------------------------------------------------------ | +| `PKG_NAME` | Package name including variant (e.g. `foobar-full`) | +| `PKG_VERSION` | Upstream version without the OpenWrt release suffix | +| `CI_HELPERS` | Path to `ci_helpers.sh` providing colored output utils | The following snippet shows a script that tests different binaries depending on -what IPK package was installed. The `gpsd` Makefile produces both a `gpsd` and -a `gpsd-clients` IPK packages. +what package was installed. The `gpsd` Makefile produces both a `gpsd` and +a `gpsd-clients` package. ```shell - #!/bin/sh +#!/bin/sh case "$1" in "gpsd") @@ -221,3 +255,47 @@ case "$1" in ;; esac ``` + +#### `test-version.sh` — Version check override + +When the generic version check cannot detect the version automatically (e.g. +the binary does not support `--version` or reports it in a non-standard way), +a `test-version.sh` script can override the version check logic. When this +script is present, the generic version detection is skipped entirely and the +script is responsible for verifying the version. + +The script receives `PKG_NAME` and `PKG_VERSION` as environment variables and +should use a `case` statement to handle individual sub-packages. Packages that +have no executable or cannot report their version should `exit 0`. Unknown +packages should `exit 1`. + +```shell +#!/bin/sh + +# shellcheck shell=busybox + +case "$PKG_NAME" in +tor) + tor --version | grep -F "$PKG_VERSION" + ;; +tor-geoip) + # Data-only package, no version to check + exit 0 + ;; +*) + echo "Untested package: $PKG_NAME" >&2 + exit 1 + ;; +esac +``` + +#### `pre-test.sh` — Pre-test setup + +A script that runs **before** the package is installed. This can be used to +install additional dependencies required for testing that are not part of the +package itself. + +```shell +#!/bin/sh +apk add openssl-util +```