With CONFIG_TARGET_IMAGES_GZIP enabled, which is the default, ext4
images are gzipped when they are copied to $(BIN_DIR), unless their
name ends with .gz or .dtb. This causes several problems:
- The JSON info contains the uncompressed name and hash, so
profiles.json lists images which don't exist on the download server:
profiles.json: openwrt-at91-sam9x-atmel_at91sam9263ek-ext4-root.ubi
sha256sums: openwrt-at91-sam9x-atmel_at91sam9263ek-ext4-root.ubi.gz
- The image is gzipped as a whole, including the metadata and signature
added by append-metadata. fwtool looks for them at the end of the
file, so sysupgrade rejects these images. Explicitly gzipped images
run append-metadata after gzip instead.
- A rule for the gzipped name is defined for every image, which clashes
with images that are built both plain and gzipped, as it was the case
for malta.
The previous commits removed the ext4 images which can't be used and
gzip the remaining ones explicitly, so drop the implicit gzip. The file
name in $(BIN_DIR) is now always the one from IMAGES. Targets wanting
gzipped images define them explicitly, e.g. sdcard.img.gz, or select
them via CONFIG_TARGET_IMAGES_GZIP like x86, armsr, loongarch64 and
malta. Limit the option to these targets.
The ext4 kernel.bin and factory.bin of ixp4xx, which only contain the
kernel, and the optional ext4 VM images of x86 and armsr are now stored
uncompressed, like their squashfs counterparts.
Link: https://github.com/openwrt/openwrt/pull/25658
Signed-off-by: Paul Spooren <mail@aparcar.org>
Image/Gzip lost its last user with commit df9ffedb75 ("archs38: drop
target") and Image/gzip-ext4-padded-squashfs with commit 8dfa38b82c
("malta: convert to Device macro image building"). The latter also
calls Image/Build/gzip, which isn't defined anywhere.
Remove both, along with Image/pad-root-squashfs, which was only used by
Image/gzip-ext4-padded-squashfs.
Link: https://github.com/openwrt/openwrt/pull/25658
Signed-off-by: Paul Spooren <mail@aparcar.org>
Every image is built for all filesystems in FILESYSTEMS. A device that
needs an image only for some of them, e.g. a flash image for squashfs
and a disk image for ext4, can only restrict FILESYSTEMS for the whole
device, which drops its other images as well.
Add FILESYSTEMS/<image> to build an image only for some filesystems of
the device, similar to IMAGE/<image>/<fs> for per filesystem recipes.
Like the recipes, it is not reset between devices, so it should be set
next to the recipe of the image.
Link: https://github.com/openwrt/openwrt/pull/25658
Signed-off-by: Paul Spooren <mail@aparcar.org>
The aarch64 toolchain is built without --with-cpu, so the
CPU_CFLAGS_<type> entry in target.mk is the only place a -mcpu option
comes from. The cortex-a72 (bcm27xx/bcm2711, mvebu/cortexa72) and
cortex-a76 (bcm27xx/bcm2712) CPU types have no entry, so userspace for
these subtargets is built with plain "-Os -pipe", which produces
generic armv8-a code. The "doesn't correspond to a known type" warning
is not visible because the target scan discards stderr.
Add the missing entries. For cortex-a76 this also enables the ARMv8.2
features such as LSE atomics, which replace the out of line atomic
helper calls.
Fixes: 2534141322 ("mvebu: add support for MACCHIATObin (cortex-a72)")
Fixes: 5e928acf22 ("brcm2708: add bcm2711 subtarget (RPi 4B aarch64)")
Fixes: 8b63d9ac4a ("bcm27xx: add new bcm2712 subtarget")
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Link: https://github.com/openwrt/openwrt/pull/25627
Signed-off-by: Robert Marko <robimarko@gmail.com>
Build/elx-header and Build/znet-header convert a hex string into \xHH
escape sequences using `fold -s2`. This relies on the obsolete -WIDTH
option syntax of GNU fold, which parses "-s2" as "-s -2". uutils
coreutils does not support this syntax and fails with:
error: unexpected argument '-2' found
tip: to pass '-2' as a value, use '-- -2'
Since fold writes nothing to stdout in this case, the command
substitution expands to an empty string. The following
`dd bs=N count=1 conv=sync` then writes zero bytes, because conv=sync
only pads partial input blocks and never pads empty input. The header
field is dropped entirely, and all subsequent fields move to lower
offsets.
The build still succeeds, because the failure happens inside the
subshell which is then handed to echo.
The malformed images are only rejected when a device tries to
boot them.
For example, on the Zyxel WSM20, the missing payload size field in the
znet-header moves the 0x12345678 magic and the payload to unexpected
offsets. The bootloader then rejects the image, even though sysupgrade
accepted it.
This affects all builds on Ubuntu 25.10 and later, because these
releases use uutils coreutils by default.
Replace `fold -s2` with `fold -w2`, which is the documented way to set
the line width and produces identical output with GNU fold.
Affected devices:
- Build/znet-header is used in all images of the Zyxel WSM20 and the
ELECOM WRC-X1800GS
- Build/elx-header is used in the factory images of the ELECOM
WAB-I1750-PS, WAB-S1167-PS, WAB-S600-PS, WRC-2533GHBK-I and
WRC-2533GHBK2-T, the I-O DATA WN-AC1167GR, WN-AC733GR3 and WNPR2600G,
the Edimax RA21S and RG21S, and the D-Link DAP-X1860 A1
Signed-off-by: Grische <github@grische.xyz>
Assisted-by: Claude:claude-opus-5
Link: https://github.com/openwrt/openwrt/pull/25410
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
UBIFS root file systems differ between builds, e.g. the one of the
Microchip SAMA7G5-EK: the superblock gets a random UUID, the inodes get
the ctime of the build host, which can't be reset with touch, and the
entries of each directory are added in readdir() order, which depends on
the host file system and decides inode numbers and layout.
Add a patch that, when SOURCE_DATE_EPOCH is set, uses a time-based
(version 1) UUID with that time and clamps all inode times to it. The
node ID of the UUID is taken from the new --uuid-node option, which the
image build sets from IMG_PART_DISKGUID, itself derived from
SOURCE_DATE_EPOCH and the kernel vermagic.
Directory entries are added sorted by name. Two trees with the same
contents but different ctimes and readdir() order now give identical
images; without SOURCE_DATE_EPOCH the UUID stays random.
Link: https://github.com/openwrt/openwrt/pull/25430
Signed-off-by: Paul Spooren <mail@aparcar.org>
The temporary directory used to create a kernel UBIFS image and its
kernel file receive wall-clock mtimes. mkfs.ubifs preserves these
times, making images differ between builds.
Set their timestamps to SOURCE_DATE_EPOCH before invoking mkfs.ubifs.
Signed-off-by: Robert Marko <robimarko@gmail.com>
Kernel/SetInitramfs writes the initramfs configuration into the kernel
tree and deletes .config.prev, so that the next configure stage puts the
regular configuration back. The check that decides if the configure stage
must run ignored this signal. The next build then compiled the regular
kernel with the initramfs configuration, and the squashfs images got a
kernel with an embedded root filesystem.
Run the configure stage if .config.prev is missing.
Fixes: e97e3f36bd ("kernel: configure the kernel only when an input changed")
Signed-off-by: John Crispin <john@phrozen.org>
Nothing tests either symbol any more. AUDIO_SUPPORT was also selected by
PCI_SUPPORT and USB_SUPPORT, which stood for "a sound card could be
plugged in here": audio hardware sits behind MMC, SPI, a Bluetooth UART
or an SoC PCM block just as often, so the inference never held. The
display feature was derived from a literal CONFIG_VGA_CONSOLE or CONFIG_FB
in a kernel fragment, which misses every target whose fbdev came from DRM
emulation.
Delete both symbols, the two selects, the feature translations and the
derivation.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The U-Boot script embedded in the GL.iNet QSDK factory image hardcodes
the GL-B3000 flash layout. Replace the UBI offset and size with
placeholders filled in by the gl-qsdk-factory recipe, so other GL.iNet
boards with a different layout can use the same script.
The script is copied under a per-device name, which keeps the FIT
description unchanged and avoids devices overwriting each other's
substituted copy in KDIR_TMP.
Signed-off-by: Gilberto Conde <gilbertorconde@gmail.com>
CompileBPF calls clang with -g but without any prefix map. Stripping debug info
keeps .BTF and .BTF.ext, since libbpf needs them, and their line info carries
the absolute path of each source file. This makes BPF objects like qosify-bpf.o
depend on the build directory and breaks reproducibility:
-/builder/shared-workdir/build/build_dir/.../qosify-bpf.c
+/tmp/rebuilderd-openwrt.juCsuQ/src/build_dir/.../qosify-bpf.c
IREMAP_CFLAGS is not enough here: without CONFIG_REPRODUCIBLE_DEBUG_INFO it
only sets -fmacro-prefix-map, which leaves the line info alone, and it is empty
for external toolchains, which have no say over the host clang. Unlike regular
debug info, BTF ships with the package, so always pass -ffile-prefix-map for
the build and staging directories.
Add an fremap helper next to iremap for that, and let IREMAP_STAGING_DIR and
CompileBPF share the map of the staging directory, so they cannot drift apart.
Link: https://github.com/openwrt/openwrt/pull/25359
Signed-off-by: Paul Spooren <mail@aparcar.org>
Instead of carrying 100-no-testing.patch, pass -DBUILD_TESTING=OFF
directly to CMake's bootstrap script. This avoids refreshing the patch
on every CMake update.
The patch also changed the default in the installed CTest.cmake module.
Preserve that behavior by passing -DBUILD_TESTING=OFF by default in
cmake.mk for target and host builds. Packages can override it via
CMAKE_OPTIONS or CMAKE_HOST_OPTIONS.
OpenWrt never runs CMake tests, so building them only wastes build time.
This replaces the patch introduced in 8af80e79bd ("tools/cmake:
update to 3.17.0").
Signed-off-by: Josef Schlehofer <pepe.schlehofer@gmail.com>
Link: https://github.com/openwrt/openwrt/pull/24403
Signed-off-by: Robert Marko <robimarko@gmail.com>
Three cases kept a cached answer that no longer held:
A new configure script. A package with an autoreconf fixup regenerates
configure with the staged autoconf, automake and libtool, and a new
version of one of them can change what a cache variable means. The site
script now keeps a checksum of configure beside the cache and drops the
cache when the checksum changes. A new patch to configure is covered too.
A changed precious variable. configure aborts when LIBS, CPP or another
precious variable differs from the value in the cache, and because
nothing removes the cache, it aborts on every later run as well:
configure: error: 'LIBS' was not set in the previous run
configure: error: changes in the environment can compromise the build
The site script now makes the same comparison before configure does and
drops the cache on a difference.
Paths under other names. The libtool answers lt_cv_path_*, acl_cv_path_*
and lt_cv_sys_global_symbol_*, and gmp_cv_prog_m4, hold the path of a
system tool or of the staged sed and m4, and survived a wipe of
staging_dir. The site script now drops them with the ac_cv_path_ answers.
m4 and gmp still configure the same way with a warm cache as they do
without one, and still read it: the m4 configure stage takes 1.5 s warm
against 23 s cold.
Signed-off-by: John Crispin <john@phrozen.org>
The initramfs step copied /init into the root filesystem directory and
never removed it again. With CONFIG_TARGET_PER_DEVICE_ROOTFS the
initramfs kernel and the rootfs image of a device come from that same
directory. Nothing orders the two steps, so /init entered the squashfs
image or not, depending on which step ran first. The reproducible build
service reports this as a difference between the published image and a
rebuild.
Add /init to the initramfs from outside the root filesystem instead. A
built in initramfs gets it through a gen_init_cpio list file. A separate
cpio gets it appended to the finished archive. The mode is now fixed at
0755 and no longer follows the umask of the checkout.
The initramfs step also normalised the mtime of the root filesystem
directory. Do that in prepare_rootfs, so no step writes to the directory
after the image build can start.
Signed-off-by: John Crispin <john@phrozen.org>
iremap is called with BUILD_DIR and with PKG_BUILD_DIR, never with
STAGING_DIR. A header taken from the staging directory therefore reaches
__FILE__ and the debug information with its absolute path, and the result
depends on the directory the tree was built in.
Measured in the published snapshot of lantiq/xway: mt76-connac-lib.ko,
mt76x02-lib.ko and mt7925-common.ko hold
/builder/shared-workdir/build/staging_dir/target-mips_24kc_musl/usr/include/mac80211/net/cfg80211.h
That path is __FILE__ in .rodata, so the strip before the package is built
cannot reach it and the module ships it.
Map STAGING_DIR as well. Keep the staging_dir element in the replacement:
the last element of STAGING_DIR and of BUILD_DIR carry the same name, so
the bare name would make the two roots indistinguishable. The map of
BUILD_DIR does not change, so only a path that leaks today changes.
Verified with the target compiler on a header below a mapped directory:
-ffile-prefix-map removes the absolute path from both __FILE__ and the
debug information, -fmacro-prefix-map removes it from __FILE__ alone. The
shipped half is __FILE__, so both variants cover it.
A local build of mt76 also holds nine staging paths, but those are debug
information: the build directory copy of mt76-connac-lib.ko is 653456
bytes with 13 debug sections and nine such paths, the shipped copy is
84688 bytes with none. They are named here so that the next reader does
not take them for the same defect.
Signed-off-by: John Crispin <john@phrozen.org>
TF-A embeds __TIME__ and __DATE__ in its build message, so bl2 and bl31
differ on every build, and so do all images that contain them.
Pass BUILD_MESSAGE_TIMESTAMP, derived from the source date of the
package, in the same format.
Signed-off-by: John Crispin <john@phrozen.org>
A configure run spends its time on the compiler invocations behind its
checks, not on the shell. tools/m4 runs the compiler 633 times and takes
24 seconds. ccache does not help, because it refuses to cache an
autoconf conftest compile at all; it counts these separately as
"Autoconf compile/link" under its uncacheable calls.
A package that sets PKG_CONFIGURE_CACHE or HOST_CONFIGURE_CACHE keeps a
config.cache under tmp/, so a package that is cleaned and built again
reads the answers instead of running the checks. The configure stage of
tools/m4 drops from 24.5 s to 1.5 s. dirclean removes tmp/, so a full
reset of the tree starts from cold answers.
A package has to opt in, because a cache is only correct for a configure
script that keeps its side effects outside the AC_CACHE_VAL body.
ncurses appends -D_XOPEN_SOURCE and -D_DEFAULT_SOURCE to CPPFLAGS inside
that body, so a cached run restores the value, skips the append, and
then fails to compile wcwidth. ncurses_cfg.h is identical in both cases;
the difference is in the generated Makefile.
The path covers everything else that changes an answer. A target package
is keyed on the toolchain directory, which carries the gcc version. A
host package is keyed on the version string of the host compiler,
because autoconf cannot detect a new compiler behind an unchanged name;
gcc 14 turned an implicit function declaration into an error and changed
many answers that way. Both are keyed on a hash of the configure
arguments and the build flags, and autoconf refuses a cache written with
different flags on top of that.
A cached program path outlives a wipe of staging_dir, so ac_cv_path_ and
ac_cv_prog_ answers are dropped when the cache loads and configure
searches PATH again. Without that, a cache written before staging_dir
held sed pins /usr/bin/sed for good. A search runs no compiler, so this
costs nothing that matters.
The cache arrives through a site script rather than --cache-file, so a
hand written configure script never sees a new argument. The site script
removes the file of the previous arguments, so a different configuration
gets a cold cache instead of an autoconf error. It also unsets the
variable, and make exports it for the configure stage alone, so a nested
configure run keeps its own cache.
Set CONFIGURE_CACHE_DIR to keep the cache elsewhere, or to the empty
string to run every check again.
Host builds now read a site script on Linux as well. Before this change
only Darwin had one.
CONFIG_SITE holds one path name: autoconf before 2.70 reads the whole
value as that one name, so a second path would silence both scripts. It
names the cache script alone, and the cache script loads the site script
of the architecture from CONFIG_SITE_BASE.
Signed-off-by: John Crispin <john@phrozen.org>
libdeflate-gzip -12 of a kernel takes about a second, and its input
usually did not change. Run it through cache-run.sh, as Build/lzma.
libdeflate-gzip stores no mtime or file name, and -k -S .new gives the
same output as -c.
Signed-off-by: John Crispin <john@phrozen.org>
On a rebuild, the initrd and the initramfs kernel image usually have
the same content as before, but xz and lzma compress them again on
every build. For the OpenWrt One this takes about 6 s of the build.
Add scripts/cache-run.sh. It runs a command that creates one file from
another, and stores the output together with a hash of the input, the
command line and the tool. If they are the same in the next run, it
copies the stored output instead. The commands are deterministic, so
the result is identical.
On a hit the script compares the stored output with the file that is
already there and copies only when the two differ. The output then keeps
its timestamp, so a later step that compares timestamps does not treat it
as new work.
Use it for the initrd compression and for Build/lzma. bzip2 and lzop
stay as they are: bzip2 writes to stdout, and lzop stores the mtime of
its input. xz and gzip now keep the cpio, as the other tools do, so a
cache hit leaves the same files behind as a normal run.
The slot name follows the path of the output below $(KDIR), because two
targets in different directories can share a basename. cache-run.sh holds
a lock over the slot and swaps the stored output in with a rename, so two
image recipes on one slot never read a half written pair.
Signed-off-by: John Crispin <john@phrozen.org>
check_download_integrity hashes the downloaded file on every parse of
every Makefile that uses it. With large archives this costs seconds per
build, for example about 1.2 s for linux-firmware, which is on the
critical path of the package build.
Remember the identity (inode, size, mtime, ctime) of a file that
matched its expected hash, and hash it again only after it changed.
The identity comes from the stat of tools/coreutils, which a Linux host
does not build. Fall back to the stat of the host, and drop the cache
when neither one answers, so that a missing stat costs time and never
trust.
Signed-off-by: John Crispin <john@phrozen.org>
prepare-tmpinfo runs the metadata scan for the packages and for the targets
on every build. Each run starts two more make processes and walks the whole
directory, which takes 0.15 s per scan even when it finds nothing to do.
Keep a stamp per scan and skip the scan while no Makefile, no .mk file and
no directory below the scanned tree is newer than the stamp, and while
include, rules.mk and .config are unchanged as well.
package/kernel/linux reads target/linux/*/modules.mk through SCAN_DEPS, so
the package scan watches those files although they sit in the target tree.
This takes the time of the prereq step from 0.45 s to 0.21 s, and the time
of a full no-op build from 3.40 s to 3.25 s for the OpenWrt One.
Signed-off-by: John Crispin <john@phrozen.org>
The configure stage rewrites the kernel configuration on every build. It
runs kconfig.pl twice and package-metadata.pl once, which takes an eighth of
a second. target/compile and target/install both pay it.
The rule carries FORCE because the kernel configuration also depends on the
package metadata, which is not a prerequisite. Add a check file that the
rule creates only when the configure stage must run, and make the stamp
depend on it. The check covers the kernel configuration files, .config, the
package metadata, the scripts that read them, the two build system files
that hold the recipe, the target directory and the user headers.
This takes the time of target/linux install on an unchanged tree from 1.33 s
to 1.15 s, and the time of a full no-op build from 3.68 s to 3.40 s for the
OpenWrt One.
Signed-off-by: John Crispin <john@phrozen.org>
target/linux install runs the whole image build on every build. It creates
the root file system image, the kernel images, the FIT containers and the
factory images again, although the inputs are the same. This takes 0.7 s for
the OpenWrt One and much more for a target with many devices.
Record a state after a successful image build:
- a hash of the target directory, because the build normalises the
timestamps of the root file system and they cannot report a change,
- the name and the size of every image in the output directory, so that a
deleted image brings the build back,
- the name and the size of every file below json_info_files, because
image_prepare drops that directory and profiles.json comes from it.
Skip the image build while the state is the same and no file of the kernel
build directory, the kernel tree, the image directory of the staging
directory, the host tools, the target directory of the tree, include,
scripts or .config is newer than the state file.
This takes the time of target/linux install on an unchanged tree from 1.77 s
to 1.33 s, and the time of a full no-op build from 4.11 s to 3.72 s for the
OpenWrt One.
Signed-off-by: John Crispin <john@phrozen.org>
Every build runs conf --defconfig on a copy of .config to warn if the
configuration is out of sync. This takes about a second before the
build starts.
Skip the check if the last one passed, and neither .config nor a
Kconfig file, the generated Kconfig files or conf changed since then.
If the check fails, it runs and warns again on every build, as before.
Signed-off-by: John Crispin <john@phrozen.org>
prepare-tmpinfo runs on every build and calls package-metadata.pl
three times, although tmp/.packageinfo usually did not change. It also
writes tmp/.config-feeds.in on every build: the check against
tmp/.packageauxvars never passes, as that file is written again later
in the same recipe.
Generate the package metadata files only if tmp/.packageinfo or the
scripts are newer, and replace tmp/.config-feeds.in only if its content
changed.
Signed-off-by: John Crispin <john@phrozen.org>
kbuild runs twice per build, for the modules and for the image, and
takes about half a second each time even if there is nothing to do.
Store a hash of the kbuild command line after a successful run, and
skip the next run if the hash is the same and no file in the kernel
tree or the toolchain is newer than the stamp. The OpenWrt files at the
top of the kernel tree are ignored, apart from .config. Directories are
ignored, as kbuild changes them with temporary files, so a deleted file
alone does not trigger a run.
With CONFIG_STRIP_KERNEL_EXPORTS the image link reads symtab.h, which
sits above the kernel tree and follows the modules of the packages. The
scan of the image run names that file as well.
The workaround that deletes vmlinux now only runs together with kbuild.
Signed-off-by: John Crispin <john@phrozen.org>
With TARGET_ROOTFS_INITRAMFS_SEPARATE, the initramfs kernel differs
from the regular kernel only in CONFIG_RD_<ALGO>. Both are built in the
same tree, so every build switches the kernel config twice and relinks
vmlinux twice, even if nothing changed.
Give the regular kernel the same ramdisk support and use its image for
the initramfs as is. The initramfs step now only creates the initrd.
Remove the separate initramfs branches of Kernel/SetInitramfs, as it is
no longer called in this mode.
Signed-off-by: John Crispin <john@phrozen.org>
Set BUILD_TIME_LOG=<file> to record a begin and an end event for every
prepare, configure, compile and install stage of every tool, toolchain
component, kernel and package. scripts/build-time-report.pl turns the log
into a report.
The report divides each second of the build between the stages that run
in it, so the numbers show which stages hold up the wall clock rather
than which ones burn the most CPU. Nothing is recorded and no extra
process runs when BUILD_TIME_LOG is unset.
Signed-off-by: John Crispin <john@phrozen.org>
subdir calls diralias twice for every target of every build directory,
and diralias splits the path into words twice to find its last
component. This takes a quarter of the parse time of the main Makefile.
Use notdir, which returns the same last component. The rules do not
change.
Signed-off-by: John Crispin <john@phrozen.org>
subdir expands the recipe of every target of every build directory when
it creates the rules, although make runs only a few of them. The main
Makefile creates these rules on every make call, and a build calls it
several times in a row.
Put only a call of the recipe into the rules, so that make expands it
when the recipe runs. The commands do not change. The parse of the main
Makefile takes 0.10 s instead of 0.15 s, with 318 package directories.
Signed-off-by: John Crispin <john@phrozen.org>
A target defines more than 100 devices, and the image Makefile
expands the kernel, image and artifact rules of every one of them. The
rules of a device that is not selected are only reachable through
install-disabled and compile-disabled, so nothing uses them. The image
Makefile is parsed twice per build, 0.3 s each for mediatek.
Only expand Device/Build for selected devices, and for all devices if
CONFIG_IB is set, as the ImageBuilder compiles them all.
Signed-off-by: John Crispin <john@phrozen.org>
BuildPackage expands the full BuildTarget/ipkg for every package in a
Makefile, whether it is selected or not. package/kernel/linux defines
more than 1000 kmods, and this takes about 0.9 s of its 1.2 s parse,
which is on the critical path of every build.
Move the clean rule into BuildTarget/ipkg/disabled, and only expand it
for packages that are not selected, unless DEVELOPER is set. The other
rules of such a package can not be reached, as its compile target only
depends on them if the package is selected. Target types without a
disabled variant are expanded as before.
Signed-off-by: John Crispin <john@phrozen.org>
host-build.mk runs the YEAR_2038 compiler test while it parses, so every
package with a host build runs it on every make call, also for a target
build or a build that has nothing to do.
Add the result to HOST_CONFIGURE_ARGS without expanding it, so the test
runs only when the host build is configured.
Signed-off-by: John Crispin <john@phrozen.org>
Make expands PKG_SOURCE_DATE_EPOCH for every recipe of a package, as
package-pack.mk exports it to the recipes. Every expansion runs
get_source_date_epoch.sh, which runs git.
Keep the result for each directory. The directory still depends on
whether the build directory has a version.date file.
Signed-off-by: John Crispin <john@phrozen.org>
Build/CoreTargets fixes the value of STAMP_PREPARED, but its expansion
references STAMP_PREPARED several times before that happens, and every
reference hashes all files of the package again with find and mkhash.
HOST_STAMP_PREPARED carries the same hash and the same cost.
Compute the hash on first use in PKG_FILES_MD5 and keep it. Both stamps
read it.
Signed-off-by: John Crispin <john@phrozen.org>
feeds.mk runs "scripts/feeds list -n" when it is included, which
happens in every package Makefile. The list is only needed to find the
package directories of the feeds, which a package Makefile only does
when it creates or cleans a package.
Evaluate FEEDS_AVAILABLE on first use.
Signed-off-by: John Crispin <john@phrozen.org>
The top-level Makefile calls itself for prereq and for the build, and
every call runs config.guess again. config.guess runs the host C
compiler three times, which takes about 40 ms.
Take GNU_HOST_NAME from the environment in recursive calls, as for
REVISION and SOURCE_DATE_EPOCH.
Signed-off-by: John Crispin <john@phrozen.org>
The top-level Makefile calls itself for prereq and for the build, and
every call runs getver.sh and get_source_date_epoch.sh again, although
the values are already exported by the first call.
Take them from the environment in recursive calls. The first call still
computes them, so a value in the environment of the user is ignored as
before.
Signed-off-by: John Crispin <john@phrozen.org>
version_filter calls package-metadata.pl for every list that contains
a versioned item such as "foo@ge6.18". Parsing package/kernel/linux
runs it about 40 times, which takes 1.3 s of 2.2 s, and this parse is
on the critical path of every build.
Filter items of the form <name>@<op><version> in make. The version
comparison stays in package-metadata.pl, but runs only once per
version. Other forms are passed to package-metadata.pl as before.
The only difference in the output: items without a version condition
are no longer passed through the shell, so quotes in a KCONFIG value
are kept. That list is only used for variable names, so the result is
the same. All kmod packages and images are identical.
Signed-off-by: John Crispin <john@phrozen.org>
The variants of a package are built one after the other in one recipe.
For packages with many variants, such as the boot loaders, this makes
the variants the longest chain of the package build, also if nothing
has to be done.
Add PKG_PARALLEL_VARIANTS. If a package sets it, every variant gets its
own make target with the dependencies of the package, and the package
target depends on these targets. A package must only set it if its
variants share no files outside of their build directories.
Signed-off-by: John Crispin <john@phrozen.org>
All variants of a package collect their InstallDev files in the same
directory below $(TMP_DIR) and remove it before and after use. Add the
build variant to its name, as for the staging file list, so that the
variants of a package do not share it.
Signed-off-by: John Crispin <john@phrozen.org>
find_md5 sorts the list of package files with the locale of the user.
The top-level Makefile sets LC_ALL=C, but a make call in a package
directory, such as make -C package/kernel/linux, uses the locale of
the shell. With en_GB.UTF-8, some file names sort in a different
order, so the hash and the name of the prepared stamp differ, and the
package is built again. The next top-level build then builds it again.
Always sort in the C locale.
Signed-off-by: John Crispin <john@phrozen.org>
Commit 501123eb7a ("include/subdir: on build failure always print
error") passes $(2) instead of $(1) to ERROR for the builds of a build
type, such as host builds. subdir has no second parameter, so with
BUILD_LOG set a failed host build is written to logs/error.txt instead
of logs/package/error.txt, where the other errors of the subdir go.
Fixes: 501123eb7a ("include/subdir: on build failure always print error")
Signed-off-by: John Crispin <john@phrozen.org>
stampfile assigns the clean rule of its stamp to <subdir>//clean with
:=, so every call replaces the one before. toolchain and tools define
a compile and a check stamp, and <subdir>/clean only removes the check
stamp. After "make toolchain/clean", the compile stamp is newer than the
toolchain sources, the next build takes the toolchain as built, and the
target build fails because the compiler is missing.
Append the clean rule instead, so that all stamps are removed.
Signed-off-by: John Crispin <john@phrozen.org>
The prepare stamp of a Makefile depends on the library version files
of every package it defines, selected or not. Packages that are not
selected are not in the dependency graph of the top-level build, so
their libraries can be built after the package in a parallel build,
and the package is prepared and built again on the next run.
The own packages of a Makefile are only filtered out if they are
defined before the packages that depend on them. mac80211 defines
cfg80211 and mac80211 after the drivers, so it depends on its own
version files, and every clean build is followed by a rebuild of
mac80211, mt76 and linux-firmware.
Only add the library dependencies of selected packages, or of all
packages if DEVELOPER is set.
Signed-off-by: John Crispin <john@phrozen.org>
The rdep check on PKG_BUILD_DIR excludes the ipkg-* staging dirs, but
not the apk-* ones. The package step writes the apk scripts after
.built is touched. If this crosses a second boundary, the next build
sees newer files and packs the package again, which starts the loop
again. The kernel package with its kmods takes several seconds to pack
and is repacked on every build.
Anchor the pattern to PKG_BUILD_DIR, as a plain "*/apk-*" also matches
the build dir of the apk package itself.
Signed-off-by: John Crispin <john@phrozen.org>
Packages that use 64-bit atomics carry open-coded architecture lists
such as +(arm||armeb||mips||mipsel||powerpc):libatomic. Those lists
drift (ruby still names arc), over-include ARMv6K and ARMv7 targets
whose compiler inlines 64-bit atomics via ldrexd/strexd, and
under-include ARMv5 in other packages. Add HAS_ATOMIC64, selected on
the architectures where GCC emits 64-bit atomics inline (all 64-bit
ones, x86 via cmpxchg8b, ARMv6K and ARMv7), and ATOMIC64_DEPENDS as
the single dependency expression to use instead. The condition is one
negated symbol because the Depends: evaluator in package-pack.mk only
handles flat conjunctions and disjunctions. 16-byte and size-generic
atomics need libatomic on every architecture and keep a plain
+libatomic dependency.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
arm_v6 and arm_v7 are taken from the merged kernel config, which
describes the kernel and can differ from the userspace ISA:
sunxi/arm926ejs inherits CONFIG_CPU_V7=y from its target config and is
advertised as ARMv7 although it builds for arm926ej-s, and a promptless
symbol such as CPU_V7 cannot be unset in a subtarget config in a way
that survives kernel_oldconfig. Derive arm_v6, arm_v6k and arm_v7 from
CPU_TYPE, the core the toolchain targets, and keep ARMv6K, which has
ldrexd/strexd, apart from plain ARMv6.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
before:
```
Applying patch 0001-configure-allow-disable-fortify_source.patch
patching file configure
Now at patch 0001-configure-allow-disable-fortify_source.patch
Patch 0001-configure-allow-disable-fortify_source.patch is unchanged
Applying patch 0002-util-mmap-alloc-fix-missing-MAP_SYNC.patch
patching file util/mmap-alloc.c
Now at patch 0002-util-mmap-alloc-fix-missing-MAP_SYNC.patch
Patch 0002-util-mmap-alloc-fix-missing-MAP_SYNC.patch is unchanged
Applying patch 0004-fix-meson-cross-build-compiler-sanity-check.patch
patching file configure
Now at patch 0004-fix-meson-cross-build-compiler-sanity-check.patch
Patch 0004-fix-meson-cross-build-compiler-sanity-check.patch is unchanged
```
after:
```
Applying patch 0001-configure-allow-disable-fortify_source.patch
patching file configure
Now at patch 0001-configure-allow-disable-fortify_source.patch
Patch 0001-configure-allow-disable-fortify_source.patch is unchanged
Applying patch 0002-util-mmap-alloc-fix-missing-MAP_SYNC.patch
patching file util/mmap-alloc.c
Now at patch 0002-util-mmap-alloc-fix-missing-MAP_SYNC.patch
Patch 0002-util-mmap-alloc-fix-missing-MAP_SYNC.patch is unchanged
Applying patch 0004-fix-meson-cross-build-compiler-sanity-check.patch
patching file configure
Now at patch 0004-fix-meson-cross-build-compiler-sanity-check.patch
Patch 0004-fix-meson-cross-build-compiler-sanity-check.patch is unchanged
```
Signed-off-by: Andy Chiang <AndyChiang_git@outlook.com>
Link: https://github.com/openwrt/openwrt/pull/24965
Signed-off-by: Robert Marko <robimarko@gmail.com>
The GL.iNet specific metadata contains a `date`-field which takes the current
timestamp instead of using SOURCE_DATE_EPOCH. This commits make use of the
variable, if available.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Paul Spooren
Hauke Mehrtens
Grische
Robert Marko
John Crispin
Daniel Golle
Gilberto Conde
Josef Schlehofer
Andy Chiang