Currently, we depend on building the host Ruby support just for the two
FWU scripts that are written in Ruby.
Use Codex 5.6 to convert the scripts to Python and drop Ruby dependency.
Signed-off-by: Robert Marko <robert.marko@sartura.hr>
The initramfs step copied /init into the root filesystem directory and
never removed it again. With CONFIG_TARGET_PER_DEVICE_ROOTFS the
initramfs kernel and the rootfs image of a device come from that same
directory. Nothing orders the two steps, so /init entered the squashfs
image or not, depending on which step ran first. The reproducible build
service reports this as a difference between the published image and a
rebuild.
Add /init to the initramfs from outside the root filesystem instead. A
built in initramfs gets it through a gen_init_cpio list file. A separate
cpio gets it appended to the finished archive. The mode is now fixed at
0755 and no longer follows the umask of the checkout.
The initramfs step also normalised the mtime of the root filesystem
directory. Do that in prepare_rootfs, so no step writes to the directory
after the image build can start.
Signed-off-by: John Crispin <john@phrozen.org>
u-boot 2013.10 builds timestamp_autogenerated.h with a bare date call, so
U_BOOT_DATE and U_BOOT_TIME hold the wall clock and the banner of every
image changes on each build:
U-Boot 2013.10-openwrt4 (Sep 21 2026 - 20:47:50) ARV7506PW11
Add a patch that uses SOURCE_DATE_EPOCH when the environment provides it,
which the build already exports, and keeps the wall clock otherwise. Later
u-boot versions do the same in their own timestamp rule, which is why only
this package needs the change.
Found with two lantiq/xway trees whose directory names differ in length.
Three bytes of u-boot.bin differed, all inside the banner. After the
change the three u-boot images of arv7506pw11 are identical in both trees
and the banner follows the source date.
Signed-off-by: John Crispin <john@phrozen.org>
gcc builds the target libraries with CFLAGS_FOR_TARGET, which carries no
prefix map. __FILE__ therefore expands to an absolute path, and assert in
libstdc++ and the ryu code put seven of them into .rodata, where the strip
before packaging cannot reach them. The libstdcpp6 package differs between
two builds in different directories, and so does every image that ships it.
The published package shows the same thing. libstdcpp6-14.4.0-r5.apk of
bcm47xx/mips74k holds
/builder/shared-workdir/build/build_dir/toolchain-mipsel_74kc_gcc-14.4.0_musl/gcc-14.4.0/libstdc++-v3/src/c++11/debug.cc
so no rebuild of that package can match today.
iremap is called with BUILD_DIR for the kernel, with PKG_BUILD_DIR for a
package and with STAGING_DIR for the target headers. The toolchain build
directory has no such call. Add one for the three FOR_TARGET flag sets.
The replacement keeps the build_dir element, because the last element of
the toolchain build directory and of the toolchain staging directory is
the same string and one replacement for both would make the two roots
indistinguishable.
Tested with two bcm47xx/mips74k trees whose directory names differ in
length. The shipped libstdc++.so.6.0.33 holds no absolute path any more
and the libstdcpp6 package is now identical in both trees. It differed by
165 bytes before.
The toolchain staging directory still appears in the debug information of
the same library, 244 times. Those are DWARF include directories, they do
not survive the strip, and no shipped file carries them.
Signed-off-by: John Crispin <john@phrozen.org>
iremap is called with BUILD_DIR and with PKG_BUILD_DIR, never with
STAGING_DIR. A header taken from the staging directory therefore reaches
__FILE__ and the debug information with its absolute path, and the result
depends on the directory the tree was built in.
Measured in the published snapshot of lantiq/xway: mt76-connac-lib.ko,
mt76x02-lib.ko and mt7925-common.ko hold
/builder/shared-workdir/build/staging_dir/target-mips_24kc_musl/usr/include/mac80211/net/cfg80211.h
That path is __FILE__ in .rodata, so the strip before the package is built
cannot reach it and the module ships it.
Map STAGING_DIR as well. Keep the staging_dir element in the replacement:
the last element of STAGING_DIR and of BUILD_DIR carry the same name, so
the bare name would make the two roots indistinguishable. The map of
BUILD_DIR does not change, so only a path that leaks today changes.
Verified with the target compiler on a header below a mapped directory:
-ffile-prefix-map removes the absolute path from both __FILE__ and the
debug information, -fmacro-prefix-map removes it from __FILE__ alone. The
shipped half is __FILE__, so both variants cover it.
A local build of mt76 also holds nine staging paths, but those are debug
information: the build directory copy of mt76-connac-lib.ko is 653456
bytes with 13 debug sections and nine such paths, the shipped copy is
84688 bytes with none. They are named here so that the next reader does
not take them for the same defect.
Signed-off-by: John Crispin <john@phrozen.org>
Both SBOM generators produce a different file on every run, so the
published .bom.cdx.json of an image and of a package feed never match a
rebuild.
dump_cyclonedxsbom_json() in scripts/package-metadata.pl has three
causes. It draws the serial number from rand(), it takes the timestamp
from the wall clock, and encode_json() follows the Perl hash order, which
changes with the hash seed of each run. The last one alone reorders every
key in the file.
scripts/make-sbom.py has the first two causes, through uuid.uuid4() and
datetime.utcnow().
Take the timestamp from SOURCE_DATE_EPOCH, which the build exports. Derive
the serial number from the timestamp and the component list, so it stays a
valid RFC 4122 identifier and still changes when the content changes. Sort
the keys of the Perl output.
Tested with the package index and the image manifest of an ipq40xx build.
Two runs with the same SOURCE_DATE_EPOCH are now identical, two runs with
different values differ, and the scripts exit with an error when the value
is not a number. Both files differ between two runs without the change.
Signed-off-by: John Crispin <john@phrozen.org>
configure.in of drv_tapi and drv_vmmc compiles the whole configure command
line into the module. drv_tapi_linux.c prints it through a proc file. The
command line holds the absolute paths of the build machine, among them
--enable-kernelbuild, --with-ifxos-incl, --enable-tapiincl, CPPFLAGS and
LDFLAGS, so the module depends on the directory it was built in.
Measured in the published snapshot modules of lantiq/xway: drv_tapi.ko and
drv_vmmc.ko both hold
--enable-kernelbuild=/builder/shared-workdir/build/build_dir/...
The existing 020-not-leak-cflags.patch covers the Kbuild CFLAGS, not this
string.
Drop every argument that holds a path and keep the feature flags. The loop
repeats the idiom the file already uses for CONFIGURE_OPTIONS.
Signed-off-by: John Crispin <john@phrozen.org>
Build/boot-overlay copies the device trees and builds the boot script in
$@.boot, so every file carries the build time. mkfs.ubifs writes st_mtime
straight into the inode and has no SOURCE_DATE_EPOCH support, and the two
tar calls pass neither --sort=name nor the options that pin the owner and
the mode, so both nand.ubi and bootfs.tar.gz differ between builds.
The diffoscope of the published bootfs.tar.gz shows both: wall clock
mtimes on the device tree files, and a member order that follows readdir.
Set the times in $@.boot once, before the UBIFS image and the tar are
made. Sort the members of both archives and give them a fixed owner and
mode, as the other archive recipes do.
Signed-off-by: John Crispin <john@phrozen.org>
Build/cfe-jffs2 is the only mkfs.jffs2 call in the tree that passes
neither --squash-uids nor --squash-perms. The recipe creates the marker
files with touch and cp, so the JFFS2 inodes carry the uid, the gid and
the umask of the build machine. The published images hold uid and gid
1000.
mkfs.jffs2 already takes the timestamp from SOURCE_DATE_EPOCH, so the
owner and the mode are the remaining inputs from the environment.
Pass both options, as mvebu and ath79 already do.
Signed-off-by: John Crispin <john@phrozen.org>
mtk_image_set_mt7621_header() reads the wall clock and writes it to the
timestamp of the legacy image header, so the MT7621 boot loader image
differs on every build.
Add a patch that uses imagetool_get_source_date(), which the other image
tools already use, and keep the wall clock as the fallback.
Signed-off-by: John Crispin <john@phrozen.org>
7zr stores the modification time of the input file. The kernel image
carries the build time, so the H3C and HPE images differ between builds.
Set the time of the input to SOURCE_DATE_EPOCH before the archive is
created.
Signed-off-by: John Crispin <john@phrozen.org>
Several image recipes call tar without the options that pin the member
order, the owner, the mode and the modification time. The archive then
holds the build time, the umask of the build user and the readdir order of
the build directory, so two builds of the same source produce different
images.
Pass the same options that Image/mkfs/targz and sysupgrade-tar.sh already
use. The recipes define no member order, so sorting the members takes
nothing away. The diffoscope of the published WAX610 archive shows the
order differing between the build machines, which is what a readdir order
produces.
--mode=go-w clears the write bit of the group and of the others. tar keeps
the mode of the file on disk, which follows the umask of the build user,
so without it the archive differs between a host with umask 022 and one
with umask 002.
Covers the Netgear factory archives of ipq40xx, ipq60xx and ipq807x, the
Creality factory archive of mt76x8, the D-Link DWL8610AP archive of
bcm53xx, the Freecom archive of ixp4xx and the Linksys switch archive of
realtek.
Signed-off-by: John Crispin <john@phrozen.org>
The LoongArch vDSO Makefile builds its objects with a flag set of its own,
in the same way as the MIPS one. It drops -ffile-prefix-map, adds -g and
links with --build-id, so the build id of the vDSO follows the absolute
path of the build directory. The kernel embeds the vDSO, so two builds in
different directories produce different images.
Take the prefix map flags over from KBUILD_CFLAGS.
Signed-off-by: John Crispin <john@phrozen.org>
arch/mips/vdso/Makefile takes a few flags from KBUILD_CFLAGS and adds -g,
so the vDSO always carries debug information. -ffile-prefix-map is not in
that list, so DW_AT_comp_dir of the vDSO objects holds the absolute build
directory. The vDSO is linked with --build-id=sha1, which hashes the debug
information, and objcopy -S then removes the debug information and keeps
the note. The kernel embeds that image.
Every MIPS image therefore depends on the name of the build directory. The
reproducible build service works in a directory with a random name, so it
reports all MIPS targets as different. The ARM architectures add their vDSO
flags on top of KBUILD_CFLAGS and are not affected.
Add a patch to 6.12 and to 6.18 that takes the two prefix map flags over
from KBUILD_CFLAGS.
Tested with two ath79/nand builds in directories of different name. Before
the patch the vDSO build ids differ and all 91 images differ. After the
patch the build ids match and all 91 images are identical.
Signed-off-by: John Crispin <john@phrozen.org>
mkimage -T mxsimage builds u-boot.sb. sb_get_timestamp() reads the wall
clock and sb_prefill_image_header() writes it to the boot image header.
The header digest covers that field, so the sdcard images of mxs differ on
every build.
Add a patch that uses imagetool_get_source_date(), which the other image
tools already use, and keep the wall clock as the fallback.
Tested with the mxsimage configuration of u-boot. Two images are identical
with the same SOURCE_DATE_EPOCH, differ with different values, and still
differ when the variable is not set.
Signed-off-by: John Crispin <john@phrozen.org>
yaffsfs_CurrentTime() returns the wall clock, and Yaffs writes that value
to the access, modification and change time of every object header. The
RouterBOARD images of ipq40xx/mikrotik therefore differ on every build,
and so does the sysupgrade metadata checksum that follows them.
Add a patch that reads SOURCE_DATE_EPOCH, which the build already exports.
Tested with two images written one second apart. They are identical with
SOURCE_DATE_EPOCH set and differ without it.
Signed-off-by: John Crispin <john@phrozen.org>
TF-A embeds __TIME__ and __DATE__ in its build message, so bl2 and bl31
differ on every build, and so do all images that contain them.
Pass BUILD_MESSAGE_TIMESTAMP, derived from the source date of the
package, in the same format.
Signed-off-by: John Crispin <john@phrozen.org>
The host apk signs with LibreSSL, which uses a random ECDSA nonce. A
package that is signed again has different bytes and often a different
size, even if its content did not change. The rootfs records the
package size in /lib/apk/db/installed, so a rebuild of one package
changes all images.
Add a patch that derives the nonce as specified in RFC 6979. Signed
packages and images are now identical after a rebuild.
Signed-off-by: John Crispin <john@phrozen.org>
These ten hold up the wall clock on their own. Their configure stages
run while nothing else does, because the tools build in dependency
order. Measured over a full build, cold cache against warm:
sed 21.68 s -> 1.22 s
m4 21.48 s -> 1.22 s
findutils 19.47 s -> 1.56 s
tar 17.75 s -> 1.47 s
cpio 14.28 s -> 1.26 s
bison 12.81 s -> 1.29 s
patch 12.16 s -> 1.31 s
elfutils 26.58 s -> 15.47 s
Each of the ten configures the same way with a warm cache as it does
without one. The check runs configure three times, without a cache, with
a cold cache and with a warm cache, and compares every file that
configure generates between the first and the last. ncurses fails that
check; these do not.
The only difference left is ac_ct_AR and ac_ct_RANLIB, which a cold run
fills through the fallback path of AC_CHECK_TOOL and a warm run leaves
empty. AR and RANLIB are correct in both, and no generated Makefile
reads the ac_ct_ form.
tools/cmake keeps its 33 s, the longest configure stage in the build. It
bootstraps with its own script rather than autoconf, so this does not
reach it.
Signed-off-by: John Crispin <john@phrozen.org>
A configure run spends its time on the compiler invocations behind its
checks, not on the shell. tools/m4 runs the compiler 633 times and takes
24 seconds. ccache does not help, because it refuses to cache an
autoconf conftest compile at all; it counts these separately as
"Autoconf compile/link" under its uncacheable calls.
A package that sets PKG_CONFIGURE_CACHE or HOST_CONFIGURE_CACHE keeps a
config.cache under tmp/, so a package that is cleaned and built again
reads the answers instead of running the checks. The configure stage of
tools/m4 drops from 24.5 s to 1.5 s. dirclean removes tmp/, so a full
reset of the tree starts from cold answers.
A package has to opt in, because a cache is only correct for a configure
script that keeps its side effects outside the AC_CACHE_VAL body.
ncurses appends -D_XOPEN_SOURCE and -D_DEFAULT_SOURCE to CPPFLAGS inside
that body, so a cached run restores the value, skips the append, and
then fails to compile wcwidth. ncurses_cfg.h is identical in both cases;
the difference is in the generated Makefile.
The path covers everything else that changes an answer. A target package
is keyed on the toolchain directory, which carries the gcc version. A
host package is keyed on the version string of the host compiler,
because autoconf cannot detect a new compiler behind an unchanged name;
gcc 14 turned an implicit function declaration into an error and changed
many answers that way. Both are keyed on a hash of the configure
arguments and the build flags, and autoconf refuses a cache written with
different flags on top of that.
A cached program path outlives a wipe of staging_dir, so ac_cv_path_ and
ac_cv_prog_ answers are dropped when the cache loads and configure
searches PATH again. Without that, a cache written before staging_dir
held sed pins /usr/bin/sed for good. A search runs no compiler, so this
costs nothing that matters.
The cache arrives through a site script rather than --cache-file, so a
hand written configure script never sees a new argument. The site script
removes the file of the previous arguments, so a different configuration
gets a cold cache instead of an autoconf error. It also unsets the
variable, and make exports it for the configure stage alone, so a nested
configure run keeps its own cache.
Set CONFIGURE_CACHE_DIR to keep the cache elsewhere, or to the empty
string to run every check again.
Host builds now read a site script on Linux as well. Before this change
only Darwin had one.
CONFIG_SITE holds one path name: autoconf before 2.70 reads the whole
value as that one name, so a second path would silence both scripts. It
names the cache script alone, and the cache script loads the site script
of the architecture from CONFIG_SITE_BASE.
Signed-off-by: John Crispin <john@phrozen.org>
The rootfs is installed from scratch on every build: apk add, then the
post-install and init scripts of every package, one after the other.
This takes about 1.5 s, even if no package changed.
Build a key from the package index, which holds a hash of every
package, the install list, .config, SOURCE_DATE_EPOCH, the files
overlay, apk, tar, gzip and the Makefiles that create the rootfs. Save
a copy of the rootfs with the key, and restore it from the copy if the
key did not change. Later build steps modify the rootfs, so it is
always restored from the copy and never reused in place.
prepare_rootfs runs tar and gzip over the apk script archive, so the
key holds those two host tools as well. With mklibs it also runs
rstrip.sh, which the key then holds too.
The install list now holds the base-files, libc and kernel versions
too, so that the key can use it. Only the APK path uses the cache.
The copy costs disk: the build directory carries the root filesystem
twice, once as root and root.orig and once in the cache.
Signed-off-by: John Crispin <john@phrozen.org>
libdeflate-gzip -12 of a kernel takes about a second, and its input
usually did not change. Run it through cache-run.sh, as Build/lzma.
libdeflate-gzip stores no mtime or file name, and -k -S .new gives the
same output as -c.
Signed-off-by: John Crispin <john@phrozen.org>
On a rebuild, the initrd and the initramfs kernel image usually have
the same content as before, but xz and lzma compress them again on
every build. For the OpenWrt One this takes about 6 s of the build.
Add scripts/cache-run.sh. It runs a command that creates one file from
another, and stores the output together with a hash of the input, the
command line and the tool. If they are the same in the next run, it
copies the stored output instead. The commands are deterministic, so
the result is identical.
On a hit the script compares the stored output with the file that is
already there and copies only when the two differ. The output then keeps
its timestamp, so a later step that compares timestamps does not treat it
as new work.
Use it for the initrd compression and for Build/lzma. bzip2 and lzop
stay as they are: bzip2 writes to stdout, and lzop stores the mtime of
its input. xz and gzip now keep the cpio, as the other tools do, so a
cache hit leaves the same files behind as a normal run.
The slot name follows the path of the output below $(KDIR), because two
targets in different directories can share a basename. cache-run.sh holds
a lock over the slot and swaps the stored output in with a rename, so two
image recipes on one slot never read a half written pair.
Signed-off-by: John Crispin <john@phrozen.org>
check_download_integrity hashes the downloaded file on every parse of
every Makefile that uses it. With large archives this costs seconds per
build, for example about 1.2 s for linux-firmware, which is on the
critical path of the package build.
Remember the identity (inode, size, mtime, ctime) of a file that
matched its expected hash, and hash it again only after it changed.
The identity comes from the stat of tools/coreutils, which a Linux host
does not build. Fall back to the stat of the host, and drop the cache
when neither one answers, so that a missing stat costs time and never
trust.
Signed-off-by: John Crispin <john@phrozen.org>
The -n caller of timestamp.pl only needs to know whether the stamp is the
newest file. The script reads the timestamp of every file to answer that,
which costs 0.09 s for the build directory of U-Boot with its 44000 files.
The rebuild check runs the script twice per package and per build.
Collect the paths first, and for -n ask find for the first file that is
newer than the stamp. find stops at that file, so the walk ends early when
there is work to do. The other modes keep the old code, and so does -n with
-follow, because find and the script treat a symbolic link to a file
differently there.
The check over the U-Boot build directory now takes 0.02 s. The result is
the same for all 44 build directories of the OpenWrt One. The time of a full
no-op build does not change on a machine with 24 cores, where the package
step waits for the dependency chain rather than for the checks.
Signed-off-by: John Crispin <john@phrozen.org>
buildversion, feedsversion and json_overview_image_info rewrite
version.buildinfo, feeds.buildinfo and profiles.json on every build. The
content is the same, but the new timestamp makes the checksum step hash
every image again, which takes 0.18 s for the OpenWrt One and grows with the
number of devices.
Write the three files to the temporary directory and copy them over the
result of the last build only when the content differs.
This takes the time of the index, image info and checksum step from 0.36 s
to 0.10 s, and the time of a full no-op build from 3.25 s to 3.00 s.
Signed-off-by: John Crispin <john@phrozen.org>
prepare-tmpinfo runs the metadata scan for the packages and for the targets
on every build. Each run starts two more make processes and walks the whole
directory, which takes 0.15 s per scan even when it finds nothing to do.
Keep a stamp per scan and skip the scan while no Makefile, no .mk file and
no directory below the scanned tree is newer than the stamp, and while
include, rules.mk and .config are unchanged as well.
package/kernel/linux reads target/linux/*/modules.mk through SCAN_DEPS, so
the package scan watches those files although they sit in the target tree.
This takes the time of the prereq step from 0.45 s to 0.21 s, and the time
of a full no-op build from 3.40 s to 3.25 s for the OpenWrt One.
Signed-off-by: John Crispin <john@phrozen.org>
The configure stage rewrites the kernel configuration on every build. It
runs kconfig.pl twice and package-metadata.pl once, which takes an eighth of
a second. target/compile and target/install both pay it.
The rule carries FORCE because the kernel configuration also depends on the
package metadata, which is not a prerequisite. Add a check file that the
rule creates only when the configure stage must run, and make the stamp
depend on it. The check covers the kernel configuration files, .config, the
package metadata, the scripts that read them, the two build system files
that hold the recipe, the target directory and the user headers.
This takes the time of target/linux install on an unchanged tree from 1.33 s
to 1.15 s, and the time of a full no-op build from 3.68 s to 3.40 s for the
OpenWrt One.
Signed-off-by: John Crispin <john@phrozen.org>
target/linux install runs the whole image build on every build. It creates
the root file system image, the kernel images, the FIT containers and the
factory images again, although the inputs are the same. This takes 0.7 s for
the OpenWrt One and much more for a target with many devices.
Record a state after a successful image build:
- a hash of the target directory, because the build normalises the
timestamps of the root file system and they cannot report a change,
- the name and the size of every image in the output directory, so that a
deleted image brings the build back,
- the name and the size of every file below json_info_files, because
image_prepare drops that directory and profiles.json comes from it.
Skip the image build while the state is the same and no file of the kernel
build directory, the kernel tree, the image directory of the staging
directory, the host tools, the target directory of the tree, include,
scripts or .config is newer than the state file.
This takes the time of target/linux install on an unchanged tree from 1.77 s
to 1.33 s, and the time of a full no-op build from 4.11 s to 3.72 s for the
OpenWrt One.
Signed-off-by: John Crispin <john@phrozen.org>
package/merge deletes the symlink farm of every built package and creates
it again on every build. package/merge-index then reads all of those
packages to create the index. With two thousand packages both steps take a
third of a second, and the cost grows with the size of the configuration.
Record the name, the size and the timestamp of every package file. Skip
both steps while that list is unchanged and the index is present. A rebuild
of the symlinks drops the index, so merge-index creates it again.
This takes the time of package/install on an unchanged tree from 1.05 s to
0.60 s for the OpenWrt One.
Signed-off-by: John Crispin <john@phrozen.org>
diffconfig.sh runs the configuration parser over the whole tree twice,
which takes 2 s of every build, and buildinfo runs it on each build.
The result depends only on .config and the Kconfig files.
Skip it while neither is newer than the file it wrote last time.
include/toplevel.mk guards the sync check of .config the same way.
This alone changes nothing on a parallel build, because buildinfo runs
beside the compile stamps. It pays once the other phases become shorter.
Signed-off-by: John Crispin <john@phrozen.org>
sha256sums hashes every file of a directory on each build. package/index
calls it over the package output of this tree, which holds 1258 files,
and spends 0.7 s of a no-op build on a file that does not change.
Skip the pass while the file exists and nothing in the directory is
newer than it.
The checksum target of a build calls sha256sums as well, but it keeps
writing the file on every build. buildversion and feedsversion rewrite
version.buildinfo and feeds.buildinfo beside it without a guard, so the
directory always holds something newer. Only package/index gains here.
Signed-off-by: John Crispin <john@phrozen.org>
package/index runs apk mkndx and make-index-json.py over every package
on each build, which takes 1.2 s of a no-op build of this tree with 1258
packages. The result depends only on the package files.
Skip the pass while packages.adb and index.json exist and no package
file, and no directory, is newer than index.json. The directory covers a
package that was added or removed, because neither changes a file that
is left. The comparison uses index.json rather than packages.adb,
because apk mkndx writes packages.adb through a rename, which sets the
mtime of the directory to that of the file it wrote. The directory would
then tie with packages.adb and the skip would rest on the two comparing
equal. index.json is written after it, so the directory is older.
Signed-off-by: John Crispin <john@phrozen.org>
timestamp.pl decides whether a subdir has work to do. It scans the stamp
file and the source directories, keeps the newest, and reports that the
work is done when the newest one is the stamp. The scan reads the mtime
through the core stat of perl, which truncates it to whole seconds, and
the comparison is strict with the stamp scanned first. Two files that
share a second therefore count as a tie, and a tie says that the work is
done.
include/subdir.mk touches tmp/.build at the start of a build and the
stamp of a subdir at the end, so the two land in the same second once a
build takes less than a second between them. The target install stamp is
the last one of a build, roughly 0.4 s before the build ends, so it is
the one that ties first: the next build then skips target/install and
ships the images of the previous build. Around one build in five was
affected once the build became fast enough.
Read the mtime through Time::HiRes, which reports the fraction as well.
Two touches around a fork no longer share a timestamp. Keep the core
stat where the module is missing, which restores the old behaviour
rather than failing.
Signed-off-by: John Crispin <john@phrozen.org>
The stamp of package/cleanup depends on tmp/.build, which is touched on
every make call, so every build deletes $(STAGING_DIR_ROOT) and every
package copies its files into it again, one after the other under a
lock. With all packages selected, this takes 5 s of a build that has
nothing to do, most of it in the 1102 kmods of package/kernel/linux.
Only clean it if .config, the package metadata or a file below package/
changed since the last cleanup. An edit of the source in a build
directory alone does not clean it, so a file that such an edit removes
from a package stays until then. make package/cleanup cleans it at any
time.
Signed-off-by: John Crispin <john@phrozen.org>
Every build runs conf --defconfig on a copy of .config to warn if the
configuration is out of sync. This takes about a second before the
build starts.
Skip the check if the last one passed, and neither .config nor a
Kconfig file, the generated Kconfig files or conf changed since then.
If the check fails, it runs and warns again on every build, as before.
Signed-off-by: John Crispin <john@phrozen.org>
prepare-tmpinfo runs on every build and calls package-metadata.pl
three times, although tmp/.packageinfo usually did not change. It also
writes tmp/.config-feeds.in on every build: the check against
tmp/.packageauxvars never passes, as that file is written again later
in the same recipe.
Generate the package metadata files only if tmp/.packageinfo or the
scripts are newer, and replace tmp/.config-feeds.in only if its content
changed.
Signed-off-by: John Crispin <john@phrozen.org>
kbuild runs twice per build, for the modules and for the image, and
takes about half a second each time even if there is nothing to do.
Store a hash of the kbuild command line after a successful run, and
skip the next run if the hash is the same and no file in the kernel
tree or the toolchain is newer than the stamp. The OpenWrt files at the
top of the kernel tree are ignored, apart from .config. Directories are
ignored, as kbuild changes them with temporary files, so a deleted file
alone does not trigger a run.
With CONFIG_STRIP_KERNEL_EXPORTS the image link reads symtab.h, which
sits above the kernel tree and follows the modules of the packages. The
scan of the image run names that file as well.
The workaround that deletes vmlinux now only runs together with kbuild.
Signed-off-by: John Crispin <john@phrozen.org>
With TARGET_ROOTFS_INITRAMFS_SEPARATE, the initramfs kernel differs
from the regular kernel only in CONFIG_RD_<ALGO>. Both are built in the
same tree, so every build switches the kernel config twice and relinks
vmlinux twice, even if nothing changed.
Give the regular kernel the same ramdisk support and use its image for
the initramfs as is. The initramfs step now only creates the initrd.
Remove the separate initramfs branches of Kernel/SetInitramfs, as it is
no longer called in this mode.
Signed-off-by: John Crispin <john@phrozen.org>
Set BUILD_TIME_LOG=<file> to record a begin and an end event for every
prepare, configure, compile and install stage of every tool, toolchain
component, kernel and package. scripts/build-time-report.pl turns the log
into a report.
The report divides each second of the build between the stages that run
in it, so the numbers show which stages hold up the wall clock rather
than which ones burn the most CPU. Nothing is recorded and no extra
process runs when BUILD_TIME_LOG is unset.
Signed-off-by: John Crispin <john@phrozen.org>
subdir calls diralias twice for every target of every build directory,
and diralias splits the path into words twice to find its last
component. This takes a quarter of the parse time of the main Makefile.
Use notdir, which returns the same last component. The rules do not
change.
Signed-off-by: John Crispin <john@phrozen.org>
subdir expands the recipe of every target of every build directory when
it creates the rules, although make runs only a few of them. The main
Makefile creates these rules on every make call, and a build calls it
several times in a row.
Put only a call of the recipe into the rules, so that make expands it
when the recipe runs. The commands do not change. The parse of the main
Makefile takes 0.10 s instead of 0.15 s, with 318 package directories.
Signed-off-by: John Crispin <john@phrozen.org>
A target defines more than 100 devices, and the image Makefile
expands the kernel, image and artifact rules of every one of them. The
rules of a device that is not selected are only reachable through
install-disabled and compile-disabled, so nothing uses them. The image
Makefile is parsed twice per build, 0.3 s each for mediatek.
Only expand Device/Build for selected devices, and for all devices if
CONFIG_IB is set, as the ImageBuilder compiles them all.
Signed-off-by: John Crispin <john@phrozen.org>
BuildPackage expands the full BuildTarget/ipkg for every package in a
Makefile, whether it is selected or not. package/kernel/linux defines
more than 1000 kmods, and this takes about 0.9 s of its 1.2 s parse,
which is on the critical path of every build.
Move the clean rule into BuildTarget/ipkg/disabled, and only expand it
for packages that are not selected, unless DEVELOPER is set. The other
rules of such a package can not be reached, as its compile target only
depends on them if the package is selected. Target types without a
disabled variant are expanded as before.
Signed-off-by: John Crispin <john@phrozen.org>
host-build.mk runs the YEAR_2038 compiler test while it parses, so every
package with a host build runs it on every make call, also for a target
build or a build that has nothing to do.
Add the result to HOST_CONFIGURE_ARGS without expanding it, so the test
runs only when the host build is configured.
Signed-off-by: John Crispin <john@phrozen.org>
Make expands PKG_SOURCE_DATE_EPOCH for every recipe of a package, as
package-pack.mk exports it to the recipes. Every expansion runs
get_source_date_epoch.sh, which runs git.
Keep the result for each directory. The directory still depends on
whether the build directory has a version.date file.
Signed-off-by: John Crispin <john@phrozen.org>
confvar runs a shell and mkhash every time make expands a stamp name that
contains it, 21 times in a make call for a package with a host build,
also if nothing has to be done.
The config symbols do not change during a make call, so keep the hash
of each list of symbols after the first use.
Signed-off-by: John Crispin <john@phrozen.org>
Build/CoreTargets fixes the value of STAMP_PREPARED, but its expansion
references STAMP_PREPARED several times before that happens, and every
reference hashes all files of the package again with find and mkhash.
HOST_STAMP_PREPARED carries the same hash and the same cost.
Compute the hash on first use in PKG_FILES_MD5 and keep it. Both stamps
read it.
Signed-off-by: John Crispin <john@phrozen.org>
Daniel Golle
Robert Marko
John Crispin