subdir calls diralias twice for every target of every build directory,
and diralias splits the path into words twice to find its last
component. This takes a quarter of the parse time of the main Makefile.
Use notdir, which returns the same last component. The rules do not
change.
Signed-off-by: John Crispin <john@phrozen.org>
subdir expands the recipe of every target of every build directory when
it creates the rules, although make runs only a few of them. The main
Makefile creates these rules on every make call, and a build calls it
several times in a row.
Put only a call of the recipe into the rules, so that make expands it
when the recipe runs. The commands do not change. The parse of the main
Makefile takes 0.10 s instead of 0.15 s, with 318 package directories.
Signed-off-by: John Crispin <john@phrozen.org>
A target defines more than 100 devices, and the image Makefile
expands the kernel, image and artifact rules of every one of them. The
rules of a device that is not selected are only reachable through
install-disabled and compile-disabled, so nothing uses them. The image
Makefile is parsed twice per build, 0.3 s each for mediatek.
Only expand Device/Build for selected devices, and for all devices if
CONFIG_IB is set, as the ImageBuilder compiles them all.
Signed-off-by: John Crispin <john@phrozen.org>
BuildPackage expands the full BuildTarget/ipkg for every package in a
Makefile, whether it is selected or not. package/kernel/linux defines
more than 1000 kmods, and this takes about 0.9 s of its 1.2 s parse,
which is on the critical path of every build.
Move the clean rule into BuildTarget/ipkg/disabled, and only expand it
for packages that are not selected, unless DEVELOPER is set. The other
rules of such a package can not be reached, as its compile target only
depends on them if the package is selected. Target types without a
disabled variant are expanded as before.
Signed-off-by: John Crispin <john@phrozen.org>
host-build.mk runs the YEAR_2038 compiler test while it parses, so every
package with a host build runs it on every make call, also for a target
build or a build that has nothing to do.
Add the result to HOST_CONFIGURE_ARGS without expanding it, so the test
runs only when the host build is configured.
Signed-off-by: John Crispin <john@phrozen.org>
Make expands PKG_SOURCE_DATE_EPOCH for every recipe of a package, as
package-pack.mk exports it to the recipes. Every expansion runs
get_source_date_epoch.sh, which runs git.
Keep the result for each directory. The directory still depends on
whether the build directory has a version.date file.
Signed-off-by: John Crispin <john@phrozen.org>
confvar runs a shell and mkhash every time make expands a stamp name that
contains it, 21 times in a make call for a package with a host build,
also if nothing has to be done.
The config symbols do not change during a make call, so keep the hash
of each list of symbols after the first use.
Signed-off-by: John Crispin <john@phrozen.org>
Build/CoreTargets fixes the value of STAMP_PREPARED, but its expansion
references STAMP_PREPARED several times before that happens, and every
reference hashes all files of the package again with find and mkhash.
HOST_STAMP_PREPARED carries the same hash and the same cost.
Compute the hash on first use in PKG_FILES_MD5 and keep it. Both stamps
read it.
Signed-off-by: John Crispin <john@phrozen.org>
world runs package/index, json_overview_image_info and checksum as
three separate calls of the top-level Makefile, and each call parses it
again. The calls run without -j, so one call with all three goals runs
them in the same order.
Signed-off-by: John Crispin <john@phrozen.org>
feeds.mk runs "scripts/feeds list -n" when it is included, which
happens in every package Makefile. The list is only needed to find the
package directories of the feeds, which a package Makefile only does
when it creates or cleans a package.
Evaluate FEEDS_AVAILABLE on first use.
Signed-off-by: John Crispin <john@phrozen.org>
The top-level Makefile calls itself for prereq and for the build, and
every call runs config.guess again. config.guess runs the host C
compiler three times, which takes about 40 ms.
Take GNU_HOST_NAME from the environment in recursive calls, as for
REVISION and SOURCE_DATE_EPOCH.
Signed-off-by: John Crispin <john@phrozen.org>
The top-level Makefile calls itself for prereq and for the build, and
every call runs getver.sh and get_source_date_epoch.sh again, although
the values are already exported by the first call.
Take them from the environment in recursive calls. The first call still
computes them, so a value in the environment of the user is ignored as
before.
Signed-off-by: John Crispin <john@phrozen.org>
NPROC is a recursive variable, so every use starts a shell. The image
Makefile expands the initramfs rule once per device, which runs nproc
about 200 times and makes the parse of the mediatek image Makefile take
about 1 s. It is parsed twice per build.
Evaluate it on first use and keep the result.
Signed-off-by: John Crispin <john@phrozen.org>
version_filter calls package-metadata.pl for every list that contains
a versioned item such as "foo@ge6.18". Parsing package/kernel/linux
runs it about 40 times, which takes 1.3 s of 2.2 s, and this parse is
on the critical path of every build.
Filter items of the form <name>@<op><version> in make. The version
comparison stays in package-metadata.pl, but runs only once per
version. Other forms are passed to package-metadata.pl as before.
The only difference in the output: items without a version condition
are no longer passed through the shell, so quotes in a KCONFIG value
are kept. That list is only used for variable names, so the result is
the same. All kmod packages and images are identical.
Signed-off-by: John Crispin <john@phrozen.org>
A package builds with -j1 unless it sets PKG_BUILD_PARALLEL, and 171 of
the 220 packages do not set it. These three gain the most from the job
server on a cold build of this tree:
mtd-utils 24.77 s -> 6.43 s
libsepol 17.99 s -> 2.18 s
libselinux 7.82 s -> 1.89 s
mtd-utils runs while nothing else does, so its 18 s count against the
wall clock in full.
Each one builds three times out of three with the job server, and each
build writes a tree that is identical to the one a serial build writes.
nftables gains as well, 17.9 s to 13.0 s, but it is left alone: its
build directory differs between two serial builds, so the check that
covers the other three says nothing about it.
Signed-off-by: John Crispin <john@phrozen.org>
A tool builds with -j1 unless it sets HOST_BUILD_PARALLEL, and 45 of the
73 tools do not set it. These three gain the most from the job server on
a cold build of this tree:
erofs-utils 23.11 s -> 15.73 s
mtd-utils 11.70 s -> 6.42 s
cpio 8.46 s -> 3.56 s
Each one builds three times out of three with the job server, and each
build writes a tree that is identical to the one a serial build writes.
Signed-off-by: John Crispin <john@phrozen.org>
Commit 30fb675847 ("gettext-full: disable parallel compilation")
disabled parallel builds of gettext 0.21, as they failed with make -j12.
With gettext 0.24.2, the host and the target build passed four times
each with make -j24, and libintl-full does not change. The host build
takes 172 s instead of 231 s, and the target build 142 s instead of
214 s. With all packages selected, both are on the longest chain of a
clean build.
Signed-off-by: John Crispin <john@phrozen.org>
The variants only write to their own build directories. The packages
without a variant are only built by the first variant.
With all packages selected, a clean build of the 31 variants takes 46 s
instead of 114 s, and the packages do not change. Together with the
commit that lists every build variant only once, a build that has
nothing to do takes 17 s instead of 33 s, as the variants were the
longest chain of it.
Signed-off-by: John Crispin <john@phrozen.org>
The variants of arm-trusted-firmware-mediatek and uboot-mediatek only
write to their own build directories and to files in the staging and
bin directories that have the variant in their name.
For the OpenWrt One, a clean build of the 9 TF-A variants takes 2 s
instead of 11 s, and of the 2 U-Boot variants 12 s instead of 21 s.
The images do not change.
Signed-off-by: John Crispin <john@phrozen.org>
The variants of a package are built one after the other in one recipe.
For packages with many variants, such as the boot loaders, this makes
the variants the longest chain of the package build, also if nothing
has to be done.
Add PKG_PARALLEL_VARIANTS. If a package sets it, every variant gets its
own make target with the dependencies of the package, and the package
target depends on these targets. A package must only set it if its
variants share no files outside of their build directories.
Signed-off-by: John Crispin <john@phrozen.org>
With FIP_COMPRESS set, the fip step compresses the BL31 image next to
the original in the staging directory. All U-Boot variants for the same
SoC, boot device and DRAM type write the same file.
Write the compressed BL31 to the build directory of the variant instead.
The fip does not change.
Signed-off-by: John Crispin <john@phrozen.org>
All variants of a package collect their InstallDev files in the same
directory below $(TMP_DIR) and remove it before and after use. Add the
build variant to its name, as for the staging file list, so that the
variants of a package do not share it.
Signed-off-by: John Crispin <john@phrozen.org>
ltq-adsl-mei and ltq-deu call make directly after a cd into the kernel
directory, and ltq-ifxos builds its module through autotools. None of them
pass KERNEL_MAKE_FLAGS, so the module compile runs without the
-ffile-prefix-map that iremap adds. WARN_ON_ONCE() in check_copy_size()
expands to warn_slowpath_fmt(__FILE__, ...), so the absolute build path
lands in the .ko.
Measured on the published snapshot packages of lantiq/xway and
xrx200_legacy:
- drv_mei_cpe.ko holds
/builder/shared-workdir/build/build_dir/target-mips_24kc_musl/linux-lantiq_xway/linux-6.18.52/include/linux/ucopysize.h
- drv_ifxos.ko holds the same path for linux-lantiq_xrx200_legacy
- ltq_atm_danube.ko, built with the flags, holds no such string
The module size follows the length of the build path, so the apk record
and every image that carries the module differ between two builds in
different directories.
ltq_deu_vr9.ko holds no such string today, but the package has the same
defect.
Pass the flags in the same way as ltq-atm and ltq-vdsl-vr9-mei already do.
Signed-off-by: John Crispin <john@phrozen.org>
/etc/uci-defaults/30_uboot-envtools runs in preinit, before the board
file is generated. Two of its helpers write to the console:
* uci batch prints the name of every anonymous section that "add"
creates, and nothing redirects stdout.
* grep -q hides a match, but it still reports a file that does not
exist. /etc/fw_env.config is absent until the first append creates it.
On the first boot after a flash this puts three lines on the console:
cfg016cb7
cfg026cb7
grep: /etc/fw_env.config: No such file or directory
Redirect the batch output and add -s to the grep.
Signed-off-by: John Crispin <john@phrozen.org>
The configure script of xdp-tools sets ARCH_INCLUDES only from the
multiarch directory of a native compiler. For a cross compiler it stays
empty, and clang takes <linux/types.h> for the BPF programs from
/usr/include of the host. On hosts that have <asm/types.h> only in the
multiarch directory, such as Debian without gcc-multilib, the build
fails:
/usr/include/linux/types.h:5:10: fatal error: 'asm/types.h' file not found
Add the include directories of the toolchain with -idirafter. They are
searched after the headers of xdp-tools and of the system, as the
multiarch directory of a native build, so they do not replace the Linux
headers that xdp-tools ships. The target compiler already has them as
system directories.
The configure script also tests for bpftool with the bash redirection &>.
/bin/sh may be dash, which runs the command in the background instead, so
configure drops a working bpftool and takes the output of whereis. Add a
second patch that uses >/dev/null 2>&1, which every shell supports.
Signed-off-by: John Crispin <john@phrozen.org>
find_md5 sorts the list of package files with the locale of the user.
The top-level Makefile sets LC_ALL=C, but a make call in a package
directory, such as make -C package/kernel/linux, uses the locale of
the shell. With en_GB.UTF-8, some file names sort in a different
order, so the hash and the name of the prepared stamp differ, and the
package is built again. The next top-level build then builds it again.
Always sort in the C locale.
Signed-off-by: John Crispin <john@phrozen.org>
YEAR_2038 writes its test program to $(TMP_DIR)/year2038.c and deletes
it after the test. Packages that are built in parallel run the test at
the same time and use the same file, so one of them can compile a
partly written file or find no file. The test then fails, and the host
build is configured with --disable-year2038.
Pass the test program to the compiler on stdin instead.
Signed-off-by: John Crispin <john@phrozen.org>
Commit 501123eb7a ("include/subdir: on build failure always print
error") passes $(2) instead of $(1) to ERROR for the builds of a build
type, such as host builds. subdir has no second parameter, so with
BUILD_LOG set a failed host build is written to logs/error.txt instead
of logs/package/error.txt, where the other errors of the subdir go.
Fixes: 501123eb7a ("include/subdir: on build failure always print error")
Signed-off-by: John Crispin <john@phrozen.org>
stampfile assigns the clean rule of its stamp to <subdir>//clean with
:=, so every call replaces the one before. toolchain and tools define
a compile and a check stamp, and <subdir>/clean only removes the check
stamp. After "make toolchain/clean", the compile stamp is newer than the
toolchain sources, the next build takes the toolchain as built, and the
target build fails because the compiler is missing.
Append the clean rule instead, so that all stamps are removed.
Signed-off-by: John Crispin <john@phrozen.org>
If several packages of a source use the same build variant, the
variant is added to the list of variants once for each selected
package, and the variant is built as often, one after the other. With
all packages selected, hostapd builds 4 variants twice, and f2fs-tools
its 2 variants four times each.
Add each variant once, with the condition of all packages that use it.
Signed-off-by: John Crispin <john@phrozen.org>
gen_package_auxiliary drops the condition of a dependency such as
+BUILD_NLS:libintl-full, so the ABI version check of a package also
counts packages that it does not depend on in this configuration.
With all packages selected and BUILD_NLS not set, libelf depends on
libintl-full. gettext-full is built after elfutils, and its new
libintl-full.version causes elfutils, and binutils, gdb, strace,
iproute2 and others through it, to be built again on the next make
call.
Write the condition as a make expression with get_conditional_dep, as
gen_package_mk does.
Signed-off-by: John Crispin <john@phrozen.org>
The prepare stamp of a Makefile depends on the library version files
of every package it defines, selected or not. Packages that are not
selected are not in the dependency graph of the top-level build, so
their libraries can be built after the package in a parallel build,
and the package is prepared and built again on the next run.
The own packages of a Makefile are only filtered out if they are
defined before the packages that depend on them. mac80211 defines
cfg80211 and mac80211 after the drivers, so it depends on its own
version files, and every clean build is followed by a rebuild of
mac80211, mt76 and linux-firmware.
Only add the library dependencies of selected packages, or of all
packages if DEVELOPER is set.
Signed-off-by: John Crispin <john@phrozen.org>
pkgaux writes Package/<name>/depends for every name that a package
provides. If a package provides the name of another real package, it
overwrites the dependencies of that package: dnsmasq-full provides
dnsmasq, so dnsmasq gets the dependencies of dnsmasq-full, among them
nftables-json and libnftnl.
These lists are used for the library dependencies of the prepare
stamp. dnsmasq thus depends on libnftnl without being ordered after it,
and is built again in the build that follows a clean build.
Skip provided names that belong to another real package.
Signed-off-by: John Crispin <john@phrozen.org>
The rdep check on PKG_BUILD_DIR excludes the ipkg-* staging dirs, but
not the apk-* ones. The package step writes the apk scripts after
.built is touched. If this crosses a second boundary, the next build
sees newer files and packs the package again, which starts the loop
again. The kernel package with its kmods takes several seconds to pack
and is repacked on every build.
Anchor the pattern to PKG_BUILD_DIR, as a plain "*/apk-*" also matches
the build dir of the apk package itself.
Signed-off-by: John Crispin <john@phrozen.org>
The fip contains BL31 from trusted-firmware-a, but the U-Boot build does
not depend on it. After a rebuild of trusted-firmware-a, the fip and the
images that contain it keep the old BL31.
Make the build stamp depend on the BL31 image.
Signed-off-by: John Crispin <john@phrozen.org>
The unlink of a local file with a hash mismatch comes after the die,
so it never runs. The message says that the file is deleted, but it
stays, and every following build fails in the same way.
Delete the file before the die, so that the next run downloads it
again.
Fixes: 4e19cbc553 ("download: handle possibly invalid local tarballs")
Signed-off-by: John Crispin <john@phrozen.org>
The RTL9607C SoC is another Otto family member with very identical indirect
table access as the rest of existing switch SoCs. It has two registers that
control tables: TBL_ACCESS_CTRL for LUT, VLAN, ACL and Classification tables
and NAT_TBL_ACCESS_CTRL for HWNAT related tables.
Add these two main control registers to OTTO_REG_LIST and all of their tables
to otto_table_maps and otto_table_id in header file.
Signed-off-by: Rustam Adilov <adilov@tutamail.com>
Link: https://github.com/openwrt/openwrt/pull/25136
Signed-off-by: Markus Stockhausen <markus.stockhausen@gmx.de>
Add the offset parameter to otto_table_map struct since some RTL9607C tables
like ACL_DATA and CF_RULE_T* require addition of an offset to their index to
differentiate from their MASK table counterpart.
Signed-off-by: Rustam Adilov <adilov@tutamail.com>
Link: https://github.com/openwrt/openwrt/pull/25136
Signed-off-by: Markus Stockhausen <markus.stockhausen@gmx.de>
The RTL9607C has access method bit field that is used as a look-up
method for read queries of L2 tables and hit bit for whether or not
the table query found an entry. It is exactly the same as how L2
tables are interfaced on RTL8365MB switch as per rtl8365mb_table in
its DSA driver.
For now only the otto_table sruct is changed for description purposes
without adding the functionality.
Signed-off-by: Rustam Adilov <adilov@tutamail.com>
Link: https://github.com/openwrt/openwrt/pull/25136
Signed-off-by: Markus Stockhausen <markus.stockhausen@gmx.de>
The RTL9607C NAT_TBL_ACCESS_CTRL register has 2 split execute bits for read and
write operations and thus does not have command bit. Add the read/write execute
bit to otto_table struct and change c_bit to signed where -1 is unused to make
NAT_TBL_ACCESS_CTRL work.
Signed-off-by: Rustam Adilov <adilov@tutamail.com>
Link: https://github.com/openwrt/openwrt/pull/25136
Signed-off-by: Markus Stockhausen <markus.stockhausen@gmx.de>
The address bit field of TBL_ACCESS_CTRL register on RTL9607C is not located
at 0 lsb. Introduce new addr lsb and addr msb properties to otto_table struct.
This also accidentally fixes the address mask of OTTO_REG_9300_L2 table as the
"BIT(t_bit) - 1" would result in an extra 2 bits.
Signed-off-by: Rustam Adilov <adilov@tutamail.com>
Link: https://github.com/openwrt/openwrt/pull/25136
Signed-off-by: Markus Stockhausen <markus.stockhausen@gmx.de>
The TBL_ACCESS_CTRL register of RTL9607C does not have an execute
bit and thus relies on a separate busy bit in its TBL_ACCESS_STS
register. Add a busy bit with -1 describing the lack of that bit
in the table registers.
Signed-off-by: Rustam Adilov <adilov@tutamail.com>
Link: https://github.com/openwrt/openwrt/pull/25136
Signed-off-by: Markus Stockhausen <markus.stockhausen@gmx.de>
The RTL931X utilizes the otto_table_read_bytes function in its DSA driver
as it needs field and mib offsets for stat reading.
This can be replaced by a proper otto_table_offset_read macro which takes
4th variable as a word offset for otto_table_read_bytes.
Signed-off-by: Rustam Adilov <adilov@tutamail.com>
Link: https://github.com/openwrt/openwrt/pull/25136
Signed-off-by: Markus Stockhausen <markus.stockhausen@gmx.de>
The FIB notifier lets the two multicast families through its filter, but
the branch that fills the work item runs only for AF_INET. An entry
event from ipmr or ip6mr therefore queues a work item whose fib_info
pointer is still the zero kzalloc() left there, and the add handler the
worker calls hands that NULL to fib_info_nh(), whose first act is to
read fi->nh. The box oopses there and panics.
The events are real ones: mr_call_mfc_notifiers() sends ENTRY_ADD,
ENTRY_REPLACE and ENTRY_DEL down the same chain this driver registers
on, once per multicast cache entry a routing daemon installs or removes.
What it takes is a kernel built with multicast routing, a daemon that
installs a cache entry, and a switch whose L3 setup is in place. The mfc
add and delete paths and mroute_clean_tables() emit these notifiers, in
ipmr and ip6mr alike, and mr_dump() replays every existing entry to a
notifier that has just registered - so the driver can also meet one at
its own probe, if a daemon installed it first. All of them start from
userspace. CONFIG_IP_MROUTE and CONFIG_IPV6_MROUTE are off in the generic
config and no realtek subtarget turns them on, so a stock image never
gets there either.
Given both, no command and no route added by hand are needed: an
RTL9303 with the offload compiled in and smcroute installed dies 58
seconds into its first boot, as the daemon opens its interfaces during
the ordinary network bring-up.
Measured the other way round as well: an image built the same way and
differing by this commit and one unrelated L3 fix runs on, 273 seconds
against a 58 second failure window, with nothing in the log. The RTL839x
carries its L3 setup outside any config symbol, so a build of that
family with multicast routing on is enough as well; the RTL930x
additionally needs the offload symbol, which no subtarget config sets.
Take only the two families the driver decodes.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Gennaro Cimmino <gcimmino@rayonra.net>
Link: https://github.com/openwrt/openwrt/pull/25309
Signed-off-by: Markus Stockhausen <markus.stockhausen@gmx.de>
John Crispin
Rustam Adilov
Gennaro Cimmino
DSP ALU32
Robert Marko