iremap is called with BUILD_DIR and with PKG_BUILD_DIR, never with
STAGING_DIR. A header taken from the staging directory therefore reaches
__FILE__ and the debug information with its absolute path, and the result
depends on the directory the tree was built in.
Measured in the published snapshot of lantiq/xway: mt76-connac-lib.ko,
mt76x02-lib.ko and mt7925-common.ko hold
/builder/shared-workdir/build/staging_dir/target-mips_24kc_musl/usr/include/mac80211/net/cfg80211.h
That path is __FILE__ in .rodata, so the strip before the package is built
cannot reach it and the module ships it.
Map STAGING_DIR as well. Keep the staging_dir element in the replacement:
the last element of STAGING_DIR and of BUILD_DIR carry the same name, so
the bare name would make the two roots indistinguishable. The map of
BUILD_DIR does not change, so only a path that leaks today changes.
Verified with the target compiler on a header below a mapped directory:
-ffile-prefix-map removes the absolute path from both __FILE__ and the
debug information, -fmacro-prefix-map removes it from __FILE__ alone. The
shipped half is __FILE__, so both variants cover it.
A local build of mt76 also holds nine staging paths, but those are debug
information: the build directory copy of mt76-connac-lib.ko is 653456
bytes with 13 debug sections and nine such paths, the shipped copy is
84688 bytes with none. They are named here so that the next reader does
not take them for the same defect.
Signed-off-by: John Crispin <john@phrozen.org>
A configure run spends its time on the compiler invocations behind its
checks, not on the shell. tools/m4 runs the compiler 633 times and takes
24 seconds. ccache does not help, because it refuses to cache an
autoconf conftest compile at all; it counts these separately as
"Autoconf compile/link" under its uncacheable calls.
A package that sets PKG_CONFIGURE_CACHE or HOST_CONFIGURE_CACHE keeps a
config.cache under tmp/, so a package that is cleaned and built again
reads the answers instead of running the checks. The configure stage of
tools/m4 drops from 24.5 s to 1.5 s. dirclean removes tmp/, so a full
reset of the tree starts from cold answers.
A package has to opt in, because a cache is only correct for a configure
script that keeps its side effects outside the AC_CACHE_VAL body.
ncurses appends -D_XOPEN_SOURCE and -D_DEFAULT_SOURCE to CPPFLAGS inside
that body, so a cached run restores the value, skips the append, and
then fails to compile wcwidth. ncurses_cfg.h is identical in both cases;
the difference is in the generated Makefile.
The path covers everything else that changes an answer. A target package
is keyed on the toolchain directory, which carries the gcc version. A
host package is keyed on the version string of the host compiler,
because autoconf cannot detect a new compiler behind an unchanged name;
gcc 14 turned an implicit function declaration into an error and changed
many answers that way. Both are keyed on a hash of the configure
arguments and the build flags, and autoconf refuses a cache written with
different flags on top of that.
A cached program path outlives a wipe of staging_dir, so ac_cv_path_ and
ac_cv_prog_ answers are dropped when the cache loads and configure
searches PATH again. Without that, a cache written before staging_dir
held sed pins /usr/bin/sed for good. A search runs no compiler, so this
costs nothing that matters.
The cache arrives through a site script rather than --cache-file, so a
hand written configure script never sees a new argument. The site script
removes the file of the previous arguments, so a different configuration
gets a cold cache instead of an autoconf error. It also unsets the
variable, and make exports it for the configure stage alone, so a nested
configure run keeps its own cache.
Set CONFIGURE_CACHE_DIR to keep the cache elsewhere, or to the empty
string to run every check again.
Host builds now read a site script on Linux as well. Before this change
only Darwin had one.
CONFIG_SITE holds one path name: autoconf before 2.70 reads the whole
value as that one name, so a second path would silence both scripts. It
names the cache script alone, and the cache script loads the site script
of the architecture from CONFIG_SITE_BASE.
Signed-off-by: John Crispin <john@phrozen.org>
Set BUILD_TIME_LOG=<file> to record a begin and an end event for every
prepare, configure, compile and install stage of every tool, toolchain
component, kernel and package. scripts/build-time-report.pl turns the log
into a report.
The report divides each second of the build between the stages that run
in it, so the numbers show which stages hold up the wall clock rather
than which ones burn the most CPU. Nothing is recorded and no extra
process runs when BUILD_TIME_LOG is unset.
Signed-off-by: John Crispin <john@phrozen.org>
BuildPackage expands the full BuildTarget/ipkg for every package in a
Makefile, whether it is selected or not. package/kernel/linux defines
more than 1000 kmods, and this takes about 0.9 s of its 1.2 s parse,
which is on the critical path of every build.
Move the clean rule into BuildTarget/ipkg/disabled, and only expand it
for packages that are not selected, unless DEVELOPER is set. The other
rules of such a package can not be reached, as its compile target only
depends on them if the package is selected. Target types without a
disabled variant are expanded as before.
Signed-off-by: John Crispin <john@phrozen.org>
Make expands PKG_SOURCE_DATE_EPOCH for every recipe of a package, as
package-pack.mk exports it to the recipes. Every expansion runs
get_source_date_epoch.sh, which runs git.
Keep the result for each directory. The directory still depends on
whether the build directory has a version.date file.
Signed-off-by: John Crispin <john@phrozen.org>
Build/CoreTargets fixes the value of STAMP_PREPARED, but its expansion
references STAMP_PREPARED several times before that happens, and every
reference hashes all files of the package again with find and mkhash.
HOST_STAMP_PREPARED carries the same hash and the same cost.
Compute the hash on first use in PKG_FILES_MD5 and keep it. Both stamps
read it.
Signed-off-by: John Crispin <john@phrozen.org>
All variants of a package collect their InstallDev files in the same
directory below $(TMP_DIR) and remove it before and after use. Add the
build variant to its name, as for the staging file list, so that the
variants of a package do not share it.
Signed-off-by: John Crispin <john@phrozen.org>
The prepare stamp of a Makefile depends on the library version files
of every package it defines, selected or not. Packages that are not
selected are not in the dependency graph of the top-level build, so
their libraries can be built after the package in a parallel build,
and the package is prepared and built again on the next run.
The own packages of a Makefile are only filtered out if they are
defined before the packages that depend on them. mac80211 defines
cfg80211 and mac80211 after the drivers, so it depends on its own
version files, and every clean build is followed by a rebuild of
mac80211, mt76 and linux-firmware.
Only add the library dependencies of selected packages, or of all
packages if DEVELOPER is set.
Signed-off-by: John Crispin <john@phrozen.org>
The rdep check on PKG_BUILD_DIR excludes the ipkg-* staging dirs, but
not the apk-* ones. The package step writes the apk scripts after
.built is touched. If this crosses a second boundary, the next build
sees newer files and packs the package again, which starts the loop
again. The kernel package with its kmods takes several seconds to pack
and is repacked on every build.
Anchor the pattern to PKG_BUILD_DIR, as a plain "*/apk-*" also matches
the build dir of the apk package itself.
Signed-off-by: John Crispin <john@phrozen.org>
Packages that use 64-bit atomics carry open-coded architecture lists
such as +(arm||armeb||mips||mipsel||powerpc):libatomic. Those lists
drift (ruby still names arc), over-include ARMv6K and ARMv7 targets
whose compiler inlines 64-bit atomics via ldrexd/strexd, and
under-include ARMv5 in other packages. Add HAS_ATOMIC64, selected on
the architectures where GCC emits 64-bit atomics inline (all 64-bit
ones, x86 via cmpxchg8b, ARMv6K and ARMv7), and ATOMIC64_DEPENDS as
the single dependency expression to use instead. The condition is one
negated symbol because the Depends: evaluator in package-pack.mk only
handles flat conjunctions and disjunctions. 16-byte and size-generic
atomics need libatomic on every architecture and keep a plain
+libatomic dependency.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Extend the git-src source tree override to host builds. It is opt-in via
HOST_USE_GIT_SRC, since by default the host build keeps using the tarball.
Move the git-src detection and checkout recipe into unpack.mk so both the
target and host build reuse the same code.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Previously PKG_SOURCE_DATE_EPOCH came from a git log on the OpenWrt
package directory. That fails in shallow feed clones and in the SDK
(no .git), collapsing to the script's mtime and breaking .apk
reproducibility across SDK rebuilds and between buildbot and SDK.
With this comment, PKG_UNPACK generates a version.date file, later used to
determine a reproducible SOURCE_DATE_EPOCH. Since unpack happens after
download, the evaluation of SOURCE_DATE_EPOCH is now lazy, invoking the
`get_source_date_epoch.sh` script on every use.
While at it, drop export of PKG_SOURCE_DATE_EPOCH and clean it from the
ipkg-build script.
Link: https://github.com/openwrt/openwrt/issues/21579
Link: https://github.com/openwrt/openwrt/pull/21587
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Link: https://github.com/openwrt/openwrt/pull/23576
Signed-off-by: Paul Spooren <mail@aparcar.org>
ELF binaries embed the maximum page size used during linking in the
PT_LOAD segment alignment. If this value is larger than the actual
system page size, the kernel will still load the binary but wastes
memory due to over-alignment. If it is smaller, the binary may fail
to load or cause incorrect behavior with features like RELRO.
Most architectures use 4KB pages, but loongarch64 uses 16KB pages
(CONFIG_PAGE_SIZE_16KB). Explicitly pass -z max-page-size to the
linker so that binaries are built with the correct page alignment for
the target, avoiding issues with RELRO segment padding and ensuring
correct memory mapping on the target system.
Before binutils 2.39 RELRO sections were padded to common page size
which is 4K on MIPS, with binutils 3.29 the padding was changed to
max page size which is 64K on MIPS. With this change we will pad them to
4K again.
This reduces the size of user space binaries in OpenWrt.
The size of the uncompressed root file system of a mips malta be build
decreased by about 20%.
old file size:
```
$ du -s build_dir/target-mips_24kc_musl/root-malta/
15844 build_dir/target-mips_24kc_musl/root-malta/
```
New file size:
```
$ du -s build_dir/target-mips_24kc_musl/root-malta/
12564 build_dir/target-mips_24kc_musl/root-malta/
```
The size of the image did not decrease much because of the strong
compression used by OpenWrt.
Link: https://github.com/openwrt/openwrt/pull/22800
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
DESCRIPTION field has been deprecated since 2007, yet internal exporting
causes intermittent package build failures due to the deprecation check,
even with packages that don't have that fields set. Replace all
DESCRIPTION exports with alternative variables defined through shexport
and shvar helpers and remove the check.
Fixes: 547b127d ("make kernel module packaging code reusable and use it in madwifi")
Fixes: 7558f028 ("DESCRIPTION:= is obselete, so complain if it is used and use TITLE if no description is set")
Fixes: cc435322 ("build: optimize target metadata dump")
Fixes: d081edf7 ("build: clean up and optimize ipkg control generator code")
Signed-off-by: George Sapkin <george@sapk.in>
Link: https://github.com/openwrt/openwrt/pull/21668
Signed-off-by: Robert Marko <robimarko@gmail.com>
When a package variant changes from =y to =m, its .install stamp file
was not being cleaned up, causing both variants to appear in the
package install list.
Fix by making compile depend on prepare-package-install, ensuring
the stamp file is always cleared. Only =y packages recreate it.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Fix setting implicit self-provides for packages when they don't have any
PROVIDES specified.
Remove redundant self-provide for kmods, since kmods are packages and
will have a self-provide added already.
Fixes: 5ed650a ("build: add support for virtual provides")
Fixes: 9b37b71 ("build: provide virtual self in kmods")
Signed-off-by: George Sapkin <george@sapk.in>
Link: https://github.com/openwrt/openwrt/pull/21369
Signed-off-by: Robert Marko <robimarko@gmail.com>
Allow defining virtual provides using the PROVIDES field by prefixing
them with @, e.g.:
PROVIDES:=@ca-certs
Virtual provides don't own the provided name and multiple packages with
the same virtual provides can be installed side-by-side. Packages must
still take care not to override each other's files.
Add an implicit self-provide to packages. apk can't handle self
provides, be it versioned or virtual, so opt for a suffix instead. This
allows several variants to provide the same virtual package without
adding extra provides to the default one, e.g. wget implicitly provides
wget-any and is marked as default, so wget-ssl can explicitly provide
@wget-any as well.
Filter out virtual provides when generating metadata.
Filter out virtual provides prefix and self provide where appropriate.
Signed-off-by: George Sapkin <george@sapk.in>
Link: https://github.com/openwrt/openwrt/pull/21288
Signed-off-by: Robert Marko <robimarko@gmail.com>
Rework the package SOURCE entry handling to account for the --root feeds
script feature.
Move the SOURCE entry string manipulation logic outside package-defaults.mk
in package.mk and limit only to non DUMP scenario to not pollute the .mk
too much.
Restructure the previous logic and add a new additional condition.
If we detect the package comes from a feed, replace any feed path that
have the _root prefix to the feed name with the non-root variant (the
feeds script create a symbolic link to it) and point the package SOURCE
entry to what the symbolic link points to.
Example:
Feed link: feeds/base_root/package -> feeds/base
Package: feeds/base_root/package/system/uci -> feeds/base/system/uci
Link: https://github.com/openwrt/openwrt/pull/20459
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Add support for overriding the SUBDIRS variable while invoking Make
by defining it after Make in the command line.
This is useful for builds that have previously patched out
the building of certain subdirectories in projects
that use recursive and independent Makefiles,
for example, to block the building of docs or test suites.
The wildcard function is used in the case of there being
subdirectories within any of the subdirectories,
for example, in the building of gengetopt,
in order to avoid the problem where Make will attempt
to execute a Makefile in a subdirectory that does not exist
within the subdirectory it is currently running from
because it really exists at the top-level, or one that exists
within one of the subdirectories when ran from top-level.
There are also cases where the Makefiles in the subdirectories
have the recursive building rules even though there are no more
subdirectories beyond that point, for example, with gnulib.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Link: https://github.com/openwrt/openwrt/pull/15853
Signed-off-by: Robert Marko <robimarko@gmail.com>
A new option called `USE_APK` is added which generated APK packages
(.apk) instead of OPKG packages (.ipk).
Some features like fstools `snapshot` command are not yet ported
Signed-off-by: Paul Spooren <mail@aparcar.org>
If CONFIG_USE_MOLD is set, all target packages will use the mold linker.
Except the ones which opted-out via setting PKG_BUILD_FLAGS:=no-mold.
Signed-off-by: Andre Heider <a.heider@gmail.com>
A package may run git as part of its build process, and if the package
source code is not from a git checkout, then git may traverse up the
directory tree to find buildroot's repository directory (.git).
For instance, Poetry Core, a Python build backend, will read the
contents of .gitignore for paths to exclude when creating a Python
package. If it finds buildroot's .gitignore file, then Poetry Core will
exclude all of the package's files[1].
This exports GIT_CEILING_DIRECTORIES for both package and host builds so
that git will not traverse beyond $(BUILD_DIR)/$(BUILD_DIR_HOST).
[1]: https://github.com/python-poetry/poetry/issues/5547
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This reduces open coding and allows to easily add a knob to enable
it treewide, where chosen packages can still opt-out via "no-lto".
Some packages used LTO, but not the linker plugin. This unifies 'em
all to attempt to produce better code.
Quoting man gcc(1):
"This improves the quality of optimization by exposing more code to the
link-time optimizer."
Also use -flto=auto instead of -flto=jobserver, as it's not guaranteed
that every buildsystem uses +$(MAKE) correctly.
Signed-off-by: Andre Heider <a.heider@gmail.com>
This reduces open coding and allows to easily add a knob to
enable it treewide, where chosen packages can still opt-out via
"no-gc-sections".
Note: libnl, mbedtls and opkg only used the CFLAGS part without the
LDFLAGS counterpart. That doesn't help at all if the goal is to produce
smaller binaries. I consider that an accident, and this fixes it.
Note: there are also packages using only the LDFLAGS part. I didn't
touch those, as gc might have been disabled via CFLAGS intentionally.
Signed-off-by: Andre Heider <a.heider@gmail.com>
PKG_BUILD_FLAGS is a new variable for package Makefiles similar to
PKG_FLAGS. It's a whitespace separated list of flags to control various
aspects of how a package is build.
The build system and/or .config defines the default for each, but
every package has the means to override it. Using $flagname enables
a flag, no-$flagname disables it.
Start with PKG_IREMAP as "iremap". That's easy as no package here
nor in any package feed uses it. The default is unchanged: enabled.
Packages can opt-out via:
PKG_BUILD_FLAGS:=no-iremap
(Not that any should, just to illustrate how to use it)
Signed-off-by: Andre Heider <a.heider@gmail.com>
While experimenting with the AUTOREMOVE option in search of a way to use
prebuilt host tools in different buildroot, it was discovered that the
md5 generated by find_md5 in depends.mk is not reproducible.
Currently the hash is generated by the path of the file in addition to
the file mod time. Out of confusion, probably, there was an idea that
such command was used on the package build_dir. Reality is that this
command is run on the package files. (Makefile, patches, src)
This is problematic because the package Makefile (for example) change at
each git clone and base the hash on the Makefile mtime doesn't really
reflect if the Makefile actually changes across a buildroot or not.
A better approach is to generate an hash of each file and then generate
an hash on the sort hash list. This way we remove the problem of git
clone setting a wrong mtime while keeping the integrity of checking if a
file changed for the package as any change will result in a different
hash.
Introduce a new kind of find_md5 function, find_md5_reproducible that
apply this new logic and limit it only with AUTOREMOVE option set to
prevent any kind of slowdown due to additional hash generation.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Package with whitespace in their build directory are not correctly
removed when CONFIG_AUTOREMOVE is enabled. This is caused by xargs that
use whitespace as delimiters. To handle this use \0 as the delimiter and
set find to use \0 as the delimiter.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
4e19cbc553: [download: handle possibly invalid local tarballs] added a
FORCE rule to downloaded files, so that they will be always checked by
download.pl.
As a side-effect, check-compile will fail, forcing unnecessary package
rebuilds.
The check-compile.txt log shows (for libxml2 for example):
Considering target file '.../dl/libxml2-2.9.12.tar.gz'.
...
prerequisite 'FORCE' of target '.../dl/libxml2-2.9.12.tar.gz' does
not exist.
Must remake target '.../dl/libxml2-2.9.12.tar.gz'.
...
Giving up on target file '...libxml2-2.9.12/.prepared_...'.
Giving up on target file '...libxml2-2.9.12/.configured_...'.
Giving up on target file '...libxml2-2.9.12/.built'.
Giving up on target file '...stamp/.libxml2_installed'.
Giving up on target file '.compile'.
Then the package is rebuilt even if it is not otherwise needed.
To fix this, instead of always forcing the download target to be remade,
check its hash first: if it matches, then the FORCE is not added.
Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
This was accidentally dropped in 27a4a71c24
("metadata: handle ABI version rebuild tracking for transient dependencies")
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This reverts commit b12288fa69.
The patchelf approach is too fragile, and the only users of this have been
converted to make patching unnecessary
Leave the abi_version_str variable in place in rules.mk
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This makes it possible to declare a package ABI_VERSION independent from the
upstream soname by setting PKG_ABI_VERSION in the package makefile.
The library filename is fixed up for files installed to packages and to the
staging dir. References to the original from executables within the same
package are also fixed up
Signed-off-by: Felix Fietkau <nbd@nbd.name>
A stray comma was being appended to the last package version dependency,
causing it to be missed for ABI version checks
Signed-off-by: Felix Fietkau <nbd@nbd.name>
The license folder is a core part of OpenWrt and all GPL-2.0 licensed.
Use SPDX license tags to allow machines to check licenses.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebase, keep some Copyright lines, sharpen commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
With commit 2ca084cc ("build: improve ccache support") these variables
are being set globally and we don't need them for specific targets.
Signed-off-by: Sven Wegener <sven.wegener@stealer.net>
Currently it's assumed, that already downloaded tarballs are always
fine, so no checksum checking is performed and the tarball is used even
if it might be corrupted.
From now on, we're going to always check the downloaded tarballs before
considering them valid.
Steps to reproduce:
1. Remove cached tarball
rm dl/libubox-2020-08-06-9e52171d.tar.xz
2. Download valid tarball again
make package/libubox/download
3. Invalidate the tarball
sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile
4. Now compile with corrupt tarball source
make package/libubox/{clean,compile}
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Adding inline shell invocations in per-target variables causes them to be
executed over and over again, which causes a significant slowdown.
Fix this by evaluating it only once per package directory
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Set CCACHE_DIR to $(TOPDIR)/.ccache and CCACHE_BASEDIR to $(TOPDIR).
This allows to do clean and dirclean. Cache hit rate for test build
after dirclean is ~65%.
If CCACHE is enabled stats are printed out at the end of building process.
CCACHE_DIR config variable allows to override default, which could be useful
when sharing cache with many builds.
cacheclean make target allows to clean the cache.
Changes from v1:
- remove ccache directory using CCACHE_DIR variable
- remove ccache leftovers from sdk and toolchain make files
- introduce CONFIG_CCACHE_DIR variable
- introduce cacheclean make target
Signed-off-by: Roman Yeryomin <roman@advem.lv>
Build/InstallDev no longer places a file list in
$(STAGING_DIR_HOST)/packages; this change removes the creation of
$(STAGING_DIR_HOST)/packages and the attempted removal of a
STAGING_DIR_HOST file list during package clean.
This also changes the host directory passed to Build/UninstallDev from
$(STAGING_DIR_HOST) to $(STAGING_DIR)/host, to match the directory
passed to Build/InstallDev.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Revert "mac80211: add new minstrel_ht patches to improve probing on mt76x2" (9861050b85)
Revert "kernel: use bulk free in kfree_skb_list to improve performance" (98b654de2e)
Revert "ramips: add preliminary support for WIO ONE" (085141dc5b)
Revert "ramips: add preliminary support for SGE AP-MTKH7-0006 developer board" (b1db6d0539)
Revert "build: use config.site generated by autoconf-lean, drop hardcoded sitefiles" (363ce4329d)
Revert "toolchain: add autoconf-lean" (fdb30eed03)
Revert "build: allow overriding the filename on the remote server when downloading" (6fa0e07758)
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This changes the default PKG_BUILD_DIR to take BUILD_VARIANT into
account (if set), so that packages do not need to manually override
PKG_BUILD_DIR just to handle variants.
This also updates most base packages with variants to use the updated
default PKG_BUILD_DIR.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This feature is similar to the SRC_TREE_OVERRIDE. However instead of having
to manually create a symlink inside the package folder, the buildsystem
will check if $(TOP_DIR)/src-checkout/$(PKG_NAME)/.git is present and use
this instead.
Signed-off-by: John Crispin <john@phrozen.org>
John Crispin
Daniel Golle
Felix Fietkau
Paul SpoorenandClaude Opus 4.7
Hauke Mehrtens
George Sapkin
Christian Marangi
Michael Pratt
Andre Heider
Jeffery To
Eneas U de Queiroz
Sven Wegener
Petr Štetiar
Roman Yeryomin