Repository navigation
Cross compiling on Linux is untested, undocumented, and broken #90039
Description
| BPO | 45881 |
|---|---|
| Nosy | @brettcannon, @tiran, @moreati |
| PRs | |
| Dependencies |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2021-11-23.16:37:45.271>
labels = ['3.9', '3.10', '3.11', 'build', 'tests', 'docs']
title = 'Cross compiling on Linux is untested, undocumented, and broken'
updated_at = <Date 2021-11-29.16:23:38.440>
user = 'https://github.com/tiran'bugs.python.org fields:
activity = <Date 2021-11-29.16:23:38.440>
actor = 'christian.heimes'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Build', 'Documentation', 'Tests', 'Cross-Build']
creation = <Date 2021-11-23.16:37:45.271>
creator = 'christian.heimes'
dependencies = ['45886', '45909']
files = []
hgrepos = []
issue_num = 45881
keywords = ['patch']
message_count = 7.0
messages = ['406853', '406860', '406935', '406942', '407016', '407165', '407292']
nosy_count = 4.0
nosy_names = ['brett.cannon', 'christian.heimes', 'docs@python', 'Alex.Willmer']
pr_nums = ['29752', '29753', '29754', '29835']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue45881'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
Activity
tiran commentedon Nov 23, 2021
Cross compiling is an approach to compile a program for a different CPU architecture and platform, e.g. compile for an ARM64 (aarch64) or WASM on a x86_64 build system.
Python configure script, Makefile, and setup.py have multiple references to cross compiling. However I could not find any documentation in the devguide or Python docs how to cross compile. We also lack CI (buildbot) to test cross compiling. This lack of awareness and testing leads to breakage of the feature. For example the design of Programs/_freeze_module in main (3.11-dev) is incompatible with cross compiling.
I kinda got cross compiling working with 3.10, but only with some additional hacks and patches. I also ran into other problems like _PYTHON_HOST_PLATFORM env var is not automatically forwarded to setup.py. The helper functions add_multiarch_paths() and add_cross_compiling_paths() break builds for me, too. Cross compiling only works when the methods are commented out.
tiran commentedon Nov 23, 2021
I have uploaded a reproducer at https://github.com/tiran/cpython_builddep/compare/cross-aarch64
$ podman run -ti --rm -v $(pwd):/cpython:Z quay.io/tiran/cpythonbuild:ubuntu-impish-aarch64...
aarch64-linux-gnu-gcc -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I/cpython/Include/internal -I/usr/include/x86_64-linux-gnu -I. -IObjects -IPython -I/usr/include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/cpython/Include -I/cpython/builddep/ubuntu-impish-x86_64 -c /cpython/Modules/_ctypes/_ctypes.c -o build/temp.linux-aarch64-3.10/cpython/Modules/_ctypes/_ctypes.o -DPy_BUILD_CORE_MODULE -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1
In file included from /usr/aarch64-linux-gnu/include/features.h:508,
from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
from /usr/aarch64-linux-gnu/include/limits.h:26,
from /usr/lib/gcc-cross/aarch64-linux-gnu/11/include/limits.h:203,
from /usr/lib/gcc-cross/aarch64-linux-gnu/11/include/syslimits.h:7,
from /usr/lib/gcc-cross/aarch64-linux-gnu/11/include/limits.h:34,
from /cpython/Include/Python.h:11,
from /cpython/Modules/zlibmodule.c:8:
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: gnu/stubs-32.h: No such file or directory
7 | # include <gnu/stubs-32.h>
tiran commentedon Nov 24, 2021
bpo-45886 addresses the cross build issue with freeze_module command.
The wrong header files come from the fact that setup.py uses CC variable from sysconfig instead of environment. The sysconfig variable contains the C compiler of the build interpreter instead of the host interpreter. The correct value is in os.environ.
tiran commentedon Nov 24, 2021
New changeset b30bf45 by Christian Heimes in branch 'main':
bpo-45881: Use CC from env first for cross building (GH-29752)
b30bf45
tiran commentedon Nov 25, 2021
New changeset cd6d257 by Christian Heimes in branch '3.9':
[3.9] bpo-45881: Use CC from env first for cross building (GH-29752) (GH-29754)
cd6d257
tiran commentedon Nov 27, 2021
New changeset 545aebd by Christian Heimes in branch '3.10':
[3.10] bpo-45881: Use CC from env first for cross building (GH-29752). (GH-29753)
545aebd
tiran commentedon Nov 29, 2021
New changeset 992565f by Christian Heimes in branch 'main':
bpo-45881: configure --with-freeze-module --with-build-python (GH-29835)
992565f
erlend-aasland commentedon Jan 4, 2024
Some improvements were applied:
I've created follow-up issues on the devguide and buildbot repos. Suggesting to close this, as there is no more actionable items at the moment.