Repository navigation
ctypes CDLL search path issue on MacOS #88130
Description
| BPO | 43964 |
|---|---|
| Nosy | @ronaldoussoren, @ned-deily, @vlazzarini |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2021-04-28.08:39:49.501>
labels = ['OS-mac', '3.9', '3.10', '3.11']
title = 'ctypes CDLL search path issue on MacOS'
updated_at = <Date 2021-06-26.19:26:05.420>
user = 'https://github.com/vlazzarini'bugs.python.org fields:
activity = <Date 2021-06-26.19:26:05.420>
actor = 'ccabrera'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['macOS']
creation = <Date 2021-04-28.08:39:49.501>
creator = 'Victor.Lazzarini'
dependencies = []
files = []
hgrepos = []
issue_num = 43964
keywords = []
message_count = 5.0
messages = ['392173', '394592', '394782', '394788', '396555']
nosy_count = 4.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'Victor.Lazzarini', 'ccabrera']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue43964'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
Activity
vlazzarini commentedon Apr 28, 2021
With Python 3.9.4 ctypes.CDLL does not appear to find framework libraries installed in /Library/Frameworks.
--- With Python 3.6.5:
--- With Python 3.9.4
This happens with all frameworks I have installed in /Library/Frameworks.
The full path seems to work OK:
>>> ctypes.CDLL('/Library/Frameworks/CUDA.framework/CUDA') <CDLL '/Library/Frameworks/CUDA.framework/CUDA', handle 7fa358404180 at 0x7fa3580eeaf0> >>>but that's suboptimal as in MacOS you might have frameworks installed elsewhere.
vlazzarini commentedon May 27, 2021
I have looked at this closely and it appears Python 3.6.5 has a search path for libraries and frameworks that is missing in the newer versions (3.7 onwards).
I can load libraries from /usr/local/lib and ~/lib without any difficulties in 3.6.5, just by passing the file name. In later versions this does not work, you need to pass a full path. Furthermore if these libraries have dependencies which are set with an rpath, the loading fails because the dependencies are not found. None of this is a problem in 3.6.5. This needs to be resolved because packages that are based on ctypes may be broken in versions >= 3.7
This is the case of Csound and ctcsound, for instance.
ned-deily commentedon May 30, 2021
Thanks for the report! I've spent some time investigating it and the story behind it turns out to be a bit complicated, so bear with me. It's all tied in to Apple's attempts to improve the security of macOS.
As of macOS 10.15 Catalina, Apple introduced new requirements for downloadable installer packages, like those we provide for macOS on python.org; in order for such packages to be installed with the macOS installer, they would now have to be "notarized" by Apple. The notarization process is somewhat similar in concept to the process that an app has to go through to be submitted to the Mac App Store but with less stringent requirements. In particular, the installer package is automatically inspected to ensure that all executables are codesigned, are linked with the more-secure "hardened runtime", and do not request certain less-secure entitlements. Although originally announced for enforcement starting with the release of Catalina in the fall of 2019, Apple delayed the enforcement until February 2020 to give application developers more time to modify their packages to meet the new requirements. (See, for example, https://developer.apple.com/news/?id=12232019a).
The first python.org macOS installers that conformed to the new requirements and were notarized were for the 3.8.2 and 3.7.7 releases, staring in 2020-02. In those first releases, we used two entitlements:
com.apple.security.cs.disable-library-validation
com.apple.security.cs.disable-executable-page-protection
Some issues were reported (like bpo-40198) when using those first releases, so we added two additional entitlements for subsequent releases:
com.apple.security.automation.apple-events
com.apple.security.cs.allow-dyld-environment-variables
While we didn't realize it until your issue, that did have an effect on ctype's behavior when trying to find shared libraries and frameworks. Using the hardened runtime, as now required for notarization, causes the system dlopen() interface, which ctypes uses, to no longer search relative paths. The dlopen man page (for macOS 11, at least) includes this warning:
Note: If the main executable is a set[ug]id binary or codesigned with
entitlements, then all environment variables are ignored, and only a
full path can be used.
After some experimentation, it looks like that statement isn't exactly correct at least on macOS 11 Big Sur: unprefixed paths can still be used to find libraries and frameworks in system locations, i.e. /usr/lib and /System/Library/Frameworks. But it is true that, when using the hardened runtime, dlopen() no longer searches the user-controlled paths /usr/local/lib or /Library/Frameworks by default. And there apparently is no way for a notarized executable to revert to the previous behavior by adding other entitlements (https://developer.apple.com/forums/thread/666066).
With the allow-dyld-environment-variables entitlement included after the initial releases, it *is* now possible to change this behavior externally, if necessary, by explicitly setting the DYLD_LIBRARY_PATH and/or DYLD_FRAMEWORK_PATH environment variables (see man 1 dyld).
To demonstrate using a third-party library in /usr/local/lib (similar results with third-party frameworks in /Library/Frameworks):
# ------ python.org 3.7.6, not codesigned ------
# ------ python.org 3.7.7, first notarized release ------
# ------ python.org 3.7.8 and beyond including 3.9.x ------
It is rather unfortunate that, when including the allow-dyld-environment-variables entitlement, you now receive a less helpful error message "image not found" rather than "file system relative paths not allowed in hardened programs".
So, what to do?
That would mean that python.org installer packages could not be notarized and thus would no longer be installable without going through various onerous steps to override Gatekeeper protections (and would presumably also reduce somewhat system security). Since the primary target of python.org installers is for inexpert users, this option seems unacceptable.
So something like: if the path passed to ctypes is not absolute and the initial call to dlopen() fails with an "image not found" error, try again by making an absolute path by prepending '/usr/local/lib' and/or '/Library/Frameworks' and retrying. I assume we could make that work somehow but the question is should we? That seems very kludgey and would be re-opening a potential security hole that Apple clearly thinks is significant and is trying to discourage.
In particular, document this change in behavior for codesigned installations using the hardened runtime (like python.org installers) and suggest avoid using relative paths on macOS or, if necessary, have the DYLD_LIBRARY_PATH or DYLD_FRAMEWORK_PATH environment variables set appropriately when launching the interpreter. (I did a quick experiment and it seemed that it was not possible to change dlopen()'s behavior by setting those variables from inside the running interpreter process by using OS.ENVIRON, an unsurprising result.)
It seems to me that changing the ctypes documentation (and adding a changelog blurb), option 3, is the least bad of the available options. One would hope that this does not affect *too* many third-party packages and user applications. Obviously it does some :(
@ronald, what do you think?
7 remaining items
mesubash commentedon Nov 7, 2026
This still reproduces on macOS 27.2. I built a small test library that links only to libSystem, put it in
~/lib, and loaded it from three Pythons:CDLL("libgh88130.dylib")DYLD_LIBRARY_PATH=~/libfind_library("gh88130")OSErrorOSError/usr/bin/python33.9.6OSErrorOSErrorWith
DYLD_PRINT_SEARCHING=1, the python.org build tries only the bare name, the cryptex prefix and/usr/lib(listed as "default fallback"). It never tries~/libor/usr/local/lib. Homebrew's Python is only ad-hoc signed, without the hardened runtime, and it doesn't try~/libeither (it adds its own rpath,/opt/homebrew/lib). So this doesn't look specific to the hardened runtime. That matches what #160985 found.DYLD_LIBRARY_PATHworks for the python.org build, which is signed withcom.apple.security.cs.allow-dyld-environment-variables. It has no effect on Apple's/usr/bin/python3.Frameworks behave the same way. A test framework in
~/Library/Frameworksfails to load asCDLL("GH88130.framework/GH88130")in all three Pythons, thoughfind_libraryfinds it. With the framework in another folder,DYLD_FRAMEWORK_PATHmakes the bare name work in the python.org and Homebrew builds, but not in Apple's.I'd like to write the docs note. My plan is a short note under
CDLLinDoc/library/ctypes.rst: on macOS, a bare library name is only searched for in system locations, so pass a full path. It would also mentionDYLD_LIBRARY_PATHandDYLD_FRAMEWORK_PATH, with the caveat that some binaries ignore them.One question:
find_library()found the library in all three cases, but the docs now call it soft deprecated. Should the note mention it as an option, or leave it out?ned-deily commentedon Nov 9, 2026
@encukou Any thoughts on this issue and the related PR (#161624) ?
encukou commentedon Nov 9, 2026
cc @ZeroIntensity
Documenting what happens now is useful, but I'd frame it differently.
I think the docs should more clearly imply that ctypes calls
dlopen, and inherits its quirks -- current and future ones, i.e. if Apple changes the details,CDLLwill follow, without deprecation. IgnoringDYLD_LIBRARY_PATHis a consequence, notCDLL-on-macOS's contract.I think the docs should say that all notes in
man dlopenapply. (The earlier:manpage:`dlopen(3)`link doesn't go to a macOS-specific version, and there's a bunch of notes. Maybe we should link to the macOS man page, if Apple provides stable URL for it?)I'd also quote the note from
man dlopendirectly, and mention that the python.org builds are affected (because readers might not be aware how their CPython was built).BTW, the manpage suggests that
CDLL("@rpath/libfoo.dylib")could work; does it?