Repository navigation
Missing DTrace probes #104280
Description
Bug report
On Linux, you can list available DTrace probes by running /bcc/tools/tplist.py tool;
$ ~/bcc/tools/tplist.py -l cpython/python
cpython/python python:function__entry
cpython/python python:function__return
cpython/python python:line
cpython/python python:import__find__load__done
cpython/python python:import__find__load__start
cpython/python python:audit
cpython/python python:gc__start
cpython/python python:gc__doneThe function__entry, function__return and line probes appear to be missing after the GH-103082: Implementation of PEP 669: Low Impact Monitoring for CPython update.
$ ~/bcc/tools/tplist.py -l cpython/python
cpython/python python:import__find__load__done
cpython/python python:import__find__load__start
cpython/python python:audit
cpython/python python:gc__start
cpython/python python:gc__doneYour environment
- CPython versions tested on: Python 3.12.0a7+
- Operating system and architecture: Arch Linux x86_64
Linked PRs
Reacted by Aryeh Hillman and Mark Smith
Activity
sunmy2019 commentedon May 8, 2023
CC: @markshannon @gaogaotiantian
gaogaotiantian commentedon May 9, 2023
I think DTrace was intentionally removed because it was not being used. I forgot where I read that, but Mark will have a better idea of it.
brandtbucher commentedon May 10, 2023
I think a bigger issue is that I'm pretty sure it isn't tested on any CI jobs or buildbots, so stuff can break (like #98894) without us noticing.
If it's not tested, it effectively isn't supported. Not sure whether the right path forward is for some motivated person to fix and maintain it, for us to remove the broken stuff and keep the working stuff (with a big disclaimer in the docs), or to just rip it out entirely.
Anecdotally, it's always been a bit uncomfortable to touch this stuff when modifying the interpreter, since I'm not very sure how it works or how to test it.
@ambv might have an idea of what the right path forward is here.
furkanonder commentedon May 25, 2023
Dtrace support was actually being used. DTrace markers, also known as "probes", are not only used by DTtrace. For example, these probes can be used with bpftrace.
There are tools written using these probes. I will even give a talk about these probes at EuroPython 2023 :)
kchoudhu commentedon Jul 10, 2023
Just chiming in here: we use these probes in anger in production, and their absence is a blocker preventing us from upgrading to 3.11.
If tests are needed to make sure dtrace support is accounted for in the release process, I am happy to contribute time to write tests and set them up.
furkanonder commentedon Jul 29, 2023
@ambv @brandtbucher I wrote a test case to check for the presence of probes. Could you review the PR?
gh-104280: Add test cases for DTrace probes (#107125)
pythongh-104280: Add test cases for DTrace probes (pythonGH-107125)
[3.11] pythongh-104280: Add test cases for DTrace probes (pythonGH-10…
[3.12] gh-104280: Add test cases for DTrace probes (GH-107125) (#107489)
[3.11] gh-104280: Add test cases for DTrace probes (GH-107125) (#107492)
encukou commentedon Aug 8, 2023
The newly added tests fail in Fedora:
Full log: https://kojipkgs.fedoraproject.org//work/tasks/6927/104486927/build.log
Here's a readelf output printed out:
I'm unfortunately unfamiliar with DTrace. Is this expected? How can I debug this?
ambv commentedon Aug 8, 2023
Are you building your Python
--with-dtrace? If not, it's a surprise thatsysconfig.get_config_var('WITH_DTRACE')returns 1 in your case...When I build
--with-dtracelocally, I get1but when I don't, that config var returns0. In this case the test is just skipped:ambv commentedon Aug 8, 2023
BTW, this says "dtrace" but those probes are also SystemTap / eBPF.
7 remaining items
Temporarily skip test_check_probes in CI tests
markshannon commentedon Oct 5, 2023
dtrace is supposed to have zero (or very close to it) overhead when not in use.
The CPython implementation failed in this. If it had, perhaps PEP 669 would not have been needed.
I think it is best to push users towards using
sys.monitoringnow that 3.12 has been released.I realize that's not the ideal solution for existing users of
dtrace, butsys.monitoringdoesn't need a special build and is tested on CI.markshannon commentedon Oct 5, 2023
It also looks like dtrace function entry robes are broken in 3.11, as they are only invoked if
sys.settracetracing is also turned on.The probe point is present, but the normal code path skips it.
kchoudhu commentedon Oct 5, 2023
With dtrace support effectively removed in 3.12, a few thoughts.
My initial read of
sys.monitoringis that while it is great for bespoke, python-specific monitoring and tooling, it doesn't quite work in a world that is increasingly moving towards one-size fits all distributed tracing via ebpf (i.e. dtrace).Is there room in Python for both visions of tracing? If not, why? If yes, what kind of resourcing would be needed to make sure that dtrace isn't removed again from the build?
Zheaoli commentedon Oct 9, 2023
This is a little bit weird, the DTrace and the PEP 669 are different features.
the DTrace API is for people who want to trace the Python process without restarting the process and injecting some code for the PEP 669
We can solve the process tracing problem by using #96123 in some circumstances. But there are still some issues.
If we decide to remove DTrace Support officially, we may need to update the documentation https://docs.python.org/3/howto/instrumentation.html
avioligo commentedon Jan 9, 2024
When building python 3.11.7 from source, I ran into this error:
python configure: error: dtrace command not found on $PATHFixed using:
sudo apt-get install systemtap-sdt-devBy installing dtrace (well almost) toolkit - the binary is not really used for tracing but only to list probes if I am correct.
Bill-Sommerfeld commentedon Feb 9, 2024
I took a look at this yesterday in 3.12 as part of my current effort to package 3.12 for OpenIndiana (illumos) and I got the function entry/exit probes restored and working well enough to pass the test_dtrace.py test.
I'll share when I get a chance - I'm doing this with original dtrace on illumos (the open-source continuation of opensolaris) but it would likely also work for those who want to use these probes via systemtap.
I'll also look at the line probes when I get a chance.
Bill-Sommerfeld commentedon Feb 9, 2024
https://github.com/Bill-Sommerfeld/cpython/tree/dtrace-3.12 has patches that restore the
function-entryandfunction-returnprobes for me on illumos. The tests now pass for me, except fortest.test_dtrace.DTraceNormalTests.test_lineandtest.test_dtrace.DTraceOptimizedTests.test_linewhich depend on thelineprobe which I haven't restored yet.I'd appreciate it if others could give these a whirl on other platforms.
Temporarily skip test_check_probes in CI tests