[localzet]

Summary

Copy live entries directly into a compact keys table when cloning a sparse combined dictionary, rather than inserting each surviving key/value pair through the generic merge path.

• Preserve insertion order, key/value identities, and cached hashes without invoking user-defined __hash__ methods again.
• Use the compact-copy path for dict.copy() and copying into empty dictionaries (including frozendict construction).
• Keep the existing path for very sparse tables (fewer than approximately 1/64 live entries), where scanning tombstones is less beneficial.
• Preserve dictionary watcher notifications when an empty destination clones a source.

Tests

The patch was tested in the contributor fork with:

• Debug builds: test_dict, test_capi.test_dict, and test_capi.test_watchers (249 tests).
• Debug free-threaded build: the same test modules (249 tests).
• Reference-leak checks: -R 3:3 on these modules.
• AddressSanitizer build: the same test modules (detect_leaks=0, matching CPython's ASan CI configuration).
• Additional randomized sparse-dictionary stress tests, including hash collisions and concurrent copies, in a separate research branch.

Verification: https://github.com/localzet/cpython/actions/runs/42510556632

Performance

Linux x86-64, GCC PGO + LTO; median of repeated source.copy() measurements against the unmodified baseline (3c92a06e). Times are ns per operation.

Keys (live / original)	Key type	Baseline	Patch	Speedup
30,000 / 100,000	str	891,644	235,320	3.79x
30,000 / 100,000	int	592,779	206,562	2.87x
3,000 / 100,000	str	118,462	46,237	2.56x
3,000 / 100,000	int	111,283	76,323	1.46x
1,500 / 100,000	int	87,039	89,021	0.98x

These are results from one CI benchmark run, not a statistically controlled multi-machine study. Extremely sparse integer-key cases are approximately unchanged; some cases show a small measured slowdown.

Benchmark: https://github.com/localzet/cpython/actions/runs/42497448717

AI assistance was used for source exploration, patch development, and test/benchmark automation. The patch and its behavior were examined and tested against the configurations listed above.

• Issue: Optimising sparse dict copy #161524

[methane]

Keep the existing path for very sparse tables (fewer than approximately 1/64 live entries), where scanning tombstones is less beneficial.

Could you benchmark the case where can_clone_sparse_dict() always returns true and compare the results?

I'm interested in whether this branch is really necessary. If the performance is comparable, there's no need to keep the existing code path. If there's a significant difference, I'd also like to reconsider the similar code in dictresize().

[localzet]

I tried making can_clone_sparse_dict() always return true and compared it with the current 1/64 threshold. The regular-build benchmark shows a clear regression for very sparse tables.

Some dict.copy() results (ns/op, averages of two measurement passes, lower is better):

Live / original entries	Key type	1/64 threshold	Always clone
1 / 1,000	str	442	724
100 / 100,000	str	36,892	66,633
1 / 100,000	str	35,066	58,266
1,000 / 500,000	str	299,353	570,655
1 / 100,000	int	57,736	71,241

The difference is small for less sparse cases, but the fallback is noticeably faster once almost all entries have been deleted. Both implementations passed the targeted dict tests, watcher tests, and the sparse-dict stress checks.

I used the same Ubuntu 24.04 runner for each comparison, with measurements in A–B–B–A order. The benchmark and workflow are available here: https://github.com/localzet/cpython/actions/runs/42595205328

The GCC PGO/LTO comparison is still running, so these numbers are from the regular build. For now, I'd keep the density check. I'll look at dictresize() separately, since its resize path has different constraints.