正文
- Notifications You must be signed in to change notification settings
- Fork 3.5k
Conversation
Summary
On main, we support content-addressed caching, but only at the wheel-level. That is, if you download the same wheel twice from different sources, they share a cache entry. But files within or across wheels are not deduplicated at all.
This PR adds deduplication at the file level: every file is now stored under its BLAKE3 hash in a files-v0 bucket. We hardlink these objects into their original locations in archive-v0, so the installation step doesn't change at all -- we're just deduping within the cache (and cache cleanup removes file objects when their hardlink count drops to one).
In the prior proposal (#19694), we included the following table:
| File selection | Additional savings | Files hardlinked | Distinct files-v0 objects |
|---|---|---|---|
| Executables and native libraries | 275.7 MiB | 3,336 | 3,088 |
| Any payload file ≥ 10 MiB | 235.0 MiB | 80 | 78 |
| Any payload file ≥ 1 MiB | 279.7 MiB | 373 | 349 |
| Any payload file ≥ 100 KiB | 353.4 MiB | 2,830 | 2,458 |
| Any payload file ≥ 10 KiB | 475.7 MiB | 23,767 | 18,722 |
| Any payload file ≥ 1 KiB | 537.4 MiB | 95,156 | 66,422 |
| All payload files | 545.2 MiB | 134,222 | 87,129 |
So we're saving 545.2 MiB on my local machine, or about 10% of the cache.
In return, the net effect seems to be something like a <4% slowdown for cold installs (and no effect on warm installs), which I think is probably worthwhile here.
GitHub Actions
Inactive
GitHub Actions
Inactive
|
N.B. Slop comment for benchmark data. We benchmarked the optimization in 85c3f7485 against the binary-only cache at a3f977ece, with
Before this optimization, the measured cold regressions were +4.02% for AnyIO 4.9.0, +19.41% for SymPy 1.14.0, +12.75% for NumPy 2.2.6, +15.30% for PyTorch 2.7.1+cpu. All eight median regressions are below 5%. The PyTorch cold 95% interval still reaches 5.18%, so its upper bound is not below 5%. Every other upper bound is below 5%. We retained all 1,860 measured installs, including outliers, and combined every confirmation run for this candidate. Cold measurements have 360 paired rounds for AnyIO, 120 each for SymPy and NumPy, and 90 for PyTorch; warm measurements have 60 paired rounds per wheel. Baseline/candidate order alternates, with three warmups per case. Intervals use a paired percentile bootstrap of the ratio of medians with 10,000 resamples. These measurements ran on Linux/ext4, an AMD EPYC-Milan VM pinned to eight CPUs, with Python 3.12.13. Both binaries use Rust 1.98.0 and the same All-file deduplication, content/executable identities, the cache layout, complete archives, and copy fallbacks are preserved. Inode checks confirmed that every archived file shares its file-store object for all four wheels. The five targeted integration tests passed ten stress iterations (50 executions), including local and streamed wheels with one and four workers, RECORD handling, cache cleanup, and cross-filesystem installation. Formatting and Clippy with warnings denied also passed. |
GitHub Actions
Inactive
Merging this PR will not alter performance
Comparing Footnotes
|
a3f977e to
1f28fd5
Compare
August 28, 2026 00:08
0a8edc0 to
a108fab
Compare
August 28, 2026 00:08
GitHub Actions
Inactive
charliermarsh
added
enhancement
New feature or improvement to existing functionality
preview
Experimental behavior
labels
Aug 28, 2026
GitHub Actions
Inactive
uv test inventory changesThis PR changes the tests when compared with the
uv-extract: +1 / -0
Added:
Removed: none uv::pip_install: +4 / -0
Added:
Removed: none |
## Summary When content hashing is enabled, we currently allocate and zero a new 64 KiB buffer for every file we copy and hash during streaming extraction. This PR reuses one buffer across the wheel instead. For the PyTorch wheel used in the benchmarks, that reduces buffer allocations for hashing from 11,120 to one, while keeping the buffer size at 64 KiB per active wheel. The following measurements compare #21327 at `a188b8e833aef3c3b4b60a32ed9fafe6ac74186a` with this optimization applied on top, before moving the change onto `main`. They are not measurements against `main`. The Linux benchmarks alternate base and candidate, using pinned wheels served over local HTTP with content-addressed caching enabled: | Cold install | #21327 | #21327 + buffer reuse | Change | | --- | ---: | ---: | ---: | | AnyIO | 110 ms | 107 ms | -2.6% | | SymPy | 845 ms | 775 ms | -8.3% | | NumPy | 627 ms | 567 ms | -9.5% | | PyTorch CPU | 6.50 s | 5.99 s | -7.8% | | 14-package environment, concurrency 4 | 6.95 s | 6.47 s | -7.0% | The individual results above use 16 paired rounds; the full environment uses 12. AnyIO, SymPy, and NumPy were repeated after an initial 20-pair run: the initial AnyIO timings were noisy, while the initial SymPy and NumPy improvements were 7.8% and 6.9%. All original samples were retained. Cached installs and local-wheel controls showed no consistent change. Across the initial runs, repeats, and controls, we measured 672 installs, excluding warmups and cache priming. Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
a188b8e to
694af80
Compare
August 28, 2026 16:56
GitHub Actions
Error
|
Okay, I experimented with a bunch of alternatives to try and make this more performant:
I also considered something like "guess whether a file is executable" during extraction (and then, if we're wrong, copy it and change the mode after unzipping). I guess this could end up being more performant, but I wasn't very happy with the heuristics. Ultimately, I think what we have here is good. |
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Error
3eb2a66 to
73c621b
Compare
August 28, 2026 17:59
GitHub Actions
Inactive
| pub fn hardlink_count(path: &Path) -> io::Result<u64> { | ||
| Ok(fs_err::metadata(path)?.nlink()) | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-->Does this work for reflinks? Or are we exclusively using hard links for this mode?
| #[tokio::test] | ||
| async fn binary_payloads_stay_in_archive_without_preview() -> Result<()> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-->I feel like these test cases ought to be moved outside of pip_install and into something to do with the cache since it's asserting cache details here
| .with_filter(( | ||
| r"\.venv[\\/](?:bin[\\/]python3|Scripts[\\/]python\.exe)", | ||
| ".venv/[BIN]/[PYTHON]", | ||
| )) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-->What's going on here? This shouldn't be needed
| Ok(wheel) | ||
| } | ||
|
|
||
| fn cache_files(root: &Path) -> Result<Vec<PathBuf>> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-->It might make sense to expose this on the TestContext and have it take a bucket
Labels
3 participants
