[Anurag-S1ngh]

Rationale for this change

Planning a table repeatedly read the same immutable files from object storage every time: the manifest list on every plan_files, and every manifest on every plan. _manifest_cache only keeps the parsed ManifestFile records, not the file contents.

This adds a byte cache to FileIO, keyed by location. Parsing is unchanged; the bytes are cached, so a ManifestFile record parsed from one manifest list cannot be served values from another, which a parsed-object cache keyed by path would allow.

It follows Iceberg Java's io.manifest.cache-* properties and is off by default:

• io.manifest.cache-enabled (default false)
• io.manifest.cache.max-total-bytes (default 100 MiB)
• io.manifest.cache.max-content-length (default 8 MiB)
• io.manifest.cache.expiration-interval-ms (default 60 s)

ContentCache is a bounded, least-recently-used map with lazy expiry, matching the get(location) -> bytes | None and put(location, data) shape from the issue. A FileIO that wants a different backing store, a disk cache for example, can override FileIO.content_cache, which is read-only and returns None when the properties do not enable caching.

Only manifest lists and manifests go through the cache. Data files and metadata.json are unchanged, so nothing that can be rewritten is cached.

Are these changes tested?

Yes. tests/io/test_content_cache.py covers:

• caching is off unless io.manifest.cache-enabled is truthy, including the values false, 0 and the empty string
• the limits are read from FileIO properties, including table properties
• entries over max-content-length, or over max-total-bytes on their own, are not cached
• least-recently-used eviction and expiry
• a manifest list and a manifest are each read once when the cache is enabled, and once per call when it is not

Full unit suite: 4586 passed, 5 skipped. prek run -a passes.

Are there any user-facing changes?

No behavior change by default. With io.manifest.cache-enabled=true, repeated planning against the same FileIO reuses the manifest bytes instead of reading them from storage again, which also keeps planning working through a short outage for files already read.

Closes #4300

[Fokko]

@Anurag-S1ngh Thanks for working on this @Anurag-S1ngh, is this something that you ran into? In the past we had some issues with caching, since it can easily take a lot of memory.

[Anurag-S1ngh]

Thanks Fokko. No, I have not run into it myself. I picked this up from #4300, where the repeated reads of the manifest list and of every manifest on each plan are described.

On memory, it is off by default and bounded: 100 MiB total, 8 MiB per entry, LRU eviction when full, and entries expire after 60s. Anything over the per-entry limit is not cached at all, so the cache cannot go over the limit, it just stops caching. Only manifest lists and manifests go through it, not data files or metadata.json, and all four limits are configurable.

If the defaults are still too generous I can lower them, or drop the in-memory cache and keep only the override point on FileIO.