[1fanwang]

Rationale for this change

Age-based snapshot expiration leaves stale branches and tags holding snapshots. This adds an opt-in step to expire those refs so their snapshots can expire too.

Ref age uses the snapshot's timestamp and its retention limit or the table-property fallback. Main never expires; dangling refs are removed. Each removal asserts the snapshot its ref had when it was judged expired, so a branch that advanced meanwhile fails the commit instead of being dropped.

Builds on #3246.

Are these changes tested?

Python 3.12, PyArrow 25.0.1. The probe builds an expired branch in SQLite with the test helper and commits the expiration. With RACE set, a second writer commits to that branch after expiry was planned. It skips ref expiry on code without that API. In the PR checkout, run uv sync --group dev --extra pyarrow --extra sql-sqlite first.

Reproducer source: probe.py

from os import environ

import pyarrow as pa

from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.exceptions import CommitFailedException
from tests.table.test_expire_snapshots import _table_with_expired_branch

root = environ["PROOF_ROOT"]
catalog = SqlCatalog("p", uri=f"sqlite:///{root}/c.db", warehouse=f"file://{root}")
table, _ = _table_with_expired_branch(catalog_with_warehouse=catalog, max_ref_age_ms=1, namespace="n")
expire = table.maintenance.expire_snapshots()
if hasattr(expire, "remove_expired_refs"):
    expire.remove_expired_refs()
if "RACE" in environ:
    catalog.load_table("n.tbl").append(pa.table({"id": [9]}), branch="audit")
try:
    expire.commit()
except CommitFailedException as error:
    print(error)
print(sorted(catalog.load_table("n.tbl").metadata.refs))

$ mkdir a b c d; F=pyiceberg/table/update/snapshot.py
$ git checkout 68898e5a -- $F; PROOF_ROOT=$PWD/a uv run python probe.py
['audit', 'main']
$ git checkout b1bd6ed -- $F; PROOF_ROOT=$PWD/b uv run python probe.py
['main']
$ git checkout 52542453 -- $F; RACE=1 PROOF_ROOT=$PWD/c uv run python probe.py
['main']
$ git checkout b1bd6ed -- $F; RACE=1 PROOF_ROOT=$PWD/d uv run python probe.py
Requirement failed: branch audit has changed: expected id 1992796208241795715, found 5365421225648146846
['audit', 'main']

REST and Hive cases were not rerun.

Are there any user-facing changes?

Yes: opt-in ref expiry and history.expire.max-ref-age-ms, documented in the API guide.

AI disclosure: GitHub Copilot CLI (GPT-6 Astra, Claude Sonnet 5.5) wrote this change and its tests, and an automated coordinator reviewed the diff and test output before submission.

[Copilot]

Pull request overview

This PR adds support for expiring stale snapshot refs (branches/tags) based on max-ref-age-ms, bringing PyIceberg’s snapshot expiration behavior closer to Iceberg’s retention policy semantics and preventing refs from pinning snapshots indefinitely.

Changes:

• Add ExpireSnapshots.remove_expired_refs() to drop stale refs based on per-ref max-ref-age-ms with fallback to history.expire.max-ref-age-ms.
• Make older_than() resolve at commit time so chaining order with remove_expired_refs() does not affect which snapshots become eligible for expiration.
• Add unit tests and API documentation for expiring branches/tags and the new table property.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
tests/table/test_expire_snapshots.py	Adds coverage for ref expiry semantics (expired/unexpired refs, opt-in behavior, order independence, table-property fallback, main exemption).
pyiceberg/table/update/snapshot.py	Implements ref expiry staging and defers older_than() evaluation to commit to make builder chaining order-independent.
pyiceberg/table/init.py	Introduces TableProperties.MAX_REF_AGE_MS and its default.
mkdocs/docs/api.md	Documents how to expire branches/tags and reclaim pinned snapshots.

Suppressed comments (2)

tests/table/test_expire_snapshots.py:409

• This test creates a fixed namespace name in a session-scoped warehouse; if another test already created it, create_namespace will raise NamespaceAlreadyExistsError. Use create_namespace_if_not_exists (or randomize the namespace) to keep the suite order-independent.

    catalog_with_warehouse.create_namespace("expire_refs")

tests/table/test_expire_snapshots.py:433

• This test creates a fixed namespace name in a session-scoped warehouse; if another test already created it, create_namespace will raise NamespaceAlreadyExistsError. Use create_namespace_if_not_exists (or randomize the namespace) to keep the suite order-independent.

    catalog_with_warehouse.create_namespace("expire_refs")

[1fanwang]

Fixed in ca43097. sys.maxsize is 2**31-1 on a 32-bit build, so the default would have been ~25 days rather than unbounded — pinned to 2**63-1 to match Java's Long.MAX_VALUE.

[1fanwang]

Fixed in ca43097 — each test now uses its own namespace. The collision does not actually occur today, since catalog_with_warehouse calls destroy_tables() on teardown, but the tests should not depend on that for isolation.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[1fanwang]

@Fokko could you review when you have time?

[rambleraptor]

One quick note after looking at the Java implementation.

[rambleraptor]

I think we need to also add a AssertRefSnapshotId requirement. This helps ensure that if a ref is expired, it's never checked against what's in the catalog at commit time.

[1fanwang]

Done in b1bd6ed. Each removed ref now asserts the snapshot it pointed at when it was judged expired, so a branch that advanced meanwhile fails the commit.