[rahulsmahadev]

Rationale for this change

Scan planning gets uuid identity partitions wrong. On a table partitioned by identity(u) on a uuid column:

tbl.scan(row_filter=EqualTo("u", str(first))).plan_files()   # no files; count() is 0
tbl.scan(row_filter=In("u", {str(first), str(second)})).plan_files()  # no files
tbl.scan(row_filter=NotEqualTo("u", str(first))).count()     # counts the excluded partition too

Manifests are read with uuid partition values as uuid.UUID, while a UUIDLiteral holds the 16 bytes, so the partition evaluator and the residual evaluator compare a UUID with bytes and never find them equal. EqualTo and In plan no files, and NotEqualTo and NotIn plan every file. Engines that use PyIceberg's planning (plan_files()) get the same wrong set of files, and count() is wrong because it trusts the residual.

This change gives partition values the representation literals use (a uuid becomes its 16 bytes) before the partition and residual evaluators compare them. It only applies to specs with a uuid partition field; other partition values are passed through unchanged.

Follow-up to #4360, which noted this as a separate fix. Reading the matching files still hits the PyArrow ArrowNotImplementedError of #2372: to_arrow() raises it, and so does count() when it has to read the files (a row predicate that the partition does not resolve, or delete files). For = and in filters this replaces silently returning no rows.

Are these changes tested?

Yes. test_scan_plans_uuid_identity_partitions in tests/io/test_pyarrow.py, next to the append test from #4360, checks the planned partitions and count() for =, in, != and not in filters on a uuid identity partition. It fails on main (the = filter plans no files) and passes with this change.

Are there any user-facing changes?

Bug fix: the partition and residual evaluators compare uuid partition values with uuid literals correctly. No API changes.

This pull request and its description were written by Isaac.

[rahulsmahadev]

@dossett could you take a look when you have a chance? It is the uuid partition filter follow-up from #4360: a small change in ManifestGroupPlanner, next to the residual cache from #3666.

[Fokko]

Thanks for digging into this! I think the root cause is that we have two in-memory representations of a uuid: the Avro UUIDReader returns uuid.UUID (and #4360 asserts that for partition[0]), while UUIDLiteral and from_bytes(UUIDType) use bytes. Rather than converting at the evaluator call sites, I'd prefer to standardize on uuid.UUID. This is also what Java does: UUIDLiteral is a Literal<UUID> and Conversions.fromByteBuffer returns a UUID.

Some pointers:

• UUIDLiteral: wrap a uuid.UUID instead of bytes. ser_model already returns a UUID.
• literal() and StringLiteral.to(UUIDType): stop calling .bytes.
• FixedLiteral.to(UUIDType) and BinaryLiteral.to(UUIDType): build UUID(bytes=...), and convert back with .bytes in UUIDLiteral.to(FixedType) / UUIDLiteral.to(BinaryType).
• from_bytes(UUIDType): return uuid.UUID(bytes=b), so the metrics evaluators also compare the lower and upper bounds as UUID.
• _to_partition_representation(UUIDType): keep identity values as UUID instead of converting them to bytes.
• _convert_scalar: convert a UUID to .bytes when building the pa.uuid() scalar for the Arrow filter.

Ordering is preserved: UUID compares by its integer value, which matches unsigned big endian byte order, so range predicates and min/max pruning keep working. The visible change is that UUIDLiteral.value becomes a UUID, which is worth a note in the changelog.

This also covers the projected identity values, where Record(field_value) is built from the partition value and compared against the literal. That path has the same mismatch and isn't touched by this PR.

[Fokko]

I don't think we should patch the Record here. It feels like we're working around the actual issue: partition values are uuid.UUID while UUIDLiteral holds bytes. Converting at two call sites in the planner means every other place that evaluates a partition Record still has to know about this. For example, the projected identity values in visitors.py build Record(field_value) from the partition value and run into the same mismatch. It also allocates a new Record for every data file on uuid specs, and adds another spec keyed cache to plan_files. If we make both representations agree (see my comment above), this helper can go away.