Repository navigation
Conversation
Use the original request filter for file scan tasks without a server residual so count() applies the same filter as Arrow scans.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
When a REST server omits
residual-filterfrom a file scan task, PyIceberg treats the task as unfiltered. For a Parquet file withx = [1, 2, 1],scan(row_filter="x != 1").count()returns 3 whileto_arrow()returns 1.Use the original scan filter as the fallback residual for both the initial plan and any fetched task batches, as required by the Iceberg REST specification. Explicit server residuals still take precedence.
This addresses the REST case reported in #4278.
Are these changes tested?
The regression uses a local Parquet file and exercises both the initial response and pagination. It fails on
mainand the current head of #4278, and passes with this change. The planning tests also verify that an explicit true residual is preserved and that an unfiltered request still gets an unfiltered task.make test: 4,685 passed, 5 skipped.make lintpassed.Are there any user-facing changes?
Filtered counts now agree with Arrow scans when a server omits the residual. Affected files may be read to apply the filter.
AI assistance
An AI assistant helped trace the planning code and draft the tests; I reviewed and ran everything and take responsibility for the change.