Repository navigation
Order -0.0 below 0.0 in data file metrics bounds - #4331
Merged
Fokko merged 2 commits intoNov 12, 2026
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's changed
StatsAggregator.update_min/update_maxaggregate parquet row-group statistics intoDataFile.lower_bounds/upper_bounds. On a tie between+0.0and-0.0,min(val, current)andmax(val, current)keep the first argument — so whichever zero arrives last wins:-0.0lowers an existing+0.0upper bound (e.g. row group 1 contains+0.0, a later row group's max is-0.0),+0.0raises an existing-0.0lower bound.The serialized bounds then order above a value that is present in the file, violating the two's-comparable bound ordering.
#4324 fixed the same class in the partition-field summary path (
PartitionFieldStats.updateinmanifest.py); this applies the equivalent guard to the data-file metrics path. The argument order here is flipped (min(val, current)vsmin(self._min, value)), so the guarded directions are mirrored, and the guard is split across the two update methods. Verified against #4324's pinned bar: for both arrival orders the bounds must serialize aslower = to_bytes(-0.0),upper = to_bytes(+0.0).NaN needs no handling here: pyarrow row-group statistics exclude NaN (
has_min_maxisFalsefor all-NaN chunks), so the aggregator never sees it on this path.Test
test_stats_aggregator_orders_negative_zero_below_zero— parametrized over both arrival orders andFloatType/DoubleType, asserting the serialized bounds match the #4324 canonical pair. Verified failing-first: 4/4 fail onmain(533a617) and pass with the guard; the existingStatsAggregatortests (18) still pass.AI disclosure: code and tests drafted with AI assistance, verified locally by running the full failing-first cycle.