Skip to content

Order -0.0 below 0.0 in data file metrics bounds - #4331

Merged
Fokko merged 2 commits into
apache:mainfrom
AP24110011695:fix/stats-aggregator-negative-zero-bounds
Nov 12, 2026
Merged

Fokko merged 2 commits into
apache:mainfrom
AP24110011695:fix/stats-aggregator-negative-zero-bounds

Conversation

@AP24110011695

Copy link
Copy Markdown
Contributor

What's changed

StatsAggregator.update_min/update_max aggregate parquet row-group statistics into DataFile.lower_bounds/upper_bounds. On a tie between +0.0 and -0.0, min(val, current) and max(val, current) keep the first argument — so whichever zero arrives last wins:

  • a late -0.0 lowers an existing +0.0 upper bound (e.g. row group 1 contains +0.0, a later row group's max is -0.0),
  • a late +0.0 raises an existing -0.0 lower bound.

The serialized bounds then order above a value that is present in the file, violating the two's-comparable bound ordering.

#4324 fixed the same class in the partition-field summary path (PartitionFieldStats.update in manifest.py); this applies the equivalent guard to the data-file metrics path. The argument order here is flipped (min(val, current) vs min(self._min, value)), so the guarded directions are mirrored, and the guard is split across the two update methods. Verified against #4324's pinned bar: for both arrival orders the bounds must serialize as lower = to_bytes(-0.0), upper = to_bytes(+0.0).

NaN needs no handling here: pyarrow row-group statistics exclude NaN (has_min_max is False for all-NaN chunks), so the aggregator never sees it on this path.

Test

test_stats_aggregator_orders_negative_zero_below_zero — parametrized over both arrival orders and FloatType/DoubleType, asserting the serialized bounds match the #4324 canonical pair. Verified failing-first: 4/4 fail on main (533a617) and pass with the guard; the existing StatsAggregator tests (18) still pass.

AI disclosure: code and tests drafted with AI assistance, verified locally by running the full failing-first cycle.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants