Repository navigation
fix(expressions): reject fractional strings converting to int/long - #4383
Open
Shahzeb-DataEngineer wants to merge 6 commits into
Open
Shahzeb-DataEngineer wants to merge 6 commits into
Shahzeb-DataEngineer wants to merge 6 commits into
Conversation
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
1 of 3 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4382
Rationale for this change
StringLiteral.to(IntegerType)andStringLiteral.to(LongType)parsed withDecimalthen truncated with plainint(), so "2.6" became 2 and a bound predicate such asx < "2.6"on an int column silently becamex < 2, wrongly dropping rows with x = 2 and feeding the same wrong literal into partition pruning. The siblingDecimalLiteralpath already rejects fractional values via_to_integral, and the Java reference implementation refuses the conversion entirely. This change reuses_to_integralin both StringLiteral paths.Are these changes tested?
Yes. New test
test_fractional_string_to_integral_type_raisesmirrorstest_fractional_decimal_to_integral_type_raises; it fails on unpatched code and passes with the fix.test_string_literalupdated to no longer assert the truncating behavior.tests/expressions/test_literals.py: 180 passed.tests/expressions/: 594 passed, 2 failed (pre-existing pyparsing-version failures on clean main, verified viagit stash).ruff checkandruff format --checkclean on both changed files.Are there any user-facing changes?
Yes, behavior change for the better: converting a fractional string such as "2.6" to an int/long column now raises ValueError at bind time instead of silently truncating. Integral strings ("2.0", "1e3") still convert exactly.