[rahulsmahadev]

Rationale for this change

The row filter parser rejects two kinds of valid numeric literals:

tbl.scan(row_filter="price > 1e-3")
# ParseException: Expected end of text, found 'e'  (at char 9), (line:1, col:10)
tbl.scan(row_filter="quantity in (1, 2.5)")
# ParseException: Expected ')', found '.'  (at char 17), (line:1, col:18)

Decimal literals used pyparsing's common.real(), which has no exponent, so 1e-3 parses as 1 and the rest is left over. A literal set had to be all decimals or all integers, so a list that mixes 1 and 2.5 did not parse either.

This change uses common.sci_real(), which accepts everything common.real() does plus an exponent (1e-3, -1.5E3, 2.5e1), and lets a literal set mix integers and decimals. Integers become LongLiterals and decimal or exponent forms become DecimalLiterals; binding uses the existing conversion rules (a decimal column needs a literal with its scale, and an integer column rejects fractional values). The decimal parse action already builds the Decimal from the matched text, so exponents are exact. A list that mixes numbers with strings still raises a ParseException (test_in_different_types).

Are these changes tested?

Yes. test_scientific_notation and test_in_mixed_numbers in tests/expressions/test_parser.py cover exponents on either side of a comparison and in BETWEEN, and mixed integer/decimal IN lists (in both orders) and NOT IN lists. Both fail on main with the ParseExceptions above and pass with this change.

Are there any user-facing changes?

Bug fix: row filter strings accept scientific notation and IN lists that mix integers and decimals. Expressions that parsed before parse to the same result.

This pull request and its description were written by Isaac.

[rahulsmahadev]

@jackylee-ch could you take a look when you have a chance? This small change to the expression parser lets 1e-3 and IN (1, 2.5) parse. You last updated this parser in #3917.

[Fokko]

Thanks for picking this up! I think the scientific notation part should follow ANSI SQL. In ANSI, <mantissa>E<exponent> is an approximate numeric literal (so a double), while only digits with an optional decimal point are exact numeric (decimal). Spark 3+, Trino and DuckDB all parse 1e-3 as a double.

Parsing it as a DecimalLiteral also causes a binding problem: Decimal("1e3") is 1E+3 with scale -3, so dec_col = 1e3 cannot bind to any decimal column, and price > 1e-3 only binds when price has exactly scale 3. The tests don't catch this because Decimal("2E+2") == Decimal("200").

Could we emit a DoubleLiteral for the exponent forms instead? That would need a DoubleLiteral.to(LongType) conversion so long_col = 1e3 keeps working. It would also be great to add a few tests that bind against a schema with decimal, long and double columns.

The mixed integer/decimal IN list change looks good to me and is in line with ANSI.