Skip to content

Core: Decode dictionary arrays before applying Arrow partition transforms - #4340

Open
Anurag-S1ngh wants to merge 1 commit into
apache:mainfrom
Anurag-S1ngh:fix/3633-bucket-transform-dictionary-arrays
Open

Anurag-S1ngh wants to merge 1 commit into
apache:mainfrom
Anurag-S1ngh:fix/3633-bucket-transform-dictionary-arrays

Conversation

@Anurag-S1ngh

Copy link
Copy Markdown
Contributor

Rationale for this change

Appending a dictionary-encoded Arrow table to a table partitioned by bucket, truncate, year, month, day or hour raised:

ValueError: FeatureUnsupported => Unsupported data type for bucket transform: Dictionary(Int32, Utf8)

The partition transforms run on Arrow arrays through _pyiceberg_transform_wrapper, which forwarded the array as-is. The Arrow kernels behind the transforms have no implementation for dictionary types, so any dictionary-encoded column used in a partition transform failed even though Iceberg has no dictionary type and reads such a column as its value type.

The wrapper now decodes dictionary arrays with dictionary_decode() before calling the transform, per chunk for chunked arrays. The result matches what the same values produce without dictionary encoding, so partitions, and therefore partition pruning, stay the same.

Are these changes tested?

Yes.

  • tests/test_transforms.py::test_pyarrow_transform_decodes_dictionary_arrays covers both pa.Array and pa.ChunkedArray and asserts the dictionary-encoded input gives the same buckets as the plain input, including a null.
  • tests/catalog/test_catalog_behaviors.py::test_write_dictionary_encoded_column_with_bucket_partition appends a dictionary-encoded table to a bucket-partitioned table, checks the rows read back and that both partitions were created. It runs against the memory, sql, and sql_without_rowcount catalogs.

Both fail on main and pass with the change.

Full unit suite: 4573 passed, 5 skipped. prek run -a passes.

Are there any user-facing changes?

Yes. Appending dictionary-encoded columns to a table with a partition transform now works instead of raising FeatureUnsupported.

Closes #3633

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BucketTransform fails on dictionary-encoded pyArrow arrays

1 participant