Repository navigation
Fix moving a column that was deleted and added back in the same update - #4365
Open
chenjian2664 wants to merge 1 commit into
Open
chenjian2664 wants to merge 1 commit into
chenjian2664 wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The added-column lookup ignores case_sensitive=False, so moving differently cased replacement columns or using them as anchors can resolve deleted field IDs and raise StopIteration.
1 open issue
What changed in this PR
This PR fixes schema updates that delete a column, add it back under the same name, and then move it by prioritizing newly added columns during move lookup.
Changes:
- Resolve move targets against columns added in the current update before the existing schema.
- Add a regression test for replacing a column’s type and moving it after another column.
| File | Description |
|---|---|
| tests/table/test_init.py | Tests column replacement and reordering within one schema update. |
| pyiceberg/table/update/schema.py | Prioritizes newly added columns when resolving move targets. |
Review effort: Balanced
Comment on lines
+507
to
+508
| if (added_id := self._added_name_to_id.get(name)) is not None: | ||
| return added_id |
Deleting a column, adding it back under the same name and then moving it in one update_schema() raised a bare StopIteration: the move resolved the name to the deleted field. Check the columns added in the update first, and track their names in case-aware form so case_sensitive=False also finds them, as Java's SchemaUpdate.findForMove does.
chenjian2664
force-pushed
the
chenjian2664/bytequay/fix-move-of-deleted-and-re-added-column
branch
from
November 14, 2026 13:18
a42f2a9 to
d9be5f2
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Within one
update_schema(), deleting a column, adding it back under the same name and then moving it withmove_first,move_beforeormove_afterraises a bareStopIteration. This is a natural way to replace a column with an incompatible type while keeping its position. The move resolved the name against the existing schema first and picked up the deleted field.The lookup now checks columns added in the same update first, as Java's
SchemaUpdate.findForMovedoes.Are these changes tested?
Are there any user-facing changes?