Repository navigation
Preserve U+FEFF when Wtf8Decoder reads payload text - #65460
Open
Josué Barros (joshbarros) wants to merge 2 commits into
Open
Josué Barros (joshbarros) wants to merge 2 commits into
Josué Barros (joshbarros) wants to merge 2 commits into
Conversation
TextDecoder drops a leading U+FEFF. Wtf8Decoder decodes each segment on its own, so the character was lost at the start of a string, a template, and the span after a lone surrogate. The constructor now defaults ignoreBOM to true. Passing false still opts out.
typescript-automation
Bot
added
the
For Uncommitted Bug
Contributor
|
This PR doesn't have any linked issues. Please open an issue that references this PR. From there we can discuss and prioritise. |
Josué Barros (joshbarros)
added
the
For Uncommitted Bug
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the change preserving U+FEFF during native TypeScript text decoding.
No open issues 🎉
What changed in this PR
Fixes #65458 by preserving U+FEFF in text decoded by the native TypeScript API, including literal values that begin with it or contain it immediately after a lone surrogate.
Changes:
- Defaults
Wtf8DecodertoignoreBOM: true, while preserving explicitignoreBOM: falsebehavior. - Adds regression coverage for leading, standalone, post-surrogate, middle, and template-text U+FEFF values.
| File | Description |
|---|---|
packages/typescript/test/wtf8.test.ts |
Adds U+FEFF preservation regression cases. |
packages/typescript/src/api/node/wtf8.ts |
Adds a constructor that preserves U+FEFF by default. |
Review effort: Balanced
Review feedback: the decoder should always set ignoreBOM to true, so the constructor no longer accepts decoder options. transport.ts no longer passes the redundant option.
Author
|
@microsoft-github-policy-service agree |
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #65458
Wtf8DecoderextendsTextDecoderwithoutignoreBOM: true.TextDecoderremoves U+FEFF from the start of a decode. This decoder also splits a buffer on a lone surrogate and decodes the following bytes as a new segment, so the same character is removed there.A string or template value that starts with U+FEFF, or that has U+FEFF immediately after a lone surrogate, comes back from the API without that character. A U+FEFF in the middle of a segment stays. The input in the report has no file-level BOM. The Go filesystem reader already strips a UTF-8 BOM from source text before the compiler sees the file. This default restores the character inside payload text.
The WASM file decoder already constructed
Wtf8Decoderwith{ ignoreBOM: true }. The constructor now defaultsignoreBOMto true. An explicitignoreBOM: falsestill opts out.The new test in
packages/typescript/test/wtf8.test.tscovers a leading U+FEFF, a value that is only U+FEFF, U+FEFF after a lone surrogate, a middle U+FEFF, and the template value from the report.node --conditions=@typescript/source --test ./test/wtf8.test.tspassed, 8 tests.I also ran the report's async API script on
typescript@7.1.0-dev.20261112.1. Before the change the code points wereleading ["78"],only [],afterSurrogate ["d800"],middle ["61","feff","62"],template ["74","61","69","6c"]. With this constructor on that package's decoder, they areleading ["feff","78"],only ["feff"],afterSurrogate ["d800","feff"],middle ["61","feff","62"],template ["feff","74","61","69","6c"]. Node 22.19 cannot parseusingin the local API sources, so that run used the published package.npx hereby test,npx hereby lint, andnpx hereby check:formatwere not run. The issue has no Backlog milestone.Drafted with assistance from Grok. I reviewed the diff and ran those tests.