Repository navigation
Respect Unicode boundaries when scanning symbol references - #65478
Open
Jake Bailey (jakebailey) wants to merge 2 commits into
Open
Jake Bailey (jakebailey) wants to merge 2 commits into
Jake Bailey (jakebailey) wants to merge 2 commits into
Conversation
Reference prefilters must classify neighboring Unicode code points, not UTF-8 bytes, so editor recovery does not drop real symbol uses. Substring matches also need their container's source offset; otherwise the first reference can be skipped or a local search can escape its scope. Keep imports and reference spans for Unicode-adjacent uses without matching parts of longer identifiers.
Separate copies of the reference scanners can drift and require the same boundary fixes in two places. Keep node and position lookup in astnav so compiler services and language-server searches use one implementation.
typescript-automation
Bot
added
Author: Team
For Uncommitted Bug
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the Unicode-aware reference-scanning consolidation.
No open issues 🎉
What changed in this PR
This PR makes symbol-reference scanning Unicode-aware and consolidates duplicated scanning logic used by the language service and checker.
Changes:
- Checks identifier boundaries using decoded Unicode characters while preserving byte offsets.
- Moves reference-scanning helpers into
astnavand corrects offsets for searches within a scope. - Adds reference, rename, and organize-imports regression tests and updates affected baselines.
| File | Description |
|---|---|
tsc/testdata/baselines/reference/fourslash/findRenameLocations/referenceScanningUnicodeBoundaries.baseline.jsonc |
Adds expected rename results around Unicode boundaries. |
tsc/testdata/baselines/reference/fourslash/findRenameLocations/processInvalidSyntax1.baseline.jsonc |
Records another rename match beside invalid Unicode syntax. |
tsc/testdata/baselines/reference/fourslash/findAllReferences/referenceScanningUnicodeBoundaries.baseline.jsonc |
Adds expected Unicode-boundary reference results. |
tsc/testdata/baselines/reference/fourslash/findAllReferences/findAllRefsWithLeadingUnderscoreNames9.baseline.jsonc |
Includes the triple-underscore function declaration in references. |
tsc/testdata/baselines/reference/fourslash/findAllReferences/findAllRefsWithLeadingUnderscoreNames8.baseline.jsonc |
Includes the double-underscore function declaration in references. |
tsc/testdata/baselines/reference/fourslash/findAllReferences/findAllRefsParameterPropertyDeclaration_inheritance.baseline.jsonc |
Updates scoped parameter-property reference results. |
tsc/testdata/baselines/reference/fourslash/documentHighlights/getOccurrencesOfAnonymousFunction.baseline.jsonc |
Includes the named function-expression declaration in highlights. |
tsc/internal/ls/findallreferences.go |
Replaces local scanning helpers with shared astnav calls. |
tsc/internal/fourslash/tests/referenceScanningUnicodeBoundaries_test.go |
Tests Unicode boundaries for references, rename, and import retention. |
tsc/internal/checker/services.go |
Uses shared scanning for symbol references and import-use detection. |
tsc/internal/astnav/tokens.go |
Adds shared scanning helpers with Unicode checks and corrected scoped offsets. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
And while here, extract a func that was copy pasted.