Repository navigation
fix(data_masking): mask what the docs promise in the edge cases of erase - #8781
Merged
Merged
Conversation
- fields and masking_rules together now raise instead of ignoring fields - Pydantic models are dumped in JSON mode: secret types stay masked, UUID, datetime and Decimal fields no longer fail the fields path - masking_rules accept JSON strings like fields does, and recurse into dict and list values instead of masking their Python repr - prepare_data tracks only the ancestors of the current path, so an object referenced twice is converted twice, and handles __slots__ objects - tampered or malformed ciphertext surfaces as DataMaskingDecryptValueError - regex cache bounded with lru_cache; docs for the behaviours above
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #8781 +/- ##
===========================================
+ Coverage 97.03% 97.18% +0.14%
===========================================
Files 323 323
Lines 16511 16537 +26
Branches 1488 1494 +6
===========================================
+ Hits 16022 16071 +49
+ Misses 349 326 -23
Partials 140 140 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…d key commitment errors
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Issue number: closes #8780, closes #8636, closes #8672
Summary
Changes
Follow-up to #8780, the edge cases where
erase()returned without masking what the docs say it masks, plus the error-handling and docs items from the same list:erase(data, fields=[...], masking_rules={...})raisesDataMaskingError; before,fieldswas silently ignored.prepare_datadumps Pydantic models withmodel_dump(mode="json"):SecretStr/SecretByteskeep Pydantic's masked rendering instead of leaking throughvars(), andUUID,datetime,Decimalfields no longer fail thefieldspath withTypeError.masking_rulesaccept a JSON string likefieldsdoes (Bug: DataMasking.erase with masking_rules ignores JSON string input #8636, fix(data-masking): parse JSON string input when erasing with masking_rules #8637), and a rule pointing at a dict or list recurses into it instead of maskingstr(value)and changing the type.prepare_datakeeps only the ancestors of the current recursion path in the visited set, so the same object referenced twice is converted twice instead of being treated as a cycle (Bug: data masking prepare_data leaves dataclasses and Pydantic models unconverted #8672); objects using__slots__are converted too. Real cycles are still cut.regex_pattern+mask_formatrule whose pattern does not match the value now masks the whole value with the dynamic mask. Before, the value came back untouched.Enummembers are reduced to their.valuebefore masking. Before,vars()dragged the enum class and its internals into the output.SerializationError,MasterKeyProviderErroron a tampered header, ...),InvalidSignature,TypeErrorandValueErrorintoDataMaskingDecryptValueError, and non-base64 input too.lru_cache(256)instead of an unbounded module dict.fieldsandmasking_rulesare exclusive, rules recurse and accept JSON strings,raise_on_missing_fieldsemantics for list expressions, a warning about regex patterns on untrusted input, and the JSON-mode note under data serialization.One existing test changed:
test_erase_dictionary_with_masking_rules_with_listexpected a list under a rule to collapse into one masked string. That was the rule being applied to the Python repr of the list, which also turned the value into astr; thefieldspath next to it already masks per element, so the test now expects the same._mask_nested_fieldinbase.pyis dead code but has unit tests; left for a separate cleanup.Behaviour change: regex rule without a match
This is a security fix and I am shipping it in a minor on purpose. A field under a
regex_patternrule is declared sensitive; returning it in clear because the value did not have the expected shape (card with spaces, email in another format) is a leak, not a contract worth keeping until v4. If you used a regex rule on free text expecting non-matching text to pass through, check the match before callingerase, or accept the whole field masked. Documented in the page and called out in the release notes.User experience
Before, these calls returned with the sensitive field still in the output and no error: both parameters together, a Pydantic secret field, a JSON string with rules in lenient mode, a payload referencing one customer object twice. After, they either mask or raise. One visible change for people already using rules on list or dict values: the output keeps the structure and type instead of becoming a masked string.
Verification
tests/functional/data_maskingandtests/unit/data_masking: 101 passed, with 18 new tests covering every item above throughDataMasking.erase,prepare_dataand the provider'sdecrypt.tests/unit,tests/performance/data_masking: 742 passed;ruff,mypyandxenonclean; docs build and markdownlint clean; agent skill references regenerated.erasescenarios (fields, rules, every mask option, JSON strings, dataclass, Pydantic secrets nested and in lists,__dict__/__slots__objects referenced twice, cycles, Enum, 900 levels deep, 10k records, invalid regex and backrefs).By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.