Skip to content

fix(data_masking): mask what the docs promise in the edge cases of erase - #8781

Merged
leandrodamascena merged 2 commits into
developfrom
fix/data-masking-improvements
Nov 9, 2026
Merged

leandrodamascena merged 2 commits into
developfrom
fix/data-masking-improvements

Conversation

@leandrodamascena

@leandrodamascena leandrodamascena commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Issue number: closes #8780, closes #8636, closes #8672

Summary

Changes

Follow-up to #8780, the edge cases where erase() returned without masking what the docs say it masks, plus the error-handling and docs items from the same list:

  • erase(data, fields=[...], masking_rules={...}) raises DataMaskingError; before, fields was silently ignored.
  • prepare_data dumps Pydantic models with model_dump(mode="json"): SecretStr/SecretBytes keep Pydantic's masked rendering instead of leaking through vars(), and UUID, datetime, Decimal fields no longer fail the fields path with TypeError.
  • masking_rules accept a JSON string like fields does (Bug: DataMasking.erase with masking_rules ignores JSON string input #8636, fix(data-masking): parse JSON string input when erasing with masking_rules #8637), and a rule pointing at a dict or list recurses into it instead of masking str(value) and changing the type.
  • prepare_data keeps only the ancestors of the current recursion path in the visited set, so the same object referenced twice is converted twice instead of being treated as a cycle (Bug: data masking prepare_data leaves dataclasses and Pydantic models unconverted #8672); objects using __slots__ are converted too. Real cycles are still cut.
  • A regex_pattern + mask_format rule whose pattern does not match the value now masks the whole value with the dynamic mask. Before, the value came back untouched.
  • Enum members are reduced to their .value before masking. Before, vars() dragged the enum class and its internals into the output.
  • KMS decrypt wraps every AWS Encryption SDK error (SerializationError, MasterKeyProviderError on a tampered header, ...), InvalidSignature, TypeError and ValueError into DataMaskingDecryptValueError, and non-base64 input too.
  • The regex cache is an lru_cache(256) instead of an unbounded module dict.
  • Docs: fields and masking_rules are exclusive, rules recurse and accept JSON strings, raise_on_missing_field semantics for list expressions, a warning about regex patterns on untrusted input, and the JSON-mode note under data serialization.

One existing test changed: test_erase_dictionary_with_masking_rules_with_list expected a list under a rule to collapse into one masked string. That was the rule being applied to the Python repr of the list, which also turned the value into a str; the fields path next to it already masks per element, so the test now expects the same.

_mask_nested_field in base.py is dead code but has unit tests; left for a separate cleanup.

Behaviour change: regex rule without a match

This is a security fix and I am shipping it in a minor on purpose. A field under a regex_pattern rule is declared sensitive; returning it in clear because the value did not have the expected shape (card with spaces, email in another format) is a leak, not a contract worth keeping until v4. If you used a regex rule on free text expecting non-matching text to pass through, check the match before calling erase, or accept the whole field masked. Documented in the page and called out in the release notes.

User experience

Before, these calls returned with the sensitive field still in the output and no error: both parameters together, a Pydantic secret field, a JSON string with rules in lenient mode, a payload referencing one customer object twice. After, they either mask or raise. One visible change for people already using rules on list or dict values: the output keeps the structure and type instead of becoming a masked string.

Verification

  • tests/functional/data_masking and tests/unit/data_masking: 101 passed, with 18 new tests covering every item above through DataMasking.erase, prepare_data and the provider's decrypt.
  • tests/unit, tests/performance/data_masking: 742 passed; ruff, mypy and xenon clean; docs build and markdownlint clean; agent skill references regenerated.
  • Real KMS in us-east-1, with this branch: round trips for dict and Pydantic payloads, then context mismatch/missing/extra key, wrong key, one byte flipped in header, middle and tail, truncation, garbage base64, non-base64, empty string and plain JSON as ciphertext. Every failure surfaces as a Data Masking exception, no plaintext and no raw SDK error. Plus 56 local erase scenarios (fields, rules, every mask option, JSON strings, dataclass, Pydantic secrets nested and in lists, __dict__/__slots__ objects referenced twice, cycles, Enum, 900 levels deep, 10k records, invalid regex and backrefs).
  • Written with an AI assistant; I ran everything above myself.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

- fields and masking_rules together now raise instead of ignoring fields
- Pydantic models are dumped in JSON mode: secret types stay masked, UUID,
  datetime and Decimal fields no longer fail the fields path
- masking_rules accept JSON strings like fields does, and recurse into dict
  and list values instead of masking their Python repr
- prepare_data tracks only the ancestors of the current path, so an object
  referenced twice is converted twice, and handles __slots__ objects
- tampered or malformed ciphertext surfaces as DataMaskingDecryptValueError
- regex cache bounded with lru_cache; docs for the behaviours above
@leandrodamascena
leandrodamascena requested a review from a team as a code owner November 9, 2026 09:40
@leandrodamascena leandrodamascena added this to the 3.40.0 milestone Nov 9, 2026
@leandrodamascena leandrodamascena added the data-masking Sensitive Data Masking feature label Nov 9, 2026
@boring-cyborg boring-cyborg Bot added documentation Improvements or additions to documentation tests labels Nov 9, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 9, 2026
@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.22222% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 97.18%. Comparing base (111f93b) to head (75bd53b).

Files with missing lines Patch % Lines
...s_lambda_powertools/utilities/data_masking/base.py 96.15% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #8781      +/-   ##
===========================================
+ Coverage    97.03%   97.18%   +0.14%     
===========================================
  Files          323      323              
  Lines        16511    16537      +26     
  Branches      1488     1494       +6     
===========================================
+ Hits         16022    16071      +49     
+ Misses         349      326      -23     
  Partials       140      140              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sonarqubecloud

sonarqubecloud Bot commented Nov 9, 2026

Copy link
Copy Markdown

@leandrodamascena
leandrodamascena merged commit 9348364 into develop Nov 9, 2026
15 checks passed
@leandrodamascena
leandrodamascena deleted the fix/data-masking-improvements branch November 9, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data-masking Sensitive Data Masking feature documentation Improvements or additions to documentation size/L Denotes a PR that changes 100-499 lines, ignoring generated files. tests

Projects

None yet

1 participant