[iacsha]

Fixes #566.

Problem

ObjectJSONSerializer converts an object to XML with XStream, then parses that XML to build the JSON. XStream writes characters that XML 1.0 doesn't allow, such as U+FFFE, U+FFFF or a C0 control other than tab, LF or CR, as character references like . The XML parser in JsonXmlUtil.xmlToJson rejects those references, so one code template with such a character in its code fails GET /api/codeTemplates as JSON for the whole list:

javax.xml.stream.XMLStreamException: ParseError at [row,col]:[...]
Message: Character reference "&#xfffe" is an invalid XML character.

The same path serves channels, global scripts and anything else with free text, so they were exposed too.

Fix

In ObjectJSONSerializer, before parsing:

1. Find each character reference to a code point that XML 1.0 doesn't allow (up to U+FFFF).
2. Replace it with a placeholder: a private-use character that doesn't occur anywhere in that document, followed by the four hex digits.
3. After the conversion, turn each placeholder into a JSON \uXXXX escape.

The characters come through unchanged instead of being stripped. Documents without such references take the same path as before. Legal character references are left alone.

This doesn't add the other option from the issue, rejecting such characters on save. That could still be useful, but it's a separate change.

Tests

Two new ObjectJSONSerializerTest cases:

• A code template whose code contains U+FFFE, U+FFFF, U+0001, U+001F and U+0008 serializes, and the parsed JSON contains the code unchanged.
• A channel description that contains U+E000–U+E002 and text that looks like a placeholder, next to a U+FFFE, comes back unchanged. This shows the placeholder can't collide with existing text.

Both fail on main with the parse error above. :server:test is green.

I haven't changed or tested the reverse direction, JSON → XML when saving through the JSON API.

[github-actions]

Test Results

  142 files  ±0    142 suites  ±0   3m 30s ⏱️ +16s
  939 tests +2    939 ✅ +2  0 💤 ±0  0 ❌ ±0 
1 101 runs  +2  1 095 ✅ +2  6 💤 ±0  0 ❌ ±0 

Results for commit e7517ff. ± Comparison against base commit cd802ef.

[mgaffigan]

This is a lot, in a core area. We need a cleaner solution - probably disallowing the character in the first place, if it is not representable.

We should not have to be doing conversions or hacks.