Skip to content

test(testkit): keep the JUnit report current during a run - #5458

Open
elezar wants to merge 1 commit into
test/5333-surface-test-stalls/elezarfrom
test/5416-incremental-junit/elezar
Open

elezar wants to merge 1 commit into
test/5333-surface-test-stalls/elezarfrom
test/5416-incremental-junit/elezar

Conversation

@elezar

@elezar elezar commented Nov 10, 2026

Copy link
Copy Markdown
Member

Stacked on #5455, which is stacked on #5450. This diff is against #5455's branch.

Summary

The testkit program wrote junit.xml only after its last test, so a run-level timeout (45 minutes) lost the report and only the log survived.

  • Rewrite the report after each test, so a run that is stopped from outside still leaves the results so far. A write error during the run is logged; the final write still decides the exit status.
  • Write to <path>.partial and rename, so a kill during a write leaves the previous report, not a truncated one.

The test that is running when the run is stopped is not in the report, since it has no result yet.

Related Issue

Part of #5416 (this item came from the review of #5332; the issue stays open). Also resolves the junit.xml note on #5333.

Testing

  • cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test pass for tests/testkit.
  • A new unit test checks that writing replaces the report and leaves no partial file.
  • No e2e lane was run.

The program wrote junit.xml only after its last test, so a run-level timeout lost the report. Rewrite it after each test, replacing the file in one step so a kill mid-write leaves the previous report.

Part of #5416.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar elezar added area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 10, 2026

@Karthik-Chowdary Karthik-Chowdary left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Disposition::Skip branch pushes an Outcome::Skipped and then continues before the new incremental JUnit checkpoint. That leaves a completed skip out of the surviving report if the supervisor is killed during the next executed test.

On head 31cd8bd488b10997f4dd3a2c52b46a233ba64dee, I exercised the unchanged program/run and registry modules with a pass → deliberate failure → ignored test → later blocking test. SIGKILL after the later test started left parseable XML with tests="2", failures="1", skipped="0": the pass and failure survived, including failure output, but the already-processed ignored test did not. This repeated three times. A skip before a completed test was retained; --include-ignored, normal completion, and graceful SIGTERM behaved as controls.

Could skipped and executed results both reach the incremental checkpoint, with a process-level regression for a skipped result immediately before the in-flight test? The in-flight test itself can remain absent as documented. Checkpointing the skip branch in a scratch-only correction retained tests="3", failures="1", skipped="1" in the same cases.

The reproduction used an isolated fixture over the upstream runner modules; it does not qualify gateway readiness or an e2e lane.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants