Repository navigation
repo-assist: Task 8 (Monthly Activity Summary) does not run deterministically despite being marked mandatory #372
Description
Summary
In repo-assist.md, Task 8 (Update Monthly Activity Summary Issue) is documented as mandatory every run:
"Always do Task 8 (Update Monthly Activity Summary Issue) every run."
In practice, on the tenstorrent/tt-metal deployment, it is not running deterministically. Over the last 7 daily scheduled runs (2026-07-24 through 2026-07-30, all completed successfully), the monthly summary issue (#50956) was only actually updated on 2 of them.
Evidence
| Date | Run | Task 8 landed? |
|---|---|---|
| 07-24 | 30072973467 | ❌ Called update_issue; safe-outputs manifest logged it as processed, but no edit ever appears on the issue's timeline. Silently dropped. |
| 07-25 | 30147764461 | ✅ add_comment landed |
| 07-26 | 30191579550 | ❌ Trace shows it read the issue and planned "Task 8: Update monthly activity summary," then never called it |
| 07-27 | 30244126524 | ✅ add_comment landed |
| 07-28 | 30335618590 | ❌ state.json memory records "task8_monthly_summary" as done; actual add_comment calls that run targeted other issues/PRs (51294, 51296, 51024) instead |
| 07-29 | 30429064534 | ❌ Same pattern — reads #50956, reasons through "Task 8," then spends its comment budget on stale-PR nudges |
| 07-30 | 30520369864 | ❌ Same again |
Issue #50956's updated_at has been stuck at 2026-07-27T07:02:56Z since — three consecutive daily runs produced zero visible output on it, despite each run's own reasoning trace explicitly citing "Task 8" as a planned step.
Root cause (best guess)
"Always do Task 8" is a prose instruction competing for attention/turn budget against the round-robin task selection in the same prompt, not a structurally-guaranteed step in the compiled workflow. Nothing fails or flags the run if the write doesn't happen. The agent's own repo-memory (state.json) isn't validated against ground truth either — it records task8_monthly_summary: done even on runs where no comment/edit actually reached the issue. On 07-24, the update_issue safe-output specifically appears to fail silently (logged as processed, but no visible effect and no error).
Question
Is there a supported pattern for making a task like this deterministic — e.g.:
- Splitting it into its own workflow/job that always runs, decoupled from the round-robin agent turn
- A safe-outputs guarantee/verification step that fails the run (or retries) if a declared "always do" output type wasn't actually emitted
- Reconciling agent-reported
state.jsonprogress against actual GitHub state rather than trusting the agent's self-report
Happy to help test a fix if there's a preferred approach.
Activity
dsyme commentedon Nov 8, 2026
Additional reproduction on current canonical revision
139fa09ef17cde4135e696ff8e7af09bc5ddfa6e:Paket run 42530420658, manually dispatched on 2026-11-08 with no command, completed successfully and applied four real labels:
dotnet publishfails with duplicate FSharp.Core.xml error fsprojects/Paket#4149:bug, 22:34:00 UTCneeds triage, 22:34:01 UTCbug, 22:34:02 UTCneeds triage, 22:34:03 UTCLive issue events confirm all four were applied by
github-actions[bot]. Memory was also persisted in commit 53954471975296a568b844a9359100e629f98cf2.The agent then emitted
noop, treating no PR/comment as no substantive work, and explicitly skipped the monthly activity update. Safe outputs successfully processed four label outputs plus the noop; no monthly issue/comment output was produced.The reporting task is now Task 11, and the current instruction requires it after work in non-command mode, including manual dispatch without a command. Genuine noops are exempt, but successful label mutations are not a genuine noop.
Suggested deterministic enforcement: derive substantive-work status from actual successful output types (including labels and existing-PR updates), rather than the agent's noop string; flag or repair a missing monthly update after such work. Preserve the genuine-noop and exclusive-command exemptions.
This evidence concerns omitted reporting, not a silently dropped monthly update: no such update was emitted. Eleven other reviewed successful runs made no external changes; missing reporting alone is not an independent violation for those, although their task assessment was incomplete.