[bcanvural]

Summary

Fixes #3508.

getPreviousAlertHistories looks up each group's newest AlertHistory row before every alert check. The {alert, group, createdAt} index is meant to make that one seek per group, but MongoDB fetched every row of the 7-day lookback for the alert. A 1-minute alert with ~1,300 groups writes ~1.9 M rows a day, so its check got slower every day. On our deployment that reached 4 minutes per run and an out-of-memory crash loop on catch-up.

Two details of the pipeline prevented the per-group plan (DISTINCT_SCAN): the compound $group key {alert, group}, and the state: {$ne: ERROR} predicate, which the old index could not answer.

This PR:

• adds state as the last field of the index: {alert: 1, group: 1, createdAt: -1, state: 1}. The ERROR exclusion becomes an index bound, so the planner keeps the DISTINCT_SCAN and steps over ERROR keys without fetching those documents;
• keeps main's single aggregation, with state: {$ne: 'ERROR'} in $match. The only change is the $group key: _id: '$group' instead of {alert, group}, because a compound _id rules out the DISTINCT_SCAN;
• returns the same map as before (callers read group || '') and uses only operators DocumentDB supports; callers are unchanged;
• drops the replaced {alert, group, createdAt} index at API startup (dropReplacedAlertHistoryIndex in runStartupMigrations), after Mongoose has built the new one. Mongoose creates indexes but never drops them.

explain() on 20 groups × 200 rows plus 200 group-less ERROR rows, the same on MongoDB 5.0.32 (CI) and 8.0.34:

	plan	keys examined	docs examined
main	IXSCAN + FETCH	4,201	4,200
this PR	DISTINCT_SCAN	221	20

At a larger scale (MongoDB 5.0.32): 1,500 groups × 300 one-minute evaluations, plus a group-less ERROR row every minute (450,300 rows): main examines 450,300 docs in ~700 ms; this PR examines 1,500 docs and 1,800 keys in 10–18 ms. Over the full 7-day window main grows with groups × evaluations (about 15 million docs for this alert); this PR stays at one doc per group, plus one key per ERROR row in the window.

Index change (for operators)

• The API builds {alert: 1, group: 1, createdAt: -1, state: 1} at startup (Mongoose autoIndex), like the index added in perf: optimize AlertHistory queries (HDX-3706) #1928. On a large AlertHistory collection this takes a while. The build holds an exclusive lock only briefly at the start and the end, and the 30-day TTL bounds the collection's size. The API serves requests during the build.
• After the build, the API drops alert_1_group_1_createdAt_-1. Until then, checks keep using it. If the new index is missing (for example, its build failed), the old index is kept.
• The check-alerts task connects with autoIndex: false and doesn't touch indexes; the API owns them.
• Downgrade: an older API recreates the old index at startup.

How to reproduce

Starting point: make dev-int-style services (MongoDB and ClickHouse from docker-compose.ci.yml); no UI involved.

On main:

1. Insert AlertHistory rows for one alert: 1,500 groups × 300 evaluations one minute apart, state: 'OK', plus a group-less ERROR row every minute.
2. explain('executionStats') on the getPreviousAlertHistories pipeline: FETCH > IXSCAN with totalDocsExamined = 450,300.

On this branch:

1. Same data: DISTINCT_SCAN, totalDocsExamined = 1,500.

How to test on Vercel preview

N/A — non-UI change.

Tests

• getPreviousAlertHistories returns the newest non-ERROR row per group, with ERROR rows with and without a group, a group that stopped reporting while in ALERT, rows without a group (OK, then a newer ERROR), and a group with only ERROR rows. It passes on main too: same results.
• The perf guard inserts group-less ERROR rows too, and asserts a single aggregate call that examines at most one document per group. It fails on main's index (4,200 docs examined vs ≤ 21).
• New: dropReplacedAlertHistoryIndex drops the old index once the new one exists, is idempotent, and keeps the old index while the new one is missing.
• The checkAlerts, alerts API and startup-migration integration tests: 522 passed.

References

• Issue: Grouped alerts: getPreviousAlertHistories reads every history row of the last 7 days, so check-alerts slows down and runs out of memory as history grows #3508
• Related PRs: none

---

Written with help from an AI assistant (Claude Code). The numbers come from the integration test setup on main and on this branch.