Add load tests, a browser swarm and committed benchmark results by T4rk1n · Pull Request #4055 · plotly/dash

T4rk1n · 2026-10-07T17:37:12Z

Load tests for streaming and callbacks, a headless-browser swarm for large runs, and CI that commits the numbers to the repo, with a short table in the README.

Changes

benchmarks/streaming/: how many browsers each server setup can stream to (Flask/gunicorn, FastAPI and Quart on uvicorn, 1 or 4 workers). Simulated browsers follow the renderer's StreamClient and the count climbs until p95 frame latency or errors give out.
benchmarks/callbacks/: the same sweep for plain callbacks, HTTP vs websocket, by backend (Flask HTTP only; FastAPI and Quart both). --kind async and --work-ms add per-call work.
benchmarks/loadkit.py: the plumbing both share. It runs the server and the clients on separate CPUs, flags points where the clients ran out of CPU, and makes the server import the checkout under test, not the installed dash.
benchmarks/swarm/: real headless Chromium users (Playwright), many per machine and many machines, against a deployed app. Users click an HTTP callback, a websocket callback or a stream, timed in the page from click to DOM update, so machines only share a start time, not clocks.
- Agents run locally or over SSH in a Docker image. They report mergeable histograms.
- Steps stop when the server saturates or when the swarm stops delivering its load: failed agents or users, agents over 85% CPU, late starts.
- Meant for a one-off cloud run; see benchmarks/README.md.
benchmarks/publish.py + .github/workflows/benchmarks-publish.yml: one place for the numbers, in this repo.
- Weekly and on dispatch only, CI measures on dev. It opens a pull request that updates benchmarks/results/ and the README table only when the numbers moved significantly: a load test's capacity changes step, or a renderer median moves by over 25% and 5 ms. The pull request lists what moved.
- The per-PR renderer regression comment (benchmarks.yml) is unchanged.
- benchmarks/results/report.json has every chart as a Plotly figure plus the tables, for the docs to render with dcc.Graph. The raw numbers and their history sit next to it.
- The repo needs "Allow GitHub Actions to create and approve pull requests" turned on. A pull request opened with GITHUB_TOKEN starts no checks, so close and reopen it before merging.
- The README table says "No published run yet" until the first run.

Numbers

Callbacks, sync, 1 worker, 8-core laptop:

	p95 at 1000 browsers	p95 at 4000
Flask HTTP	3.7 ms	4.6 s (saturated from 2000)
FastAPI HTTP / websocket	1.2 / 0.9 ms	165 / 27 ms
Quart HTTP / websocket	1.9 / 1.0 ms	511 / 469 ms

With 4 workers no setup saturated at 4000 browsers; the swarm is how to find those limits. The multi-worker Redis streaming numbers need #4054.

Tests

Unit tests for the publisher (results, history, figures, README table) and for histogram merging.
A report from real runs rendered in a Dash app, headless: all 9 figures draw, no console errors, and the trend dropdown switches scenario.
Every load test and every setup ran locally.
The swarm ran locally and through a fake ssh/docker fan-out. One agent with 50 users measured about 0.05 core and up to 170 MB per user.
The swarm's Docker image builds, and one agent in it ran 10 real browsers against a local app with no errors.

benchmarks/streaming/ measures how many browsers each server setup can stream to (Flask/gunicorn, FastAPI and Quart on uvicorn, 1 or 4 workers), with simulated browsers that follow the renderer's StreamClient. benchmarks/publish.py builds a static site from the renderer timings and the streaming sweep (embeddable chart pages, raw data with history, shields badges), and benchmarks-publish.yml pushes it to gh-pages from dev.

benchmarks/callbacks/ sweeps browser counts for plain callbacks over HTTP and over the websocket on Flask (HTTP only), FastAPI and Quart, 1 and 4 workers, recording round trip, throughput, errors and CPU. The server and client plumbing it shares with the streaming test moves to benchmarks/loadkit.py.

benchmarks/swarm/ drives real Chromium users (Playwright) against a deployed app: plain HTTP callbacks, websocket callbacks and streaming, timed in the page from click to DOM update. Agents run locally or on remote hosts over SSH in a Docker image, start together at a shared wall-clock time, and report mergeable histograms; steps stop when the server saturates or when the swarm stops delivering its load (failed agents or users, agents over 85% CPU, late starts).

The benchmark site gains a callbacks section: browsers served per setup coloured by transport, p95 round trip for 1 and 4 workers (websocket dashed), server CPU per 1,000 calls a second, the full table, badges and data with history. The publish workflow runs the callback sweep weekly and on demand, like the streaming one.

github-actions · 2026-10-07T17:42:47Z

Dash performance benchmarks

⚠️ regressions to review

	scenario	metric	p90 (ms)	median	growth	baseline p90	note
⚠️	full_children_replace	replace_ms	4882.2	1612.5	21.36x	4697.1	1.3x baseline (norm)
✅	callback_chain	chain_ms	361.9	346.2	0.95x	499.1
✅	callback_chain	graph_ms	5.8	5.8	1.0x	2.4
✅	callback_fanout	fanout_ms	71.5	63.1	0.88x	92.5
✅	deep_nesting	render_ms	51.8	43.8	0.99x	56.8
✅	initial_render_large	render_ms	448.3	439.1	0.98x	694.4
✅	initial_render_small	render_ms	93.7	76.4	0.89x	104.0
✅	patch_append_nested	append_ms	128.1	92.8	2.33x	192.3
✅	patch_append_toplevel	append_ms	115.9	85.3	2.41x	140.2
✅	patch_scalar_update_large	update_ms	145.3	128.4	1.04x	202.9
✅	wildcard_all_resolve	wildcard_ms	238.1	231.2	0.95x	313.6
✅	wildcard_all_resolve	graph_ms	1.1	1.1	1.0x	1.3

growth = late-third / early-third per-op time; ~1 is flat, a large value means the per-op cost scales with accumulated state.

machine scale vs baseline: 0.80x - divided out of the baseline ratios so they compare like for like (the absolute warn/fail ceilings are left un-scaled); calibrated on initial_render_small.

Share the sweep loop between the streaming and callback runners (loadkit.sweep), split the callback client's and swarm agent's long loops into small steps, write the swarm report outside the event loop, dedupe the site builder's sections and literals, run the agent image as its unprivileged user, and pin the third-party setup-chrome action to a commit. The site output is byte-identical for the same inputs.

The playwright/python image ships the browsers but not the Python package, so the agent failed to import it.

The numbers live in benchmarks/results/ for the docs to render (report.json holds every chart as a Plotly figure), and a short table in README.md shows how many browsers each server setup keeps interactive. The workflow now runs weekly and opens a pull request with the update.

The weekly run commits a load test only when a setup's capacity changes step, and the renderer only when a median moves by over 25% and 5 ms. The pull request lists what moved.

Times dash._utils.to_json on a large figure alone and next to one value orjson cannot serialize (issue #4065), each engine in its own process. Published weekly with the other benchmarks.

sonarqubecloud · 2026-11-05T15:20:42Z

Quality Gate passed

Issues
58 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

KoolADE85

It looks like this benchmark tooling will be bundled with the published dash whl. Is that intended? Maybe we should omit benchmark code as we do with tests?

KoolADE85 · 2026-11-10T22:32:37Z

The load tests import aiohttp and require gunicorn, but these don't come with a stock venv. We should have a "how to get benchmark" section that says exactly what users need to install.. Maybe

T4rk1n added 4 commits October 7, 2026 12:10

T4rk1n mentioned this pull request Oct 7, 2026

Fix streaming on multi-worker ASGI with Redis shared storage #4054

Merged

T4rk1n added 2 commits October 7, 2026 13:51

Split the shared sweep loop into a per-setup climb and its log line

4008669

camdecoster marked this pull request as draft October 8, 2026 19:32

T4rk1n added 5 commits October 13, 2026 10:54

Install playwright in the swarm agent image

05f0792

The playwright/python image ships the browsers but not the Python package, so the agent failed to import it.

Open a benchmark results PR only when the numbers move

7768d5b

The weekly run commits a load test only when a setup's capacity changes step, and the renderer only when a median moves by over 25% and 5 ms. The pull request lists what moved.

Benchmark callback response serialization with and without orjson

a9493e3

Times dash._utils.to_json on a large figure alone and next to one value orjson cannot serialize (issue #4065), each engine in its own process. Published weekly with the other benchmarks.

Note what is left of serialization cost without orjson

f1e9fe7

T4rk1n mentioned this pull request Oct 14, 2026

Serialize only the values the JSON encoder cannot take #4067

Draft

Merge remote-tracking branch 'origin/dev' into feat/benchmarks

193596d

T4rk1n changed the title ~~Add load tests, a browser swarm and a public benchmark site~~ Add load tests, a browser swarm and committed benchmark results Nov 5, 2026

T4rk1n marked this pull request as ready for review November 5, 2026 15:19

T4rk1n mentioned this pull request Nov 6, 2026

Only reset descendant layout hashes when there is one to clear #4124

Open

4 tasks done

KoolADE85 reviewed Nov 10, 2026

View reviewed changes

Navigation Menu

Repository navigation

Add load tests, a browser swarm and committed benchmark results - #4055

Add load tests, a browser swarm and committed benchmark results#4055
T4rk1n wants to merge 12 commits into
devfrom
feat/benchmarks

T4rk1n commented Oct 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

sonarqubecloud Bot commented Nov 5, 2026

Uh oh!

KoolADE85 left a comment

Uh oh!

KoolADE85 Nov 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

T4rk1n commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Numbers

Tests

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dash performance benchmarks

Uh oh!

sonarqubecloud Bot commented Nov 5, 2026

Quality Gate passed

Uh oh!

KoolADE85 left a comment

Choose a reason for hiding this comment

Uh oh!

KoolADE85 Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

T4rk1n commented Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 7, 2026 •

edited

Loading