Fix zero feedback intervals and swallowed status update failures by Brar · Pull Request #6719 · npgsql/npgsql

Brar · 2026-10-25T21:40:04Z

Depends on #6715 and #6716, which aren't merged yet; their commits show up here until they are.

A WalReceiverStatusInterval or WalReceiverTimeout of zero made the feedback timers fire over and over, which flooded the server with status updates (for the timeout with reply requests, which the server answered every time). Like wal_receiver_status_interval and wal_receiver_timeout in PostgreSQL, zero now disables them, and so do intervals longer than a timer supports (about 49 days), which failed once the server had switched to copy mode. Positive intervals below a millisecond fire every millisecond. Waiting for the end of START_REPLICATION after CopyDone is bounded for such long timeouts too.
Negative values other than Timeout.InfiniteTimeSpan are now rejected by the setters. Before, starting replication failed with them once the server had switched to copy mode, which left the connection unusable.
SendStatusUpdate logged write failures without reporting them, so callers that push their flush position (e.g. for synchronous replication) couldn't tell that it never reached the server. It now throws them; like any failed write, they break the connection. A status update that waited for one that broke the connection fails with InvalidOperationException instead of a NullReferenceException, and so does one after the stream ended. The reply to a keepalive also reports its failure, through the stream, instead of a NullReferenceException at the next read.
A log message gets its missing space ("SendStatusUpdatewas called").

How a real server gets there

On PostgreSQL 18.6 and 14.24, main against this PR:

Zero intervals: a user sets WalReceiverStatusInterval or WalReceiverTimeout to TimeSpan.Zero to switch it off, which is what 0 means for PostgreSQL's own settings; no server setting is involved. With a zero status interval, the walsender logged 100,556 status updates in 2 s. With a zero timeout, the client sent about 69,000 reply-requesting updates per second while the consumer held a message, and 224,473 keepalives came back during 1 s of reading. With this PR there were none.
Negative and too-long intervals: e.g. -1 s from a configuration in which -1 means "off", as for several PostgreSQL settings, or TimeSpan.MaxValue as .NET's "infinite". On main the first MoveNextAsync threw ArgumentOutOfRangeException from the Timer constructor after the server had entered copy mode, the next IDENTIFY_SYSTEM failed with "Received backend message CopyData while expecting RowDescriptionMessage", and the server logged FATAL "invalid standby message type". With this PR the setter throws for -1 s, and with TimeSpan.MaxValue the timer is off, and streaming and IDENTIFY_SYSTEM work.
A failed SendStatusUpdate: the server closes the connection while the consumer holds a message, e.g. because wal_sender_timeout passed without a message from the client, which can happen with a busy consumer when it is shorter than WalReceiverStatusInterval (CI uses 3 s; Npgsql's default interval is 10 s), or through pg_terminate_backend, a shutdown or a network failure. The consumer then pushes its position with SetReplicationStatus and SendStatusUpdate. Main returned normally and only logged the failure (event 50103); this PR throws NpgsqlException with the IOException inside.
An update that waited for a failing one: two status updates are in flight when the connection dies, e.g. SendStatusUpdate from two threads, or together with Npgsql's keepalive reply or timers. Forced with a suspended walsender, so that the first write blocked and the second update waited: main returned normally from both calls and only logged an IOException and a NullReferenceException; with this PR the first call threw the write failure and the second InvalidOperationException. Without forcing, the window is the duration of one failing write.
A keepalive reply that can't be written: the consumer holds a message while the server's reply request sits in the buffer behind it, and then the walsender goes away. When the consumer resumes, Npgsql answers the request, and the write fails. Main logged the failure, and the next MoveNextAsync threw NullReferenceException; this PR throws NpgsqlException with the IOException inside, and the connection is broken.

Tests

ReplicationFeedbackMockTests (8 cases, mock): all of them fail on #6716's code. #6630's live test SendStatusUpdate_racing_replication_teardown_throws_InvalidOperationException passes on top of this PR.

For the reviewer

No public API change. Behavior changes, also in the 11.0 release notes: zero disables the interval and the timeout, negative values throw ArgumentOutOfRangeException, and SendStatusUpdate throws write failures.
This builds on Fix teardown NRE/ODE in ReplicationConnection's SendStatusUpdate #6630 (Hawxy's fix of SendStatusUpdate racing the end of a stream); SendFeedback's new throwOnFailure replaces its OperationCanceledException catch.
The handling of zero and negative intervals is meant to go to the 10.0 hotfix branch once this is merged.

Dependencies and merge order

Contains the commits of Handle the end of a replication stream at timeline switches and shutdown #6715 and Stop replication streams properly when they are disposed early #6716; merge those first. Only the last commit is this PR's own; the others drop out when the branch is rebased onto main after they land.
Land it together with Report why the connection broke while a replication message was held #6720: after SendStatusUpdate threw a write failure, the consumer's next MoveNextAsync throws a NullReferenceException, as it already does on main after a failed status update; that PR fixes it.

Fixes #6698

- At the end of a historic timeline (e.g. after the standby we stream from got promoted), the server ends copy mode with CopyDone. Npgsql expected only CopyData and broke the connection ("Received backend message CopyDone while expecting CopyDataMessage"). It now answers with CopyDone, reads the next timeline the server sends, both CommandCompletes and ReadyForQuery, and ends the enumeration normally, leaving the connection usable. Status updates stop before our CopyDone, since the protocol doesn't allow CopyData after it. The next timeline is logged, and the StartReplication docs explain how to continue: IDENTIFY_SYSTEM returns the current timeline and TIMELINE_HISTORY the position where it branched off, which can be before the end of the WAL received on the old timeline. - Asked to start at the end of a historic timeline, the server doesn't enter copy mode but sends the next timeline right away, which Npgsql treated as an unexpected message. The enumeration now ends right away. - A walsender that exits after everything has been replicated (e.g. at server shutdown, npgsql#5752) sends CommandComplete, which also broke the connection with a misleading error. It now throws an NpgsqlException saying that the server ended the replication stream. - In logical replication, unknown CopyData message types are skipped instead of breaking the connection, like the logical replication worker, which ignores them on purpose. Physical replication keeps rejecting them, like walreceiver, since it must not lose WAL. Keepalive fields that newer servers may add are skipped in both. - In physical replication, the received position reported to the server was the server's own WAL end, not the end of the WAL received, so pg_stat_replication.write_lsn and synchronous_commit = remote_write treated WAL as written before it arrived. It is now the end of the data the consumer is done with, like walreceiver reports what it wrote. - LastReceivedLsn starts over at the start position of each stream instead of keeping the maximum of all previous streams, which was wrong when continuing on a new timeline at a lower position. - The XLogData stream is disposed after each message instead of later in cleanup, where it could skip data based on a stale position: a server error that ended the stream after a keepalive (e.g. after pg_cancel_backend) turned into a timeout and a broken connection. The new cluster tests stream physically across a standby's promotion and through a fast shutdown, using the throwaway clusters from NPGSQL_TEST_PG_BIN. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Breaking out of a replication enumeration, or an exception in the consumer, left the inner enumerator undisposed. The connection stayed in replication mode with its feedback timers running, and every other command failed with NpgsqlOperationInProgressException. The enumerators are now disposed, which stops streaming: physical replication ends copy mode with CopyDone, which keeps the walsender ready for the next START_REPLICATION and keeps temporary slots; logical replication cancels the command, since a logical walsender can't stream again after CopyDone. Like any error, the cancellation drops the temporary slots of the session (documented on the logical StartReplication methods). Servers before PostgreSQL 10 ignore cancellation requests on replication connections, so with them the logical stop breaks the connection instead. A final status update reports the processed position first; for logical replication this is best effort, since the walsender may handle the cancellation before it reads the update. If stopping fails, the connection is broken; the exception that made the consumer stop is never replaced. On PostgreSQL 14-17, the server may not send pgoutput's Relation messages again in the next stream on the same connection after such a cancellation (on 14 after any error, on 15-17 if the connection created the slot with an exported snapshot; fixed in PostgreSQL 18 by 80b727eb9de). This is documented on the pgoutput StartReplication, and the test avoids it. - Status updates that were still on their way when a stream ended (e.g. from a timer or another thread) could be written after the end of the command, interleaved with the next one. They are now dropped. - Whatever the consumer didn't read of the message it held is skipped first, also when a plugin like pgoutput reads the data directly from the buffer. Reading beyond the end of a message breaks the connection. - After cancellation, messages that were already on their way were still handed out until the server's error arrived. They are now skipped. This makes the physical replication tests reliable, so they run again (some of them also used colliding slot names). - Starting a second stream while one was in progress overwrote the state of the first one before failing, so disposing the connection didn't stop the first stream anymore. It now fails before touching anything, and a start that fails leaves no state behind. - A message bigger than the read buffer (e.g. a long notice) made the connector switch to a bigger buffer while the stream kept reading from the old one. This is fixed for the replication stream itself; the pgoutput decoder still reads from the old buffer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- A WalReceiverStatusInterval or WalReceiverTimeout of zero made the feedback timers fire over and over, which flooded the server with status updates (in the case of the timeout with reply requests, which the server answered every time). Like wal_receiver_status_interval and wal_receiver_timeout in PostgreSQL, zero now disables them, and so do intervals longer than a timer supports (~49 days), which failed once the server had switched to copy mode. Positive intervals below a millisecond fire every millisecond. Waiting for the end of START_REPLICATION after CopyDone is bounded for such long timeouts too. - Negative values other than Timeout.InfiniteTimeSpan are now rejected by the setters. Before, starting replication failed with them once the server had switched to copy mode, which left the connection unusable. - SendStatusUpdate logged write failures without reporting them, so callers that push their flush position (e.g. for synchronous replication) couldn't tell that it never reached the server. It now throws them; like any failed write, they break the connection. A status update that waited for one that broke the connection fails with InvalidOperationException instead of a NullReferenceException, and so does one after the stream ended. The reply to a keepalive also reports its failure, through the stream, instead of a NullReferenceException at the next read. - Add a missing space to a log message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

Active findings identify cross-stream feedback races, incorrect WAL acknowledgement timing that can stall synchronous replication or acknowledge unread data, and an unbounded payload drain that can hang cleanup.

3 open issues

What changed in this PR

This PR fixes replication feedback timing and exposes status-update write failures in Npgsql. It also includes pending dependency changes for stream termination, early disposal, and physical WAL position reporting.

Changes:

Disable feedback timers for zero or oversized intervals, reject invalid negative values, and propagate explicit status-update failures.
Handle timeline endings and early stream disposal while coordinating feedback with teardown.
Add mock and live regression tests, lifecycle logging, and replication usage documentation.

File	Description
`test/Npgsql.Tests/Support/PgServerMock.cs`	Adds a nonblocking check for pending client data.
`test/Npgsql.Tests/Replication/ReplicationStreamEndMockTests.cs`	Tests stream endings, protocol handling, and received positions.
`test/Npgsql.Tests/Replication/ReplicationStopTests.cs`	Tests early disposal and connection reuse against PostgreSQL.
`test/Npgsql.Tests/Replication/ReplicationStopMockTests.cs`	Tests cleanup, cancellation, and buffer changes.
`test/Npgsql.Tests/Replication/ReplicationFeedbackMockTests.cs`	Tests disabled intervals, validation, and feedback failures.
`test/Npgsql.Tests/Replication/PhysicalReplicationTests.cs`	Re-enables tests without parallel execution and avoids slot-name collisions.
`test/Npgsql.Tests/Replication/PhysicalReplicationClusterTests.cs`	Tests standby promotion and server shutdown.
`src/Npgsql/Replication/TestDecoding/TestDecodingConnectionExtensions.cs`	Documents early-stop behavior and temporary-slot loss.
`src/Npgsql/Replication/ReplicationConnection.cs`	Implements timer validation, failure propagation, and stream lifecycle handling.
`src/Npgsql/Replication/PhysicalReplicationConnection.cs`	Disposes inner streams and documents timeline continuation.
`src/Npgsql/Replication/PgOutput/PgOutputConnectionExtensions.cs`	Documents early stopping and server-version reuse limitations.
`src/Npgsql/Replication/Internal/LogicalReplicationConnectionExtensions.cs`	Disposes inner streams and initializes their received position.
`src/Npgsql/NpgsqlEventId.cs`	Adds replication lifecycle event identifiers.
`src/Npgsql/LogMessages.cs`	Adds timeline-ending and stream-stopping log messages.

_{Review effort: Balanced}

Brar · 2026-11-08T11:00:33Z

+        // What we received on a previous stream doesn't count for this one, which may e.g. continue on another timeline at a
+        // lower position
+        Interlocked.Exchange(ref _lastReceivedLsn, unchecked((long)(ulong)walLocation));
+        _statusUpdatesStopped = false;


That's #6716's code, whose commit shows up here until it's merged. Fixed there in a4659cf: status updates stay stopped until the server is in copy mode.

Brar · 2026-11-08T11:00:34Z

+                    var physicalReceivedLsn = startLsn + (ulong)dataLen;
+                    if (ReplicationMode == ReplicationMode.Physical && unchecked((ulong)Interlocked.Read(ref _lastReceivedLsn)) < physicalReceivedLsn)
+                        Interlocked.Exchange(ref _lastReceivedLsn, unchecked((long)physicalReceivedLsn));


That's #6715's code, whose commit shows up here until it's merged. Both halves are fixed there in 1d76446: a message's data counts as received as it arrives, so a fully read message counts before the next MoveNextAsync, and unread data only once skipping it has received it.

Brar · 2026-11-08T11:00:36Z

+            // Skip whatever the consumer didn't read of the message it held
+            if (bypassingStream)
+                await SkipToPayloadEnd(connector, payloadEnd).ConfigureAwait(false);
+            else
+                await columnStream.DisposeAsync().ConfigureAwait(false);


That's #6716's code, whose commit shows up here until it's merged. Fixed there in a4659cf: the skip now waits at most WalReceiverTimeout, or 30 s if that's infinite, like the rest of stopping.

Brar requested review from roji and vonzshik as code owners October 25, 2026 21:40

This was referenced Oct 25, 2026

Report why the connection broke while a replication message was held #6720

Open

Document the replication and COPY fixes in 11.0 npgsql/doc#478

Draft

Add copy-both and replication support to the test server mock #6689

Merged

Brar and others added 3 commits November 8, 2026 00:08

Brar force-pushed the repl/feedback-fixes branch from c9bf1df to daf84b2 Compare November 7, 2026 23:34

Copilot AI balanced review requested due to automatic review settings November 7, 2026 23:34

Copilot started reviewing on behalf of Brar November 7, 2026 23:38

View session

Copilot AI reviewed Nov 7, 2026

View reviewed changes

Navigation Menu

Repository navigation

Fix zero feedback intervals and swallowed status update failures - #6719

Fix zero feedback intervals and swallowed status update failures#6719
Brar wants to merge 3 commits into
npgsql:mainfrom
Brar:repl/feedback-fixes

Brar commented Oct 25, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Brar Nov 8, 2026

Uh oh!

Brar Nov 8, 2026

Uh oh!

Brar Nov 8, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Brar commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How a real server gets there

Tests

For the reviewer

Dependencies and merge order

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Brar Nov 8, 2026

Choose a reason for hiding this comment

Uh oh!

Brar Nov 8, 2026

Choose a reason for hiding this comment

Uh oh!

Brar Nov 8, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Brar commented Oct 25, 2026 •

edited

Loading