Skip to content

Apply rest.client timeouts to the SigV4 adapter - #4099

Open
David-Banquet wants to merge 1 commit into
apache:mainfrom
David-Banquet:fix-sigv4-timeout
Open

David-Banquet wants to merge 1 commit into
apache:mainfrom
David-Banquet:fix-sigv4-timeout

Conversation

@David-Banquet

Copy link
Copy Markdown
Contributor

Closes #4076

Rationale for this change

rest.client.connection-timeout-ms and rest.client.socket-timeout-ms are ignored when rest.sigv4-enabled is set. SigV4Adapter is mounted on the catalog URI, so requests picks it over the timeout adapter mounted on http:// and https://, and it never sets a timeout.

SigV4Adapter now extends _RetryTimeoutHTTPAdapter and gets the same timeout, computed by a small _connection_timeout helper shared with _create_connection_adapter. Its retries still come from rest.sigv4.max-retries.

Are these changes tested?

Yes. test_sigv4_adapter_applies_connection_timeout checks the adapter on the catalog URI carries the summed timeout and the SigV4 retry default. It fails without the change.

With the repro from the issue (server answering after 8 s, 1 s + 1 s timeout):

without SigV4: ConnectionError after 2.0 s
with SigV4: ConnectionError after 22.1 s   # was: response received after 8.1 s

The 22 s are 11 attempts of 2 s: SigV4Adapter retries 10 times by default with no backoff. That is the retry behavior tracked in #3008, left unchanged here.

Are there any user-facing changes?

Yes: the rest.client.* timeouts now apply to SigV4 catalogs (S3 Tables, Glue REST). Those properties are not released yet (#3418), so no released behavior changes.

#3120 replaces SigV4Adapter with a plain HTTPAdapter; if it lands first I'll rebase this onto it.

The SigV4 adapter is mounted on the catalog URI and wins over the timeout adapter, so
rest.client.connection-timeout-ms and socket-timeout-ms were ignored with SigV4 (apache#4076).
@David-Banquet

Copy link
Copy Markdown
Contributor Author

@kevinjqliu @Fokko rebased on main; the conflict with #4310 is resolved by keeping its new retry_post parameter. Could one of you take a look? The SigV4 adapter now applies the rest.client.* timeouts like the default adapter does (#4076).

@rambleraptor rambleraptor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like an easy PR. Thanks so much for the contribution!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REST: rest.client.* timeouts are ignored when rest.sigv4-enabled is set

2 participants