EOF occurred in violation of protocol starting Python3.10 on large requests

Bug report

Bug description:

We have identified a regression starting with Python 3.10 when making a large request over TLS.

import requests
requests.post('https://google.com', data=b'A'*1000000)

urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:2426)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))

The same request works fine on Python 3.9.

CPython versions tested on:

3.10, 3.11, 3.12

Operating systems tested on:

Linux

Linked PRs

gh-110467: Fix EOF occurred in violation of protocol starting Python3.10 on large requests #115273

This issue tracker is for CPython. This looks like a bug in either https://github.com/psf/requests or https://github.com/urllib3/urllib3, please report to one of their trackers.

My mistake. Here is the same issue without urllib3 or requests:

import http.client
http.client.HTTPSConnection("www.google.com").request('GET', '/', body=b'A'*1000000)

No issue on Python3.9, crashes on 3.10 onward with the following statck trace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1077, in _send_output
    self.send(chunk)
  File "/usr/local/lib/python3.10/http/client.py", line 999, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.10/ssl.py", line 1270, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/lib/python3.10/ssl.py", line 1239, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2426)

Thanks, reopening as reproducible using the stdlib.

Is the same version of OpenSSL used with Python 3.9 and Python 3.10?

Looking a bit more closely, sending a body using a GET request is not supported by the HTTP standard. www.google.com is just forcefully closing the socket at some point because the request cannot be valid. The cutoff appears to be between 100KB and 1MB. While it could be interesting to look at the difference between Python 3.9 and Python 3.10 (and the version of OpenSSL used), nothing seems wrong here.

In urllib3, sending multiple gigabytes of data over OpenSSL is part of our test suite, and it does work well on Linux. (We do have an issue with that test being slow on macOS.)

The Python 3.10 release notes also mention changes around ssl.OP_IGNORE_UNEXPECTED_EOF that seem relevant.

To help understand the difference between Python3.9 and Python 3.10, SSLKEYLOGFILE can be used to decrypt the TLS traffic. I am using requests because it is convenient to use.

import os
import requests
os.environ["SSLKEYLOGFILE"] = "secrets.log"
req = requests.get('https://google.com')
req = requests.post('https://google.com', data=b'A'*100000)

The secrets.log can then be opened Wireshark, in Edit > Preferences > Protocols > TLS and set in (Pre)-Master-Secret log filename.

From the traffic, the TLS does break on Python 3.10.

Is there a way to solve this other than moving to a version of Python other than 3.10?

import http.client
import requests

try :
    print("START send http.client.HTTPSConnection")
    http.client.HTTPSConnection("google.com").request('POST', '/', body=b'A'*1000000)
    print("END send http.client.HTTPSConnection")
except Exception as e:
    print(e)

try :
    print("START send requests.post")
    requests.post('https://google.com/', data=b'A'*1000000)
    print("END send requests.post")
except Exception as e:
    print(e)

result :

orange@32thread-server:~/project/pythonssleof$ python3.9 main.py
START send http.client.HTTPSConnection
[Errno 32] Broken pipe
START send requests.post
END send requests.post
orange@32thread-server:~/project/pythonssleof$ python3.10 main.py
START send http.client.HTTPSConnection
EOF occurred in violation of protocol (_ssl.c:2426)
START send requests.post
HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))
orange@32thread-server:~/project/pythonssleof$

Hello,

It seems that it is rather the right situation for an error to occur in the above test case situation.

Python 3.9 also encountered errors when sending large amounts of data to Google with HTTPS Connection. However, 'requests' does not cause an error. Python 3.10 is error in both transmission methods. This is not an error that occurred in python3.10, but simply seems to have failed to deal with exceptions in the implementation of requests or urlib3 as the type of exception changed.

In this regard, I think the problem is that requests in python3.9 version did not return the exception. Even in python 3.9, the data sent by the user through requests would have been interrupted without being fully transmitted. Users should also be aware that their data has not been sent successfully. This is vulnerable to tls truncation attack.

Now that this won't be picked up, is there a way we can circumvent it?

Like they have done here: websocket-client/websocket-client#942

I encountered the error while using Google API python client as it uses http.Client under the hood.

I think the original python 3.9 error format changed from 3.10 is the cause of the issue now. I need to investigate why the error type changed from 3.10. I'll check if there's a good reason for the change or if it's simply a side effect by the upgrade and upload the investigation back here.

Bump Pulp squeezer to latest version

Add retry/backoff to YouTube credential check for transient SSL failures

Navigation Menu

Repository navigation

EOF occurred in violation of protocol starting Python3.10 on large requests #110467

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

15 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

EOF occurred in violation of protocol starting Python3.10 on large requests #110467

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

hugovk commented on Oct 6, 2023

3asm commented on Oct 6, 2023

hugovk commented on Oct 8, 2023

pquentin commented on Oct 8, 2023

pquentin commented on Oct 8, 2023

pquentin commented on Oct 8, 2023

3asm commented on Oct 10, 2023

narayanacharya6 commented on Nov 6, 2023

keepworking commented on Jan 5, 2024

ankitmaloo commented on Jan 19, 2024

keepworking commented on Jan 23, 2024

15 remaining items

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions