SMTP EmailPolicy not using the correct line length for RCF 2045 encoded data (is 78, should be 76)

@warsaw

BPO	34025
Nosy	@warsaw, @bitdancer, @maxking, @AldarisPale, @dougthor42

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-07-02.16:18:14.042>
labels = ['type-bug', 'expert-email']
title = 'SMTP EmailPolicy not using the correct line length for RCF 2045 encoded data (is 78, should be 76)'
updated_at = <Date 2021-06-14.18:03:15.960>
user = 'https://github.com/dougthor42'

bugs.python.org fields:

activity = <Date 2021-06-14.18:03:15.960>
actor = 'Douglas Thor'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['email']
creation = <Date 2018-07-02.16:18:14.042>
creator = 'Douglas Thor'
dependencies = []
files = []
hgrepos = []
issue_num = 34025
keywords = []
message_count = 3.0
messages = ['320898', '320905', '324150']
nosy_count = 5.0
nosy_names = ['barry', 'r.david.murray', 'maxking', 'AldarisPale', 'dougthor42']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34025'
versions = ['Python 3.5']

It appears that the SMTP EmailPolicy object does not correctly set max_line_length.

RFC 2045 (https://www.ietf.org/rfc/rfc2045.txt) requires a max_line_length of 76 characters, while email._policybase.Policy sets it to 78 (which typically is correct).

This causes email attachments to be truncated/corrupted.

While the workaround is quite trivial, debugging the root cause was not as easy. Thus I think this should be fixed in at the std lib level, or at the very least mentioned in the docs.

This was tested on Python 3.5.2. I did not test on 3.7, but I did check the source code and it appears that the bug still exists.

To reproduce:

    import smtplib
    from email.message import EmailMessage
    from email.policy import SMTP

    msg = EmailMessage()
    msg['Subject'] = "Test message"
    msg['To'] = "to@valid_address.com"
    msg['From'] = "from@valid_address.com"
    content = "hello world, I should have an attachment"
    msg.set_content(content)

    maintype = 'image'
    subtype = 'png'
    test_file = './aaa.png'

    with open(test_file, 'rb') as fp:
        data = fp.read()
    msg.add_attachment(data,
                       maintype=maintype,
                       subtype=subtype,
                       filename='aaa.png')

    with smtplib.SMTP_SSL('smtp.address.com', port=465) as s:
        s.ehlo()
        user = 'smtp_user@valid_address.com'
        pw = 'smtp_password'
        s.login(user, pw)
        s.send_message(msg)

Check the raw email message received. You'll see that lines have a length of 78 and are padded with "==" which makes the total length 80 chars:

...
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1585876390557366757=="

--===============1585876390557366757==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

hello world, I should have an attachment

--===============1585876390557366757==
Content-Type: image/png
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="aaa.png"
MIME-Version: 1.0

iVBORw0KGgoAAAANSUhEUgAABE0AAALFCAYAAAA2rB2uAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjw==
C/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAD/jklEQVR4Xg==
7P1rsHVJWecBvl4RFFAuIpcCEYQuLkUDBcgBoQsULLnJXaoaEApe5FLSTVFUC4JACQ1NiQdsQoFBAQ==
iW4aQSWEt4cQwwB6xqC+CPYHqX5jQno+jHyYCCMmZsKJmYjZs3/rnOe8uXNnrpW5LrlzrfVfVSfOeQ==
986Vl/+T678y//nkk9+3OXdu+78uISAEhIAQEAJCQAgIASEgBISAEBACQkAICIEdBBBN9CMM1AfUBw==
...

Next, modify the code with:

    from email.policy import SMTP
    smtp_pol = SMTP.clone(max_line_length=76)
    msg = EmailMessage(policy=smtp_pol)
    msg['Subject'] = ...

And check the raw email again. This time, lines are 76 chars long and have no padding. Data is the same, just wrapped differently as expected.

...
MIME-Version: 1.0
Content-Type: multipart/mixed;
 boundary="===============4874323426669347622=="

--===============4874323426669347622==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

hello world, I should have an attachment

--===============4874323426669347622==
Content-Type: image/png
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="aaa.png"
MIME-Version: 1.0

iVBORw0KGgoAAAANSUhEUgAABE0AAALFCAYAAAA2rB2uAAAAAXNSR0IArs4c6QAAAARnQU1BAACx
jwv8YQUAAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAAOpgAABdwnLpRPAAA/45JREFU
eF7s/WuwdUlZ5wG+XhEUUC4ilwIRhC4uRQMFyAGhCxQsucldqhoQCl7kUtJNUVQLgkAJDU2JB2xC
gUEBiW4aQSWEt4cQwwB6xqC+CPYHqX5jQno+jHyYCCMmZsKJmYjZs3/rnOe8uXNnrpW5LrlzrfVf
...

This fix *should* be extremely easy:

diff --git "a/policy.py" "b/policy.py"
index 6ac64a5..046b788 100644
--- "a/policy.py"
+++ "b/policy.py"
@@ -209,6 +209,6 @@ default = EmailPolicy()
 # Make the default policy use the class default header_factory
 del default.header_factory
 strict = default.clone(raise_on_defect=True)
-SMTP = default.clone(linesep='\r\n')
+SMTP = default.clone(linesep='\r\n', max_line_length=76)
 HTTP = default.clone(linesep='\r\n', max_line_length=None)
 SMTPUTF8 = SMTP.clone(utf8=True)

The default maximum line length is indeed supposed to be 78 (characters..and I can't now remember whether I implemented it in terms of characters or octets), that's per RFC 5322. What is wrong is that content encoded text is supposed to use a max line length of 76.

A "simple" fix would be to hardcode the maximum line length at 76 when doing content encoding. An arguably better fix would be a separate policy control for the encoded text line length.

The former can be the bug fix, but it would be nice to have the latter as an enhancement.

Hi!

I hit this bug on python 3.4.

For the reference here are two email sources. The first one works while the second one does not. Both were produced using smtplib the only difference being the workaround proposed by Douglas Thor.
I hit this bug when using K-9 mail for reading e-mails.
Relevant github link: thunderbird/thunderbird-android#1659 (comment)

This one works:
---------------

Return-Path: <sender@senderdomain.com>
X-Original-To: receiver@receiverdomain.com
Delivered-To: receiver@receiverdomain.com
Received: from [127.0.0.1] (smtp [127.0.0.1])
by smtp.receiverdomain.com (Postfix) with ESMTP id 2697F23C0063
for <receiver@receiverdomain.com>; Mon, 27 Aug 2018 09:20:15 +0300 (EEST)
Subject: Malformed base64
From: sender@senderdomain.com
To: receiver@receiverdomain.com
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Message-Id: <20180827062015.2697F23C0063@smtp.receiverdomain.com>
Date: Mon, 27 Aug 2018 09:20:15 +0300 (EEST)

Q09OVEVOVF9MRU5HVEg6IDEyMzQ1Njc4OTBhYmNkZWYNCkNPTlRFTlRfVFlQRTogMTIzNDU2Nzg5
MGFiY2RlZg0KQ09OVEVYVF9ET0NVTUVOVF9ST09UOiAxMjM0NTY3ODkwYWJjZGVmDQpDT05URVhU
X1BSRUZJWDogMTIzNDU2Nzg5MGFiY2RlZg0KRE9DVU1FTlRfUk9PVDogMTIzNDU2Nzg5MGFiY2Rl
Zg0KR0FURVdBWV9JTlRFUkZBQ0U6IDEyMzQ1Njc4OTBhYmNkZWYNCkhUVFBTOiAxMjM0NTY3ODkw
YWJjZGVmDQpIVFRQX0FDQ0VQVDogMTIzNDU2Nzg5MGFiY2RlZg0KSFRUUF9BQ0NFUFRfRU5DT0RJ
Tkc6IDEyMzQ1Njc4OTBhYmNkZWYNCkhUVFBfQUNDRVBUX0xBTkdVQUdFOiAxMjM0NTY3ODkwYWJj
ZGVmDQpIVFRQX0NBQ0hFX0NPTlRST0w6IDEyMzQ1Njc4OTBhYmNkZWYNCkhUVFBfQ09OTkVDVElP
TjogMTIzNDU2Nzg5MGFiY2RlZg0KSFRUUF9ETlQ6IDEyMzQ1Njc4OTBhYmNkZWYNCkhUVFBfSE9T
VDogMTIzNDU2Nzg5MGFiY2RlZg0KSFRUUF9PUklHSU46IDEyMzQ1Njc4OTBhYmNkZWYNCkhUVFBf
UkVGRVJFUjogMTIzNDU2Nzg5MGFiY2RlZg0KSFRUUF9VUEdSQURFX0lOU0VDVVJFX1JFUVVFU1RT
OiAxMjM0NTY3ODkwYWJjZGVmDQpIVFRQX1VTRVJfQUdFTlQ6IDEyMzQ1Njc4OTBhYmNkZWYNClBB
VEg6IDEyMzQ1Njc4OTBhYmNkZWYNClFVRVJZX1NUUklORzogMTIzNDU2Nzg5MGFiY2RlZg0KUkVE
SVJFQ1RfSFRUUFM6IDEyMzQ1Njc4OTBhYmNkZWYNClJFRElSRUNUX1NTTF9UTFNfU05JOiAxMjM0
NTY3ODkwYWJjZGVmDQpSRURJUkVDVF9TVEFUVVM6IDEyMzQ1Njc4OTBhYmNkZWYNClJFRElSRUNU
X1VOSVFVRV9JRDogMTIzNDU2Nzg5MGFiY2RlZg0KUkVESVJFQ1RfVVJMOiAxMjM0NTY3ODkwYWJj
ZGVmDQpSRU1PVEVfQUREUjogMTIzNDU2Nzg5MGFiY2RlZg0KUkVNT1RFX1BPUlQ6IDEyMzQ1Njc4
OTBhYmNkZWYNClJFUVVFU1RfTUVUSE9EOiAxMjM0NTY3ODkwYWJjZGVmDQpSRVFVRVNUX1NDSEVN
RTogMTIzNDU2Nzg5MGFiY2RlZg0KUkVRVUVTVF9VUkk6IDEyMzQ1Njc4OTBhYmNkZWYNClNDUklQ
VF9GSUxFTkFNRTogMTIzNDU2Nzg5MGFiY2RlZg0KU0NSSVBUX05BTUU6IDEyMzQ1Njc4OTBhYmNk
ZWYNClNFUlZFUl9BRERSOiAxMjM0NTY3ODkwYWJjZGVmDQpTRVJWRVJfQURNSU46IDEyMzQ1Njc4
OTBhYmNkZWYNClNFUlZFUl9OQU1FOiAxMjM0NTY3ODkwYWJjZGVmDQpTRVJWRVJfUE9SVDogMTIz
NDU2Nzg5MGFiY2RlZg0KU0VSVkVSX1BST1RPQ09MOiAxMjM0NTY3ODkwYWJjZGVmDQpTRVJWRVJf
U0lHTkFUVVJFOiAxMjM0NTY3ODkwYWJjZGVmDQpTRVJWRVJfU09GVFdBUkU6IDEyMzQ1Njc4OTBh
YmNkZWYNClNTTF9UTFNfU05JOiAxMjM0NTY3ODkwYWJjZGVmDQpVTklRVUVfSUQ6IDEyMzQ1Njc4
OTBhYmNkZWYNCg==
----------------

This one does not:
------------------

Return-Path: <sender@senderdomain.com>
X-Original-To: receiver@receiverdomain.com
Delivered-To: receiver@receiverdomain.com
Received: from [127.0.0.1] (smtp [127.0.0.1])
by smtp.receiverdomain.com (Postfix) with ESMTP id 0D25523C0063
for <receiver@receiverdomain.com>; Mon, 27 Aug 2018 09:21:13 +0300 (EEST)
Subject: Malformed base64
From: sender@senderdomain.com
To: receiver@receiverdomain.com
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Message-Id: <20180827062113.0D25523C0063@smtp.receiverdomain.com>
Date: Mon, 27 Aug 2018 09:21:13 +0300 (EEST)

Q09OVEVOVF9MRU5HVEg6IDEyMzQ1Njc4OTBhYmNkZWYKQ09OVEVOVF9UWVBFOiAxMjM0NTY3ODkwYQ==
YmNkZWYKQ09OVEVYVF9ET0NVTUVOVF9ST09UOiAxMjM0NTY3ODkwYWJjZGVmCkNPTlRFWFRfUFJFRg==
SVg6IDEyMzQ1Njc4OTBhYmNkZWYKRE9DVU1FTlRfUk9PVDogMTIzNDU2Nzg5MGFiY2RlZgpHQVRFVw==
QVlfSU5URVJGQUNFOiAxMjM0NTY3ODkwYWJjZGVmCkhUVFBTOiAxMjM0NTY3ODkwYWJjZGVmCkhUVA==
UF9BQ0NFUFQ6IDEyMzQ1Njc4OTBhYmNkZWYKSFRUUF9BQ0NFUFRfRU5DT0RJTkc6IDEyMzQ1Njc4OQ==
MGFiY2RlZgpIVFRQX0FDQ0VQVF9MQU5HVUFHRTogMTIzNDU2Nzg5MGFiY2RlZgpIVFRQX0NBQ0hFXw==
Q09OVFJPTDogMTIzNDU2Nzg5MGFiY2RlZgpIVFRQX0NPTk5FQ1RJT046IDEyMzQ1Njc4OTBhYmNkZQ==
ZgpIVFRQX0ROVDogMTIzNDU2Nzg5MGFiY2RlZgpIVFRQX0hPU1Q6IDEyMzQ1Njc4OTBhYmNkZWYKSA==
VFRQX09SSUdJTjogMTIzNDU2Nzg5MGFiY2RlZgpIVFRQX1JFRkVSRVI6IDEyMzQ1Njc4OTBhYmNkZQ==
ZgpIVFRQX1VQR1JBREVfSU5TRUNVUkVfUkVRVUVTVFM6IDEyMzQ1Njc4OTBhYmNkZWYKSFRUUF9VUw==
RVJfQUdFTlQ6IDEyMzQ1Njc4OTBhYmNkZWYKUEFUSDogMTIzNDU2Nzg5MGFiY2RlZgpRVUVSWV9TVA==
UklORzogMTIzNDU2Nzg5MGFiY2RlZgpSRURJUkVDVF9IVFRQUzogMTIzNDU2Nzg5MGFiY2RlZgpSRQ==
RElSRUNUX1NTTF9UTFNfU05JOiAxMjM0NTY3ODkwYWJjZGVmClJFRElSRUNUX1NUQVRVUzogMTIzNA==
NTY3ODkwYWJjZGVmClJFRElSRUNUX1VOSVFVRV9JRDogMTIzNDU2Nzg5MGFiY2RlZgpSRURJUkVDVA==
X1VSTDogMTIzNDU2Nzg5MGFiY2RlZgpSRU1PVEVfQUREUjogMTIzNDU2Nzg5MGFiY2RlZgpSRU1PVA==
RV9QT1JUOiAxMjM0NTY3ODkwYWJjZGVmClJFUVVFU1RfTUVUSE9EOiAxMjM0NTY3ODkwYWJjZGVmCg==
UkVRVUVTVF9TQ0hFTUU6IDEyMzQ1Njc4OTBhYmNkZWYKUkVRVUVTVF9VUkk6IDEyMzQ1Njc4OTBhYg==
Y2RlZgpTQ1JJUFRfRklMRU5BTUU6IDEyMzQ1Njc4OTBhYmNkZWYKU0NSSVBUX05BTUU6IDEyMzQ1Ng==
Nzg5MGFiY2RlZgpTRVJWRVJfQUREUjogMTIzNDU2Nzg5MGFiY2RlZgpTRVJWRVJfQURNSU46IDEyMw==
NDU2Nzg5MGFiY2RlZgpTRVJWRVJfTkFNRTogMTIzNDU2Nzg5MGFiY2RlZgpTRVJWRVJfUE9SVDogMQ==
MjM0NTY3ODkwYWJjZGVmClNFUlZFUl9QUk9UT0NPTDogMTIzNDU2Nzg5MGFiY2RlZgpTRVJWRVJfUw==
SUdOQVRVUkU6IDEyMzQ1Njc4OTBhYmNkZWYKU0VSVkVSX1NPRlRXQVJFOiAxMjM0NTY3ODkwYWJjZA==
ZWYKU1NMX1RMU19TTkk6IDEyMzQ1Njc4OTBhYmNkZWYKVU5JUVVFX0lEOiAxMjM0NTY3ODkwYWJjZA==
ZWYK
-----------------

The reported base64 problem was fixed in 3.6.0 (bpo-28047)

On 3.16.0a1+dev 'quoted-printable' bodies still use max_line_length as 78, while RFC 2045 limits it to 76.

>>> from email.message import EmailMessage
>>> m = EmailMessage()
>>> m.set_content("a" * 200, cte="quoted-printable")
>>> print(len(m.get_payload().splitlines()[0]))
78

As David suggested above, I can just cap the encoded body lines at 76 in email.contentmanager.
max_line_length would still apply to header folding, where 78 is correct per RFC 5322.

I'm happy to open a PR.

Navigation Menu

Repository navigation

SMTP EmailPolicy not using the correct line length for RCF 2045 encoded data (is 78, should be 76) #78206

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

SMTP EmailPolicy not using the correct line length for RCF 2045 encoded data (is 78, should be 76) #78206

Description

Activity

dougthor42 commented on Jul 2, 2018

bitdancer commented on Jul 2, 2018

AldarisPale commented on Aug 27, 2018

vasiliyk commented on Nov 9, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions