Skip to content

Exception in BytesGenerator.flatten #71445

New issue
New issue
Closed
Closed
Exception in BytesGenerator.flatten#71445
Labels
topic-email

Description

@frispete
frispete
mannequin
opened on Jun 7, 2016
BPO 27258
Nosy @warsaw, @bitdancer, @berkerpeksag, @pslacerda, @iritkatriel
Files
  • flatten-exception.mail
  • email_flatten.py
  • flatten-no-exception.mail
  • flatten-no-exception.mail
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-12-12.16:35:16.824>
created_at = <Date 2016-06-07.19:52:43.623>
labels = ['expert-email']
title = 'Exception in BytesGenerator.flatten'
updated_at = <Date 2021-12-12.16:35:16.823>
user = 'https://bugs.python.org/frispete'

    bugs.python.org fields:

    activity = <Date 2021-12-12.16:35:16.823>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-12.16:35:16.824>
closer = 'iritkatriel'
components = ['email']
creation = <Date 2016-06-07.19:52:43.623>
creator = 'frispete'
dependencies = []
files = ['43288', '43415', '43456', '43457']
hgrepos = []
issue_num = 27258
keywords = []
message_count = 8.0
messages = ['267736', '268645', '268658', '268673', '268759', '268807', '268810', '408397']
nosy_count = 6.0
nosy_names = ['barry', 'frispete', 'r.david.murray', 'berker.peksag', 'Pedro Lacerda', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue27258'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']

    Activity

    1. frispete commented on Jun 7, 2016

      @frispete
      frispetemannequin
      on Jun 7, 2016
      MannequinAuthor

      Attached mail, parsed with email.message_from_binary_file results in:

      Traceback (most recent call last):
  File "./mail_filter.py", line 616, in <module>
    ret = main.run()
  File "./mail_filter.py", line 605, in run
    self.process(fp)
  File "./mail_filter.py", line 589, in process
    self.save_message(msg, self._fname + '.out')
  File "./mail_filter.py", line 103, in save_message
    ofd.write(msg.as_bytes())
  File "/usr/lib64/python3.4/email/message.py", line 179, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib64/python3.4/email/generator.py", line 115, in flatten
    self._write(msg)
  File "/usr/lib64/python3.4/email/generator.py", line 195, in _write
    self._write_headers(msg)
  File "/usr/lib64/python3.4/email/generator.py", line 422, in _write_headers
    self._fp.write(self.policy.fold_binary(h, v))
  File "/usr/lib64/python3.4/email/policy.py", line 190, in fold_binary
    folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
  File "/usr/lib64/python3.4/email/policy.py", line 204, in _fold
    return self.header_factory(name, ''.join(lines)).fold(policy=self)
  File "/usr/lib64/python3.4/email/headerregistry.py", line 255, in fold
    return header.fold(policy=policy)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 300, in fold
    self._fold(folded)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 1228, in _fold
    rest._fold(folded)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 338, in _fold
    if folded.append_if_fits(part, tstr):
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 149, in append_if_fits
    token._fold(self)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 324, in _fold
    for part in self.parts:
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 254, in parts
    if token.startswith_fws():
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws
    return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws
    return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws
    return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws
    return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws
    return self[0].startswith_fws()
IndexError: list index out of range

      when flattened with BytesGenerator.

    2. frispete
      added
      topic-email
      on Jun 7, 2016

    3. pslacerda commented on Jun 16, 2016

      @pslacerda
      pslacerdamannequin
      on Jun 16, 2016
      Mannequin

      I was unable to reproduce this bug using the following snippet

          import email, sys
    from email.generator import BytesGenerator
    from email.mime.text import MIMEText

    fp = open('flatten-exception.mail', 'rb')
    email.message_from_binary_file(fp)
    bs = BytesGenerator(sys.stdout.buffer)
    bs.flatten(MIMEText('msg', 'plain', 'utf-8'))

    4. berkerpeksag commented on Jun 16, 2016

      @berkerpeksag
      berkerpeksag
      on Jun 16, 2016
      Member

      Thanks for helping to triage this, Pedro. I think there is a typo in your example: email.message_from_binary_file(fp) needs to be passed to bs.flatten().

      With the following script I'm also unable to reproduce the issue in Python 3.4+:

          import email
    import email.generator
    import sys

    with open('flatten-exception.mail', 'rb') as f:
        msg = email.message_from_binary_file(f)
        gen = email.generator.BytesGenerator(sys.stdout.buffer)
        gen.flatten(msg)

      Hans-Peter, could you share a reproducer with us? Thanks!

    5. frispete commented on Jun 16, 2016

      @frispete
      frispetemannequin
      on Jun 16, 2016
      MannequinAuthor

      Sorry guys for not providing this earlier.

      It turned out, that the sub optimal behaviour is related to a unfortunate policy choice: email.policy.SMTP.

    6. pslacerda commented on Jun 18, 2016

      @pslacerda
      pslacerdamannequin
      on Jun 18, 2016
      Mannequin

      Seems that token.has_fws evaluates to True in the following condition

          if token.has_fws:

      causing token._fold(self) where isn't needed and raising the exception. Hope it helps!

      By the way, why the _header_value_parser.py was removed from the repository?
      https://github.com/python/cpython/blob/master/Lib/email/_header_value_parser.py#L144

    7. pslacerda commented on Jun 18, 2016

      @pslacerda
      pslacerdamannequin
      on Jun 18, 2016
      Mannequin

      Now the file is back! If any previous header has a newline before the value the error will not happen. But even with the output correct it isn't as expected.

    8. pslacerda commented on Jun 18, 2016

      @pslacerda
      pslacerdamannequin
      on Jun 18, 2016
      Mannequin

      Now the file is back! If any previous header has a newline before the value the error will not happen. But even with the output correct it isn't as expected.

    9. iritkatriel commented on Dec 12, 2021

      @iritkatriel
      iritkatriel
      on Dec 12, 2021
      Member

      I am also unable to reproduce the issue on 3.11 with the attached script and input.

      Please create a new issue if this problem still exists in a current python version (>= 3.9).

    10. iritkatriel
      closed this as completedon Dec 12, 2021
    11. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      topic-email

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions