QUICK LINKS

Postfix Workarounds

This webpage documents workarounds for bugs in non-Postfix mail software. Workarounds may vary from "dumbing down" Postfix, to an optional source code patch that automatically enables the workaround.

(Resolved: December 9, 2010) Mail FROM FrontBridge customers does not go through while Postfix repeatedly logs "lost connection" errors

(Resolved: December 9, 2010) Mail FROM FrontBridge customers does not go through while Postfix repeatedly logs "lost connection" errors.

summary | demonstration | workarounds | background

Problem summary

The SMTP service at mail.global.frontbridge.com does not fully conform to RFC 2920 which defines the ESMTP PIPELINING extension. When the service receives a group of SMTP commands ending with QUIT, it ignores all commands in that group except QUIT. Needless to say, this can break SMTP sessions in unexpected ways.

As one symptom of this defect, mail from FrontBridge customers does not go through to Postfix sites, when a Postfix site is configured to verify that the sender address is valid (this involves connecting to the FrontBridge service).

Due to the FrontBridge service's non-conformance to RFC 2920, the attempt to verify the sender address fails, and Postfix logs "lost connection with mail.global.frontbridge.com[x.x.x.x] while sending RCPT TO". As Postfix is unable to verify the sender address, mail from the FrontBridge customer will not go through.

Problem demonstration

The following is a slightly-edited transcript of an SMTP session that demonstrates the problem. The transcript was obtained by executing the following commands:

# postconf -e debug_peer_list=frontbridge.com
# postfix reload
# sendmail -bv receiver@example.com

Instead of receiver@example.com specify an email address that is hosted by FrontBridge.

S: 220 VA3EHSMHS018.bigfish.com Microsoft ESMTP MAIL Service ready at
       Sun, 28 Nov 2010 19:01:22 +0000

C: EHLO amnesiac.example.com

S: 250-VA3EHSMHS018.bigfish.com Hello [192.0.2.1]
S: 250-SIZE 157286400
S: 250-PIPELINING
S: 250-ENHANCEDSTATUSCODES
S: 250-STARTTLS
S: 250-AUTH
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250 CHUNKING

Note: the following client commands are sent in one single TCP segment,
not in multiple segments back-to-back.

C: MAIL FROM:<sender@example.com> SIZE=338
C: RCPT TO:<receiver@example.com>
C: RSET
C: QUIT

S: 221 2.0.0 Service closing transmission channel

  <Client sees premature TCP close while expecting three more responses>

The client sees a premature TCP close, while it expects three more server responses. Specifically, the server responses to the "MAIL FROM" and "RCPT TO" and "RSET" commands are "lost", and the "QUIT" response arrives out-of-order. One may speculate that the issue is not in the FrontBridge server itself, but rather in proxy software in front of the FrontBridge server.

As discussed below, the correct behavior would be:

C: MAIL FROM:<sender@example.com> SIZE=338
C: RCPT TO:<receiver@example.com>
C: RSET
C: QUIT

S: 250 2.1.0 Sender OK
S: 250 2.1.5 Recipient OK
S: 250 2.0.0 Resetting
S: 221 2.0.0 Service closing transmission channel

Workarounds

A Postfix patch may be made available that turns on workarounds automatically, by looking at the content of the "220 welcome" server greeting. Until that patch is available, workarounds will need to be turned on by hand.

Disable sender address verification. The problem with the FrontBridge service triggers with "MAIL FROM/RCPT TO/RSET/QUIT", which appears with Postfix sender address verification probes.

Disable SMTP command pipelining for FrontBridge servers, Unfortunately, Postfix supports this only by IP address and the FrontBridge servers appear to use many IP addresses.

/etc/postfix/main.cf:
    smtp_discard_ehlo_keyword_address_maps = 
	cidr:/etc/postfix/discard_ehlo_keywords

/etc/postfix/discard_ehlo_keywords:
    # This is likely to be incomplete.
    216.32.0.0/16 silent-discard, pipelining
    213.199.0.0/16 silent-discard, pipelining
    65.55.0.0/16  silent-discard pipelining
    94.245.0.0/16 silent-discard pipelining

Disable SMTP command pipelining for all mail. This sledgehammer solution will reduce performance for all outbound mail, by causing extra network round-trip times.
```
/etc/postfix/main.cf:
    smtp_discard_ehlo_keywords = silent-discard pipelining
```

Background information

The SMTP service at mail.global.frontbridge.com does not fully conform to RFC 2920 which defines the ESMTP PIPELINING extension. When the service receives a group of pipelined SMTP commands ending with QUIT, it ignores all commands in that group except QUIT.

In http://tools.ietf.org/html/rfc2920#section-3.1 we see that "RSET", "MAIL FROM", and "RCPT TO" are examples of commands that can appear anywhere in a pipelined group, while "EHLO", "DATA" and "QUIT" are examples of commands that can only appear as the last command in a group.

Section 3.2 http://tools.ietf.org/html/rfc2920#section-3.2 specifies required server behavior:

(1) MUST respond to commands in the order they are received from the client.

...

(9) MUST NOT flush or otherwise lose the contents of the TCP input buffer under any circumstances whatsoever.

A correct implementation of RFC 2920 is recorded below from an intranet Microsoft Exchange 2007 server.

S: 220 exchange.example.com Microsoft ESMTP MAIL Service ready at
       Sun, 28 Nov 2010 02:41:13 -0500

C: EHLO amnesiac.example.com

S: 250-exchange.example.com Hello [192.0.2.1]
S: 250-SIZE 29999104
S: 250-PIPELINING
S: 250-DSN
S: 250-ENHANCEDSTATUSCODES
S: 250-STARTTLS
S: 250-AUTH
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250-CHUNKING
S: 250 XEXCH50

Note: the following client commands are sent in one single TCP segment,
not in multiple segments back-to-back.

C: MAIL FROM:<sender@example.com>
C: RCPT TO:<recipient@example.com>
C: RSET
C: QUIT

S: 250 2.1.0 Sender OK
S: 250 2.1.5 Recipient OK
S: 250 2.0.0 Resetting
S: 221 2.0.0 Service closing transmission channel

The four pipelined client commands "MAIL FROM", "RCPT TO", "RSET" and "QUIT" are all processed by the Exchange server which correctly returns four responses.

The same test against the public MX host for Microsoft's Exchange engineering group yields similar results:

S: 220 mail7.exchange.microsoft.com Microsoft ESMTP MAIL Service ready
       at Sat, 27 Nov 2010 23:43:19 -0800

C: EHLO amnesiac.example.com

S: 250-mail7.exchange.microsoft.com Hello [192.0.2.1]
S: 250-SIZE 104857600
S: 250-PIPELINING
S: 250-DSN
S: 250-ENHANCEDSTATUSCODES
S: 250-STARTTLS
S: 250-X-ANONYMOUSTLS
S: 250-AUTH
S: 250-X-EXPS NTLM
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250-CHUNKING
S: 250-XEXCH50
S: 250 XSHADOW

Note: the following client commands are sent in one single TCP segment,
not in multiple segments back-to-back.

C: MAIL FROM:<sender@example.com>
C: RCPT TO:<recipient@example.com>
C: RSET
C: QUIT

S: 250 2.1.0 Sender OK
S: 250 2.1.5 Recipient OK
S: 250 2.0.0 Resetting
S: 221 2.0.0 Service closing transmission channel

Finally, testing mail.global.frontbridge.com we get non-conformant results as discussed above.

S: 220 DB3EHSMHS006.bigfish.com Microsoft ESMTP MAIL Service ready at
       Sun, 28 Nov 2010 07:47:08 +0000

C: EHLO amnesiac.example.com

S: 250-DB3EHSMHS006.bigfish.com Hello [192.0.2.1]
S: 250-SIZE 157286400
S: 250-PIPELINING
S: 250-ENHANCEDSTATUSCODES
S: 250-STARTTLS
S: 250-AUTH
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250 CHUNKING

Note: the following client commands are sent in one single TCP segment,
not in multiple segments back-to-back.

C: MAIL FROM:<sender@example.com>
C: RCPT TO:<recipient@example.com>
C: RSET
C: QUIT

S: 221 2.0.0 Service closing transmission channel

   <Client sees premature TCP close while expecting three more responses>

In this case the client's "MAIL FROM", "RCPT TO" and "RSET" commands are "lost", and the "QUIT" response arrives out-of-order. One may speculate that the issue is not in the FrontBridge server itself, but rather in proxy software in front of the FrontBridge server.

Significantly, the FrontBridge SMTP servers correctly handle pipelining of DOT + QUIT at the end of a delivery transaction, perhaps because this "group" is treated specially.

S: 220 DB3EHSMHS003.bigfish.com Microsoft ESMTP MAIL Service ready at
       Sun, 28 Nov 2010 07:27:55 +0000

C: EHLO amnesiac.example.com

S: 250-DB3EHSMHS003.bigfish.com Hello [192.0.2.1]
S: 250-SIZE 157286400
S: 250-PIPELINING
S: 250-ENHANCEDSTATUSCODES
S: 250-STARTTLS
S: 250-AUTH
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250 CHUNKING

Note: the following client commands are sent in one single TCP segment,
not in multiple segments back-to-back.

C: MAIL FROM:<sender@example.com>
C: RCPT TO:<recipient@example.com>
C: DATA

S: 250 2.1.0 Sender OK
S: 250 2.1.5 Recipient OK
S: 354 Start mail input; end with <CRLF>.<CRLF>

C: [message-content]
C: .
C: QUIT

S: 250 2.6.0 <20101128070828.AB036504373@amnesiac.example.com>
             [InternalId=11073071] Queued mail for delivery
S: 221 2.0.0 Service closing transmission channel