Postfix Performance Results

Table of Contents | Back: Postfix Architecture | Next: Postfix Queue Management

Note: in the USENET news article below I am still referring to the Postfix project by its obsolete MX code name. In addition, some people pointed out to me that Qmail should be spelled as qmail. I stand corrected.

Path: news.porcupine.org!news.porcupine.org!not-for-mail
From: wietse@porcupine.org (Wietse Venema)
Newsgroups: comp.mail.sendmail,comp.mail.misc
Subject: Wietse's mailer project: first results
Followup-To: poster
Date: 9 Jun 1997 14:26:06 -0400
Organization: Wietse's hangout while on sabattical in the USA
Lines: 117
Sender: wietse@spike.porcupine.org
Message-ID: <5nhhru$1ng@spike.porcupine.org>
NNTP-Posting-Host: spike.porcupine.org

Thought you might be interested. MX is the code name of the freeware mailer project that I've been working on for the last few months while visiting IBM TJ Watson Research.

Initial results are promising: MX is up to three times faster than its nearest competitor, Qmail. Discussion follows below the signature; tables are at the end of this message.

The mailer is based on a dozen semi-resident daemon programs, almost all of which can run inside a chroot/nobody jail. I'm using tricks from web servers to reduce fork/exec overhead. And, just like web servers, my mailer supports real virtual hosting. You can give each virtual host its own limit on, for example, the number of processes.

My mailer is designed to be sendmail compatible at the user level so it supports /etc/aliases, NIS aliases, ~/.forward files and can deliver to /var[/spool]/mail. However, it has no sendmail.cf file, and none of its constituent programs is set-uid.

MX is the current project code name. I understand that this name has already been taken by a VMS mail product. Suggestions for a better name are welcome. The best suggestion so far: Blackmail, suggested by Dan Farmer :-).

First release to the net about two months from now. I'll try to set up a fact sheet web page as soon as I have switched to the mailer myself.

Let there be some friendly competition.

Wietse

This week my mailer finally became complete enough that I could do some first performance tests. I wrote a few tools to generate and receive arbitrary amounts of SMTP messages and did some measurements on my local 10MHz home ethernet. For a first experiment I used my tools to compare Qmail performance to that of my own MX mailer.

The setup involved a message generating program, the mail software under test, and a message sink program.

The message generator and sink programs ran on BSD/OS 2.1, P166. The mail systems ran on FreeBSD 2.2.1, P133 Thinkpad. When I used the generator and sink programs with no mail system in-between, these two machines were able to pump 250 SMTP messages per second over the 10MHz ethernet. Of course no real mail system can achieve that, it just shows that the overhead of the generator and sink tools is small compared to the overhead of Qmail and MX.

One test could not be completed because my MX mailer saturated the message sink machine. The P166 is my main machine and has to stay up, so I was unable to reconfigure it for a larger network load.

About the only test where Qmail performance approaches that of MX is when sending one single message to a lot of different places: 76 deliveries per second (see tables below). Note that this is still far below the 250 messages/second that I get when my message generator and sink talk to each other over ethernet without mail system in-between. Some speed-up may still be possible.

However, Qmail is three times slower with delivery of messages that have only one (local or remote) recipient. With one recipient per message, MX does 5.3 remote deliveries/second (4.2 local/second) while Qmail does only 1.8 remote deliveries/second (1.7 local/second).

With the latter tests, speed is dominated completely by disk I/O. It is obvious that Qmail hits the disk much harder than MX does. Fixing that would require a complete redesign of Qmail's database structure: Qmail uses three files per mail message, whereas my mailer uses only one. This explains the factor of three in speed. All other differences in architecture vanish in the face of this disk I/O bottleneck.

List performance: time in seconds to forward one SMTP message with 1000, 2000 and 5000 different remote destinations. Outbound SMTP concurrency = 100 (limit on simultaneous outgoing deliveries).

Conclusion: MX is 20% faster than Qmail.

        1000    2000    5000    avg time/1000
---------------------------------------------
Qmail   15      32      80      16
MX      13      25      (*)     13

(*) message sink host saturated under the load :-(

Local delivery performance: time in seconds to deliver 100 incoming SMTP messages to one local recipient. Inbound SMTP concurrency = 5, 10 and 20 (number of incoming messages being pumped into the system simultaneously).

Conclusion: MX is three times faster than Qmail.

        5       10      20      avg time/100
--------------------------------------------
Qmail   62      59      58      60
MX      24      24      25      24

Relay performance: time in seconds to forward 100 incoming SMTP messages with one remote recipient. Outbound SMTP concurrency = 100 (limit on simultaneous outgoing deliveries), inbound SMTP concurrency = 5, 10 and 20 (number of incoming messages being pumped into the system simultaneously).

Conclusion: MX is three times faster than Qmail.

        5       10      20      avg time/100
--------------------------------------------
Qmail   56      54      54      55
MX      17      19      20      19

Table of Contents | Back: Postfix Architecture | Next: Postfix Queue Management