Exim stress test with 100,000 messages in queue
One morning on an Exchange mail server, more than 100,000 messages were found in the queue. It doesn't matter how they got there. The point is that removing the messages from the queue took approximately 4-5 hours using PowerShell. Selecting 20,000 records from the queue took more than 30 minutes... we didn't wait until the end, reduced the count to 1,000 then 2,000... but the queries were still executing very slowly.
I'm running a test to understand how Exim will behave in a similar situation and what time losses to expect.
# uname -a
FreeBSD exim-test.domain.local 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 18:38:15 UTC 2016 [email protected]:/usr/obj/usr/src/sys/GENERIC amd64
# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/da0p2 8.7G 5.4G 2.6G 67% /
devfs 1.0K 1.0K 0B 100% /dev
# exim -bV
Exim version 4.87 #0 (FreeBSD 10.3) built 16-Aug-2016 15:40:34
Copyright (c) University of Cambridge, 1995 - 2016
(c) The Exim Maintainers and contributors in ACKNOWLEDGMENTS file, 2007 - 2016
Probably Berkeley DB version 1.8x (native mode)
Support for: crypteq iconv() use_setclassresources PAM Perl Expand_dlfunc Content_Scanning Old_Demime DNSSEC I18N PRDR Experimental_SPF
Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb dsearch ldap ldapdn ldapm mysql passwd
Authenticators: cram_md5 dovecot plaintext spa
Routers: accept dnslookup ipliteral manualroute queryprogram redirect
Transports: appendfile/maildir/mailstore/mbx autoreply pipe smtp
Fixed never_users: 0
Size of off_t: 8
Configuration file is /usr/local/etc/exim/configure
Start:
20,000 messages in queue
118,000 messages in queue
Full test period:
Results:
- the server remained operational, ssh client connects normally and works adequately.
- disk space did not run out (though it probably depends on message size).
- memory usage did not increase critically.
- CPU load increased massively.
We'll clear the message queue with the server running:
exim -bpc # response is instant
118765
exim -bp | exiqsumm # took about 1:51
Count Volume Oldest Newest Domain
----- ------ ------ ------ ------
100 14MB 26h 9h test.domain.com
1 2150 25h 25h exim-test.net
11865 7775KB 25h 8h exim-test000.net
11866 7775KB 25h 8h exim-test111.net
11867 7776KB 25h 8h exim-test222.net
11867 7776KB 25h 8h exim-test333.net
11867 7776KB 25h 8h exim-test444.net
11867 7776KB 25h 8h exim-test555.net
11867 7776KB 25h 8h exim-test666.net
11866 7775KB 25h 8h exim-test777.net
11866 7775KB 25h 8h exim-test888.net
11866 7775KB 25h 8h exim-test999.net
---------------------------------------------------------------
118765 90MB 26h 8h TOTAL
# WHO the messages are being sent from
exim -bp | awk '{print $4}' | sort | uniq -c | sort -rn
237530
118668 <[email protected]>
97 <>
# Delete messages from and to EMPTY recipient
exim -bpr |grep "<>" | awk '{print $3}' | xargs exim –Mrm
# Delete messages to a SPECIFIC recipient
# Response started after 2 minutes 24 seconds
exiqgrep -i -r [email protected] | xargs exim -Mrm
# 83,000 messages in queue:
exiqgrep -i -r [email protected] | xargs exim -Mrm
# Selection took 1:15, deletion itself took 19 seconds.