SpamAssassin/Integrating SpamAssassin with qmail
(Initial conversion from Docbook)
(Initial conversion from Docbook)
qmail is a mail transport agent written by cryptography researcher Dan Bernstein and designed to provide a highly secure mail system. It consists of several components, each of which runs with least privilege and none of which trusts data from the other without validating it itself. qmail works best in concert with several other systems designed by Bernstein that take over other functions traditionally performed by standard system utilities.
This chapter explains how to integrate SpamAssassin into a qmail-based mail server to perform spam-checking for local recipients or to create a spam-checking mail gateway.
qmail is a complex piece of software and, like most MTAs, offers scores of configuration choices. This chapter assumes that you are running the netqmail 1.05 version of qmail 1.03 and does not cover how to securely install, configure, or operate qmail itself. For that information, see the qmail documentation, David Sill's Life with qmail web site (http://www.lifewithqmail.org) and The qmail Handbookby David Sill (Apress) or qmail by John Levine (O'Reilly).
This chapter assumes that you have set up your qmail system as described in Life with qmail and that you are using the recommended daemontools and ucspi-tcp packages.
Several different qmail components play roles in receiving messages from the Internet. Messages from the Internet typically enter the mail server via the qmail-smtpd daemon, which listens on port 25 and conducts the SMTP transaction with the remote sender. qmail-smtpd passes the messages to the qmail-queue program, which stores them in an outgoing queue for further processing. The qmail-send daemon reads the messages in the outgoing queue and attempts to deliver them using either the qmail-lspawn daemon (which passes it to the qmail-local program for local delivery) or the qmail-rspawn daemon (which passes them to the qmail-remote program for relaying to remote hosts). Figure 7-1 illustrates the flow of email through qmail components.
Most systems keep all of qmail's files in /var/qmail. Configuration files reside in /var/qmail/control.
Spam-Checking During Local Delivery
The easiest way to add SpamAssassin to a qmail system is to configure qmail to pipe messages through SpamAssassin during local delivery.
The advantages of this approach are:
- It's easy to set up.
- You can run spamd and can use spamc for faster spam-checking.
- User preference files, autowhitelists, and Bayesian databases can be used, because qmail will invoke SpamAssassin as the user to whom it is delivering a message.
However, qmail runs a local delivery agent only for email destined for a local recipient. You cannot create a spam-checking gateway with this approach.
If you're using the installation described in the Life with qmail web site, the /var/qmail/control/defaultdeliveryfile contains a line that specifies either a directory (e.g., ./Maildir/) or a filename (e.g., ./Mailbox). The /var/qmail/rc script passes the contents of defaultdelivery to qmail-start, and thence to qmail-lspawn and qmail-local.
If you deliver to a maildir directory, change the line in your defaultdelivery file to read:
| /usr/bin/spamc | maildir ./Maildir/
In this case, be sure you've installed the safecat package, which includes the maildir script. You can get safecat at http://www.pobox.com/~lbudney/linux/software/safecat.html.
If you deliver to a mailbox file in each user's home directory, install procmail and change the line in defaultdelivery to read:
| preline procmail
In this case, the system's /etc/procmailrc file should have a default recipe that looks like this:
:0fw * <300000 |/usr/bin/spamc :0: $HOME/Mailbox
The default delivery method is used only when users don't have their own .qmail files. This permits users to override spam-checking. Conversely, if you don't do spam-checking by default during local delivery, any user can add the preceding lines to her .qmail file and have her messages checked.
Spam-Checking All Incoming Mail
If you want to set up a spam-checking gateway for all recipients, local or not, you need a way to perform spam-checking as mail is received, before final delivery. qmail provides this capability through the QMAILQUEUE patch, which is included in the netqmail distribution of qmail (and most packaged qmail distributions).
You can find out if your qmail installation has the QMAILQUEUE patch applied by executing the following commands:
# cd /var/qmail/bin # strings qmail-smtpd | grep QMAILQUEUE QMAILQUEUE
If you don't see QMAILQUEUE in response to the strings command, the patch has not been applied. You will have to recompile qmail from the netqmail source code.
With the QMAILQUEUE patch applied, the qmail-smtpd daemon checks to see if the environment variable QMAILQUEUE has been set. If so, qmail-smtpd hands the message off to the program specified in that variable instead of to the default qmail-queue program. The new program can call SpamAssassin and then pass the (possibly tagged) message to qmail-queue. Figure 7-2 illustrates this arrangement.
SpamAssassin includes a small C program called qmail-spamc by John Peacock, with its source code (in the qmail subdirectory in SpamAssassin 2.63, and in the spamc subdirectory in SpamAssassin 3.0). When compiled, qmail-spamc is suitable for use as a QMAILQUEUE program; it invokes spamc on an incoming message and pipes the result to qmail-queue. Because it's written in C and is a very simple program, it runs quickly. To set up qmail-spamc, perform the following steps:
- Compile qmail-spamc.c. On most systems, issue a command like the following in the directory containing qmail-spamc.c:
cc -O -o qmail-spamc qmail-spamc.c
- As root, install qmail-spamc in an appropriate location on your system (e.g., /var/qmail/bin or /usr/local/bin). Make it executable. For example:
install -m 755 qmail-spamc /var/qmail/bin
- Ensure that qmail-queue is on the system's default path. The easiest way to do so is usually to create a symbolic link from /var/qmail/bin/qmail-queue to /usr/bin/qmail-queue. Do the same for spamc if it is not already installed in /usr/bin. For example:
ln -s /var/qmail/bin/qmail-queue /usr/bin/qmail-queue
- Ensure that spamd is running.
- Ensure that qmail-smtpd has enough memory available to allow it to run qmail-spamc and spamc. Edit /var/qmail/supervise/qmail-smtpd/run and modify the -m and/or -a arguments of softlimit to increase the number of bytes available to qmail-smtpd and its child processes to an amount sufficient to allow all of the processes to execute completely on a large message. A setting of 10MB (roughly 10,000,000) is usually sufficient, but you may have to vary the setting and keep an eye on your logs to find the right amount. If the setting is too low, you will see errors such as the following at the end of the DATA step during SMTP transactions:
fatal: qq temporary problem (#4.3.0)
- Edit /etc/tcp.smtp. This file controls access to the SMTP service when you're using ucspi-tcp. Add or modify the line shown in bold:
This change causes the QMAILQUEUE environment variable to be set when qmail-smtpd is invoked by a connection from hosts outside the 127. network (i.e., spam-checking will be performed on email from remote hosts, but not from the local host).
With the version of qmail-spamc distributed with SpamAssassin 3.0, you can customize the way spamc is invoked by adding additional environment variables to the list in /etc/tcp.smtp, including:
- Direct spamc to use the given path to a Unix socket for connecting to spamd.
- Direct spamc to connect to spamd at the given host.
- Direct spamc to connect to spamd at the given TCP port number.
- Direct spamc to connect to spamd using SSL.
- Direct spamc not to perform spam-checking on messages that exceed number-of-bytes in size.
- Direct spamc to supply the given username to spamd.
- Update the TCP rules database by running the command qmailctl cdb, which is found in your /var/qmail/bin/ directory. At this point, all incoming remote SMTP connections should have their messages passed through qmail-spamc.
You can emulate the QMAILQUEUE approach without the QMAILQUEUE patch by renaming qmail-queue to qmail-queue.orig and writing a new qmail-queue script that pipes the message through SpamAssassin and then to qmail-queue.orig, like this:
#!/bin/sh PATH=/var/qmail/bin:$PATH | spamc | qmail-queue.orig
However, this approach is less flexible than using QMAILQUEUE and more prone to causing trouble later when you want to queue messages without spam-checking them.
Building a Spam-Checking Gateway
Several content-filtering daemons that call SpamAssassin are available for qmail. This section provides a complete sample installation of qmail-scanner, a particularly flexible filter that supports both spam-checking and virus-checking. qmail-scanner is written in Perl and available at http://qmail-scanner.sourceforge.net/. The version used in this section's example is 1.21. Some of qmail-scanner's features include:
- The filter was specifically developed and tested for qmail.
- Messages can be rejected based on MIME type or extensions of attached filenames.
- Messages can be rejected based on invalid formatting.
- Messages can be checked with multiple virus scanners, and messages carrying viruses can be refused, discarded, or quarantined.
- SpamAssassin can be invoked on a message, and spam can be refused, discarded, quarantined, or tagged.
The rest of this chapter details the installation, configuration, and operation of qmail-scanner as an example of a full-scale approach to using SpamAssassin with qmail. qmail-scanner's other functions, such as virus-checking, are mentioned but not covered in detail; read the documentation to learn more about these features.
qmail-scanner is written in Perl and invokes SpamAssassin by running spamc, so you must run spamd to use qmail-scanner. You should set up spamd before you install qmail-scanner. Install SpamAssassin (and your antivirus software) first, then install qmail-scanner. qmail-scanner also requires some other Perl modules, including: Time::HiRes, DB_File, and Sys::Syslog. You can install these Perl modules using CPAN as described in Chapter 2. You must also install the Maildrop software package (http://www.courier-mta.org/download.php), and if you plan to perform virus-checking, TNEF (http://sourceforge.net/projects/tnef/). '
qmail-scanner requires the 5.005_03 version of Perl or later. Perl must be compiled to allow setuid Perl scripts; often this means that a separate suidperl program is available on the system. If your system's Perl does not support setuid Perl scripts, you may be able to find a package for your system that does, you may build Perl from source code and enable support, or you may compile a setuid wrapper program in C (described later in this chapter).
Begin the install process by creating a new user account and group for running qmail-scanner; the usual name for both the user and group is qscand. The new user will own qmail-scanner's files, and the user (or group) must have access to SpamAssassin's configuration and database files as well. The user's home directory is traditionally /home/qscand, but you can create it anywhere that fits your system's needs.
qmail-scanner uses several important directories and files in /var/spool/qmailscan. For example, quarantined messages are stored in /var/spool/qmailscan/quarantine, and qmail-scanner logs its operations in /var/spool/qmailscan/qmail-queue.log. The directories /var/spool/qmailscan/tmp and /var/spool/qmailscan/working are temporary directories used for unpacking and processing messages. For optimal performance, these directories should be on a fast disk—even a RAM disk if your operating system supports it and you have enough memory to spare. In contrast, the quarantine directory should never be located on a RAM disk because you will often want to be sure that you can access quarantined files.
Next, download the qmail-scanner source code, unpack it, and build it. You must be root to configure and build qmail-scanner. The qmail-scanner build process uses the familiar configure command to configure and build qmail-scanner's components, which you then install.
To configure qmail-scanner, use the commands shown in Example 7-1. The example also reproduces the output you should expect.
Example 7-1. Building qmail-scanner
$ tar xfz qmail-scanner-1.21.tar.gz $ cd qmail-scanner-1.21 $ su Password: XXXXXXXX # ./configure --install Building Qmail-Scanner 1.21... This script will search your system for the virus scanners it knows about, and will ensure that all external programs qmail-scanner-queue.pl uses are explicitly pathed for performance reasons. It will then generate qmail-scanner-queue.pl - it is up to you to install it correctly. Continue? ([Y]/N) Y /usr/bin/uudecode works as expected on system... The following binaries and scanners were found on your system: mimeunpacker=/usr/local/bin/reformime uudecode=/usr/bin/uudecode unzip=/usr/bin/unzip Content/Virus Scanners installed on your System fprot=/usr/local/bin/f-prot fast_spamassassin=/usr/local/bin/spamc Qmail-Scanner details. log-details=0 fix-mime=2 ignore-eol-check=0 debug=1 notify=psender,nmlvadm redundant-scanning=no firstname.lastname@example.org local-domains='example.com' silent- viruses='klez','bugbear','hybris','yaha','braid','nimda','tanatos','sobig','winevar','pal yh','fizzer','gibe','cailont','lovelorn','swen','dumaru','sober','hawawi','holar- i','mimail','poffer','bagle','worm.galil','mydoom','worm.sco','tanx','novarg','@mm' scanners="fprot_scanner","fast_spamassassin" If that looks correct, I will now generate qmail-scanner-queue.pl for your system... Continue? ([Y]/N) Y Finished. Please read README(.html) and then go over the script to check paths/etc, and then install as you see fit. Remember to copy quarantine-attachments.txt to /var/spool/qmailscan and then run "qmail-scanner-queue.pl -g" to generate DB version. ****** FINAL TEST ****** Please log into an unpriviledged account and run /var/qmail/bin/qmail-scanner-queue.pl -g If you see the error "Can't do setuid", or "Permission denied", then refer to the FAQ. (e.g. "setuidgid qmaild /var/qmail/bin/qmail-scanner-queue.pl -g") That's it! To report success: % (echo 'First M. Last'; cat SYSDEF)|mail email@example.com Replace First M. Last with your name.
As with qmail-spamc, ensure that qmail-smtpd has enough memory available to allow it to run qmail-scanner-queue.pl, any virus checkers you have configured, and spamc. Edit /var/qmail/supervise/qmail-smtpd/run and modify the -m and/or -a arguments of softlimit to increate the number of bytes available to qmail-smtpd and its child processes to an amount sufficient to allow all of the processes to execute completely on a large message.
To enable qmail-scanner, edit /etc/tcp.smtp. Add or modify lines such as those shown in bold:
127.:allow,RELAYCLIENT="" 192.168.:allow,RELAYCLIENT="",QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl" 10.:allow,RELAYCLIENT="",QS_SPAMASSASSIN="on",QMAILQUEUE="/var/qmail/bin/qmail- scanner-queue.pl" :allow,QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl"
When you invoke qmail-scanner with qmail's RELAYCLIENT variable set, as in the line for connections from the 192.168/16 network, only virus-checking is performed, unless you also include QS_SPAMASSASSIN="on", as in the line for connections from the 10/8 network. When you invoke it without setting RELAYCLIENT, as in the line for default connections, both virus-checking and spam-checking are performed.
Be sure to run /var/qmail/bin/qmailctl cdb after updating /etc/tcp.smtp.
The first time you install qmail-scanner, you must direct it to initialize its databases. As the qscand user, run these commands:
$ /var/qmail/bin/qmail-scanner-queue.pl -z $ /var/qmail/bin/qmail-scanner-queue.pl -g perlscanner: generate new DB file from /var/spool/qmailscan/quarantine-attachments.txt perlscanner: total of 9 entries.
qmail-scanner comes with a shell script called test_installation.sh that can be used to exercise an installation. Example 7-2 shows how to run the script, along with its output.
Example 7-2. Testing qmail-scanner
# cd contrib # QMAILQUEUE="/var/qmai/bin/qmail-scanner-queue.pl" ./test_installation.sh -doit Sending standard test message - no viruses... done! Sending eicar test virus - should be caught by perlscanner module... done! Sending eicar test virus with altered filename - should only be caught by commercial anti-virus modules (if you have any)... Sending bad spam message for anti-spam testing - In case you are using SpamAssassin... Done! Finished test. Now go and check Email for root
If qmail-scanner's spam-checking is operating properly, root (or the user that receives root's email) should receive a non-spam message like this:
From MAILER-DAEMON Tue Mar 23 05:03:28 2004 From: Qmail-Scanner Test <firstname.lastname@example.org> Received: from by example.com by uid 0 with qmail-scanner-1.21 (f-prot: 3.11/. spamassassin: 2.63. Clear:RC:1(127.0.0.1):SA:0(0.0/5.0):. Processed in 5.577981 secs); 23 Mar 2004 05:03:28 -0000 To: Root Account <email@example.com> Subject: Qmail-Scanner test (1/4): inoffensive message Date: 23 Mar 2004 05:03:22 -0000 Delivered-To: firstname.lastname@example.org X-Spam-Status: No, hits=0.0 required=5.0 Message 1/4 This is a test message. It should arrive unaffected.
The same user should also receive a spam message like this:
From MAILER-DAEMON Tue Mar 23 05:03:41 2004 Received: from by example.com by uid 0 with qmail-scanner-1.21 (f-prot: 3.11/. spamassassin: 2.63. Clear:RC:1(127.0.0.1):SA:1(16.7/5.0):. Processed in 5.129358 secs); 23 Mar 2004 05:03:40 -0000 X-Spam-Status: Yes, hits=16.7 required=5.0 X-Spam-Level: ++++++++++++++++ Delivery-Date: Mon, 19 Feb 2001 13:57:29 +0000 Delivered-To: email@example.com Received: from webnote.net (mail.webnote.net [22.214.171.124]) by mail.netnoteinc.com (Postfix) with ESMTP id 09C18114095 for <firstname.lastname@example.org>; Mon, 19 Feb 2001 13:57:29 +0000 (GMT) Received: from netsvr.Internet (USR-157-050.dr.cgocable.ca [126.96.36.199] (may +be forged)) by webnote.net (8.9.3/8.9.3) with ESMTP id IAA29903 for <email@example.com>; Sun, 18 Feb 2001 08:28:16 GMT From: firstname.lastname@example.org Received: from R00UqS18S (max1-45.losangeles.corecomm.net [188.8.131.52]) by +netsvr.Internet with SMTP (Microsoft Exchange Internet Mail Service Version +5.5.2653.13) id 1429NTL5; Sun, 18 Feb 2001 03:26:12 -0500 DATE: 18 Feb 01 12:29:13 AM Message-ID: <9PS291LhupY> Subject: Qmail-Scanner anti-spam test (4/4): checking SpamAssassin [if present] +(There yours for FREE!) To: undisclosed-recipients: ; Congratulations! You have been selected to receive 2 FREE 2 Day VIP Passes to Universal Studios! Click here http://184.108.40.206 As an added bonus you will also be registered to receive vacations discounted 25%- 75%! @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ This mailing is done by an independent marketing co. We apologize if this message has reached you in error. Save the Planet, Save the Trees! Advertise via E mail. No wasted paper! Delete with one simple keystroke! Less refuse in our Dumps! This is the new way of the new millennium To be removed please reply back with the word "remove" in the subject line. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Note the bold lines in the messages. These are headers demonstrating that the messages were processed by qmail-scanner, and in the case of the spam message, that qmail-scanner can recognize spam.
qmail-scanner uses /var/spool/qmailscan as a working directory and quarantine area for viruses. By default, qmail-scanner's operations are logged to the /var/spool/qmailscan/qmail-queue.log file, which should be added to your log rotation schedule. Errors are also reported to qmail's log files.
When an SMTP session is dropped partway, temporary files may remain in /var/spool/qmailscan. These messages can be cleared out by running /var/qmail/bin/qmail-scanner-queue.pl -z. Set up a cron job to execute this command once a day to delete older files in this directory.
Per-User Spam Preferences
qmail-scanner invokes spamc with the -u recipient argument when a message has a single recipient. Accordingly, in this case, per-user spam-checking preferences (either from users' .spamassassin/user_prefs files or from an SQL or LDAP database if spamd is so configured) will be applied when qmail-scanner checks messages. When a message has multiple recipients, qmail-scanner uses the default preferences.
Although there is no way to configure qmail to force senders to send messages with one recipient at a time, qmail itself always breaks up a multirecipient message when it is sending and sends copies of the message to single recipients. Ron Culler pointed out in a December 2003 message to the qmail-scanner-general mailing list that one way to ensure that every message has only a single recipient is to run a pair of qmail gateways. The first gateway receives messages from the Internet and can perform some general scanning (e.g., refusing viruses) before forwarding messages on to the second gateway for spam-checking. Because the first qmail server will always split up multirecipient messages before sending them, the second qmail server will always receive messages with a single recipient and can apply per-user spam preferences.
If you built qmail-scanner using the default fast_spamassassin configuration (described in the qmail-scanner Configuration Options sidebar), spamc is invoked with the -c option. This limits which per-user spam preferences are applied: spam thresholds and score modifications will work, but preferences that affect the way messages or headers are rewritten will not (because spamc -c returns only a spam score, not a rewritten message). Use the verbose_spamassassin configuration if you need to enable these preferences.
Sitewide Bayesian Filtering
You can easily add sitewide Bayesian filtering to qmail-scanner. Use the usual SpamAssassin use_bayes and bayes_path directives in local.cf, and ensure that the spamduser has permission to create the databases in the directory named in bayes_path.
Adding autowhitelisting is just as easy. Add the usual SpamAssassin auto_whitelist_path directive to local.cf, and if you're using SpamAssassin 2.63, invoke spamd with the --auto-whitelist option (which is unnecessary in SpamAssassin 3.0). As with the Bayesian databases, the spamd user must have permission to create the autowhitelist database and read and write to it.
Routing Email Through the Gateway
Once you have qmail and qmail-scanner receiving messages for the local host and performing SpamAssassin checks on them, you can start accepting email for your domain and routing it to an internal mail server after spam-checking. Figure 7-3 illustrates this topology.
The following sections describe the changes you need to make to implement the topology shown in Figure 7-3.
To configure qmail to relay incoming mail for example.com to internal.example.com, add the following line to /var/qmail/control/rcpthosts:
Then, create the /var/qmail/control/smtproutes file, and add either:
or, if mail.example.com can look up an (internal) MX record for example.com that points to internal.example.com (and possibly other internal mail servers), you could use
Mail from the Internet for example.com should be sent to the spam-checking gateway mail.example.com. Add a DNS MX record for the example.com domain that points to mail.example.com.
Once received by mail.example.com, messages will be spam-checked and should then be relayed to internal.example.com by qmail. No DNS records for internal.example.com need be published to the Internet, but it's necessary that mail.example.com can resolve internal.example.com.
Internal server configuration
Once the external mail gateway is in place, you can configure the internal mail server to accept SMTP connections only from the gateway (for incoming Internet mail). If you don't have a separate server for outgoing mail, the internal mail server should also accept SMTP connections from hosts on the internal network. These restrictions are usually enforced by limiting access to TCP port 25 using a host-based firewall or a packet-filtering router.