Scaling for Performance

Many people appreciate Maia's integration--the ability to manage a complete mail filtering system on a single machine--but it is useful to remember that all of the underlying components were designed to operate independently, communicating with one another via network protocols; hosting everything on a single machine is just the special case that results when all of the network parameters are set to "localhost". There is certainly no reason that all of the components must be hosted on the same machine, and indeed there are situations in which performance can be improved dramatically by distributing some of the components across multiple hosts. For small installations there would be little to gain from such distribution, but medium- to large-scale implementations would almost certainly benefit. Here's a brief description of what's involved.

Determine the axes of scalability

There are several to consider:

1. The receiving MTA (MTA-RX). This MTA listens for SMTP connections from other mail servers on port 25, and possibly some reserved ports for submissions from local users with or without SSL (e.g. 587, 465, etc.). How many such MTA-RXes you need will obviously depend on the volume of SMTP traffic you receive, and how much of a margin you wish to reserve as a safeguard against abuse. The responsibility of the MTA-RX is to perform first-stage filtration of the mail--weed out the most obvious spam, detect common forms of abuse, consult conservative DNSBLs, perform greylisting (e.g. with policyd)--and then queue whatever mail is not rejected during the SMTP session.

2. The mail filter. The mail filter is a combination of amavisd-maia, SpamAssassin, ClamAV, and any other scanners you wish to use to analyze the mail. This is second-stage filtration, i.e. after-queue, so timing is not as critical here, and you can be reasonably thorough with your analysis. This is a very CPU- and RAM-intensive operation, so you can either make sure that the machines handling this task are powerful, or you can run multiple mail filter hosts in parallel to compensate.

3. The web server. Scaling along the web server axis allows you to keep the Maia GUI feeling fast, smooth, and responsive despite growth in your user population. When there aren't enough resources to achieve this level of performance, the user experience is diminished, and users are discouraged from participating in the feedback-driven processes that improve Maia's effectiveness. This is the side of Maia that users see, so if resources are lacking along this axis there will be complaints at the help desk.

4. The database. Maia is a database-driven system, so it is natural to want to scale along this axis as well. This is the most difficult piece to scale however, because the database plays a central role and this is a read-write system, so data consistency is paramount. Clustering solutions (e.g. MySQL Cluster) are appealing, but they usually require that the entire database fit into RAM, which is not practical with large databases like Maia's. For the moment, then, we can only recommend that you use database replication and/or tools such as DRBD or Tungsten Replicator to maintain a hot spare for failover support. Even if it is not practical to scale the database aspect, it is at least possible to eliminate this single-point-of-failure by using a live backup server that can take over gracefully if the master should fail.

Note: You may not need to scale your Maia installation in all of these dimensions, of course. There is relatively little to be gained in most cases by scaling the MTA independently of the mail filter, for example, so it is usually sufficient to bundle those two functions together on a single host which is then responsible for end-to-end mail processing.

Installing the Components

Clearly in a distributed installation not every component is required on every host. This is a rough guide to what needs to be installed where:

MTA Servers

  • Postfix or Sendmail
  • Policyd (optional)

On a host that exclusively serves the MTA function, only whatever software is necessary to run the MTA itself is strictly necessary. Optional first-stage filtering components such as policyd, other policy daemons (Postfix) or milters (Sendmail) would be installed here.

No Maia components are needed here, but of course the MTA must be configured to route mail to amavisd-maia on a mail filter host. In other words, rather than pointing to localhost:10024, it needs to point to mail-filter-host:10024.

If this host is also handling the downstream delivery of filtered mail, listening on port 10025 for connections, it must be configured to accept connections on that port only from the mail filter hosts.

Again, this dimension is usually not scaled on its own, it is usually merged with the mail filter, below, to reduce complexity.

Mail Filters

Mail filter hosts need amavisd-maia and its prerequisites, and any other filtering components you wish to add.

Note that the load-sa-rules script only needs to exist on one of these servers--whichever one is tasked with running the saupdate script. This could certainly be run on all of these hosts for simplicity, but doing so would be inefficient; it would be better to only do this on one "master" mail filter host and then use a tool such as rsync to copy the updated SpamAssassin rules to the others, and force a restart of amavisd-maia on those hosts when it is done.

Database configuration in maia.conf, amavisd.conf, and SpamAssassin's local.cf must all point to the Maia database on the database server, rather than localhost.

Web Servers

  • Apache or other web server software
  • PHP
  • PEAR modules
  • Smarty
  • HTMLPurifier
  • Maia's PHP files

The web server hosts need all of the expected PHP and PEAR modules to support the Maia GUI, and of course all of Maia's PHP files must be installed.

Database configuration in config.php must point to the Maia database on the database server, rather than localhost.

Database Servers

  • MySQL or PostgreSQL
  • maia-mysql.sql or maia-pgsql.sql
  • Maia's maintenance scripts
  • /etc/maia.conf

The maia-*.sql script should be used to initialize the database server, but it must also be configured to accept connections not merely from localhost but also from the mail filter hosts and the web server hosts. In other words, something like:

GRANT CREATE, DROP, ALTER, SELECT, INSERT, UPDATE, DELETE ON maia.* TO amavis@localhost IDENTIFIED BY 'passwd';
GRANT CREATE, DROP, ALTER, SELECT, INSERT, UPDATE, DELETE ON maia.* TO amavis@mail-filter-host IDENTIFIED BY 'passwd';
GRANT CREATE, DROP, ALTER, SELECT, INSERT, UPDATE, DELETE ON maia.* TO amavis@web-server-host IDENTIFIED BY 'passwd';

If you've got multiple mail filter and web server hosts and you've named them predictably (e.g. mail-filter-1, mail-filter-2, etc.), you can use wildcards (%) to simplify this:

GRANT CREATE, DROP, ALTER, SELECT, INSERT, UPDATE, DELETE ON maia.* TO amavis@localhost IDENTIFIED BY 'passwd';
GRANT CREATE, DROP, ALTER, SELECT, INSERT, UPDATE, DELETE ON maia.* TO amavis@mail-filter-% IDENTIFIED BY 'passwd';
GRANT CREATE, DROP, ALTER, SELECT, INSERT, UPDATE, DELETE ON maia.* TO amavis@web-server-% IDENTIFIED BY 'passwd';

Maia's maintenance scripts should also be installed on the database server, since these operate on the database itself for the most part. The exception is the load-sa-rules script, which should only be run on the mail filter hosts, as explained above. The maia.conf file is needed by these scripts, and can be configured to connect to the database at localhost.

Practical Considerations

Simple distribution of the workload across multiple servers is straightforward, but when you want to run multiple mail filters or multiple web servers you need to consider the load-balancing system as well. Round-robin DNS can be used in simple cases to alternate traffic across multiple servers, but this is not as effective or reactive as a true hardware-based load balancer, which is strongly recommended for larger installations.

Network traffic to and from the database server can become a bottleneck if not taken into consideration. Multiple (possibly bonded) NICs or fibre channel connectivity to the other hosts in the Maia system is recommended for the database server. Also consider the database server's access to storage, which must be fast and reliable--ideally a storage array with multiple fast discs which can be accessed in parallel (e.g. RAID 5, 6, 1+0, etc.). With mail and web performance depending on database reads and writes, you need to consider the hardware and network requirements of the database server above all else, since it can't scale as well as the other components.

Maintaining a distributed installation also requires a bit more diligence, due to the fact that components might be upgraded on some hosts and not others, leading to inconsistent behaviour that may be difficult to diagnose. It is best to perform all upgrades on one machine in a parallel array first, and then clone it to all of the others, rather than allow them to auto-update on their own. More fundamentally, the machines in an array should all be running identical versions of the operating system, as a hedge against inconsistency. Again, building one machine and cloning it to produce the others in the array is the most reliable way to achieve this.