Version 1 (modified by rjl, 9 years ago)

--

Why isn't Maia based on the latest version of amavisd-new?

A lot of people--particularly existing amavisd-new users--are surprised to discover that Maia Mailguard appears to be based on amavisd-new 2.2.1 and not on any of the newer generations (e.g. 2.3.x or 2.4.x), and they wonder when Maia will "catch up" to the latest version of amavisd-new. There's no short and easy answer, unfortunately, but I'll do my best to explain the situation.

Philosophical Reasons

It's important to understand that amavisd-new gets used somewhat differently when there's Maia-like functionality available. Features that were designed for amavisd-new in the pre-Maia era had to rely on hard-coded settings in amavisd.conf for things like whitelists and blacklists, and even per-user policy settings. There was no way for end-users to view or modify these settings themselves, no way for them to view or manage their quarantines, or adjust their own spam thresholds. These were things that only the administrator could do, and there are still a lot of sites out there that operate that way--a lot of old-school admins like it that way, and don't like the idea of trusting end-users to manage their own content filter settings.

Consequently there are a lot of legacy features in amavisd-new that were designed to support those administrator-manages-everything environments. Features like the @score_sender_maps, policy banks, and so on were designed for administrators to set in amavisd.conf on behalf of their users, which is antithetical to the way a user-oriented application like Maia works. Maia certainly lets administrators define default settings for users to inherit, but Maia is about letting the end-users modify their filter settings themselves. It's a difference of philosophy, really, and in general Maia tries to get away from hard-coding anything in amavisd.conf when it should really be adjusted by the users themselves, or by domain-level administrators.

It also requires a leap of faith for many administrators to trust the use of a SQL database as a key component of the mail system. Some admins are worried about adding a new point of failure into an already fragile environment, while others are simply concerned that database reads and writes will consume system resources they can't spare. In spite of the reluctant addition of SQL-based storage in more recent versions of amavisd-new, its design still favours the use of file-based quarantines and an elaborate custom protocol (AM.PDP) for allowing users to release quarantined items, largely because its designer mistrusts SQL storage in this context. Maia uses SQL storage exclusively, and its distributed design reflects this.

This leads to yet another conceptual difference. Maia uses its central SQL database not just for quarantines and users' filter settings, but to control various administrator-level features as well. This makes it possible and practical for a Maia administrator to control an entire cluster of filter servers from a single GUI, rather than having to manually edit a whole bunch of amavisd.conf files and restart all of the daemons every time a configuration change is required. The amavisd-new design expects the daemon to be running on one host, and does not take into account larger-scale installations with multiple filtering hosts and the need to consolidate their administration in a manageable way.

In short, the design goals for amavisd-new are somewhat different from those of Maia Mailguard. The designers in each case have made different basic assumptions about the environments in which their products will be used, and while the core functionality may be common to both applications, the implementation differences continue to grow. In the long term (i.e. Maia 2.0), Maia will part ways with amavisd-new completely in favour of a new design better suited to its goals and philosophy.

Technical Reasons

The design of amavisd-new has never made it easy to develop third-party add-ons. It was not designed with a "developer API" or plugin interface in mind, so any third-party developers have had to either build entirely external products that live within the limitations of amavisd-new (e.g. MailZu), or make modifications to the amavisd-new code to support additional features (e.g. Maia Mailguard).

In an ideal world, Mark Martinec and I could agree on some changes that he could incorporate into future versions of amavisd-new, which would look for a Mail::MaiaMailguard module and support it as an optional feature, all without requiring any patching of amavisd-new. Unfortunately, while Mark and I have discussed this, we failed to agree on how this should be done; his solutions would limit Maia's ability to control the mail filtering process, while mine would require fundamental changes to his design.

Rather ironically, the addition of SQL-based quarantining and release mechanisms in amavisd-new 2.3.x were motivated by Mark's employer asking him to make amavisd-new more like Maia Mailguard in that regard, as these were obviously desirable features. Unfortunately, he chose to implement these features rather differently, based on a deeply-ingrained mistrust of involving databases in the mail processing pipeline. As he expressed it to me at the time, he hoped that by adding those features he could help me reduce the size of the amavisd-maia patch, but in practice we would have had to redesign a significant chunk of Maia Mailguard to accommodate those new mechanisms, since he chose a rather different way of storing and managing the SQL quarantine.

Even so, there would still remain a need to patch the amavisd-new file, since Maia is more than just a MailZu-like quarantine release tool. Maia provides direct control over a running amavisd process, offering the administrator the ability to tweak its behaviour from the System Configuration web page rather than solely from the amavisd.conf file, for one thing, and the support for features like user auto-creation, encryption, and so on require changes that a wholly-external third-party tool cannot achieve.

Patching software is another problem entirely, since patches need to be adapted for each new release. It would mean developing patches for every available version of amavisd-new, and in the case of major version changes these patches would become significant. To develop a Maia patch for amavisd-new 2.3.x or 2.4.x, for example, we would have to significantly rework Mark's new quarantining code, bypassing it entirely to replace it with the quarantine code from the 2.2.x series. The end result would be a version of amavisd-new that looked very much like 2.2.1, so why bother?

There's also a lot of additional code in the 2.3.x and 2.4.x series that has no place in a Maia Mailguard context. The new quarantine release mechanism, for example, is a command-line based tool that makes use of a custom client/server protocol (AM.PDP, the Amavis Policy Delegation Protocol) that Mark developed to support it. Maia does all of its quarantine release via a web server with PHP scripts, so this new protocol that interacts direcly with an amavisd daemon to achieve the same result is unnecessary. It's a feature that might make sense in an environment that doesn't have a web-based GUI and release logic like Maia Mailguard, but not otherwise. As a result, Maia would never use that code, so it would waste memory needlessly--or more likely we'd remove it entirely, in which case we'd be back to something closely resembling 2.2.1 again.

The alternative, of course, is to redesign Maia Mailguard's internals to take advantage of the changes Mark has made in the amavisd-new 2.3.x and 2.4.x series. And to be clear, there are some small advantages to the way he's chosen to implement things, such as breaking messages into 16k chunks to avoid hitting any BLOB size limits. The Maia approach, by contrast, stores the mail in a single row, so there's technically a size limit imposed by the database server, though PostgreSQL and MySQL 4.x and later support BLOBs at least as large as 2 GB, which should be plenty for an e-mail. Mark's approach is more "universal," and would support a broader range of databases, but for MySQL and PostgreSQL (the two that Maia Mailguard promises to support) this implementational difference offers no benefits, and arguably hurts performance more than it helps, since mail then needs to be broken into chunks and reassembled, resulting in more database reads and writes, and more processing work at both ends of the operation.

Enter "amavisd-maia"

In the end, then, what's happened is that Maia Mailguard has effectively "forked" from the amavisd-new development stream, to become its own product. You no longer need to install and maintain amavisd-new, so there's no need to worry about compatibility with past/current/future releases of amavisd-new. The package you need to maintain instead is Maia Mailguard, which incorporates its own modified version of amavisd-new. That version (1.0.1) was based originally on amavisd-new 2.2.1, but will continue to evolve and incorporate the more useful features of newer amavisd-new releases--when they make sense in a Maia Mailguard context. Maia 1.0.1, for instance, incorporates bits of code from amavisd-new 2.3.3, and Maia 1.0.2 incorporates some code from amavisd-new 2.4.1.

As such, you can stop worrying about amavisd-new version numbers; if you've got amavisd-new 2.3.x or 2.4.x installed on your system, uninstall it and remove it from any auto-updating mechanisms you may be using. Then install Maia Mailguard and you'll have what you need.