History^

Until now, i have used a bunch of “standalone” solutions, such as postgrey, amavisd-new, policyd-weight, and so on, which are really good and achieved ample results – but they lack the capability of sharing those results with each other lead to a lot of configurations/modifications to see the “big picture”. This in turns lead to unnecessary configuration mistakes and eliding of features on my side.

Increasing spam emergences in the last months made me over-think the whole issue. The first attempt was to re-write and improve my “wrapper” server, but it would have required major changes in the used third party software. amavisd-new and all the others are written in perl and in the beginning i thought i could handle this, but because none of them are on CPAN and i don’t want to rewrite all my modifications each time they make an update, i decided to to start from scratch and build a solution myself – highly modularized this time. As soon as the documentation and the last tests are finished it will be released under the artistic license and/or the GPLs on CPAN.

How it works^

There are three components: policy server, content filter and log parser. All concrete actions are outsourced into modules to simplify configurations and extension. All configuration files are written in  in YAML. All three server are based on perl’s POE, which is a modern and well tested framework for any kind of server programming (among other). I have chosen the newish  moose object system (first release in 2006, but 1.0 this year), which allows developers to implement their own extensions very easily. For portable database access the servers use the perl DBI-API which allows nearly any relevant RDBS to be used (among others: MySQL, SQLite, Postgres, Oracle, ..) – and MongoDB, which i personally like a lot and think it fits just perfectly for this kind of application. Besides SQlite, all of those are can be used in a network with multiple instances and a shared dataset. Caching is the last issue. decency uses the perl Cache::-modules, which supports simple file-caches, faster mmap files as well as network-capable memcache-servers. Besides reinventing the anti-spam-server wheel, it uses a wide range of CPAN modules. I plan to release a pre-build module package on foaa and a debian package to simplify the installation for non-perlers (yes, there are still some).

The policy server^

.. works at the very front. The mail has not to be received (fully, as in body). There are modules for various checks like blacklisting (DNSBL as well as per-recipient custom blacklists), whitelisting (per recipient domain, for customers), SPF record checks, association proves (does the sender ip relate to the sender domain) and of course greylisting. There is also a honeypot which can be used for spam collection (training filters) and for building a tailored blacklist. Then there is throttle module, which can handle complex scenarios by introducing an “account” aspect, which associate recipient/sender domains/addresses/ips to a named account which can have throttle settings – of course simple per xy throttling works also. Each module scores the incoming mail until a configurable threshold is reached or all modules have performed their checks.

The content filter^

.. can be implemented via content_filter or client_*_access and comes into action after the mail is received and before it is delivered. Having the full mail, it can perform much “deeper” analysis. If one or multiple policy servers are superposed, it will take their scorings into account. If decency is deployed in a distributed environment, the policy server can sign it’s scoring to assure no spammer can inject positive scores. There is an interface for multiple anti-spam filters (for now: SpamAssassin, DSPAM, Bogofilter, CRM114, Vilpur’s Razor but could work with any command line tool) and virus filters (for now only ClamAV). A sendmail milter client is planed. Other then that it comes with DKIM checks and an mail archive module. The counter part of the policy’s honey pot is the honey collector which can either only collect spam in some archive directory or train it directly into the enabled spam filters.

The log parser^

.. is no really part of the spam fighting, but can act as a supporter for the policy server (for the throttle module) and can provide vital informations about what is happening on you mail server. Each of the previous server provide basic statistics about the actions they have performed (eg amount of rejects per module) but stats about the actual throughput, analyses of the spam sources an so on are only performed by the log parser.

Structure^

Two examples of how decency might be deployed in  a single-server or multi-server environment.

This is a very basic example of usage as it could be realized on a single server.

Deployed in a more complex, large scale environment, it could look like this.

Status^

Currently the whole app is in freeze, running on a testing mail server with a couple of spammy domains.