Databases and Caches

decency requires a database and a cache. The database is used for storing and querying persistent data (eg: greylisted sender->recipient pairs, custom whitelists, custom blacklists, honeypot blacklists and so on..) and caches to increase the performance by keeping results temporary (eg: custom blacklist results, spf results and so on). Within the “warm-up phase” (when the caches are empty) the load on the databases will be much bigger then in a running system. Some caches are persistent over restart (eg Memcached), if you use one of those your server will not suffer from this.

Databases^

There is a huge range of databases supported. In the DBD section there are a lot of relational databases (RDBMS) listed, then there is MongoDB..

DBD^

This is only an example of supported database. You have to install the driver yourself. The debian decency-perl-modules package ships with SQLite and MySQL support, anything else is up to you.

SQLite^

From the sqlite.org website: “SQLite is a software library that implements a self-contained, serverless, zero-configuration, transactional SQL database engine”. SQLite is very easy to setup and runs nearly everywhere. However, it is not very fast nor network capable.

database:
    type: DBD
    args:
        - 'dbi:SQLite:dbname=/var/spool/database/decency.db'

MySQL^

.. is a very widespread, easy to setup (hard to tune) relational database. It can be used in a network environment.

database:
    type: DBD
    args:
        - 'dbi:mysql:host=localhost;database=decency;port=1234'
        - 'username'
        - 'password'

PostgreSQL^

PostgresSQL is a very powerful open source relational database. Also network capable.

database:
    type: DBD
    args:
        - 'dbi:Pg:host=localhost;dbname=decency'
        - 'username'
        - 'password'
        # the following might come in handy, if you need:
        -
            pg_enable_utf8: 1
            name_sep: '.'
            quota_char: '"'
            on_connect_do:
                - 'SET search_path=my_schema'
                - 'SET client_encoding=UTF8'

And more ..^

There are a couple more usable databases ..

Whatever of those you prefer, keep in mind: read, read, read performance is what matters.

MongoDB^

Then there is MongoDB, which is highly recommended (above any above). It is fast as lightning and very – as in very, very – scalable. Redundancy is no problem if you setup multiple instances. The (memory|cpu) footprint can be kept very small, too.
One of those new, “fancy” but still beta applications, you conclude – but wait! Before you stop considering MongoDB (or decency for that matter) have a look at this list of MongoDB users (yes, source forge is among them!). Then ask yourself: do all of those people really have less demands in stability and reliability then i do ? Some have even switched from classy symbols of stability as Oracle to MongoDB. Hope this brings in another angle ..
Huh, before you run away because you think “Switched from oracle to MongoDB.. hmm, probably the same class of configuration monster”, it is NOT! Actually, it has one config file which you can nail down to less then 10 lines. Easy as pie.
And last but not least: installation is dead easy, too. For most distris there is a binary package, check out for debian and ubuntu, centos and fedora and the rest and here. Install, edit config, start, done.

database:
    type: MongoDB
    server: 'localhost'
    port: 27017
    database: 'decency'

    # or if you have a "left" and "right":
    #server: '123.123.123.123,234.234.234.234'
    #port: '27017,27018'

    # if authentication is required:
    #username: 'yourname'
    #password: 'yourpassword'

What’s the best database ?^

Huh, cannot say. Depends on your environment. If you whole infrastructure relies on PostgreSQL, you probably go best with it. If you require lot’s of speed, maybe scale large (or are already) and/or don’t want to hassle with tuning your RDBMS  then you should give MongoDB a try (read above). If you just want to try it out, or have a really small mail server use SQLite, because it is very easy and runs nearly everywhere.

I Could not find Berkeley Database nor CDB on the list.. did i miss it ?^

No, you didn’t. Berkeley DBs / CDBs are really nice and very fast. However, within perl you mostly deal with forked processes (not threads, but “kind of”). Sharing Berkeley DBs among those in write mode in a save matter is possible, but never satisfying. It took me nearly two days of reading, trial and error and stumbling in the dark to conclude that it probably might or might not work out. And finally, using above DBDs and MongoDB allows to store “complex” data, whereas Berkeley goes with plain “key = value” (i already wrote an abstraction, but it was a patchwork). Even if i could figure out how to share it between all writers, it still would not be usable in a network. In the end the huge speed of MongoDB made the difference and dropped Berkeley DBs for good (speed was my last argument why i was hasseling with that at all).

Caches^

Decency supports any Cache from CPAN using the Cache-API (Cache::*). Following i will introduce the four most interesting.

Memcached and Memcached::XS^

Speed:fast
Persistent: yes
Network: yes
Use Case: multiple servers

Memcached by Danga Interactive is a very popular and fast network cache server. Probably the way to go, if you have multiple mail servers and/or decency servers. The ::XS variant is the faster choice, but harder to install (if you don’t use the debian package). more..

cache:
    class: Memcached
    servers:
        - '127.0.0.1:11211'
        - '10.0.20.100:11211'

FastMmap^

Speed:fast
Persistent: yes
Network: no
Use Case: single server

Uses a so called mmaped file. Good choice for a single server (also multiple decency instances on the same machine). more..

cache:
    class: FastMmap
    share_file: /tmp/decency.mmap
    expire_time: 7d
    cache_size: 128m

See the new-method description on CPAN for more parameters.

File^

Speed: slow
Persistent: yes
Network: no
Use Case: testing

As the name implies, it is a cache base on the file system. You should not use it. It is only good for testing. more..

cache:
    class: File
    cache_root: /tmp/decency-cache
    default_expires: '86400 sec'

Memory^

Speed: very fast
Persistent: no
Network: no
Use Case: testing or stand alone + huge memory

You guessed right, puts everything in the memory. Very fast, very memory extensive and wiped on server shutdown. more..

cache:
    class: Memory
    default_expires: '86400 sec'

And more ..^

There are more, look for everything named “Cache::XYZ” here, but i think i’ve covered the most important and fitting.

Leave a Reply

CAPTCHA image