Content filter

What is a content filter ?^

A content filter comes in action after the mail is received by the mails server. It has access to the full mail (headers, body) and is therefore able to check the contents against any kind of filters such as DKIM (requires at least all headers), virus filters (require body, especially attachments), spam filters (analyzing the words in the body) and so on. In opposite to the policy behavior, the MTA will hand over / deliver the the whole mail to the content filter. The content filter has to re-inject the mail after filtering back to the mail server for delivery. Therefore, you have to have two running mail servers processes (with postfix, you can setup another one in the master.cf).

What kind of filters are there in decency ?^

In general, there are two different kinds: external, which are mostly command line programs or other running servers for which decency acts as a proxy, and internal, which are filters implemented directly into decency.

External filters^

For external filters there are for now:

Spam Filters: DSPAM, Bogofilter, CRM114, Razor, SpamAssassin
Virus Filters: ClamAV

Those are planned:

AnyMilter – module for using external milters
AVG – virus filter using AVG anti virus

Internal filters^

In the current release are:

DKIM – Domain Key checks
Archive – Archiving all or certain mails in directories / tar files
HoneyCollector – Counter part of HoneyPot in the policy server, collects spam mails and/or trains them into activated spam filters
MimeAttribs – simple mime attribute manipulation.

What other content filters are there ?^

There are multiple:

amavisd-new, most popular interface between multiple anti-spam and -virus filters
SpamAssassin, to some degree, cause you can run some “external tools”
Mailspect, commercial content filter AND policy server
Xamime, also commercial content filter AND policy server
and more ..

Configuration^

Most parts of the configuration is similar to the policy server configuration. Also, you can have multiple content filters “in a row”, on multiple servers, which are able to access previous scorings, as long as they can access the same cache.

In the debian and tar.gz package the configuration file can be found in /etc/decency/content-filter.yml.

server^

Allowed values: Hash
Default: -
Suggested: { host: “127.0.0.1″, port: 15000, instances: 3 }
Required: yes

This configures the TCP settings for the server and sets the amount of instances you want to run. The server host and port settings has to be adjusted in your postfix configuration (below) accordingly.

server.host

The hostname or IP address the server should be listen to. Default: 127.0.0.1 (localhost)

server.port

The port the server should be listen to. Default: 15000

server.instances

The amount of instances to be run in parallel. The more you run, the more mails can be handled in parallel and the memory will be used. Default: 3

server:
    host: 127.0.0.1
    port: 15000
    instances: 3

spool_dir^

Allowed values: String (path)
Default: /var/spool/decency
Required: yes

The directory where all processed mails are temporary stored. See the folder structure.

accept_scoring^

Allowed values: Bool
Default: 0
Required: no

Whether scoring from external policy servers will be accepted. All local, cached scoring informations (if content filter and policy server access the same caches) will be accepted automatically. This should only be enabled, if you provide a the verification key (policy_verify_key).

policy_verify_key^

Allowed values: String (path)
Default: -
Required: no

Path to the public part of the verification key. Provide an absolute path or a relative path from the configuration dir (where content-filter.yml is). This key verifies, that the signed scorings originate from one of your policy servers.

How you can generate the forward sign key and more about this matter, read here.

policy_verify_key: sign.pub

notification_from^

Allowed values: String (path)
Default: -
Required: no

From-header for all notifications. There are three possible notifications:

Virus sender: send to the sender of an infected mail
Virus recipient: send to the recipient of an infected mail
Spam recipient: send to the recipient of an as spam recognized mail

notification_from: 'Postmaster <postmaster@localhost>'

reinject^

Allowed values: Hash
Default: { host: “127.0.0.1“, port: 10250 }
Required: yes

Reinjection information for all to-be-passed mails (virus could be and so on). Remember to setup a MTA listening on this host/port (eg via your master.cf in postfix).

reinject.host

Can be IP or hostname.

reinject.port

The port for delivery.

reinject:
    host: 127.0.0.1
    port: 10250

spam^

Allowed values: Hash
Default: -
Required: yes

How to handle spam and when to recognize a mail as spam.

spam.behavior

There are three behaviors:

scoring – score all mails via modules. The scores will be accounted until the threshold is reached. Probably default.
strict – first module recognizing this mail as spam triggers spam handling. Be careful with that.
ignore – for debugging: modules will score the mail, but no actions will be performed.

spam.threshold

Only for “scoring” behavior. Should be a negative integer. The lower, the more mails will not be recognized as spam, the higher (closer to zero), the more mails will be marked as spam.

spam.handle

What to do, if a mail is recognized as spam ?

bounce – mail will be bounced to sender. ^{Careful! This is backscatter spam!}
delete – mail will be silently deleted
quarantine – mail will be quarantined in the spool dir.
ignore – mail will be proceeded. ^Suggested

spam.noisy_headers

If this is enabled (1), all spam headers will be included on any mail. The following matrix shows which header will be injected in the mail wheter noisy_headers are enabled or not:

	enabled	disabled
	Is SPAM	Not SPAM	Is SPAM	Not SPAM
X-Decency-Result	SPAM	GOOD	SPAM	-
X-Decency-Score	Number	Number	Number	-
X-Decency-SpamInfo	All	All	-	-

Enabling this would be the only possiblity to determine why a certain mail has been marked as spam. However, there might be some security concerns.

spam.notify_recipient

Enabling this will send a notification mail to the recipient of the spam mail. Of course, should only be used with spam.handle set to”delete” or “quarantine”. Not suggested.

spam.recipient_template

If notify_recipient is enabled. Path to the notification template. Variables usable in the template are:

%%reason%% – content of the X-Decency-SpamInfo header
%%to%% – original recipient
%%from%% – original sender
%%subject%% – original subject

spam.notify_subject

If notify_recipient is enabled. Subject of the mail. Default is “SPAM notification”.

spam:
    behavior: scoring
    threshold: -50
    handle: ignore
    noisy_headers: 1

    # for handle: bounce or delete:
    #notify_recipient: 1
    #recipient_template: 'templates/spam-recipient-notify.tmpl'
    #recipient_subject: 'Spam detection notification'

virus^

Allowed values: Hash
Default: -
Required: yes

What to do with recognized infected mails ?

virus.handle

There are four handle types:

bounce – bounce the mail back to the recipient ^{NOT recommended}
delete – delete the mail silently
quarantine – move the mail into a quarantine directory ^Suggested
ignore – just pass the mail. ^{Dangerous for the recipient.}

virus.notify_sender, virus.notify_recipient

For handle types: bounce, delete or quarantine. Enables notification of the sender / recipient of the mail. It is not recommended to send mails back to the sender!

sender_template, recipient_template

Templates used for the notification mail. Variables usable are:

%%virus%% – Name / information about the detected virus (provided from virus filter, eg ClamAV)
%%to%% – original recipient
%%from%% – original sender
%%subject%% – original subject

sender_subject, recipient_subject

Subjected used for the notifications. Defaults to “VIRUS notification”

virus:
    handle: quarantine
    #notify_sender: 1
    notify_recipient: 1
    #sender_template: 'templates/virus-sender-notify.tmpl'
    #sender_subject: 'Virus detection notification'
    recipient_template: 'templates/virus-recipient-notify.tmpl'
    recipient_subject: 'Virus detection notification'

filters^

Allowed values: List of Module Hashes or List of Module configuration file paths
Default: none
Required: yes

This is the most important part, because all modules are activated here. You can either use inline configuration or include external files. See the policy server “policy” section for an example.

cache, database and logging^

Read here

Full YAML example^

---

spool_dir: /var/spool/decency

accept_scoring: 1
policy_verify_key: sign.pub

notification_from: 'Postmaster <postmaster@localhost>'

enable_stats: 1

include:
    - logging.yml
    - database.yml
    - cache.yml

server:
    host: 127.0.0.1
    port: 16000
    instances: 3

reinject:
    host: 127.0.0.1
    port: 10250

spam:
    behavior: scoring
    threshold: -50
    handle: tag
    noisy_headers: 1
    spam_subject_prefix: "SPAM:"

    # for handle: bounce or delete:
    #notify_recipient: 1
    #recipient_template: 'templates/spam-recipient-notify.tmpl'
    #recipient_subject: 'Spam detection notification'

virus:
    handle: bounce

    # for handle: bounce, delete or quarantine
    notify_sender: 1
    notify_recipient: 1
    sender_template: 'templates/virus-sender-notify.tmpl'
    sender_subject: 'Virus detection notification'
    recipient_template: 'templates/virus-recipient-notify.tmpl'
    recipient_subject: 'Virus detection notification'

filters:
    #- MimeAttribs: "content-filter/mime-attribs.yml"
    - DKIM: "content-filter/dkim.yml"
    # - ClamAV: content-filter/clamav.yml
    - Bogofilter: content-filter/bogofilter.yml
    #- DSPAM: content-filter/dspam.yml
    - CRM114: content-filter/crm114.yml
    - Razor: content-filter/razor.yml
    - HoneyCollector: content-filter/honey-collector.yml
    # -
    #     SpamAssassin:
    #         disable: 0
    #         default_user:
    #         weight_translate:
    #             1: -100
    #             -2: 0
    #             -3: 100
    - Archive: content-filter/archive.yml

Shared module configuration^

Default^

Can be used in any module. See here for timeout and disable.

max_size^

Allowed values: Integer (size in bytes)
Default: 0
Required: no

Can be set in each content filter module. If the mail is bigger then max_size (and max_size is > 0) the mail will not filtered by the module. Most (external) filters might take the longer, the bigger the mail is.

Anti Spam modules^

weight_innocent^

Default: 10
Allowed values: Integer
Required: yes

The score which should be applied for innocent (good, HAM) mails. Should not be as high as the spam weight (in absolute terms).

weight_spam^

Default: -50
Allowed values: Integer
Required: yes

The score which should be applied for SPAM mails. The higher, the more weight will this module have.

weight_translate^

Default: -
Allowed values: HashRef[Int]
Required: no

Can be set instead of weight_innocent and weight_spam. Translates numeric CRM114 response to a score for decency.

Example:

CRM114 lower then -10 will be translated to -100, scores between -3 and -10 will be translated to -50 and so on ..

weight_translate:
     5: 20
     1: 10
     0: 0
     -2: 0
     -3: -50
     -10: -100

User module^

Many modules support a “user” in some kind. For some this is a configuration path (eg CRM114) and for some this is a unix user account. The “user” will be determined in the following order:

An external script/program which provide decency with the user/path (cmd_user)
A default user (global/shared) (default_user)
The recipient (MAIL TO) of the mail

default_user^

Default: -
Allowed values: String (path)

See the modules docu for further information. If this is set, it will overwrite the recipient. Common scenario: one shared configuration for CRM114, one shared user for DSPAM and so on..

cmd_user^

Default: -
Allowed values: String (path to command line)
Required: no

Command line for determining the user. Can be any executable shell script / prog. The program will get the “RCPT TO” in STDIN and has to print out the user. Example:

echo "recipient@domain.tld" | /usr/local/bin/determine-crm-user
# prints: /etc/crm114/local-user/

Integration in Postfix^

You have to edit two files: master.cf and main.cf in /etc/postfix.

master.cf^

Add the following to the end of your master.cf file:

# the decency server itself
decency	unix  -       -       n       -       4        smtp
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20
    -o smtp_send_xforward_command=yes
    -o disable_mime_output_conversion=yes
    -o smtp_destination_recipient_limit=1

# re-inject mails from decency for delivery
127.0.0.1:10250      inet  n       -       -       -       -       smtpd
    -o content_filter=
    -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
    -o smtpd_helo_restrictions=
    -o smtpd_client_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject_unauth_destination,permit
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8

main.cf^

There are two possible ways you can include this content filter into postfix. The first is via content_filter, the second via check_*_access, eg check_client_access.

content_filter^

The advantage: it is easy. The disadvantage: all mails (incoming, outgoing) will be filtered. In a one-mail-server-for-all configuration this might be ugly.

# main.cf
content_filter = decency:127.0.0.1:16000

Via check_*_access^

# main.cf
smtpd_client_restrictions =
    # ...
    check_client_access = pcre:/etc/postfix/decency-filter, reject
    # ...

Then in the /etc/postfix/decency-filter file (has to be readable by the postfix user):

# /path/to/access
/./ FILTER decency:127.0.0.1:16000

Comment (RSS) | Trackback

decency @ Foaa