Modular Python Postfix Policy Server
====================================
Modular Python Postfix Policy Server is tool for extending Postfix
checking capabilities. It uses Postfix access policy delegation
(http://www.postfix.org/SMTPD_POLICY_README.html) to check incoming
SMTP request and accept or reject it according provided data. It can
reduce mailserver load with rejecting incorrect mail during SMTP
connection. It was made with stress to height reliability and performance
by providing caching of required data and results.
Because it has modular design it can be easily extended by custom
modules (only one method has to be implemented and everything else is
handled automatically). By default it provide modules for SPF
checking, domain mailhost checking, sender/recipient verification,
black/white listing, greylisting, spam traps, DNS blacklists, ...
You can configure how to use all modules by implementing one simple
method in config file. It provide to you ability to define check
dependencies and use different modules according to the result of
previous.
1. Installation
1.1 Requirements
postfix >= 2.1 - http://www.postfix.org
python >= 2.3 - http://www.python.org
python-twisted >= 1.3 - http://twistedmatrix.com
MySQL * >= 3.23 - http://www.mysql.com
python-GeoIP * >= 1.2.1 - http://www.maxmind.com/app/python
python-ldap * >= 2.0.x - http://python-ldap.sourceforge.net
dnspython ** >= 1.3.3 - http://www.dnspython.org
pyperl *** >= 1.0.1 - http://ftp.activestate.com/Zope-Perl
* required by some check modules (but not by application core)
** version 1.3.5 is much slower then 1.3.4 because of changes in resolver
(may be that newer version is better but I did not check it)
*** required by modules that directly call perl (none module now)
you can use local patched copy (works with python 2.4 and perl 5.8.8)
http://kmlinux.fjfi.cvut.cz/~vokac/activities/ppolicy/download/pyperl
1.2 PPolicy download
Primary site for this PPolicy server is
http://kmlinux.fjfi.cvut.cz/~vokac/activities/ppolicy/download. You can
download there prebuild RPM package or sources in SRPM and tgz format.
1.3 PPolicy installation
1.3.1 Install from RPM
rpm -Uvh ppolicy-2.x.x.noarch.rpm
1.3.2 Install from sources
Rebuild SRPM package or install application directly from tgz package.
* building SRPM:
rpmbuild --rebuild ppolicy-2.x.x.src.rpm
rpm -Uvh /path/to/builded/package/ppolicy-2.x.x.noarch.rpm
* building tgz:
tar xzf ppolicy-2.x.x.tar.gz
cd ppolicy-2.x.x
python setup.py install
# if you want to use different than default python installation directory
# than you should omit last step, but you have to set basePath
# in ppolicy.conf configuration file
# installing additional required files
cp ppolicy.tap /usr/sbin/ppolicy.tap
cp ppolicy.init /etc/init.d/ppolicy # this script was tested on RedHat
cp ppolicy.conf /etc/postfix
mkdir -p /var/log/ppolicy # this path is defined in /etc/init.d/ppolicy
mysql < ppolicy.sql # if you want to use database
1.3.2 Starting ppolicy
* use startup script /etc/init.d/ppolicy
* or start ppolicy daemon manually using command
/usr/bin/python /usr/bin/twistd \
--pidfile=/var/log/ppolicy/ppolicy.pid \
--rundir=/var/log/ppolicy \
--file=/usr/sbin/ppolicy.tap \
--python=/usr/sbin/ppolicy.tap \
--logfile=/var/log/ppolicy/ppolicy.log \
--no_save
2. Configuration
2.1 PPolicy
Configuration of ppolicy daemon is done through ppolicy.conf file. Default
location for this file is /etc/postfix/ppolicy.conf. If you place this file
somewhere else than you have to change this information in ppolicy.tap
twisted bootstrap file that is installed by default to /usr/sbin.
There are plenty of comments and examples in ppolicy.conf so you
should be able to adapt it to your needs. For details about particular
check modules you can read class description in source file or
documentation exported in file MODULES.
2.2 Postfix configuration
Update configuration according following examples and adapt to your needs.
/etc/postfix/main.cf:
smtpd_recipient_restrictions =
...
reject_unauth_destination
check_policy_service inet:127.0.0.1:10030
...
127.0.0.1:10030_time_limit = 3600
2.3 Database configuration
Database is used to permanently store result of some expensive
checks. In most cases it should be possible to run ppolicy without
database backend but it is not recommended because of degraded
performance and (may be) more bug (it was not fully tested without
database).
Only MySQL database was tested with ppolicy but it should not be
difficult to adapt to any SQL database. You have to create new ppolicy
database and change database connection information in configuration
file ppolicy.conf. User you use to connect to ppolicy database must
have rights to create new tables. All required tables are created
automatically during ppolicy server startup.
New (MySQL) database can be created using ppolicy.sql template, but
you should first change predefined password. To run this SQL script
you need database user with CREATE DATABASE privileges, e.g.:
mysql --hostname=localhost --user=root --password=secret < ppolicy.sql
3. Modules
3.1 Existing modules
* Country
* Dnsbl
* DnsblDynamic
* DnsblScore
* DOS
* Dummy
* DumpDataDB
* DumpDataFile
* Greylist
* List
* ListBW
* ListDyn
* ListMailDomain
* LookupLDAP
* P0f
* Resolve
* SPF
* Trap
* Verification
* Whois
Detailed descriptions of all modules can be found in file named MODULES.
There are also information about all parameters and basic examples
how to use the module in config file.
3.2. Writing new modules
Before you start to write custom modules look at IPPolicyServerCheck,
Base and mainly Dummy module. There are plenty of comments that can be
useful to know. The heart of each module is check() method that is
called when you use module in config file. Next important is hashArg()
which influence data caching (it is key for result cache dictionary),
start() that is called before first usage of check() method. Last method
that can be implemented is stop() and it is called e.g. when application
terminating.
4. Bug reports
If you find some bug or performance bottleneck than let me know or
better send me patch. You can contact me using following address
Petr Vokac <vokac[at]kmlinux.fjfi.cvut.cz>
Module Country
--------------
Country module for recognizing country according IP address
or domain name.
Parameters:
cacheNegative (0)
maximum time for caching negative result
cachePositive (0)
maximum time for caching positive result
cacheUnknown (0)
maximum time for caching unknown result
country (None)
check if IP is in this country
dataPath (None)
path to GeoIP.dat file
factory (None)
reference to factory instance
param (None)
name of parameter in data dictionary
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
Check arguments:
data ... all input data in dict
Check returns:
1 .... ok, country match, second parameter is country name
0 .... undefined (some error)
-1 ... failed, country doesn't match, second parameter is country name
Examples:
# return country for client_address
modules['country1'] = ( 'Country', { 'param': 'client_address',
'dataPath': '/usr/share/GeoIP/GeoIP.dat' } )
# check if client_address is 'cz'
modules['country2'] = ( 'Country', { 'param': 'client_address',
'dataPath': '/usr/share/GeoIP/GeoIP.dat',
'country': 'CZ' } )
Module Dnsbl
------------
Check if client address is listed in specified DNS blacklist.
see tools/dnsbl.txt for list of valid blacklist names - original
file can be downloaded from http://moensted.dk/spam/drbsites.txt
You can also run `python tools/dnsbl.py --list` to see formated
output of configured balacklists.
Parameters:
cacheNegative (43200)
maximum time for caching negative result
cachePositive (21600)
maximum time for caching positive result
cacheUnknown (1800)
maximum time for caching unknown result
dnsbl (None)
name of DNS blacklists defined in this module
factory (None)
reference to factory instance
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
Check arguments:
data ... all input data in dict
Check returns:
1 .... listed in dnsbl
0 .... unknown error (e.g. DNS problems)
-1 ... not listed
Examples:
# check if sender mailserver is in ORDB blacklist
modules['dnsbl1'] = ( 'Dnsbl', { dnsbl="ORDB" } )
Module DnsblScore
-----------------
Check if client address and sender domain is in various DNS
blacklists and make sum of scores that are defined in spamassassin
config files. Returned result depends on parameter treshold (this
module can return -1,0,1 or exact score if treshold is None).
This module use tools.dnsbl.score to score mail. By default
it uses all available checks, but you can specify your custom by
listing their name in dnsbl parameter. You can list all valid
names by calling `python tools/dnsbl.py --list`.
Parameters:
cacheNegative (43200)
maximum time for caching negative result
cachePositive (21600)
maximum time for caching positive result
cacheUnknown (1800)
maximum time for caching unknown result
dnsbl (None)
list of DNS blacklist to use
factory (None)
reference to factory instance
params (['client_address', 'sender'])
which params we should check
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
treshold (None)
treshold that define which score mean this module fail
Check arguments:
data ... all input data in dict
Check returns:
1 (positive) .... score > treshold or positive score
0 ............... unknown error (e.g. DNS problems)
-1 (negative) ... score < treshold or negative score
Examples:
# return blacklist score for client address
modules['dnsbl1'] = ( 'DnsblScore', {} )
# check if blacklist score for client address exceed defined treshold
modules['dnsbl1'] = ( 'DnsblScore', { treshold=5 } )
Module DnsblDynamic
-------------------
Check if client address is in dynamic allocated ranges. It use
corresponding dnsbl and also domain name (e.g. format
xxx-yyy-zzz.dsl.provider.com).
There will by probably many "correct" mailservers sitting on IP
address from dynamic alocated space. Use the result as one of the
decision rule and don't reject mail only relaying on this result.
Parameters:
cacheNegative (43200)
maximum time for caching negative result
cachePositive (21600)
maximum time for caching positive result
cacheUnknown (1800)
maximum time for caching unknown result
check_name (True)
check format of client name (e.g. xxx-yyy-zzz.dsl.provider.com)
dnsbl (None)
list of DNS blacklists
factory (None)
reference to factory instance
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
Check arguments:
data ... all input data in dict
Check returns:
1 .... client address seems to be in dynamic ip range
0 .... unknown error (e.g. DNS problems)
-1 ... client address doesn't seem to be in dynamic ip range
Examples:
# check if sender mailserver is in ORDB blacklist
modules['dnsbl1'] = ( 'DnsblDynamic', { 'dnsbl': [ 'NJABLDYNA', 'SORBSDUL' ] } )
Module DOS
----------
Limit number of incomming mail that has same parameters. You
can e.g. limit number of messages that can be accepted by one
recipient from one sender in defined period of time. It will
divide time interval into several parts and agregate information
about number of incomming mails. Be carefull setting this module,
because badly choosed parameters can exhause a lot of memory.
Some mail should not be blocked (e.g. to postmaster@your.domain.com,
from <>, ...) and your configuration should take care of it.
Parameters:
cacheNegative (0)
maximum time for caching negative result
cachePositive (0)
maximum time for caching positive result
cacheUnknown (0)
maximum time for caching unknown result
factory (None)
reference to factory instance
limitCount (1000)
number of messages accepted during limitTime period
limitGran (10)
data collection granularity
limitTime (3600)
time period for limitCount messages
params (None)
parameters that will be used to test equality
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
Check arguments:
data ... all input data in dict
Check returns:
1 .... frequency reached defined limit
0 .... unrelated error (e.g. database problem)
-1 ... frequency is acceptable
Examples:
# limit number of mail from one sender to 1000/hod
modules['dos1'] = ( 'DOS', { 'params': "sender" } )
# limit number of mail from one sender
# and one mailserver to 100 in 10 minutes
modules['dos2'] = ( 'DOS', { 'params': ["sender","client_address"],
'limitCount': 100, 'limitTime': 10 } )
Module Dummy
------------
Dummy module skeleton for creating new modules
Parameters:
cacheNegative (900)
maximum time for caching negative result
cachePositive (900)
maximum time for caching positive result
cacheUnknown (900)
maximum time for caching unknown result
factory (None)
reference to factory instance
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
test1 (None)
test parameter 1
test2 (abc)
test parameter 2
Check arguments:
data ... all input data in dict
Check returns:
1 .... ok
0 .... undefined (default for this module)
-1 ... failed
Examples:
# make instance of dummy module
modules['dummy1'] = ( 'Dummy', {} )
# make instance of dummy module with parameters
modules['dummy2'] = ( 'Dummy', { 'test1': 1, 'test2': "def" } )
Module DumpDataDB
-----------------
Dump data from incomming request into the database. These informations
can be used to improve debugging of other modules or to gather
statistical data for further analysis. This module should be safe to
use in sense its check method doesn't raise any exception.
Parameters:
cacheNegative (0)
maximum time for caching negative result
cachePositive (0)
maximum time for caching positive result
cacheUnknown (0)
maximum time for caching unknown result
factory (None)
reference to factory instance
interval (None)
split interval (value depends on split type)
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
split (None)
split database (None, "records", "date")
table (dump)
database where to dump data from requests
Check arguments:
data ... all input data in dict
Check returns:
this module always return 1 and resEx is set to the new database ID
Examples:
# definition for module for saving request info in default
# database table 'dump'
modules['dumpdb1'] = ( 'DumpDataDB', {} )
# module that save info in custom defined table
modules['dumpdb2'] = ( 'DumpDataDB', { table="my_dump" } )
Module DumpDataFile
-------------------
Dump data from incomming request into the file. These informations
can be used to improve debugging of other modules or to gather
statistical data for further analysis. This module should be safe to
use in sense its check method doesn't raise any exception.
Parameters:
cacheNegative (0)
maximum time for caching negative result
cachePositive (0)
maximum time for caching positive result
cacheUnknown (0)
maximum time for caching unknown result
factory (None)
reference to factory instance
fileName (None)
file where to dump data from requests
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
Check arguments:
data ... all input data in dict
Check returns:
this module always return 0 (undefined result)
Examples:
# definition for module for saving request info in file
modules['dumpfile1'] = ( 'DumpDataFile', { fileName="/var/spool/ppolicy/dump.dat" } )
Module Greylist
---------------
Greylist implementation. Mail thats triplet (from,to,client)
was not seen before should be rejected with code 450 (temporary
failed). It relay on fact that spammer software will not try to
send mail once again and correctly configured mailservers must
try it one again (see RFC 2821).
Be carefull because some poorly configured mailserver did not
retry to send mail again and some mailing list has unique sender
name for each mail and this module delay delivery and increase
load of remote server.
You should run following cleanup tasks from cron job (may be it
will be part of this module in the future). Replace XXXX and YYYY
with same value as for "mustRetry" and "expire":
mysql 'DELETE FROM `greylist` WHERE UNIX_TIMESTAMP(`date`) + XXXX < UNIX_TIMESTAMP() AND `state` = 0
mysql 'DELETE FROM `greylist` WHERE UNIX_TIMESTAMP(`date`) + YYYY < UNIX_TIMESTAMP()
Parameters:
cacheNegative (60)
maximum time for caching negative result
cachePositive (86400)
maximum time for caching positive result
cacheUnknown (30)
maximum time for caching unknown result
delay (600)
how long to delay mail we see its triplet first time
expiration (2678400)
expiration of triplets in database
factory (None)
reference to factory instance
mustRetry (43200)
time we wait to receive next mail after we geylisted it
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
table (greylist)
greylist database table
Check arguments:
data ... all input data in dict
Check returns:
2 .... always allow postmaster
1 .... was seen before
0 .... some error occured (database, dns, ...)
-1 ... greylist in progress
-2 ... invalid sender address
Examples:
# greylisting module with default parameters
modules['greylist1'] = ( 'Greylist', {} )
# greylisting module with own database table "grey", delay set
# to 1 minute and expiration time of triplets to 1 year
modules['greylist2'] = ( 'Greylist', { table="grey", delay=60,
expiration=86400*365 } )
Module List
-----------
Check if parameter value (or array of parameter values) is in
database table. If you don't need complex cartesian product searches
then it can be more efficient to use simple LookupDB module.
If data type of input value is array, it is created cartesian product
of all input values. E.g. if input is { 'param1': [ 'a', 'b' ],
'param2': [ 'c', 'd' ] } then it tries to search following pairs
in the database: [ ('a', 'c'), ('a', 'd'), ('b', 'c'), ('b', 'd') ]
This module use additional internal result cache for each db query.
It is more efficient for further queries with similar values in the
array of incomming parameters.
Parameters:
cacheAll (False)
cache all records in memory
cacheAllRefresh (900)
refresh time in case of caching all records
cacheCaseSensitive (False)
case-sensitive cache (set to True if you are using case-sensitive text comparator on DB column
cacheNegative (900)
maximum time for caching negative result
cachePositive (900)
maximum time for caching positive result
cacheUnknown (900)
maximum time for caching unknown result
column (None)
name of database column or array of columns according "param"
factory (None)
reference to factory instance
memCacheExpire (900)
memory cache expiration - used only if param value is array
memCacheSize (1000)
memory cache max size - used only if param value is array
param (None)
name of parameter in data dictionary (value can be string or array)
retcols (None)
name of column returned by check method
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
table (None)
name of database table where to search parameter
Check arguments:
data ... all input data in dict
Check returns:
1 .... parameter was found in db, second parameter include selected row
0 .... failed to check (request doesn't include required param,
database error, ...)
-1 ... parameter was not found in db
Examples:
# module for checking if sender is in database table list
modules['list1'] = ( 'List', { 'param="sender" } )
# check if sender domain is in database table my_list
# compare case-insensitive and return whole selected row
modules['list2'] = ( 'List', {
'param': [ "sender", "recipient" ],
'table': "my_list",
'column': [ "my_column1", "my_column2" ],
'cacheCaseSensitive': False,
'retcols': "*" } )
Module ListBW
-------------
Check if array of defined parameters are in blacklist/whitelist.
Each parameter is searched in b/w list and first occurence is returned.
Parameters:
cacheAll (False)
cache all records in memory
cacheAllRefresh (900)
refresh time in case of caching all records
cacheCaseSensitive (False)
case-sensitive cache (set to True if you are using case-sensitive text comparator on DB column
cacheNegative (900)
maximum time for caching negative result
cachePositive (900)
maximum time for caching positive result
cacheUnknown (900)
maximum time for caching unknown result
factory (None)
reference to factory instance
mappingBlacklist ({})
mapping between params and database columns for blacklist
mappingWhitelist ({})
mapping between params and database columns for whitelist
param (None)
name of parameter in data dictionary (value can be string or array)
retcolsBlacklist (None)
name of blacklist columns returned by check method
retcolsWhitelist (None)
name of whitelist columns returned by check method
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
tableBlacklist (None)
name of blacklist database table where to search parameter
tableWhitelist (None)
name of whitelist database table where to search parameter
Check arguments:
data ... all input data in dict
Check returns:
1 .... parameter was found in whitelist, second parameter include selected row
0 .... parameter was not found in whitelist/blacklist or failed to check
(request doesn't include required param, database error, ...)
second parameter - None in case of error
not None in case record was not found in b/w list
1 .... parameter was found in blacklist, second parameter include selected row
Examples:
# module for checking if sender is in blacklist/whitelist
modules['list1'] = ( 'ListBW', { 'param': "sender_splitted" } )
# check if sender in blacklist/whitelist
# compare case-insensitive and return whole selected row
modules['list2'] = ( 'ListBW', {
'param': "sender_splitted",
'tableBlacklist': "my_blacklist",
'tableWhitelist': "my_whitelist",
'mappingBlacklist': { "my_param": ("my_column", "VARCHAR(50)") },
'mappingWhitelist': {"my_param": ("my_column", "VARCHAR(50)") },
'cacheCaseSensitive': False,
'retcol': "*" } )
Module ListDyn
--------------
This module provide general access (add/remove/check) to the
persistent data store in database. You can use this module for
black/white lists, persistent data cache for result from other
modules, ...
One of the most important parameter is "mapping". It is used to
map request parameters database columns. Its syntax is [
'dictName': ('columnName', 'columnType'), ...],
where dictName is attribude name comming in check method in data
dictionary, columnName is database column name, column type is
database column type in SQL syntax. If don't define mapping for
concrete column, than default is used 'dictName':
('dictName', 'VARCHAR(255)')
Second very important parametr is "operation". With this parameter
you specify what this module should do by default with passed request.
Also result returned from check method depends on this operation.
add ...... add new data to database
remove ... remove all matching data from database
check .... check if database include data from request
You can also set soft and hard expiration time for the
records. For example you can specify, that after 1 hour it will
return SOFT_NEGATIVE constant and after 2 hours
HARD_NEGATIVE. This can be usefull when you use this module as
persistent cache for some other module and you are not sure that
the its results are always reachable (e.g. when it uses DNS and it
is temporarly unreachable).
Parameters:
cacheNegative (0)
maximum time for caching negative result
cachePositive (0)
maximum time for caching positive result
cacheUnknown (0)
maximum time for caching unknown result
factory (None)
reference to factory instance
hardExpire (None)
expiration time for the record (0 == never)
mapping ({})
mapping between params and database columns
operation (check)
list operation (add/remove/check)
param (None)
names of input data keys used to identify data row(s)
retcols (None)
names of columns to be returned (None mean no related data)
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
softExpire (None)
information that record will be soon expired (0 == never))
table (list)
name of database table where to search parameter
Check arguments:
data ... all input data in dict
operation ... add/remove/check operation that overide the default
Check returns:
add, remove
1 .... operation was successfull
0 .... error (database error, ...)
check
2 .... parameters are in list, but soft expired
1 .... parameters are in list
0 .... failed to check (database error, ...)
-1 ... parameters are not in list
Examples:
# module for checking if sender is in database table list1
modules['list1'] = ( 'ListDyn', { 'table': 'list1',
'mapping': { "sender": "mail", } } )
# module for checking/getting if sender row values in database table list2
modules['list2'] = ( 'ListDyn', { 'table': 'list2',
'param': 'sender', 'retcols': [ "recip" ],
'mapping': { "sender": ("mail", "VARCHAR(50)"),
"recip": ("rmail", "VARCHAR(50)") },
} )
# module with soft/hard expiration and 'add' as default operation
modules['list3'] = ( 'ListDyn', { 'table': 'list3', 'operation': 'add',
'softExpire': 60*10, 'hardExpire': 60*30 } )
Module ListMailDomain
---------------------
Check mail address or its part is in database. Can be used for
black/white listing sender or recipient mail addresses.
Parameters:
cacheAll (True)
cache all records in memory
cacheAllRefresh (900)
refresh time in case of caching all records
cacheCaseSensitive (False)
case-sensitive cache (set to True if you are using case-sensitive text comparator on DB column
cacheNegative (900)
maximum time for caching negative result
cachePositive (900)
maximum time for caching positive result
cacheUnknown (900)
maximum time for caching unknown result
factory (None)
reference to factory instance
mappingBlacklist ({})
mapping between params and database columns for blacklist
mappingWhitelist ({})
mapping between params and database columns for whitelist
param (None)
name of parameter in data dictionary (value can be string or array)
retcolsBlacklist (None)
name of blacklist columns returned by check method
retcolsWhitelist (None)
name of whitelist columns returned by check method
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
tableBlacklist (None)
name of blacklist database table where to search parameter
tableWhitelist (None)
name of whitelist database table where to search parameter
Check arguments:
data ... all input data in dict
Check returns:
1 .... parameter was found in whitelist, second parameter include selected row
0 .... parameter was not found in whitelist/blacklist or failed to check
(request doesn't include required param, database error, ...)
second parameter - None in case of error
not None in case record was not found in b/w list
1 .... parameter was found in blacklist, second parameter include selected row
Examples:
# module for checking if sender is in database b/w list
modules['list1'] = ( 'ListMailDomain', { paramMailDomain="sender" } )
# check if sender domain is in database table my_blacklist, my_whitelist
modules['list2'] = ( 'ListMailDomain', { paramMailDomain="sender",
tableBlacklist="my_blacklist",
tableWhitelist="my_whitelist",
column="my_column",
retcol="*" } )
Module LookupDB
---------------
LookupDB module for searching records in DB.
You can map incomming data to database columns using "mapping"
parameter. Its syntax is [ 'dictName': ('columnName',
'columnType'), ...], where dictName is attribude name comming in
check method in data dictionary, columnName is database column
name, column type is database column type in SQL syntax. If don't
define mapping for concrete column, than default is used
'dictName': ('dictName', 'VARCHAR(255)')
Parameters:
cacheAll (False)
cache all records in memory
cacheAllRefresh (900)
refresh time in case of caching all records
cacheCaseSensitive (False)
case-sensitive cache (set to True if you are using case-sensitive text comparator on DB column
cacheNegative (900)
maximum time for caching negative result
cachePositive (900)
maximum time for caching positive result
cacheUnknown (900)
maximum time for caching unknown result
factory (None)
reference to factory instance
mapping ({})
mapping between params and database columns
param (None)
name of parameter in data dictionary (value can be string or array)
retcols (None)
name of column returned by check method
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
table (None)
name of database table where to search parameter
Check arguments:
data ... all input data in dict
Check returns:
1 .... parameter was found in db, second parameter include selected row
0 .... failed to check (request doesn't include required param,
database error, ...)
-1 ... parameter was not found in db
Examples:
# module for checking if sender is in database table
modules['lookup1'] = ( 'LookupDB', { 'param': "sender" } )
# check if sender domain is in database table my_list
# compare case-insensitive and return whole selected row
modules['lookup2'] = ( 'LookupDB', {
'param': [ "sender", "recipient" ],
'table': "my_list",
'mapping': { "sender": ("mail", "VARCHAR(50)"), },
'retcols': "*" } )
Module LookupLDAP
-----------------
LookupLDAP module for searching records in LDAP
Parameters:
attributes (None)
attributes returned by LDAP query
base (None)
LDAP search base
bind_dn (None)
LDAP bind DN (None == anonymous bind)
bind_method (128)
LDAP bind method (only simple is supported)
bind_pass (None)
LDAP bind password
cacheNegative (900)
maximum time for caching negative result
cachePositive (900)
maximum time for caching positive result
cacheUnknown (900)
maximum time for caching unknown result
factory (None)
reference to factory instance
filter (None)
LDAP filter (%m will be replaced by "param" value
param (None)
name of parameter in data dictionary
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
scope (2)
LDAP search scope
uri (None)
LDAP server URI
Check arguments:
data ... all input data in dict
Check returns:
1 .... ok, found records for the filter
0 .... undefined (problem with LDAP query, e.g. failing connection)
-1 ... failed to find records for the filter
Examples:
# recipient ldap lookup in ldap1/2.domain.tld with "mail" filter
modules['lookup_ldap1'] = ( 'LookupLDAP', { 'param': 'recipient',
'uri': 'ldap://ldap1.domain.tld ldap://ldap2.domain.tld',
'base': 'ou=People,dc=domain,dc=tld',
'filter': '(mail=%m)' } )
# recipient ldap lookup that returns common name attribute
modules['lookup_ldap1'] = ( 'LookupLDAP', { 'param': 'recipient',
'uri': 'ldap://ldap1.domain.tld ldap://ldap2.domain.tld',
'base': 'ou=People,dc=domain,dc=tld',
'bind_dn': 'cn=PPolicy User,ou=People,dc=domain,dc=tld',
'bind_pass': 'secret',
'scope': ldap.SCOPE_ONELEVEL,
'filter': '(mail=%m)', 'attributes': 'cn' } )
Module P0f
----------
P0f module detect sender OS using p0f and its unix socket interface.
You can use patched version of p0f that requires only sender IP to detect
OS - it is less reliable but easier to use, because you don't have to
specify "ip" and "port" parameters. It can be downloaded from
http://kmlinux.fjfi.cvut.cz/~vokac/activities/ppolicy/download/p0f
Start p0f with -0 parameter for releases >= 2.0.8, earlier releases
works only if you apply patch that can be downloaded on given URL.
This module returns tuple with information described in p0f-query.h
p0f_response structure. Most important for our purposes are
retEx[3] ... detected OS
retEx[4] ... details about detected OS (e.g. version)
retEx[5] ... distance of sender
retEx[12] .. uptime (not reliable)
Parameters:
cacheNegative (3600)
maximum time for caching negative result
cachePositive (3600)
maximum time for caching positive result
cacheUnknown (900)
maximum time for caching unknown result
factory (None)
reference to factory instance
ip (None)
IP address of destionation server
port (25)
port address of destionation server
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
socket (/var/run/p0f.socket)
p0f socket for sending requests
version (2.0.8)
p0f version
Check arguments:
data ... all input data in dict
Check returns:
1 .... success (p0f response structure in second parameter)
0 .... undefined (no data)
-1 ... failed (error getting data from p0f)
Examples:
# make instance of p0f module
modules['p0f1'] = ( 'P0f', {} )
# make instance of p0f module with different socket path
modules['p0f2'] = ( 'P0f', { 'socket': '/tmp/p0f.socket'
'ip': '192.168.0.1', 'port': 1234 } )
Module Resolve
--------------
Try to resolve ip->name, name->ip, name->mx, ip->name->ip,
name->ip->name, ip1->name->ip2, ip->name->mx, name1->ip->name2.
Parameters:
cacheNegative (21600)
maximum time for caching negative result
cachePositive (86400)
maximum time for caching positive result
cacheUnknown (1800)
maximum time for caching unknown result
factory (None)
reference to factory instance
param (None)
which request parameter should be used
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
type (None)
ip->name, name->ip, name->mx, ip->name->ip, name->ip->name, ip1->name->ip2, ip->name->mx, name1->ip->name2
Check arguments:
data ... all input data in dict
Check returns:
1 .... all tranlation were successfull and equal
0 .... problem resolving ip or name (DNS error)
-1 ... translation failed
Examples:
# check if sender domain exist
modules['resolve1'] = ( 'Resolve', { param="sender",
type="name->ip" } )
# check if remote mailserver has reverse records in DNS
modules['resolve2'] = ( 'Resolve', { param="client_address",
type="ip->name" } )
# check if remote mailserver has reverse records in DNS
# and translating back returns set of IP that contains original IP
modules['resolve3'] = ( 'Resolve', { param="client_address",
type="ip->name->ip" } )
Module SPF
----------
This module use sender address and client IP to check SPF
records in DNS if they are exist.
More informations about SPF can be found at following address
http://www.openspf.org
http://en.wikipedia.org/wiki/Sender_Policy_Framework
Parameters:
cacheNegative (86400)
maximum time for caching negative result
cachePositive (86400)
maximum time for caching positive result
cacheUnknown (21600)
maximum time for caching unknown result
factory (None)
reference to factory instance
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
Check arguments:
data ... all input data in dict
Check returns:
1 .... SPF passed, second parameter contain values returned
by tools.spf function (result, status, explanation)
0 .... undefined SPF records or exception when checking SPF
-1 ... SPF softfail or temperror
-2 ... SPF fail or permerror
Examples:
# define module for checking SPF
modules['spf1'] = ( 'SPF', {} )
Module Sleep
------------
Sleep module for debugging
Parameters:
cacheNegative (900)
maximum time for caching negative result
cachePositive (900)
maximum time for caching positive result
cacheUnknown (900)
maximum time for caching unknown result
factory (None)
reference to factory instance
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
sleep (1)
sleep interval in seconds (default 1s)
Check arguments:
data ... all input data in dict
Check returns:
1 .... ok
0 .... undefined (default for this module)
-1 ... failed
Examples:
# make instance of Sleep module
modules['Sleep1'] = ( 'Sleep', {} )
# make instance of Sleep module with parameters
modules['Sleep2'] = ( 'Sleep', { 'sleep': 10 } )
Module Trap
-----------
Trap module catches mail probing random recipient addresses. If in
defined time fall more than defined amount of mail from one client to
the trap, all mail from that client will be temporarly blocked.
Parameters:
cacheNegative (0)
maximum time for caching negative result
cachePositive (0)
maximum time for caching positive result
cacheUnknown (0)
maximum time for caching unknown result
expire (3600)
expiration time for client that send trapped email
factory (None)
reference to factory instance
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
traps (None)
comma separated list of trap email addresses
treshold (-2)
how many traps has to be touched before blacklisting (negative value mean fraction 1/x)
Check arguments:
data ... all input data in dict
Check returns:
1 .... request from client that send many mail touched our mail traps
0 .... some problem
-1 ... client is not touched mail traps
Examples:
# define trap to block client_address for one hour if we receive
# one mail with recipient spamtrap1 or spamtrap2
modules['trap1'] = ( 'Trap', { traps="spamtrap1@domain.com,spamtrap2@domain.com" } )
Module Verification
-------------------
Module for mx, connection, domain and user verification. It
check domain existence and then it try to establish SMTP
connection with mailhost for the domain (MX, A or AAAA DNS records
- see RFC2821, chapter 5).
You can check if sender/recipient or whatever reasonable has
correct DNS records as mailhost and try to connect to this server.
Second option si to check if sender/recipient is accepted by
remote mailserver. Be carefull when turning on verification and
first read http://www.postfix.org/ADDRESS_VERIFICATION_README.html
to see limitation and problem that can happen.
Parameters:
cacheDB (False)
cache results in database (almost useless for mx verification)
cacheNegative (3600)
maximum time for caching negative result
cachePositive (86400)
maximum time for caching positive result
cacheUnknown (900)
maximum time for caching unknown result
dbExpireNegative (10800)
negative result expiration time in db
dbExpirePositive (2419200)
positive result expiration time in db
factory (None)
reference to factory instance
param (None)
string key for data item that should be verified (sender/recipient)
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
table (verification)
database table with persistent cache
timeout (20)
set SMTP connection timeout
vtype (mx)
mx, connection, domain or user verification
Check arguments:
data ... all input data in dict
Check returns:
5 (15) .... user verification was successfull (hit pernament cache)
4 (14) .... domain verification was successfull (hit pernament cache)
3 (13) .... connection verification was successfull (hit pernament cache)
2 (12) .... mx verification was successfull (hit pernament cache)
1 (11) .... check succeeded according RFC (sender, recipient)
0 ......... undefined (e.g. DNS error, SMTP error, ...)
-1 (-11) .. check failed, mail format invalid
-2 (-12) .. mx verification failed (hit pernament cache)
-3 (-13) .. connection verification failed (hit pernament cache)
-4 (-14) .. domain verification failed (hit pernament cache)
-5 (-15) .. user verification failed (hit pernament cache)
Examples:
# sender domain verification
modules['verification'] = ( 'Verification', { param="sender" } )
# recipient user verification
modules['verification'] = ( 'Verification', { param="recipient",
vtype="user" } )
Module Whois
------------
Whois module can be used to return IP whois informations
Parameters:
cacheNegative (3600)
maximum time for caching negative result
cachePositive (3600)
maximum time for caching positive result
cacheUnknown (900)
maximum time for caching unknown result
factory (None)
reference to factory instance
saveResult (True)
save returned value in data hash for further modules
saveResultPrefix (result_)
prefix for saved data
test1 (client_address)
test parameter 1
Check arguments:
data ... all input data in dict
Check returns:
1 .... ok
0 .... undefined (default for this module)
-1 ... failed
Examples:
# make instance of dummy module
modules['whois1'] = ( 'Whois', {} )
# make instance of dummy module with parameters
modules['whois2'] = ( 'Whois', { 'param': "client_address" } )