Modular Python Postfix Policy Server ==================================== Modular Python Postfix Policy Server is tool for extending Postfix checking capabilities. It uses Postfix access policy delegation (http://www.postfix.org/SMTPD_POLICY_README.html) to check incoming SMTP request and accept or reject it according provided data. It can reduce mailserver load with rejecting incorrect mail during SMTP connection. It was made with stress to height reliability and performance by providing caching of required data and results. Because it has modular design it can be easily extended by custom modules (only one method has to be implemented and everything else is handled automatically). By default it provide modules for SPF checking, domain mailhost checking, sender/recipient verification, black/white listing, greylisting, spam traps, DNS blacklists, ... You can configure how to use all modules by implementing one simple method in config file. It provide to you ability to define check dependencies and use different modules according to the result of previous. 1. Installation 1.1 Requirements postfix >= 2.1 - http://www.postfix.org python >= 2.3 - http://www.python.org python-twisted >= 1.3 - http://twistedmatrix.com MySQL * >= 3.23 - http://www.mysql.com python-GeoIP * >= 1.2.1 - http://www.maxmind.com/app/python python-ldap * >= 2.0.x - http://python-ldap.sourceforge.net dnspython ** >= 1.3.3 - http://www.dnspython.org pyperl *** >= 1.0.1 - http://ftp.activestate.com/Zope-Perl * required by some check modules (but not by application core) ** version 1.3.5 is much slower then 1.3.4 because of changes in resolver (may be that newer version is better but I did not check it) *** required by modules that directly call perl (none module now) you can use local patched copy (works with python 2.4 and perl 5.8.8) http://kmlinux.fjfi.cvut.cz/~vokac/activities/ppolicy/download/pyperl 1.2 PPolicy download Primary site for this PPolicy server is http://kmlinux.fjfi.cvut.cz/~vokac/activities/ppolicy/download. You can download there prebuild RPM package or sources in SRPM and tgz format. 1.3 PPolicy installation 1.3.1 Install from RPM rpm -Uvh ppolicy-2.x.x.noarch.rpm 1.3.2 Install from sources Rebuild SRPM package or install application directly from tgz package. * building SRPM: rpmbuild --rebuild ppolicy-2.x.x.src.rpm rpm -Uvh /path/to/builded/package/ppolicy-2.x.x.noarch.rpm * building tgz: tar xzf ppolicy-2.x.x.tar.gz cd ppolicy-2.x.x python setup.py install # if you want to use different than default python installation directory # than you should omit last step, but you have to set basePath # in ppolicy.conf configuration file # installing additional required files cp ppolicy.tap /usr/sbin/ppolicy.tap cp ppolicy.init /etc/init.d/ppolicy # this script was tested on RedHat cp ppolicy.conf /etc/postfix mkdir -p /var/log/ppolicy # this path is defined in /etc/init.d/ppolicy mysql < ppolicy.sql # if you want to use database 1.3.2 Starting ppolicy * use startup script /etc/init.d/ppolicy * or start ppolicy daemon manually using command /usr/bin/python /usr/bin/twistd \ --pidfile=/var/log/ppolicy/ppolicy.pid \ --rundir=/var/log/ppolicy \ --file=/usr/sbin/ppolicy.tap \ --python=/usr/sbin/ppolicy.tap \ --logfile=/var/log/ppolicy/ppolicy.log \ --no_save 2. Configuration 2.1 PPolicy Configuration of ppolicy daemon is done through ppolicy.conf file. Default location for this file is /etc/postfix/ppolicy.conf. If you place this file somewhere else than you have to change this information in ppolicy.tap twisted bootstrap file that is installed by default to /usr/sbin. There are plenty of comments and examples in ppolicy.conf so you should be able to adapt it to your needs. For details about particular check modules you can read class description in source file or documentation exported in file MODULES. 2.2 Postfix configuration Update configuration according following examples and adapt to your needs. /etc/postfix/main.cf: smtpd_recipient_restrictions = ... reject_unauth_destination check_policy_service inet:127.0.0.1:10030 ... 127.0.0.1:10030_time_limit = 3600 2.3 Database configuration Database is used to permanently store result of some expensive checks. In most cases it should be possible to run ppolicy without database backend but it is not recommended because of degraded performance and (may be) more bug (it was not fully tested without database). Only MySQL database was tested with ppolicy but it should not be difficult to adapt to any SQL database. You have to create new ppolicy database and change database connection information in configuration file ppolicy.conf. User you use to connect to ppolicy database must have rights to create new tables. All required tables are created automatically during ppolicy server startup. New (MySQL) database can be created using ppolicy.sql template, but you should first change predefined password. To run this SQL script you need database user with CREATE DATABASE privileges, e.g.: mysql --hostname=localhost --user=root --password=secret < ppolicy.sql 3. Modules 3.1 Existing modules * Country * Dnsbl * DnsblDynamic * DnsblScore * DOS * Dummy * DumpDataDB * DumpDataFile * Greylist * List * ListBW * ListDyn * ListMailDomain * LookupLDAP * P0f * Resolve * SPF * Trap * Verification * Whois Detailed descriptions of all modules can be found in file named MODULES. There are also information about all parameters and basic examples how to use the module in config file. 3.2. Writing new modules Before you start to write custom modules look at IPPolicyServerCheck, Base and mainly Dummy module. There are plenty of comments that can be useful to know. The heart of each module is check() method that is called when you use module in config file. Next important is hashArg() which influence data caching (it is key for result cache dictionary), start() that is called before first usage of check() method. Last method that can be implemented is stop() and it is called e.g. when application terminating. 4. Bug reports If you find some bug or performance bottleneck than let me know or better send me patch. You can contact me using following address Petr Vokac <vokac[at]kmlinux.fjfi.cvut.cz>
Module Country -------------- Country module for recognizing country according IP address or domain name. Parameters: cacheNegative (0) maximum time for caching negative result cachePositive (0) maximum time for caching positive result cacheUnknown (0) maximum time for caching unknown result country (None) check if IP is in this country dataPath (None) path to GeoIP.dat file factory (None) reference to factory instance param (None) name of parameter in data dictionary saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data Check arguments: data ... all input data in dict Check returns: 1 .... ok, country match, second parameter is country name 0 .... undefined (some error) -1 ... failed, country doesn't match, second parameter is country name Examples: # return country for client_address modules['country1'] = ( 'Country', { 'param': 'client_address', 'dataPath': '/usr/share/GeoIP/GeoIP.dat' } ) # check if client_address is 'cz' modules['country2'] = ( 'Country', { 'param': 'client_address', 'dataPath': '/usr/share/GeoIP/GeoIP.dat', 'country': 'CZ' } ) Module Dnsbl ------------ Check if client address is listed in specified DNS blacklist. see tools/dnsbl.txt for list of valid blacklist names - original file can be downloaded from http://moensted.dk/spam/drbsites.txt You can also run `python tools/dnsbl.py --list` to see formated output of configured balacklists. Parameters: cacheNegative (43200) maximum time for caching negative result cachePositive (21600) maximum time for caching positive result cacheUnknown (1800) maximum time for caching unknown result dnsbl (None) name of DNS blacklists defined in this module factory (None) reference to factory instance saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data Check arguments: data ... all input data in dict Check returns: 1 .... listed in dnsbl 0 .... unknown error (e.g. DNS problems) -1 ... not listed Examples: # check if sender mailserver is in ORDB blacklist modules['dnsbl1'] = ( 'Dnsbl', { dnsbl="ORDB" } ) Module DnsblScore ----------------- Check if client address and sender domain is in various DNS blacklists and make sum of scores that are defined in spamassassin config files. Returned result depends on parameter treshold (this module can return -1,0,1 or exact score if treshold is None). This module use tools.dnsbl.score to score mail. By default it uses all available checks, but you can specify your custom by listing their name in dnsbl parameter. You can list all valid names by calling `python tools/dnsbl.py --list`. Parameters: cacheNegative (43200) maximum time for caching negative result cachePositive (21600) maximum time for caching positive result cacheUnknown (1800) maximum time for caching unknown result dnsbl (None) list of DNS blacklist to use factory (None) reference to factory instance params (['client_address', 'sender']) which params we should check saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data treshold (None) treshold that define which score mean this module fail Check arguments: data ... all input data in dict Check returns: 1 (positive) .... score > treshold or positive score 0 ............... unknown error (e.g. DNS problems) -1 (negative) ... score < treshold or negative score Examples: # return blacklist score for client address modules['dnsbl1'] = ( 'DnsblScore', {} ) # check if blacklist score for client address exceed defined treshold modules['dnsbl1'] = ( 'DnsblScore', { treshold=5 } ) Module DnsblDynamic ------------------- Check if client address is in dynamic allocated ranges. It use corresponding dnsbl and also domain name (e.g. format xxx-yyy-zzz.dsl.provider.com). There will by probably many "correct" mailservers sitting on IP address from dynamic alocated space. Use the result as one of the decision rule and don't reject mail only relaying on this result. Parameters: cacheNegative (43200) maximum time for caching negative result cachePositive (21600) maximum time for caching positive result cacheUnknown (1800) maximum time for caching unknown result check_name (True) check format of client name (e.g. xxx-yyy-zzz.dsl.provider.com) dnsbl (None) list of DNS blacklists factory (None) reference to factory instance saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data Check arguments: data ... all input data in dict Check returns: 1 .... client address seems to be in dynamic ip range 0 .... unknown error (e.g. DNS problems) -1 ... client address doesn't seem to be in dynamic ip range Examples: # check if sender mailserver is in ORDB blacklist modules['dnsbl1'] = ( 'DnsblDynamic', { 'dnsbl': [ 'NJABLDYNA', 'SORBSDUL' ] } ) Module DOS ---------- Limit number of incomming mail that has same parameters. You can e.g. limit number of messages that can be accepted by one recipient from one sender in defined period of time. It will divide time interval into several parts and agregate information about number of incomming mails. Be carefull setting this module, because badly choosed parameters can exhause a lot of memory. Some mail should not be blocked (e.g. to postmaster@your.domain.com, from <>, ...) and your configuration should take care of it. Parameters: cacheNegative (0) maximum time for caching negative result cachePositive (0) maximum time for caching positive result cacheUnknown (0) maximum time for caching unknown result factory (None) reference to factory instance limitCount (1000) number of messages accepted during limitTime period limitGran (10) data collection granularity limitTime (3600) time period for limitCount messages params (None) parameters that will be used to test equality saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data Check arguments: data ... all input data in dict Check returns: 1 .... frequency reached defined limit 0 .... unrelated error (e.g. database problem) -1 ... frequency is acceptable Examples: # limit number of mail from one sender to 1000/hod modules['dos1'] = ( 'DOS', { 'params': "sender" } ) # limit number of mail from one sender # and one mailserver to 100 in 10 minutes modules['dos2'] = ( 'DOS', { 'params': ["sender","client_address"], 'limitCount': 100, 'limitTime': 10 } ) Module Dummy ------------ Dummy module skeleton for creating new modules Parameters: cacheNegative (900) maximum time for caching negative result cachePositive (900) maximum time for caching positive result cacheUnknown (900) maximum time for caching unknown result factory (None) reference to factory instance saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data test1 (None) test parameter 1 test2 (abc) test parameter 2 Check arguments: data ... all input data in dict Check returns: 1 .... ok 0 .... undefined (default for this module) -1 ... failed Examples: # make instance of dummy module modules['dummy1'] = ( 'Dummy', {} ) # make instance of dummy module with parameters modules['dummy2'] = ( 'Dummy', { 'test1': 1, 'test2': "def" } ) Module DumpDataDB ----------------- Dump data from incomming request into the database. These informations can be used to improve debugging of other modules or to gather statistical data for further analysis. This module should be safe to use in sense its check method doesn't raise any exception. Parameters: cacheNegative (0) maximum time for caching negative result cachePositive (0) maximum time for caching positive result cacheUnknown (0) maximum time for caching unknown result factory (None) reference to factory instance interval (None) split interval (value depends on split type) saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data split (None) split database (None, "records", "date") table (dump) database where to dump data from requests Check arguments: data ... all input data in dict Check returns: this module always return 1 and resEx is set to the new database ID Examples: # definition for module for saving request info in default # database table 'dump' modules['dumpdb1'] = ( 'DumpDataDB', {} ) # module that save info in custom defined table modules['dumpdb2'] = ( 'DumpDataDB', { table="my_dump" } ) Module DumpDataFile ------------------- Dump data from incomming request into the file. These informations can be used to improve debugging of other modules or to gather statistical data for further analysis. This module should be safe to use in sense its check method doesn't raise any exception. Parameters: cacheNegative (0) maximum time for caching negative result cachePositive (0) maximum time for caching positive result cacheUnknown (0) maximum time for caching unknown result factory (None) reference to factory instance fileName (None) file where to dump data from requests saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data Check arguments: data ... all input data in dict Check returns: this module always return 0 (undefined result) Examples: # definition for module for saving request info in file modules['dumpfile1'] = ( 'DumpDataFile', { fileName="/var/spool/ppolicy/dump.dat" } ) Module Greylist --------------- Greylist implementation. Mail thats triplet (from,to,client) was not seen before should be rejected with code 450 (temporary failed). It relay on fact that spammer software will not try to send mail once again and correctly configured mailservers must try it one again (see RFC 2821). Be carefull because some poorly configured mailserver did not retry to send mail again and some mailing list has unique sender name for each mail and this module delay delivery and increase load of remote server. You should run following cleanup tasks from cron job (may be it will be part of this module in the future). Replace XXXX and YYYY with same value as for "mustRetry" and "expire": mysql 'DELETE FROM `greylist` WHERE UNIX_TIMESTAMP(`date`) + XXXX < UNIX_TIMESTAMP() AND `state` = 0 mysql 'DELETE FROM `greylist` WHERE UNIX_TIMESTAMP(`date`) + YYYY < UNIX_TIMESTAMP() Parameters: cacheNegative (60) maximum time for caching negative result cachePositive (86400) maximum time for caching positive result cacheUnknown (30) maximum time for caching unknown result delay (600) how long to delay mail we see its triplet first time expiration (2678400) expiration of triplets in database factory (None) reference to factory instance mustRetry (43200) time we wait to receive next mail after we geylisted it saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data table (greylist) greylist database table Check arguments: data ... all input data in dict Check returns: 2 .... always allow postmaster 1 .... was seen before 0 .... some error occured (database, dns, ...) -1 ... greylist in progress -2 ... invalid sender address Examples: # greylisting module with default parameters modules['greylist1'] = ( 'Greylist', {} ) # greylisting module with own database table "grey", delay set # to 1 minute and expiration time of triplets to 1 year modules['greylist2'] = ( 'Greylist', { table="grey", delay=60, expiration=86400*365 } ) Module List ----------- Check if parameter value (or array of parameter values) is in database table. If you don't need complex cartesian product searches then it can be more efficient to use simple LookupDB module. If data type of input value is array, it is created cartesian product of all input values. E.g. if input is { 'param1': [ 'a', 'b' ], 'param2': [ 'c', 'd' ] } then it tries to search following pairs in the database: [ ('a', 'c'), ('a', 'd'), ('b', 'c'), ('b', 'd') ] This module use additional internal result cache for each db query. It is more efficient for further queries with similar values in the array of incomming parameters. Parameters: cacheAll (False) cache all records in memory cacheAllRefresh (900) refresh time in case of caching all records cacheCaseSensitive (False) case-sensitive cache (set to True if you are using case-sensitive text comparator on DB column cacheNegative (900) maximum time for caching negative result cachePositive (900) maximum time for caching positive result cacheUnknown (900) maximum time for caching unknown result column (None) name of database column or array of columns according "param" factory (None) reference to factory instance memCacheExpire (900) memory cache expiration - used only if param value is array memCacheSize (1000) memory cache max size - used only if param value is array param (None) name of parameter in data dictionary (value can be string or array) retcols (None) name of column returned by check method saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data table (None) name of database table where to search parameter Check arguments: data ... all input data in dict Check returns: 1 .... parameter was found in db, second parameter include selected row 0 .... failed to check (request doesn't include required param, database error, ...) -1 ... parameter was not found in db Examples: # module for checking if sender is in database table list modules['list1'] = ( 'List', { 'param="sender" } ) # check if sender domain is in database table my_list # compare case-insensitive and return whole selected row modules['list2'] = ( 'List', { 'param': [ "sender", "recipient" ], 'table': "my_list", 'column': [ "my_column1", "my_column2" ], 'cacheCaseSensitive': False, 'retcols': "*" } ) Module ListBW ------------- Check if array of defined parameters are in blacklist/whitelist. Each parameter is searched in b/w list and first occurence is returned. Parameters: cacheAll (False) cache all records in memory cacheAllRefresh (900) refresh time in case of caching all records cacheCaseSensitive (False) case-sensitive cache (set to True if you are using case-sensitive text comparator on DB column cacheNegative (900) maximum time for caching negative result cachePositive (900) maximum time for caching positive result cacheUnknown (900) maximum time for caching unknown result factory (None) reference to factory instance mappingBlacklist ({}) mapping between params and database columns for blacklist mappingWhitelist ({}) mapping between params and database columns for whitelist param (None) name of parameter in data dictionary (value can be string or array) retcolsBlacklist (None) name of blacklist columns returned by check method retcolsWhitelist (None) name of whitelist columns returned by check method saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data tableBlacklist (None) name of blacklist database table where to search parameter tableWhitelist (None) name of whitelist database table where to search parameter Check arguments: data ... all input data in dict Check returns: 1 .... parameter was found in whitelist, second parameter include selected row 0 .... parameter was not found in whitelist/blacklist or failed to check (request doesn't include required param, database error, ...) second parameter - None in case of error not None in case record was not found in b/w list 1 .... parameter was found in blacklist, second parameter include selected row Examples: # module for checking if sender is in blacklist/whitelist modules['list1'] = ( 'ListBW', { 'param': "sender_splitted" } ) # check if sender in blacklist/whitelist # compare case-insensitive and return whole selected row modules['list2'] = ( 'ListBW', { 'param': "sender_splitted", 'tableBlacklist': "my_blacklist", 'tableWhitelist': "my_whitelist", 'mappingBlacklist': { "my_param": ("my_column", "VARCHAR(50)") }, 'mappingWhitelist': {"my_param": ("my_column", "VARCHAR(50)") }, 'cacheCaseSensitive': False, 'retcol': "*" } ) Module ListDyn -------------- This module provide general access (add/remove/check) to the persistent data store in database. You can use this module for black/white lists, persistent data cache for result from other modules, ... One of the most important parameter is "mapping". It is used to map request parameters database columns. Its syntax is [ 'dictName': ('columnName', 'columnType'), ...], where dictName is attribude name comming in check method in data dictionary, columnName is database column name, column type is database column type in SQL syntax. If don't define mapping for concrete column, than default is used 'dictName': ('dictName', 'VARCHAR(255)') Second very important parametr is "operation". With this parameter you specify what this module should do by default with passed request. Also result returned from check method depends on this operation. add ...... add new data to database remove ... remove all matching data from database check .... check if database include data from request You can also set soft and hard expiration time for the records. For example you can specify, that after 1 hour it will return SOFT_NEGATIVE constant and after 2 hours HARD_NEGATIVE. This can be usefull when you use this module as persistent cache for some other module and you are not sure that the its results are always reachable (e.g. when it uses DNS and it is temporarly unreachable). Parameters: cacheNegative (0) maximum time for caching negative result cachePositive (0) maximum time for caching positive result cacheUnknown (0) maximum time for caching unknown result factory (None) reference to factory instance hardExpire (None) expiration time for the record (0 == never) mapping ({}) mapping between params and database columns operation (check) list operation (add/remove/check) param (None) names of input data keys used to identify data row(s) retcols (None) names of columns to be returned (None mean no related data) saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data softExpire (None) information that record will be soon expired (0 == never)) table (list) name of database table where to search parameter Check arguments: data ... all input data in dict operation ... add/remove/check operation that overide the default Check returns: add, remove 1 .... operation was successfull 0 .... error (database error, ...) check 2 .... parameters are in list, but soft expired 1 .... parameters are in list 0 .... failed to check (database error, ...) -1 ... parameters are not in list Examples: # module for checking if sender is in database table list1 modules['list1'] = ( 'ListDyn', { 'table': 'list1', 'mapping': { "sender": "mail", } } ) # module for checking/getting if sender row values in database table list2 modules['list2'] = ( 'ListDyn', { 'table': 'list2', 'param': 'sender', 'retcols': [ "recip" ], 'mapping': { "sender": ("mail", "VARCHAR(50)"), "recip": ("rmail", "VARCHAR(50)") }, } ) # module with soft/hard expiration and 'add' as default operation modules['list3'] = ( 'ListDyn', { 'table': 'list3', 'operation': 'add', 'softExpire': 60*10, 'hardExpire': 60*30 } ) Module ListMailDomain --------------------- Check mail address or its part is in database. Can be used for black/white listing sender or recipient mail addresses. Parameters: cacheAll (True) cache all records in memory cacheAllRefresh (900) refresh time in case of caching all records cacheCaseSensitive (False) case-sensitive cache (set to True if you are using case-sensitive text comparator on DB column cacheNegative (900) maximum time for caching negative result cachePositive (900) maximum time for caching positive result cacheUnknown (900) maximum time for caching unknown result factory (None) reference to factory instance mappingBlacklist ({}) mapping between params and database columns for blacklist mappingWhitelist ({}) mapping between params and database columns for whitelist param (None) name of parameter in data dictionary (value can be string or array) retcolsBlacklist (None) name of blacklist columns returned by check method retcolsWhitelist (None) name of whitelist columns returned by check method saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data tableBlacklist (None) name of blacklist database table where to search parameter tableWhitelist (None) name of whitelist database table where to search parameter Check arguments: data ... all input data in dict Check returns: 1 .... parameter was found in whitelist, second parameter include selected row 0 .... parameter was not found in whitelist/blacklist or failed to check (request doesn't include required param, database error, ...) second parameter - None in case of error not None in case record was not found in b/w list 1 .... parameter was found in blacklist, second parameter include selected row Examples: # module for checking if sender is in database b/w list modules['list1'] = ( 'ListMailDomain', { paramMailDomain="sender" } ) # check if sender domain is in database table my_blacklist, my_whitelist modules['list2'] = ( 'ListMailDomain', { paramMailDomain="sender", tableBlacklist="my_blacklist", tableWhitelist="my_whitelist", column="my_column", retcol="*" } ) Module LookupDB --------------- LookupDB module for searching records in DB. You can map incomming data to database columns using "mapping" parameter. Its syntax is [ 'dictName': ('columnName', 'columnType'), ...], where dictName is attribude name comming in check method in data dictionary, columnName is database column name, column type is database column type in SQL syntax. If don't define mapping for concrete column, than default is used 'dictName': ('dictName', 'VARCHAR(255)') Parameters: cacheAll (False) cache all records in memory cacheAllRefresh (900) refresh time in case of caching all records cacheCaseSensitive (False) case-sensitive cache (set to True if you are using case-sensitive text comparator on DB column cacheNegative (900) maximum time for caching negative result cachePositive (900) maximum time for caching positive result cacheUnknown (900) maximum time for caching unknown result factory (None) reference to factory instance mapping ({}) mapping between params and database columns param (None) name of parameter in data dictionary (value can be string or array) retcols (None) name of column returned by check method saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data table (None) name of database table where to search parameter Check arguments: data ... all input data in dict Check returns: 1 .... parameter was found in db, second parameter include selected row 0 .... failed to check (request doesn't include required param, database error, ...) -1 ... parameter was not found in db Examples: # module for checking if sender is in database table modules['lookup1'] = ( 'LookupDB', { 'param': "sender" } ) # check if sender domain is in database table my_list # compare case-insensitive and return whole selected row modules['lookup2'] = ( 'LookupDB', { 'param': [ "sender", "recipient" ], 'table': "my_list", 'mapping': { "sender": ("mail", "VARCHAR(50)"), }, 'retcols': "*" } ) Module LookupLDAP ----------------- LookupLDAP module for searching records in LDAP Parameters: attributes (None) attributes returned by LDAP query base (None) LDAP search base bind_dn (None) LDAP bind DN (None == anonymous bind) bind_method (128) LDAP bind method (only simple is supported) bind_pass (None) LDAP bind password cacheNegative (900) maximum time for caching negative result cachePositive (900) maximum time for caching positive result cacheUnknown (900) maximum time for caching unknown result factory (None) reference to factory instance filter (None) LDAP filter (%m will be replaced by "param" value param (None) name of parameter in data dictionary saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data scope (2) LDAP search scope uri (None) LDAP server URI Check arguments: data ... all input data in dict Check returns: 1 .... ok, found records for the filter 0 .... undefined (problem with LDAP query, e.g. failing connection) -1 ... failed to find records for the filter Examples: # recipient ldap lookup in ldap1/2.domain.tld with "mail" filter modules['lookup_ldap1'] = ( 'LookupLDAP', { 'param': 'recipient', 'uri': 'ldap://ldap1.domain.tld ldap://ldap2.domain.tld', 'base': 'ou=People,dc=domain,dc=tld', 'filter': '(mail=%m)' } ) # recipient ldap lookup that returns common name attribute modules['lookup_ldap1'] = ( 'LookupLDAP', { 'param': 'recipient', 'uri': 'ldap://ldap1.domain.tld ldap://ldap2.domain.tld', 'base': 'ou=People,dc=domain,dc=tld', 'bind_dn': 'cn=PPolicy User,ou=People,dc=domain,dc=tld', 'bind_pass': 'secret', 'scope': ldap.SCOPE_ONELEVEL, 'filter': '(mail=%m)', 'attributes': 'cn' } ) Module P0f ---------- P0f module detect sender OS using p0f and its unix socket interface. You can use patched version of p0f that requires only sender IP to detect OS - it is less reliable but easier to use, because you don't have to specify "ip" and "port" parameters. It can be downloaded from http://kmlinux.fjfi.cvut.cz/~vokac/activities/ppolicy/download/p0f Start p0f with -0 parameter for releases >= 2.0.8, earlier releases works only if you apply patch that can be downloaded on given URL. This module returns tuple with information described in p0f-query.h p0f_response structure. Most important for our purposes are retEx[3] ... detected OS retEx[4] ... details about detected OS (e.g. version) retEx[5] ... distance of sender retEx[12] .. uptime (not reliable) Parameters: cacheNegative (3600) maximum time for caching negative result cachePositive (3600) maximum time for caching positive result cacheUnknown (900) maximum time for caching unknown result factory (None) reference to factory instance ip (None) IP address of destionation server port (25) port address of destionation server saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data socket (/var/run/p0f.socket) p0f socket for sending requests version (2.0.8) p0f version Check arguments: data ... all input data in dict Check returns: 1 .... success (p0f response structure in second parameter) 0 .... undefined (no data) -1 ... failed (error getting data from p0f) Examples: # make instance of p0f module modules['p0f1'] = ( 'P0f', {} ) # make instance of p0f module with different socket path modules['p0f2'] = ( 'P0f', { 'socket': '/tmp/p0f.socket' 'ip': '192.168.0.1', 'port': 1234 } ) Module Resolve -------------- Try to resolve ip->name, name->ip, name->mx, ip->name->ip, name->ip->name, ip1->name->ip2, ip->name->mx, name1->ip->name2. Parameters: cacheNegative (21600) maximum time for caching negative result cachePositive (86400) maximum time for caching positive result cacheUnknown (1800) maximum time for caching unknown result factory (None) reference to factory instance param (None) which request parameter should be used saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data type (None) ip->name, name->ip, name->mx, ip->name->ip, name->ip->name, ip1->name->ip2, ip->name->mx, name1->ip->name2 Check arguments: data ... all input data in dict Check returns: 1 .... all tranlation were successfull and equal 0 .... problem resolving ip or name (DNS error) -1 ... translation failed Examples: # check if sender domain exist modules['resolve1'] = ( 'Resolve', { param="sender", type="name->ip" } ) # check if remote mailserver has reverse records in DNS modules['resolve2'] = ( 'Resolve', { param="client_address", type="ip->name" } ) # check if remote mailserver has reverse records in DNS # and translating back returns set of IP that contains original IP modules['resolve3'] = ( 'Resolve', { param="client_address", type="ip->name->ip" } ) Module SPF ---------- This module use sender address and client IP to check SPF records in DNS if they are exist. More informations about SPF can be found at following address http://www.openspf.org http://en.wikipedia.org/wiki/Sender_Policy_Framework Parameters: cacheNegative (86400) maximum time for caching negative result cachePositive (86400) maximum time for caching positive result cacheUnknown (21600) maximum time for caching unknown result factory (None) reference to factory instance saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data Check arguments: data ... all input data in dict Check returns: 1 .... SPF passed, second parameter contain values returned by tools.spf function (result, status, explanation) 0 .... undefined SPF records or exception when checking SPF -1 ... SPF softfail or temperror -2 ... SPF fail or permerror Examples: # define module for checking SPF modules['spf1'] = ( 'SPF', {} ) Module Sleep ------------ Sleep module for debugging Parameters: cacheNegative (900) maximum time for caching negative result cachePositive (900) maximum time for caching positive result cacheUnknown (900) maximum time for caching unknown result factory (None) reference to factory instance saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data sleep (1) sleep interval in seconds (default 1s) Check arguments: data ... all input data in dict Check returns: 1 .... ok 0 .... undefined (default for this module) -1 ... failed Examples: # make instance of Sleep module modules['Sleep1'] = ( 'Sleep', {} ) # make instance of Sleep module with parameters modules['Sleep2'] = ( 'Sleep', { 'sleep': 10 } ) Module Trap ----------- Trap module catches mail probing random recipient addresses. If in defined time fall more than defined amount of mail from one client to the trap, all mail from that client will be temporarly blocked. Parameters: cacheNegative (0) maximum time for caching negative result cachePositive (0) maximum time for caching positive result cacheUnknown (0) maximum time for caching unknown result expire (3600) expiration time for client that send trapped email factory (None) reference to factory instance saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data traps (None) comma separated list of trap email addresses treshold (-2) how many traps has to be touched before blacklisting (negative value mean fraction 1/x) Check arguments: data ... all input data in dict Check returns: 1 .... request from client that send many mail touched our mail traps 0 .... some problem -1 ... client is not touched mail traps Examples: # define trap to block client_address for one hour if we receive # one mail with recipient spamtrap1 or spamtrap2 modules['trap1'] = ( 'Trap', { traps="spamtrap1@domain.com,spamtrap2@domain.com" } ) Module Verification ------------------- Module for mx, connection, domain and user verification. It check domain existence and then it try to establish SMTP connection with mailhost for the domain (MX, A or AAAA DNS records - see RFC2821, chapter 5). You can check if sender/recipient or whatever reasonable has correct DNS records as mailhost and try to connect to this server. Second option si to check if sender/recipient is accepted by remote mailserver. Be carefull when turning on verification and first read http://www.postfix.org/ADDRESS_VERIFICATION_README.html to see limitation and problem that can happen. Parameters: cacheDB (False) cache results in database (almost useless for mx verification) cacheNegative (3600) maximum time for caching negative result cachePositive (86400) maximum time for caching positive result cacheUnknown (900) maximum time for caching unknown result dbExpireNegative (10800) negative result expiration time in db dbExpirePositive (2419200) positive result expiration time in db factory (None) reference to factory instance param (None) string key for data item that should be verified (sender/recipient) saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data table (verification) database table with persistent cache timeout (20) set SMTP connection timeout vtype (mx) mx, connection, domain or user verification Check arguments: data ... all input data in dict Check returns: 5 (15) .... user verification was successfull (hit pernament cache) 4 (14) .... domain verification was successfull (hit pernament cache) 3 (13) .... connection verification was successfull (hit pernament cache) 2 (12) .... mx verification was successfull (hit pernament cache) 1 (11) .... check succeeded according RFC (sender, recipient) 0 ......... undefined (e.g. DNS error, SMTP error, ...) -1 (-11) .. check failed, mail format invalid -2 (-12) .. mx verification failed (hit pernament cache) -3 (-13) .. connection verification failed (hit pernament cache) -4 (-14) .. domain verification failed (hit pernament cache) -5 (-15) .. user verification failed (hit pernament cache) Examples: # sender domain verification modules['verification'] = ( 'Verification', { param="sender" } ) # recipient user verification modules['verification'] = ( 'Verification', { param="recipient", vtype="user" } ) Module Whois ------------ Whois module can be used to return IP whois informations Parameters: cacheNegative (3600) maximum time for caching negative result cachePositive (3600) maximum time for caching positive result cacheUnknown (900) maximum time for caching unknown result factory (None) reference to factory instance saveResult (True) save returned value in data hash for further modules saveResultPrefix (result_) prefix for saved data test1 (client_address) test parameter 1 Check arguments: data ... all input data in dict Check returns: 1 .... ok 0 .... undefined (default for this module) -1 ... failed Examples: # make instance of dummy module modules['whois1'] = ( 'Whois', {} ) # make instance of dummy module with parameters modules['whois2'] = ( 'Whois', { 'param': "client_address" } )