The project page is here.

Download fdm 1.7 with this link.

A mailing list is available, see this page.


fdm

Introduction

fdm is a program to fetch mail and deliver it in various ways depending on a user-supplied ruleset. Mail may be fetched from stdin, IMAP or POP3 servers, or from local maildirs, and filtered based on whether it matches a regexp, its size or age, or the output of a shell command. It can be rewritten by an external process, dropped, left on the server or delivered into maildirs, mboxes, to a file or pipe, or any combination. fdm is designed to be lightweight but powerful, with a compact but clear configuration syntax. It is primarily designed for single-user uses but may also be configured to deliver mail in a multi-user setup. In this case, it uses privilege separation to minimise the amount of code running as the root user.

Table of contents

1 Installation 2 Quick start 3 The configuration file 3.1 Including other files 3.2 Macros 3.3 Testing macros 3.4 Shell commands 4 Invoking fdm 4.1 Temporary files 4.2 Command line arguments 4.3 Running from cron 4.4 The lock file 4.5 Testing and debugging 5 Fetching mail 5.1 Mail tags 5.2 POP3 and POP3S 5.3 SSL certificate verification 5.4 The .netrc file 5.5 IMAP and IMAPS 5.6 IMAP or POP3 over a pipe or ssh 5.7 stdin and local mail 5.8 From maildirs and mboxes 5.9 Using NNTP and NNTPS 5.10 New or old mail only 6 Defining actions 6.1 Drop and keep 6.2 Maildirs 6.3 Mboxes 6.4 IMAP and IMAPS 6.5 SMTP 6.6 Write, pipe, exec and append 6.7 stdout 6.8 Rewriting mail 6.9 Adding or removing headers 6.10 Tagging 6.11 Compound actions 6.12 Chained actions 7 Filtering mail 7.1 Nesting rules 7.2 Lambda actions 7.3 The all condition 7.4 Matched and unmatched 7.5 Matching by account 7.6 Matching a regexp 7.7 Matching bracket expressions 7.8 Matching by age or size 7.9 Using a shell command 7.10 Attachments 7.11 Matching tags 7.12 Using caches 7.13 Cache commands 8 Setting options 9 Archiving and searching mail 10 Using fdm behind a proxy 11 Bug reports and queries 12 Frequently asked questions

1 Installation

fdm depends on the Trivial Database library (TDB), available at: http://sourceforge.net/projects/tdb/ Ensure it is installed, then download the source tarball and build fdm with: $ tar -zxvf fdm-?.?.tar.gz $ cd fdm-?.? $ ./configure && make Then run 'make install' to install fdm to the default locations under /usr/local. The PREFIX environment variable may be set to specify an alternative installation location: $ export PREFIX=/opt # defaults to /usr/local $ sudo make install If being run as root, fdm requires a user named "_fdm" to exist. It will drop privileges to this user and its primary group. The user may be added on OpenBSD with, for example: # useradd -u 999 -s /bin/nologin -d /var/empty -g=uid _fdm It is not necessary to add a user if fdm is always started by a non-root user. fdm can be built to use PCRE rather than standard regexps. To do so, add -DPCRE to the make command: $ make -DPCRE Or PCRE=1 if using GNU make: $ make PCRE=1

2 Quick start

A simple ~/.fdm.conf file for a single user fetching from POP3, POP3S and IMAP accounts and delivering to one maildir may look similar to: # Set the maximum size of mail. set maximum-size 128M # An action to save to the maildir ~/mail/inbox. action "inbox" maildir "%h/mail/inbox" # Accounts: POP3, POP3S and IMAP. Note the double escaping of the '\' # character in the password. If the port is omitted, the default # ("pop3", "pop3s", "imap" or "imaps" in the services(5) db) is used. account "pop3" pop3 server "my.pop3.server" user "my-username" pass "my-password-with-a-\\-in-it" account "pop3s" pop3s server "pop.googlemail.com" port 995 user "my-account@gmail.com" pass "my-password" # If the 'folder "my-folder"' argument is omitted, fdm will fetch mail # from the inbox. account "imap" imap server "my.imap.server" user "my-username" pass "my-password" folder "my-folder" # Discard mail from Bob Idiot. Note that the regexp is an extended # regexp, and case-insensitive by default. This action is a "lambda" or # unnamed action, it is defined inline as part of the match rule. match "^From:.*bob@idiot\\.net" in headers action drop # Match all other mail and deliver using the 'inbox' action. match all action "inbox" A simple initial configuration file without filtering, perhaps to replace fetchmail or getmail delivering to maildrop, may look similar to: # Set the maximum size of mail. set maximum-size 128M # Action to pipe directly to maildrop. action "maildrop" pipe "/usr/local/bin/maildrop" # Account definitions. account .... # Send all mail to maildrop. match all action "maildrop" To run fdm every half hour from cron, add something like this: */30 * * * * /usr/local/bin/fdm -l fetch See the fdm.conf(5) man page or the rest of this manual for more detail of the configuration file format.

3 The configuration file

fdm is controlled by its configuration file. It first searches for a .fdm.conf file in the invoking user's home directory. If that fails, fdm attempts to use /etc/fdm.conf. The configuration file may also be specified using the '-f' command line option, see the section on that subject below. This section gives an overview of the configuration file syntax. Further details of syntax, and specific keywords, are covered in later sections. The configuration file has the following general rules: - Keywords are specified as unadorned lowercase words: match, action, all. - Strings are enclosed in double quotes (") or single quotes ('). In double quoted strings, double quotes may be included by escaping them using the backslash character (\). Backslashes must also be escaped ("\\") - this applies to all such strings, including regexps and passwords. The special sequence '\t' is replaced by a tab character. In single quoted strings no escaping is necessary, but it is not possible to include a literal ' or a tab character. - Comments are prefixed by the hash character (#) and continue to the end of the line. - Whitespace is largely ignored. Lines may generally be split, concatenated or indented as preferred. - Lists are usually specified as 'singular item' or 'plural { item item }', for example: 'user "nicholas"', 'users { "nicholas" "bob" }'. The singular/plural distinction is not required, it is recommended only to aid readability: 'user { "nicholas "bob" }' is also accepted. - Regexps are specified as normal strings without additional adornment other than the "s (not wrapped in /s). All regexps are extended regexps. They are case insensitive by default but may be prefixed with the 'case' keyword to indicate case sensitivity is required. - Strings may be concatenated using plus: "a" + "b" is the same as "ab". This is most useful to wrap strings across multiple lines. Definition/option lines generally follow the following basic form: <keyword> <name or command> <parameters> Example lines that may appear in a configuration file are: # This is a comment. set lock-types flock account "stdin" disabled stdin action "strip-full-disclosure" rewrite "sed 's/^\\(Subject:.*\\)\\[Full-disclosure\\] /\\1/'" match "^X-Mailing-List:.*linux-kernel@vger.kernel.org" in headers or "^(To:|Cc:):.*@vger.kernel.org" in headers action "linux-kernel"

3.1 Including other files

The fdm configuration may be split into several files. Additional files may be referenced using the 'include' keyword: include "my-include-file.conf" include "/etc/fdm.d/shared-conf-1.conf"

3.2 Macros

Macros may be defined and used in the configuration file. fdm makes a distinction between macros which may hold a number (numeric macros) and those that hold a string (string macros). Numeric macros are prefixed with the percentage sign (%) and string by the dollar sign ($). Macros are defined using the equals operator (=): %nummacro = 123 $strmacro = "a string" Macros may then be referenced in either a standalone fashion anywhere a string or number is expected, depending on the type of macro: $myfile = "a-file" include $myfile %theage = 12 match age < %theage action "old-mail" Or embedded in a string by enclosing the macro name in {}s: $myfile2 = "a-file2" include "/etc/${myfile2}" %anum = 57 include "/etc/file-number-%{anum}" Macros are not substituted in strings specified using single-quotes.

3.3 Testing macros

The 'ifdef', 'ifndef' and 'endif' keywords may be used to include or omit sections of the configuration file depending on whether a macro is defined. An 'ifdef' is followed by a macro name (including $ or % type specifier) and if that macro exists, all following statements up until the next endif are evaluated (accounts created, rules added, and so on), otherwise they are skipped. 'ifndef' is the inverse: if the macro exists, the statements are skipped, otherwise they are included. An example is: ifdef $dropeverything match all action drop endif These keywords are particularly useful in conjunction with the '-D' command line option. Any statements between 'ifdef'/'ifndef' and 'endif' must still be valid syntax.

3.4 Shell commands

The value of a shell command may be used at any point in the configuration file where fdm expects a string or number. Shell commands are invoked by enclosing them in $() or %(). They are executed when the configuration file is parsed and if $() is used, any output to stdout is treated as a literal string (as if the output was inserted directly in the file enclosed in double quotes); %() attempts to convert the output to a number. For example: $mytz = $(date +%Z) %two = %(expr 1 + 1) $astring = "abc" + $(echo def) Parts of the command within double quotes (") are subject to tag and macro replacement as normal (so it is necessary to use %% if a literal % is required, see the section on tags below); parts outside double quotes or inside single quotes are not.

4 Invoking fdm

fdm accepts a number of command line arguments and may be invoked as needed from the command line or by a mail transfer agent, such as sendmail, or at regular times using a program such as cron(8).

4.1 Temporary files

As each mail is being processed, it is stored in a temporary file in /tmp, or if the TMPDIR environment variable exists in the directory it points to. fdm tries to queue a number of mails simultaneously, so that older can be delivered while waiting for the server to provide the next. The maximum length of the queue for each account is set by the 'queue-high' option (the default is two) and the maximum mail size accepted by the 'maximum-size' option (the default is 32 MB). In addition, the 'rewrite' action requires an additional temporary mail. Although fdm will fail rather than dropping mail if the disk becomes full, users should bear in mind the possibility and set the size of the temporary directory and the fdm options according to their needs.

4.2 Command line arguments

The fdm command has the following synopsis: fdm [-klmnqv] [-f conffile] [-u user] [-a account] [-x account] [-D name=value] [fetch | poll | cache ...] The meaning of the flags are covered in the fdm(1) man page, but a brief description is given below. The flags are also mentioned at relevant points in the rest of this document. Flag Meaning -k Keep all mail (do not delete it from the server). This is useful for testing delivery rules without risking mail ending up permanently in the wrong place. -l Log to syslog(3) using the 'mail' facility rather than outputting to stderr. -m Ignore the lock file. -n Run a syntax check on the configuration file and exit without fetching any mail. -q Quiet mode. Don't print anything except errors. -v Print verbose debugging output. This option may be specified multiple times for increasing levels of verbosity. Useful levels are -vv to display the result of parsing the configuration file, and -vvvv to copy all traffic to and from POP3 or IMAP servers to stdout (note that -l disables this behaviour). -f conffile Specify the path of the configuration file. -u user Use 'user' as the default user for delivering mail when started as root. -a account Process only accounts with a name matching the given pattern. Note that fnmatch(3) wildcards may be used to match multiple accounts with one option, and that the option may be specified multiple times. -x account Process all accounts except those that match the given pattern. Again, fnmatch(3) wildcards may be used, and the -x option may be specified multiple times. -D name=value Define a macro. The macro name must be prefixed with '$' or '%' to indicate if it is a string or numeric macro. Macros defined on the command line override any macros with the same name defined in the configuration file. If -n is not specified, the flags must be followed by one of the keywords 'fetch' or 'poll' or 'cache'. The 'fetch' keyword will fetch and deliver mail, the 'poll' keyword print an indication of how many mails are present in each account, and the 'cache' keyword is followed by one of a set of cache commands used to manipulate caches from the command-line (see the sections on caches below). 'fetch' or 'poll' or 'cache' may be abbreviated. Examples: $ fdm -v poll $ fdm -vvnf /etc/my-fdm.conf $ fdm -lm -a pop3\* fetch $ fdm -x stdinacct fetch # fdm -u nicholas -vv f

4.3 Running from cron

To fetch mail regularly, fdm must be run from cron. This line in a crontab(5) will run fdm every 30 minutes: */30 * * * * /usr/local/bin/fdm -l fetch The '-l' option sends fdm's output to syslog(3) rather than having cron mail it. To keep a closer eye, adding '-v' options and removing '-l' will have debugging output mailed by cron, or, using a line such as: */30 * * * * fdm -vvvv fetch >>/home/user/.fdm.log 2>&1 Will append extremely verbose fdm output to the ~/.fdm.log file. Note that this log file can become pretty large, so another cronjob may be required to remove it occasionally!

4.4 The lock file

fdm makes use of a lock file to prevent two instances running simultaneously. By default, this lock file is .fdm.lock in the home directory of the user who runs fdm, or /var/db/fdm.lock for root. This default may be overridden in the configuration file with the 'set lock-file' command: set lock-file "/path/to/my/lock-file" Or disabled altogether by being set to the empty string: set lock-file "" The '-m' command line option may be used to force fdm to ignore the lock file and run regardless of its existence and without attempting to create it.

4.5 Testing and debugging

fdm has some features to assist with testing and debugging a ruleset: The '-n' command line option. This is particularly useful in conjunction with '-vv', for example: $ cat test.conf account "pop3" pop3 server "s" user "u" pass "p" action "rw" rewrite "sed 's/\\(Subject:.*\\)\\[XYZ\\]/\1/'" action "mbox" mbox "%h/INBOX" match all actions { "rw" "mbox" } $ fdm -vvnf test.conf version is: fdm 0.6 (20061204-1433) starting at: Tue Dec 5 15:45:41 2006 user is: nicholas, home is: /home2/nicholas loading configuration from test.conf added account: name=pop3 fetch=pop3 server "s" port pop3 user "u" added action: name=rw deliver=rewrite "sed 's/\(Subject:.*\)\[XYZ\]/1/'" added action: name=mbox deliver=mbox "%h/INBOX" finished file test.conf added rule: actions="rw" "mbox" matches=all configuration loaded locking using: flock headers are: "to" "cc" domains are: "yelena" using tmp directory: /tmp Looking at the output, the parsed strings used by fdm can be seen, and it is possible to spot that an escape character has been missed in the command. If '-vvvv' is used, fdm will print all data sent to and received from remote servers to stdout. Note that this is disabled if the '-l' option is given, and includes passwords, usernames and hostnames unmodified. The 'fdm-sanitize' script provided with fdm may be used to remove passwords and usernames from this output, either while it is being collected: fdm -vvvv -a testacct f 2>&1|./fdm-sanitize|tee my-output Or afterwards: ./fdm-sanitize <vvvv-output >my-output Since fdm fetches multiple accounts simultaneously, which may intersperse debugging output, it is recommended to fetch each account seperately if running the output through fdm-sanitize. If this is not done, it may not be able to detect all usernames or passwords. The '-k' command line option (and the 'keep' keywords on actions and accounts, covered later) prevent fdm from deleting any mail after delivery. This may be used to perform any number of test deliveries without risk of losing mail.

5 Fetching mail

fdm fetches mail from a set of 'accounts', defined using the 'account' keyword. Each account has a name, a type, a number of account specific parameters and a couple of optional flags. The general form is: account <name> [<users>] [disabled] <type> [<parameters>] [keep] The <name> item is a string by which the account is referred in filtering rules, log output and for the '-a' and '-x' command line options. The <users> portion specifies the default users to use when delivering mail fetched from this account as root. It has the same syntax as discussed in detail in the section below on defining actions. If the optional 'disabled' keyword is present, fdm ignores the account unless it is specified on the command line using the '-a' flag. The optional 'keep' keyword instructs fdm to keep all mail from this account (not delete it from the server) regardless of the result of the filtering rules. The <type> item may be one of: 'pop3', 'pop3s', 'imap', 'imaps', 'stdin', 'maildir' or 'maildirs'.

5.1 Mail tags

As mail is processed by fdm, it is tagged with a number of name/value pairs. Some tags are added automatically, and mail may also be tagged explicitly by the user (see the later tagging section). Tags may be inserted in strings in a similar manner to macros, except tags are processed when the string is used rather than always as the configuration file is parsed. A tag's value is inserted by wrapping its name in %[], for example: match string "%[account]" to "myacct" action "myacctact" Most of the default tags have a single-letter shorthand which removes the needs for the []s: match string "%a" to "myacct" action "myacctact" Including a nonexistent tag in a string is equivalent to including a tag with an empty value, so "abc%[nonexistent]def" will be translated to "abcdef". The automatically added tags are: Name Shorthand Replaced with account %a The name of the account from which the mail was fetched. home %h The delivery user's home directory. uid %n The delivery user's uid. action %t The name of the action the mail has matched. user %u The delivery user's username. hour %H The current hour (00-23). minute %M The current minute (00-59). second %S The current second (00-59). day %d The current day of the month (00-31). month %m The current month (01-12). year %y The current year as four digits. year2 The current year as two digits. dayofweek %W The current day of the week (0-6, Sunday is 0). dayofyear %Y The current day of the year (000-365). quarter %Q The current quarter (1-4). rfc822date The current time in RFC822 date format. mail_hour The hour from the mail's date header, converted to local time, if it exists and is valid, otherwise the current time. mail_minute The minute from the mail's date header. mail_second The second from the mail's date header. mail_day The day from the mail's date header. mail_month The month from the mail's date header. mail_year The year from the mail's date header as four digits. mail_year2 The same as two digits. mail_rfc822date The mail date in RFC822 format. hostname The local hostname. In addition, the shorthand %% is replaced with a literal %, and %1 to %9 are replaced with the result of any bracket expressions in the last regexp (see later section on regexps). A leading ~ or ~user is expanded in strings where a path or command is expected. Some accounts add additional tags, discussed below. Tags are replaced in almost all strings (including those in single-quotes!), some when the configuration file is parsed and some when the string is used.

5.2 POP3 and POP3S

Mail may be fetched from a POP3 account. A POP3 account is defined by specifying the following parameters: the server host and optionally port, and optionally the user name and password. If the port is not specified, the default port ('pop3' in the services(5) database) is used. If the user name, password, or both is omitted, fdm attempts to look it up the .netrc file, see the next section for details. Examples of a POP3 account definition are: account "pop3acct" pop3 server "pop.isp.com" user "bob" pass "pass" account "gmx" pop3 server "pop.gmx.net" port 110 user "jim" pass "pass" account "acct" pop3 server "10.0.0.1" port "pop3" user "nicholas" keep account "lycos" disabled pop3 server $localserver port 10110 pass "password" Note that the server string is enclosed in double quotes even if it is an IP, and don't forget to escape any " and \ characters in passwords! fdm will attempt to use APOP to obscure the password, if the server offers it. If the server advertises itself as supporting APOP but subsequently refuses to accept it, fdm will not retry with a cleartext password. Use of APOP can be disabled for an account using the 'no-apop' flag, for example: account "acct" pop3 server "server" user "bob" pass "pass" no-apop POP3S is specified in exactly the same way, except using the 'pop3s' keyword for the type, and the default port is 'pop3s' rather than 'pop3': account "pop3sacct" pop3s server "pop.isp.com" user "bob" pass "pass" POP3 accounts automatically tag mail with 'server' and 'port' tags, with the value of the server and port attributes exactly as specified in the account definition. A 'server_uid' tag is also added with the server unique id (UIDL). POP3 adds 'lines', 'body_lines' and 'header_lines' tags with the number of lines in the complete mail and its body and header. These tags are not updated to reflect any changes made to the mail by fdm rules.

5.3 SSL certificate verification

fdm can verify SSL certificates before collecting mail from an SSL server. This is enabled globally with the 'verify-certificates' option: set verify-certificates And may be disabled per-account using the 'no-verify' keyword (this applies to both POP3S and IMAPS accounts): account "pop3sacct" pop3s server "pop.isp.com" no-verify For an introduction to SSL, see: http://httpd.apache.org/docs/2.0/ssl/ssl_intro.html A cert bundle is required to verify SSL certificate chains. For more information see: http://lynx.isc.org/current/README.sslcerts A pregenerated bundle is available courtesy of the MirOS project: http://cvs.mirbsd.de/src/etc/ssl.certs.shar

5.4 The .netrc file

If the user name or password is omitted in POP3 or IMAP account definitions, fdm will attempt to look it up in the .netrc file in the invoking user's home directory. The .netrc file format is shared with ftp(1) and some other programs. It consists of a number of 'machine' sections and optionally one 'default' section containing a username ('login') and password for that host. fdm accepts entries only if the machine name matches the POP3 or IMAP server string exactly. If no matches are found and a 'default' section exists, it is used. An example .netrc file is: machine "my.mail-server.com" login "nicholas" password "abcdef" machine "pop.googlemail.com" password "pass1" default login "bob" password "moo" fdm will abort if the .netrc file is world-writable or world-readable.

5.5 IMAP and IMAPS

IMAP and IMAPS accounts are defined using exactly the same syntax as for POP3 and POP3S, aside from using the 'imap' or 'imaps' keywords and that the default port is 'imap' or 'imaps'. There is also an additional, optional 'folders' option to specify the folders from which mail should be fetched. If omitted, fdm defaults to the inbox. Note that with IMAP and IMAPS, mail is still removed from the server unless the 'keep' option is given, or the '-k' command line option used. Examples of IMAP and IMAPS accounts include: account "imapacct" imap server "imap.server.ca" user "bob" pass "pass" account "oldimap" disabled imaps server "192.168.0.1" port 10993 user "nicholas" pass "pass" folders { "Saved" "MyStuff" } account "workspam" disabled imap server "my-work.ath.cx" user "Nicholas" folder "Junk" By default, fdm prefers the CRAM-MD5 authentication method, since no passwords are sent in the clear. If the server does not advertise CRAM-MD5 capability, the older LOGIN method is used. For IMAPS connections (which use SSL), the LOGIN method is just as secure. Either of these methods may be disabled with the 'no-cram-md5' and 'no-login' options. As with POP3, IMAP adds the 'server', 'port', 'server_uid' and the three line count tags to mail.

5.6 IMAP or POP3 over a pipe or ssh

Mail may be fetched using IMAP or POP3 via a pipe. This is particularly useful for fetching mail over ssh using public keys. For IMAP, a user and password may be supplied, but fdm will only use them if the server asks. If the connection is preauthenticated, the user and password are unnecessary. For POP3, a user and password must be supplied as usual: due to the lack of server name, it cannot be read from the .netrc file. Communication takes place via the pipe program's stdin and stdout. If any output is found on stderr, fdm will print it (or log it with '-l'). Examples are: account "imapssh" imap pipe "ssh jim@myhost /usr/local/libexec/imapd" account "imapssh2" imap pipe "/usr/bin/whatever" user "bob" pass "bah" account "pop3local" pop3 pipe "/usr/local/bin/ipop3d" user "me" pass "foo"

5.7 stdin and local mail

fdm may be configured to fetch mail from stdin, by specifying an account of type 'stdin', for example: account "stdin" disabled stdin This is most useful to have fdm behave as a mail delivery agent. To configure it for single-user use with sendmail, the simplest method it to add: "|/usr/local/bin/fdm -m -a stdin fetch" To the user's ~/.forward file (including the double quotes). Note the use of '-m' to prevent stdin delivery from interfering with any normal cronjob, and '-a' to specify that only the disabled "stdin" account should be fetched. stdin accounts add the three line count tags described in the POP3 section.

5.8 From maildirs and mboxes

Fetching from maildirs allows fdm to be used to filter mail on the local machine. This is covered more detail in the later section on archiving and searching. Maildir accounts are specified as follows: account "mymaildir" maildir "/path/to/dir" account "mymaildirs" maildirs { "/path/to/dir1" "/path/to/dir2" } Shell glob wildcards may be included in the path names to match multiple maildirs, but every directory found must be a valid maildir. Maildir accounts tag mail with a 'maildir' tag which is the basename of the maildir. Fetching from mboxes is similar: account "mybox" mbox "/path/to/mbox" account "mymboxes" mboxes { "/path/to/mbox1" "/path/to/mbox2" } Note that if an mbox is modified (mail is dropped from it), sufficient disk space is required to create a temporary copy of the entire mbox.

5.9 Using NNTP and NNTPS

fdm can fetch news messages from a news server using NNTP or NNTPS. News accounts are specified like so: account "news1" nntp server "news.server.sk" port 119 group "comp.unix.bsd.openbsd.misc" cache "%h/.fdm.cache/%[group]" account "mynews" nntps server "ssl.news.server" port "nntps" user "myuser" pass "mypass" groups { "alt.test" "alt.humor.best-of-usenet" } cache "%h/.fdm.cache" The cache is a file used to store details of the last article fetched. If only one group is supplied in the account definition, %[group] tags are replaced by the name of the group in the cache path. If multiple groups are provided, %[group] is removed. Note that whether a message is kept or deleted is irrelevent to NNTP, articles are always left on the server. The index and message-id of the last article is recorded in the cache file so that older articles are skipped when the a newsgroup is again fetched. This happens regardless of any 'keep' keywords or the '-k' command line option. As with POP3 and IMAP, NNTP accounts add the 'server' and 'port' tags to mail. In addition, a 'group' tag is added with the group name. This can ensure articles are matched purely on the group they are fetched from (trying to do this using headers is unreliable with cross-posted articles). For example: match account "news" { match string "%[group]" to "comp.lang.c" action "news-%[group]" match string "%[group]" to "comp.std.c" action "news-%[group]" match all action drop }

5.10 New or old mail only

With POP3 and IMAP, fdm can be set up to fetch only new or old mail. For POP3 this is achieved by recording the current state of the server in a cache file, which is updated as each mail is fetched. For IMAP it makes use of the 'seen' server flag which is updated by the server after each mail is fetched. These options are specified as in the following examples. For POP3: account "name" pop3 server "blah" new-only cache "~/.fdm-pop3-cache" account "acct" pop3s server "my-server" user "bob" new-only cache "my-server-pop3-cache" no-apop And for IMAP: account "imap" imap server "blah" new-only account "sslimap" imaps server "imaps.somewhere" user "user" pass "pass" old-only no-verify Note that currently, when using this with IMAP, the server is permitted to flag the mail as 'seen' before fdm has successfully delivered it, so there is no guarantee that mail so marked has been delivered, only that it has been fetched.

6 Defining actions

An action is a particular command to execute on a mail when it matches a filtering rule (see the next section on filtering mail). Actions are named, similar to accounts, and have a similar form: action <name> [<users>] <type> <parameters> The <users> item may be either: - the keyword 'user' followed by a single username string or uid, such as: user "nicholas" user "1000" - the keyword 'users' followed by a list of users in {}s, for example: users { "1001" "nicholas" } If users are specified, the action will be run once for each user, with fdm changing to that user before executing the action. Note that fdm will execute the action once for each user even when not started as root, but will not be able to change to the user. The user keyword is primarily of use in multiuser configurations. If users are present on an action, they override any specified by the account definition. Users are looked up in the Unix passwd file or optionally (if fdm is built with "make -DCOURIER" or "make COURIER=1") using courier-authlib. The order of lookups may be specified with the lookup-order option: set lookup-order courier passwd set lookup-order passwd If running as root and no user is specified on either the action or on the filtering rule (see the section on filtering below), the default user is used, see the '-u' command line option and the 'default-user' option in the setting options section

6.1 Drop and keep

The simplest actions are the 'drop' and 'keep' actions. They have no parameters and are specified like this: action "mydropaction" drop action "mykeepaction" keep The 'drop' action arranges for mail to be dropped when rule evaluation is complete. Note that using 'drop' does not stop further evaluation if the filtering rule contains a 'continue' keyword, and it may be overridden by a 'keep' option on the account or by the '-k' flag on the command line. The 'keep' action is similar to 'drop', but it arranges for the mail to be kept once rule evaluation is complete, rather than dropped.

6.2 Maildirs

Mails may be saved to a maildir through a 'maildir' action, defined like so: action "mymaildiraction" maildir "/path/to/maildir" If any component of the maildir path does not exist, it is created, unless the no-create option is specified. Mails saved to a maildir are tagged with a 'mail_file' tag containing the full path to the file in which they were saved.

6.3 Mboxes

An action to deliver to an mbox is defined in the same way as for a maildir: action "mymboxaction" mbox "/path/to/mbox" The same % tokens are replaced in the path. If the mbox does not exist, it is created. Mboxes may optionally be gzip compressed by adding the 'compress' keyword: action "mymboxaction" mbox "/path/to/mbox" compress fdm will append .gz to the mbox path (if it is not already present) and append compressed data. If the mbox exists but is not already compressed, uncompressed data will be appended. As with maildirs, if any component of the mbox path does not exist, it is created, unless the no-create option is set. Mails saved to an mbox are tagged with an 'mbox_file' tag with the path of the mbox.

6.4 IMAP and IMAPS

An action may be defined to store mail in an IMAP folder. The specification is similar to the IMAP action. A server host and optionally port (default 'imap' or 'imaps') must be specified. A username and password may be supplied; if they are omitted, fdm will attempt to find a .netrc entry. Examples include: action "myimapaction" imap server "imap.server" action "myimapaction" imaps server "imap.server" port "8993" user "user" pass "pass" folder "folder" action "myimapaction" imaps server "imap.server" user "user" pass "pass" no-verify no-login

6.5 SMTP

An action may be defined to pass mail on over SMTP. The server host must be specified and optionally the port and string to pass to the server with the RCPT TO and MAIL FROM commands. If the port is not specified it defaults to "smtp". Examples include: action "mysmtpaction" smtp server "smtp.server" action "mysmtpaction" smtp server "smtp.server" port 587 action "mysmtpaction" smtp server "smtp.server" port "submission" from "bob@server.com" action "mysmtpaction" smtp server "smtp.server" to "me@somewhere"

6.6 Write, pipe, exec and append

Actions may be defined to write or append a mail to a file, to pipe it to a shell command, or merely to execute a shell command. The append action appends to and write overwrites the file. % tokens are replaced in the file or command as for maildir and mbox actions. Examples are: action "mywriteaction" write "/tmp/file" action "myappendaction" append "/tmp/file" action "mypipeaction" pipe "cat > /dev/null" action "domaildirexec" exec "~/.fdm.d/my-special-script %[mail_file]" Pipe and exec commands are run as the command user (by default the user who invoked fdm).

6.7 stdout

fdm can write mails directly to stdout, using the 'stdout' action: action "so" stdout

6.8 Rewriting mail

Mail may be altered by passing it to a rewrite action. This is similar to the pipe action, but the output of the shell command to stdout is reread by fdm and saved as a new mail. This is useful for such things as passing mail through a spam filter or removing or altering headers with sed. Note that rewrite only makes sense on filtering rules where the continue keyword is specified, or where multiple actions are used (see the next section for details of this). Possible rewrite action definitions are: action "myspamaction" rewrite "bmf -p" action "mysedaction" rewrite "sed 's/x/y/'"

6.9 Adding or removing headers

Simple actions are provided to add a header to a mail: action "lines" add-header "Lines" value "%[lines]" Or to remove all instances of a header from mail: action "del-ua" remove-header "user-agent" action "rmhdr" remove-header "x-stupid-header" action "remove-headers" remove-headers { "X-*" "Another-Header" }

6.10 Tagging

Mails may be assigned one of more tags manually using the tag action type. For example, match account "my*" action tag "myaccts" match "^User-Agent:[ \t]*(.*)" action tag "user-agent" value "%1" The tag is attached to the mail with the specified value, or no value if none is provided.

6.11 Compound actions

Compound actions may be defined which perform multiple single actions. They are similar to standard single actions but multiple actions are provided using {}. For example, action "multiple" { add-header "X-My-Header" value "Yay!" mbox "mbox2" } action "myaction" users { "bob" "jim" } { rewrite "rev" maildir "%h/%u's maildir" } Compound action are executed from top-to-bottom, once for each user. Note that the effects are cumulative: the second example above would deliver a mail rewritten once to 'bob' and rewritten again (ie, twice) to 'jim'. If this is not desired, seperate actions must be used.

6.12 Chained actions

An action may call other named actions by reusing the 'action' keyword: action "abc" action "def" action "an_action" { rewrite "rev" action "another_action" action "yet_more_actions" } There is a hard limit of five chained actions in a sequence to prevent infinite loops.

7 Filtering mail

Mail is filtered by defining a set of filtering rules. These rules tie together mail fetched from an account and passed to one or more actions. Rules are evaluated from top-to-bottom of the file, and evaluation stops at the first matching rule (unless the continue keyword is specified). The general form of a filtering rule is: match <conditions> [<users>] <actions> [continue] The optional <users> item is specified as for an action definition. If users are specified on a filtering (match) rule, they override any specified on the action or account. The <conditions> item is set of conditions against which the match may be specified, each condition returns true or false. Conditions are described in the next few sections. Aside from the 'all' condition, which is a special case, conditions may be chained as an expression using 'and' and 'or', in which case they are evaluated from left to right at the same precedence, or prepended with 'not' to invert their outcome. The <actions> item is a list of actions to execute when this rule matches. It is in the same list format: 'action "name"' or 'actions { "name1" "name2" }'. It may also be a lambda (inline) action, see the section below. If a rule with the 'continue' keyword matches, evaluation does not stop after the actions are executed, instead subsequent rules are matched.

7.1 Nesting rules

Filtering rules may be nested by using the special form: match <conditions> [<accounts>] { match ... } If the conditions on the outer rule match, the inner rules are evaluated. If none of the inner rules match (or they all specify the 'continue' keyword) evaluation continues outside to rules following the nested rule, otherwise it stops.

7.2 Lambda actions

Lambda actions are unnamed actions included inline as part of the filtering rule. This can be convenient for actions which do not need to be used multiple times. Lambda actions are specified as a combination of the rule and an action definition. For example: match all action maildir "mymaildir" match all actions { rewrite "rev" tag "reversed" } continue

7.3 The all condition

The all condition matches all mail. Examples include: match all action "default" match all rewrite "rewaction" continue

7.4 Matched and unmatched

The matched and unmatched conditions are used to match mail that has matched or has not matched previous rules and been passed on with the 'continue' keyword. For example, match "myregexp" action "act1" continue # This rule will match only mails that also matched the first. match matched action "act2" # This rule will match only mails that matched neither of the first two. match unmatched action "act3"

7.5 Matching by account

The account condition matches a list of accounts from which the mail was fetched. It is specified as either a single account ('account "name"') or a list of accounts ('accounts { "name1" "name2" }'). fnmatch(3) wildcards may also be used. Examples include: match "blah" accounts { "pop3" "imap" } action "go!" match matched and account "myacc" action drop

7.6 Matching a regexp

Matching against a regexp is the most common form of condition. It takes the following syntax: [case] <regexp> [in headers|in body] The 'case' keyword instructs fdm to match the regexp case sensitively rather than the default of case insensitivity. The 'in headers' or 'in body' keywords make fdm search only the headers or body of each mail, the default is to match the regexp against the entire mail. Any multiline headers are unwrapped onto a single line before matching takes place and the process reversed afterwards. The regexp itself is an extended regexp specified as a simple string, but care must be taken to escape \s and "s properly. Examples include: match "^From:.*bob@bobland\\.bob" in headers and account "pop3" action "act" match ".*YAHOO.*BOOTER.*" in body action "junk"

7.7 Matching bracket expressions

The results of any bracket expressions within the last regexp match are remembered, and may be made use of using the 'string' condition, or used to construct an action name, maildir or mbox path, etc. The bracket expressions may be substituted using the %0 to %9 tokens. For example, match "^From:.*[ \t]([a-z]*)@domain" in headers action "all" continue match string "%1" to "bob.*" action "bobmail" match "^From:.*[ \t]([a-z]*)@domain" in headers action "all" continue match all action "%1mail" This is particularly useful in combination with nested rules (see later): bracket expressions in a regexp on the outer rule may be compared on inner rules. Note that %0 to %9 are used only for 'regexp' rules. Regexps that are part of 'command' rules use the 'command0' to 'command9' tags.

7.8 Matching by age or size

Mail may be matched based on its age or size. An age condition is specified as follows: age [<|>] <age> [hours|minutes|seconds|days|months|years] If '<' is used, mail is matched if it is younger than the specified age. If '>', if it is older. The <age> item may be a simple number of seconds, or suffixed with a unit. Examples are: match age < 3 months actions { "act1" "act2" } match age > 100 hours action "tooold" The age is extracted from the 'Date' header, if possible. To match mails for which the header was invalid, the following form may be used: match age invalid action "baddate" The size condition is similar: size [<|>] <size> [K|KB|kilobytes...] Where <size> is a simple number in bytes, or suffixed with 'K', 'M' or 'G' to specify a size in kilobytes, megabytes or gigabytes, such as: match size < 1K action "small" match size > 2G action "whoa"

7.9 Using a shell command

Mail may be matched using the result of a shell command. This condition follows the form: [exec|pipe] <command> returns (<return code>, [case] <stdout regexp>) If 'exec' is used, the command is executed. If 'pipe', the mail is piped to the command's stdin. The <command> is a simple string. % tokens are replaced as normal. Any of the <return code> or <stdout regexp> or both may be specified. The <return code> is a simple number which is compared against the return code from the command, the <stdout regexp> is a regexp that is matched case insensitively against each line output by the command on stdout. The result of any bracket expressions in the stdout regexp are saved as 'command0' to 'command9' tags on the mail. Any output on stderr is logged by fdm, so 2>&1 must be included in the command in order to apply the regexp to it. Examples: match exec "true" (0, ) action "act" match not pipe "grep Bob" (1, ) action "act" match pipe "myprogram" (, "failed") actions { "act1" "act2" } match exec "blah" (12, "^Out") action "meep"

7.10 Attachments

There are five conditions available to filter based on the size, quantity, type and name of attachments. They are all prefixed with the 'attachment' keyword. Two compare the overall number of attachments: The 'attachment count' conditions matches if the number of attachments is equal to, not equal to, less than or greater than the specified number: match attachment count == 0 action "action" match attachment count != 10 action "action" match attachment count < 2 action "action" match attachment count > 7 action "action" The 'attachment total-size' condition is similar, but compares the total size of all the attachments in a mail: match attachment total-size < 4096 kilobytes action "action" match attachment total-size > 1M action "action" There are also three conditions which matches if any individual attachment fulfils the condition: 'any-size' to match if any attachment is less than or greater than the given size, and 'any-type' and 'any-name' which compare the attachment MIME type and name attribute (if any) using fnmatch(3): match attachment any-size < 2K action "action" match attachment any-type "*/pdf" action "action" match attachment any-name "*.doc" action "action"

7.11 Matching tags

The existence of a tag may be tested for using the 'tagged' condition: match tagged "mytag" action "a" match tagged "ohno" and size >1K action drop Or the tags value matched using the 'string' match type (in a similar way to matching bracket expressions): match string "%[group]" to "comp.lang.c" action "clc" match string "%u" to "bob" action "bob"

7.12 Using caches

fdm has builtin support for maintaining a cache of string keys, including appending to a cache, checking if a key is present in a cache, and expiring keys from a cache once they reach a certain age. These caches should not be confused with the NNTP cache file. Key caches are referenced by filename and must be declared before use: cache "%h/path/to/cache" cache "~/.fdm.db" expire 1 month If the expiry time is not specified, items are never expired from the cache. Once declared, keys may be added to the cache with the 'add-to-cache' action: match all action add-to-cache "~/my-cache" key "%[message_id]" Or removed with the 'remove-from-cache' action: match all action remove-from-cache "~/my-cache" key "%[message_id]" And the existence of a key in the cache may be tested for using the 'in-cache' condition: match in-cache "~/my-cache" key "%[message_id]" action "foundincache" Any string may be used as key, but the message-id is most often useful. Note that the key may not be empty, so care must be taken with messages without message-id (such as news posts fetched with NNTP). Caches may be used to elimate duplicate messages using rules similar to those above: $db = "~/.fdm-duplicates.db" $key = "%[message_id]" cache $db expire 2 weeks match not string $key to "" { match in-cache $db key $key action maildir "%h/mail/duplicates" match all action add-to-cache $db key $key continue }

7.13 Cache commands

fdm includes a number of commands to manipulate caches from the command-line. These are invoked with the 'cache' keyword followed by a command. The following commands are supported: cache add <path> <string> cache remove <path> <string> These add or remove <string> as a key in the cache <path>. cache list [<path>] This lists the number of keys in a cache, or in all caches declared in the configuration file if <path> is omitted. cache dump <path> This dumps the contents of the cache <path> to stdout. Each key is printed followed by a space and the timestamp as Unix time. cache clear <path> Delete all keys from a cache. Examples: $ fdm cache list /export/home/nicholas/.fdm.d/duplicates: 4206 keys $ touch my-cache $ fdm cache dump my-cache $ fdm cache add my-cache test $ fdm cache dump my-cache test 1195072403

8 Setting options

fdm has a number of options that control its behaviour. These are defined using the set command: set <option> [<value>] In addition to the options below, some environment variables may be used to control fdm. If TMPDIR is present, its value will be used instead of /tmp for saving temporary files. The possible options are: - maximum-size <size> This specifies the maximum size of mail that fdm will accept. The default is 32 MB. Note that fdm may be storing a number of mails simultaneously, up to the 'queue-high' setting (doubled if rewrite is used) for each account, so care should be taken when increasing this option. If mail over the maximum size is encountered, fdm will abort with an error, without deleting the mail from the server (unless 'delete-oversized' is set). - delete-oversized If this option is set, fdm will delete oversized mail from the server rather than leaving it for the user to sort out. - queue-high <number> This sets the number of mails fdm will queue simultaneously. The default is two. Once this limit is reached, fdm will cease fetching mail until the number queued drops to the value of the 'queue-low' setting. fdm queues mails so that they may be processed while waiting for data from the server. Changing this option can increase (or decrease) performance, it's usefulness varies wildly with the ruleset, speed of connection and remote host, and the local hardware. The maximum possible value is currently 50. To ensure fdm fetches and delivers mail sequentially, set this option to one. - queue-low <number> This sets the number of mails fdm will wait for the queue to drop to before restarting fetching after the 'queue-high' limit has been reached. The default is three-quarters of the 'queue-high' setting. - allow-multiple This option makes fdm ignore the lock file, similar to the '-m' command line option. - lock-file <path> This option specifies a string to use as the path of the lock file. For example: set lock-file "/tmp/my-lock-file" - command-user <user> This specifies the user used to run exec and pipe actions. By default it is the user who invoked fdm. - default-user <user> This specifies the default user to use if run as root and no users are specified on the action or filtering rule. This option may be overriden with the '-u' flag on the command line. - lock-types [fcntl] [flock] [dotlock] These specify the type of locking to use when writing to mboxes. fcntl and flock locking are mutually exclusive, but dotlock may be used with either. Some NFS servers do not support fcntl. The default is flock only. Example: set lock-types fcntl dotlock - proxy <url> This specifies a URL to proxy outgoing connections through. See the section on proxying below. - unmatched-mail [drop|keep] This option controls how fdm should deal with mail that reaches the end of the ruleset (doesn't match any rules, or only rules with the 'continue' keyword). If 'drop' is specified, such mail is dropped. If 'keep', it is kept. The default is to keep the mail, and issue a warning. - purge-after <message count> The 'purge-after' option instructs fdm to attempt to purge deleted mail after the specified number of mails has been fetched. This is useful to limit the number of mails refetched on the next run if the connection fails. It can have a large effect on performance, particularly if the message count is set to a low number. Note that for POP3, purging deleted mail involves disconnecting and reconnecting from the server; some POP3 servers refuse reconnections if too many are made too quickly, so this option should be used with care. - no-received If this option is present, fdm will not insert a 'Received' header into fetched mail. - no-create When this option is specified, fdm will not attempt to create maildir and mboxes or directories above them. - file-umask [user|<umask>] This specifies the umask to use when creating files. 'user' means to use the umask set when fdm is started, or it may be specified as a three-digit octal number. The default is 077. - file-group [user|<group>] This option allows the default group ownership of files and directories created by fdm to be specified. 'group' may be a group name string or a numeric gid. If 'user' is used, or if this option is not set in the configuration file, fdm does not attempt to set the group of new files and directories. - timeout <time> This controls the maximum time to wait for a server to send data before closing a connection. The default is 900 seconds. - verify-certificates This instructs fdm to verify SSL certificates for all SSL connections, see the previous section on SSL certificate verification.

9 Archiving and searching mail

As fdm can fetch from maildirs, it can be used to filter mail from one maildir into another, for example to archive older mail (with drop) or to search for mail matching a particular pattern (with keep). The 'age' condition, and the %[maildir] and time/date tags are particularly useful for archiving. For example, to archive all mail older than 30 days by quarter, something like this may suffice (the account restriction can be dropped if being used in a single-purpose configuration file): account "archive" disabled maildir "source-maildir" match account "archive" and age > 30 days action mbox "%[maildir]q%Q" match account "archive" action keep Then, fdm may be run to move the mail: fdm -vaarchive fetch To search mail, similar rules may be used, but all mail should be kept, in this example by marking the account with 'keep' so that mail is kept no matter what rules it matches: account "search" disabled maildir "source-maildir" keep action "found" mbox "search-results" match "^From.*Bob" in headers account "search" action "found" match account "search" action keep All mail matching the regexp will be copied to the target mbox. There are several other ways to write this ruleset.

10 Using fdm behind a proxy

fdm may be used behind a proxy by specifying the proxy URL using the 'proxy' option. HTTP and SOCKS5 proxies are supported: set proxy "http://proxy.server/" set proxy "http://proxy.server:port/" set proxy "socks5://proxy.server/" set proxy "socks5://proxy.server:port/" set proxy "socks5://user@proxy.server/" set proxy "socks5://user:pass@proxy.server/" Authentication is not supported for HTTP proxies.

11 Bug reports and queries

Bug reports, queries, suggestions and code are best sent by email to: nicm@users.sf.net Bug reports may also be registered on Sourceforge, but they are likely to be dealt with much faster by email, particularly as it is easier to elicit further information if it is required. A mailing list is available for fdm users, see: https://lists.sourceforge.net/lists/listinfo/fdm-users

12 Frequently asked questions

12.1 Why?

Two main reasons: 1. I didn't like the existing tools. That is not to say that other tools are bad or aren't useful, merely that fdm meets my needs better. And more importantly: 2. I disliked the fact that as a home user with a relatively simple setup I had to have five programs to deal with mail: sendmail, fetchmail, procmail, archivemail, mutt; all with different, variously broken configuration file syntaxes, and weird quirks. I now have three programs: sendmail, fdm and mutt, and fewer weird configurations to learn and potential problems. I can also do some quite complex things without reaching for additional tools like formail and reformail.

12.2 How do I write regexps?

See the re_format(7) man page. It is online here (for OpenBSD): http://www.openbsd.org/cgi-bin/man.cgi?re_format There are also a number of books and probably websites on the subject.

12.3 Keep doesn't work with GMail!

This is because GMail is broken, see: http://pyropus.ca/software/getmail/faq.html#faq-notabug-gmail-bug In addition, GMail IMAP doesn't set the \Seen flag when mail is fetched with the UID FETCH BODY[] command (RFC3501 says that the "\Seen flag is implicitly set"). fdm works around this bug by setting the flag explicitly when mail is kept.

12.4 Why doesn't fdm run as a daemon?

Because that is what cron is for: it comes as standard with all sensible operating systems, and by using it I avoid a lot of code to deal with signals, configuration reload, extra configuration options and general other crap related to running all the time. Daemonising is for servers, for stuff that needs to run periodically, use cron.

12.5 Why does fdm fork child processes when not running as root?

Because one design is much cleaner and easier to work with than two. And it has a negligible practical effect on performance in any case.

12.6 Can fdm get rid of that crap in []s some mailing lists add to subjects?

Yes! Rewrite the mail using sed to remove the crap, for example: action "full-disclosure" { rewrite "sed 's/^\\(Subject:.*\\)\\[Full-disclosure\\] /\\1/'" maildir "%h/mail/full-disclosure" } match "^List-Id:.*<full-disclosure\\.lists\\.grok\\.org\\.uk>" in headers action "full-disclosure"

12.7 I'm building on Linux and it complains about REG_STARTEND.

Older versions (pre-2005) of glibc don't support the REG_STARTEND flag to regexec(3). Either upgrade to a later version of glibc, or use PCRE.

12.8 Should I use PCRE or standard POSIX regexps? What's the difference?

PCRE is a library providing "perl-compatible" regexps. These are broadly compatible with POSIX extended regexps, but with a number of extensions based on perl regexps. Unless you want or need some of the extensions, there is generally no compelling reason to choose one over the other. PCRE is faster than some POSIX regexp implementations, but few rulesets will include sufficient regexps for this to make any difference. PCRE has had some security problems, but most regexp implementations pose a similar risk for their complexity if nothing else; fdm performs regexp matching as non-root in any case. The Debian package and FreeBSD port both use PCRE by default. Because I hate maintaining and testing two sets of code, there is a strong possibility that PCRE may become the fdm default at some point. ================================================================================ $Id: MANUAL.in,v 1.80 2009/07/29 19:20:16 nicm Exp $

FDM(1)
FDM(1) OpenBSD Reference Manual FDM(1)

NAME

fdmfetch and deliver mail

SYNOPSIS

fdm [-hklmnqv] [-a account] [-D namevalue] [-f conffile] [-u user] [-x account] [fetch | poll]

DESCRIPTION

The fdm program fetches mail from a POP3 or IMAP server or from stdin and delivers it based on a ruleset in the configuration file.

The options are as follows:

-a name
Process only the specified account. This option may appear multiple times. The account name may include shell glob characters to match multiple accounts.
-D name=value
This option defines a macro for use when parsing the configuration file. The macro name must be prefixed with $ or % to specify a string or numeric macro. This option may appear multiple times.
-f conffile
Specify the configuration file location. Default is ~/.fdm.conf, or /etc/fdm.conf if that doesn't exist.
-h
Look at the HOME environment variable to ascertain the user's home directory.
-k
Keep all mail after delivery, regardless of whether it matches a drop action. Note that mails kept in this way will be refetched by fdm if it is run again on the same account.
-l
Log using syslog(3) rather than to stderr.
-m
Ignore the lock file and run regardless of other instances of fdm.
-n
Do not process any accounts, just verify the configuration file syntax and exit.
-q
Quiet mode. Only print errors.
-u user
Specify the default user for delivery. This overrides the default-user option in the configuration file.
-v
Request verbose logging. This option may be specified multiple times. -vv will print information on configuration (useful with -n). -vvvv duplicates all traffic to and from remote servers to stdout. This feature is disabled when using the -l flag.
-x name
Exclude the named account. Multiple -x options may be specified. As with -a, shell glob characters may be used.
fetch | poll | cache
The fetch command instructs fdm to fetch and deliver messages. The poll command polls the accounts in the configuration file and reports a message count for each. cache allows fdm cache files to be manipulated: see the next section.

CACHE COMMANDS

The following cache manipulation commands are supported:
cache add path string
cache remove path string
Add or remove string as a key in the cache at path.
cache list [path]
List the number of keys in the specified cache, or if path is omitted, in all caches declared in the configuration file.
cache dump path
Dump the contents of the cache path to stdout. Each key is printed followed by a space and the timestamp as Unix time.
cache clear path
Delete all keys from the cache at path.

FILES

~/.fdm.conf
default fdm configuration file
/etc/fdm.conf
default system-wide configuration file
~/.fdm.lock
default lock file
/var/db/fdm.lock
lock file for root user

SEE ALSO

mail(1), fdm.conf(5), sendmail(8)

AUTHORS

Nicholas Marriott <nicm@users.sourceforge.net>
December 22, 2008 OpenBSD 5.3

FDM.CONF(5)
FDM.CONF(5) OpenBSD Programmer's Manual FDM.CONF(5)

NAME

fdm.conffdm configuration file

DESCRIPTION

This manual page describes the fdm(1) configuration file. It defines accounts from which to fetch mail, a number of possible actions to take, and rules connecting a regexp with an action. The file is parsed once from top to bottom, so action and account definitions must appear before they are referenced in a rule. Rules are evaluated from first to last and (unless overridden by the continue keyword) evaluation stops at the first match.

The file has the following format:

Empty lines and lines beginning with the ‘#’ character are ignored.

Regexps and strings must be enclosed in double quotes. Special characters in regexps and strings (including passwords) must be escaped. Note that this may mean double-escaping in regexps.

Possible commands are covered in the following sections.

OPTIONS

Options are configured using the set command. It may be followed by the following options, one per command:
maximum-size size
This is used to set the maximum size of a mail. Mails larger than this limit are dropped and, if applicable, not deleted from the server.

The size may be specified as a plain number in bytes or with a suffix of ‘K’ for kilobytes, ‘M’ for megabytes or ‘G’ for gigabytes. The default is 32 megabytes and the maximum is one gigabyte.

delete-oversized
If this option is specified, fdm(1) attempts to delete messages which exceed maximum-size, and continue. If it is not specified, oversize messages are a fatal error and cause fdm(1) to abort.

Note that fdm(1) may have a number of messages queued (up to the queue-high setting, doubled for rewrite, per account), so this setting and the queue-high option should be set after consideration of the space available in the temporary folder and the implications should fdm(1) abort due to the space becoming full.

queue-high number
This sets the maximum number of messages fdm(1) will hold simultaneously. fdm(1) will attempt to process previously queued messages as the next is being fetched. Once this limit is reached, no further messages wil be fetched until the number of messages held drops to the queue-low value.
queue-low number
This is the length to which the message queue must drop before fetching continues after the queue-high limit has been reached.
allow-multiple
If this option is specified, fdm(1) does not attempt to create a lock file and allows multiple instances to run simultaneously.
lock-file path
This sets an alternative lock file. The default is ~/.fdm.lock for non-root users and /var/db/fdm.lock for root.
command-user user
This specifies the user used to run exec and pipe actions. By default it is the user who invoked fdm.
default-user user
This sets the default user to change to before delivering mail, if fdm(1) is running as root and no alternative user is specified as part of the action or rule. This option may be overridden with the -u switch on the command line. A default user must be given if running as root.
lookup-order location ...
This specifies the order in which to do user lookup from left to right. Possible types are passwd to use the passwd(5) file, or courier to use Courier authlib (if support is compiled).
lock-types type ...
This specifies the locks to be used for mbox locking. Possible types are fcntl, flock, and dotlock. The flock and fcntl types are mutually exclusive. The default is flock.
proxy url
This instructs fdm(1) to proxy all connections through url. HTTP and SOCKS5 proxies are supported at present (URLs of the form http://host[:port] or socks://[user:pass@]host[:port]). No authentication is supported for HTTP.
unmatched-mail drop | keep
This option controls what fdm(1) does with mail that reaches the end of the ruleset (mail that matches no rules or matches only rules with the continue keyword). drop will cause such mail to be discarded, and keep will attempt to leave the mail on the server. The default is to keep the mail and log a warning that it reached the end of the ruleset.
purge-after count
The purge-after option makes fdm(1) attempt to purge deleted mail from the server (if supported) after count mails have been retrieved. This is useful on unreliable connections to limit the potential number of mails refetched if the connetion drops, but note that it can incur a considerable speed penalty.
no-received
If this option is present, fdm(1) will not insert a ‘Received’ header into each mail.
no-create
If this option is set, fdm(1) will not attempt to create maildirs and mboxes or missing elements of their paths.
file-umask user | umask
This specifies the umask(2) to use when creating files. user means to use the umask set when fdm(1) is started, or umask may be specified as a three-digit octal number. The default is 077.
file-group user | group
This option allows the default group ownership of files and directories created by fdm(1) to be specified. group may be a group name string or a numeric gid. If user is used, or this option does not appear in the configuration file, fdm(1) does not attempt to set the group of new files and directories.
timeout time
This controls the maximum time to wait for a server to send data before closing a connection. The default is 900 seconds.
verify-certificates
Instructs fdm(1) to verify SSL certificates for all SSL connections.

INCLUDING FILES

Further configuration files may be including using the include command:
include path

The file to include is searched for first as an absolute path and then relative to the directory containing the main configuration file.

MACROS

Macros may be defined using the following syntax:
$name = string
%name = number

Macros are prefixed with $ to indicate a string value and % to indicate a numeric value. Once defined, a macro may be used in any place a string or number is expected. Macros may be embedded in strings by surrounding their name (after the $ or %) with {}s, like so:

"abc ${mymacro} %{anothermacro} def"

The ifdef, ifndef and endif keywords may be used to conditionally parse a section of the configuration file depending on whether or not the macro given exists or does not exist. ifdef and ifndef blocks may be nested.

SHELL COMMANDS

The result of a shell command may be used at any point a string or number is expected by wrapping it in $() or %(). If the former is used, the command result is used as a string; if the latter, it is converted to an integer. Shell commands are executed when the configuration file is parsed.

ACCOUNTS

The account command is used to instruct fdm(1) to fetch mail from an account. The syntax is:
account name [users] [disabled] type [args] [keep]

The name argument is a string specifying a name for the account. The optional users argument has the following form:

user user | users { user ... }

The first two options specify a user or list of users as which the mail should be delivered when an action is executed. If no users are specified, the default user (set with set default-user) is used. Users specified as part of the account definition may be overridden by similar arguments to action definitions or on match rules. If fdm(1) is run as non-root, it will still execute any actions once for each user, but will be unable to change to that user so the action will be executed multiple times as the current user.

The disabled keyword instructs fdm(1) to ignore this account unless it is explicitly enabled with a -a option on the command line. If the keep keyword is specified, all mail collected from this account is kept (not deleted) even if it matches a drop action.

Supported account types and arguments are:

stdin
This account type reads mail from stdin, if it is connected to a pipe. This may be used to deliver mail from sendmail(8), see fdm(1) for details.
pop3 server host [port port] [user user] [pass pass] [only] [no-apop] [no-uidl]
pop3s server host [port port] [userpass] [only] [no-apop] [no-uidl] [no-verify]
These statements define a POP3 or POP3S account. The userpass element has the following form:
[user user] [pass pass]

The host, user and pass arguments must be strings. If the user or pass is not provided, fdm(1) attempts to look it up in the ~/.netrc file (see ftp(1) for details of the file format). The port option may be either a string which will be looked up in the services(5) database, or a number. If it is omitted, the default port (110 for POP3, 995 for POP3S) is used.

The only option takes the form:

[new-only | old-only] cache path

new-only fetches only mail not previously fetched, and old-only is the inverse: it fetches only mail that has been fetched before. The cache file is used to save the state of the POP3 mailbox. The no-apop flag forces fdm(1) not to use the POP3 APOP command for authentication, and the no-verify keyword instructs fdm(1) to skip SSL certificate validation for this account. The no-uidl keyword makes fdm(1) not use the UIDL command to retrieve mails. This is mainly useful for broken POP3 servers.

pop3 pipe command [userpass] [only] [no-apop]
This account type uses the POP3 protocol piped through command, such as ssh(1). If the command produces any output to stderr, it is logged. For POP3 over a pipe, providing a user and password is not optional and it may not be read from ~/.netrc.
imap server host [port port] [userpass] [folder name] [only] [no-cram-md5] [no-login]
imap server host [port port] [userpass] [folders] { name ... } [only]
imaps server host [port port] [userpass] [folders] [only] [no-verify] [no-cram-md5] [no-login]
These define an IMAP or IMAPS account. The parameters are as for a POP3 or POP3S account, aside from the additional folders option which sets the name of the folder or folders to use (the default is to fetch from the inbox). This has the form:
folder name | folders { name ... }

The default ports used are 143 for IMAP and 993 for IMAPS. For IMAP, the only item consists only of one of the keywords new-only or old-only - a cache file is not required.

Options no-cram-md5 and no-login disable the given authentication method. The default is to use CRAM-MD5 if it is available, or LOGIN otherwise.

imap pipe command [userpass] [folders] [only]
As with pop3 pipe, this account type uses the IMAP protocol piped through command. If the optional IMAP user and pass are supplied, they will be used if necessary, but if one is provided, both must be - using ~/.netrc is not permitted.
maildir path
maildirs { path ... }
These account types instruct fdm(1) to fetch mail from the maildir or maildirs specified. This allows fdm(1) to be used to filter mail, fetching from a maildir and deleting (dropping) unwanted mail, or delivering mail to another maildir or to an mbox.

Mail fetched from a maildir is tagged with a maildir tag containing the basename of the mail file.

mbox path
mboxes { path ... }
These are similar to maildir and maildirs, but cause fdm(1) to fetch mail from an mbox or set of mboxes.

Mail fetched from a mbox is tagged with a mbox tag containing the basename of the mbox file.

nntp server host [port port] [userpass] group group cache cache
nntp server host [port port] [userpass] groups { group ... } cache cache
nntps server host [port port] [userpass] group group cache cache
nntps server host [port port] [userpass] groups { group ... } cache cache
An NNTP account. Articles are fetched from the specified group or groups and delivered. The index and message-id of the last article fetched in each group is saved in the specified cache file. When fdm(1) is run again, fetching begins at the cached article. Note that the keep option is completely ignored for NNTP accounts - all mail is kept, and the cache is always updated.

TAGGING

As mail is processed by fdm(1), it is tagged with a number of name/value pairs. Some tags are added automatically, and mail may also be tagged explicitly by the user using the tag action. Tags may be inserted in most strings in a similar manner to macros, except tags are processed at runtime rather than as the configuration file is parsed. A tag's value is inserted by wrapping its name in %[], for example:
abc%[account]def
%[hour]:%[minute]:%[second]

The default tags also have a single-letter shorthand. Including a nonexistent tag in a string is equivalent to including a tag with an empty value, so "abc%[nonexistent]def" will be translated to "abcdef".

The automatically added tags are:

account (%a)
The name of the account from which the mail was fetched.
home (%h)
The delivery user's home directory.
uid (%n)
The delivery user's uid.
action (%t)
The name of the last action executed for this mail.
user (%u)
The delivery user's username.
hour (%H)
The current hour (00-23).
minute (%M)
The current minute (00-59).
second (%S)
The current second (00-59).
day (%d)
The current day of the month (01-31).
month (%m)
The current month (01-12).
year (%y)
The current year.
year2
The current year as two digits.
dayofweek (%W)
The current day of the week (0-6, Sunday is 0).
dayofyear (%Y)
The current day of the year (001-366).
quarter (%Q)
The current quarter (1-4).
rfc822date
The current date in RFC822 format.
mail_hour
The hour from the mail's date header, if it exists and is valid, otherwise the current time.
mail_minute
The minute from the mail's date header.
mail_second
The second from the mail's date header.
mail_day
The day from the mail's date header.
mail_month
The month from the mail's date header.
mail_year
The year from the mail's date header.
mail_year2
The same as two digits.
mail_dayofweek
The day of the week from the mail's date header.
mail_dayofyear
The day of the year from the mail's date header.
mail_quarter
The quarter (1-4) from the mail's date header.
mail_rfc822date
The mail's date in RFC822 format.
hostname
The local hostname.

In addition, the shorthand %% is replaced with a literal %, and %0 to %9 are replaced with the result of any bracket expressions in the last regexp.

CACHES

fdm(1) can maintain a cache file with a set of user-defined strings. In order to use caches, fdm(1) must have been compiled with them enabled. Caches are declared with the cache keyword:
cache path [expire age]

The path is the location of the cache file. If the expire keyword is specified, items in the cache are removed after they reach the age specified. age may be given unadorned in seconds, or followed by one of the modifiers: seconds, hours, minutes, days, months or years.

Caches must be declared before they are used. Items are added to caches using the add-to-cache action, removed using the remove-from-cache action, and searched for using the in-cache condition; see below for information on these.

ACTIONS

The action command is used to define actions. These may be specified by name in rules (see below) to perform some action on a mail. The syntax is:
action name [users] action
action name [users] { action ... }

The name is a string defining a name for the action. The users argument has the same form as for an account definition. An action's user setting may be overridden in the matching rule.

The possible values for action are listed below. If multiple actions are specified they are executed once in the order specified, for each user.

drop
Discard the mail.
keep
Keep the mail, do not remove it from the account.
tag string [value value]
This tags mail with string, and optionally value, which may be matched using the tagged or string conditions.
maildir path
Save the mail to the maildir specified by path. If the maildir or any part of its path does not exist, it is created, unless the no-create option is set.

Mail delivered to a maildir is tagged with a mail_file tag containing the full path of the mail file.

mbox path [compress]
Append the mail to the mbox at path. If compress is specified, fdm(1) will add ‘.gz’ to path and attempt to write mail using gzip(1) compression. If the mbox or any part of its path does not exist, it is created, unless the no-create option is set.

Mail delivered to an mbox is tagged with a mbox_file tag containing the path of the mbox.

exec command
Execute command.
pipe command
Pipe the mail to command. exec and pipe commands are run as the command user.
write path
Write the mail to path.
append path
Append the mail to path.
smtp server host [port port] [from from] [to to]
Connect to an SMTP server and attempt to deliver the mail to it. If from or to is specified, they are passed to the server in the MAIL FROM or RCPT TO commands. If not, the current user and host names are used.
rewrite command
Pipe the entire mail through command to generate a new mail and use that mail for any following actions or rules. An example of the rewrite action is:

action "cat" pipe "cat" 
action "rewrite" rewrite "sed 's/bob/fred/g'" 
# this rule will rewrite the message 
match all action "rewrite" continue 
# this rule will cat the rewritten message 
match all action "cat"
add-header name value value
Add a header name with contents value.
remove-header name
remove-headers { name ... }
Remove all occurances of headers matching the fnmatch(3) pattern name.
stdout
Write the mail to stdout.
add-to-cache path key key
This action adds the string key to the cache specified by path. If key already exists in the cache, it is replaced.
remove-from-cache path key key
Remove the string key from the cache path, if a matching key is present.
action name
This invokes another named action. A maximum of five actions may be called in a sequence.

RULES

Rules are specified using the match keyword. It has the following basic form:
match condition [and | or condition ...] [users] actions [continue]

The condition argument may be one of:

all
Matches all mail.
matched
Matches only mail that has matched a previous rule and been passed on with continue.
unmatched
The opposite of matched: matches only mails which have matched no previous rules.
account name | accounts { name ... }
Matches only mail fetched from the named account or accounts. The account names may include shell glob wildcards to match multiple accounts, as with the -a and -x command line options.
tagged string
Matches mails tagged with string.
[case] regexp [in headers | in body]
Specifies a regexp against which each mail should be matched. The regexp matches may be restricted to either the headers or body of the message by specifying either in headers or in body. The case keyword forces the regexp to be matched case-sensitively: the default is case-insensitive matching.
exec command [user user] returns (return code, stdout regexp)
pipe command [user user] returns (return code, [case] stdout regexp)
These two conditions execute a command and test its return value and output. The return code argument is the numeric return code expected and stdout regexp is a regexp to be tested against the output of the command to stdout. Either of these two arguments may be omitted: if both are specified, both must match for the condition to be true. The pipe version will pipe the mail to the command's stdin when executing it. If a user is specified, fdm(1) will change to that user before executing the command, otherwise the current user (or root if started as root) is used.
size < number
size > number
Compare the mail size with number.
string string to [case] regexp
Match string against regexp.
age < time
age > time
The age condition examines the mail's date header to determine its age, and matches if the mail is older (>) or newer (<) than the time specified. The time may be given as a simple number in seconds, or followed by the word seconds, hours, minutes, days, months or years to specify a time in different units.
in-cache path key key
This condition evaluates to true if the string key is in the cache at path.
attachment count < number
attachment count > number
attachment count == number
attachment count != number
These conditions match if the mail possesses a number of attachments less than, greater than, equal to or not equal to number.
attachment total-size < size
attachment total-size > size
Matches if the total size of all attachments is smaller or larger than size.
attachment any-size < size
attachment any-size > size
Compare each individual attachment on a mail to size and match if any of them are smaller or larger.
attachment any-type string
attachment any-name string
Match true if any of a mail's attachments possesses a MIME type or filename that matches string. fnmatch(3) wildcards may be used.

Multiple conditions may be chained together using the and or or keywords. The conditions are tested from left to right. Any condition may be prefixed by the not keyword to invert it.

The optional users argument to the first form has the same syntax as for an action definition. A rule's user list overrides any users given as part of the actions.

The actions list specifies the actions to perform when the rule matches a mail. It is either of a similar form:

action name | actions { name ... }

Or may specify a number of actions inline (lambda actions):

action action
action { action ... }

In the latter case, action follows the same form as described in the ACTIONS section. The actions are performed from first to last in the order they are specified in the rule definition.

If the continue keyword is present, evaluation will not stop if this rule is matched. Instead, fdm(1) will continue to match further rules after performing any actions for this rule.

NESTED RULES

Rules may be nested by specifying further rules in braces:
match condition [and | or condition ...] {
match ...
}

The inner rules will not be evaluated unless the outer one matches. Rules may be multiply nested. Note that the outer rule does not count as a match for the purposes of the matched and unmatched conditions.

FILES

~/.fdm.conf
default fdm.conf configuration file
/etc/fdm.conf
default system-wide configuration file
~/.fdm.lock
default lock file
/var/db/fdm.lock
lock file for root user

SEE ALSO

fdm(1), re_format(7)

AUTHORS

Nicholas Marriott <nicm@users.sourceforge.net>
August 21, 2006 OpenBSD 5.3