Provided by: lire_2.1.1-2.1_all bug

NAME

       postfix2dlf - convert postfix logfiles to dlf format

SYNOPSIS

       postfix2dlf

DESCRIPTION

       postfix2dlf converts a postfix logfile to a Lire email Distilled Log Format file.  It
       calls postfix2dlf_pre(1) and postfix2dlf_main(1) to do the real job.  It expects the
       logfile on stdin, and prints the DLF to stdout.  Diagnostics are printed to stderr.  This
       script is called by lr_log2xml(1).

RATIONALE

       The postfix2dlf architecture is quite different from other email dlf convertors.  This is
       because when parsing a postfix logfile, there is no sign indicating wether a message has
       been delivered to all addressees.  (In a sendmail log, there is a rcpts= field, which can
       be used for this purpose.)

       There are several workarounds to deal with such a log.  1: keep track of all queueid's
       along with their from- info during the entire logfile processing.  This is too memory
       hungry.  2: do some 10-seconds heuristic: assume a message will never be longer in the
       queue than e.g. 10 seconds.  Or, alternatively, assume no more than
       $LR_POSTFIX_MAX_QUEUE_SIZE messages will be in the queue concurrently.  This is too
       errorprone.  3: Use sort(1) to sort the loglines on queueid.  This is what we used to do.
       However, we don't know beforehand which field will store the queueid (logfiles processed
       by Sun Solaris syslog will have their queueid on another position, e.g.)  Calculating this
       position would mean parsing the log twice.

       We've choosen a variation of the third alternative: we store the information which we
       would've gotten by sort(1) in a db file.  While building this db file, , which holds a map
       from queueids to nof-lines-with-this-id, we do as much preprocessing as we can.  The
       preprocessed log is printed to a tmpfile.  The structure of the preprocessed log is the
       same as the raw log: one raw logline gives one preprocessed line.  The preprocessed log is
       converted to dlf by lr_postfix2dlf_main(1), using the information in the db file.

       A nice sideeffect of this way of processing is: the log is processed in the original time-
       sorted order.

EXAMPLE

       A logfile

        Dec 1 04:02:56 internetsrv postfix/pickup[20919]:
         693A3578E: uid=0 from=<root>
        Dec 1 04:02:56 internetsrv postfix/cleanup[20921]:
         693A3578E: message-id=<john.doe.1@example.com>
        Dec 1 04:02:57 internetsrv postfix/qmgr[20164]: 693A3578E:
         from=<john.doe.2@example.com>, size=617 (queue active)
        Dec 1 04:02:57 internetsrv postfix/cleanup[20921]:
         E325C578D: message-id=<john.doe.1@example.com>
        Dec 1 04:02:58 internetsrv postfix/local[20924]:
         693A3578E: to=<john.doe.2@example.com>, relay=local,
         delay=3, status=sent (forwarded as E325C578D)
        Dec 1 04:02:58 internetsrv postfix/qmgr[20164]: E325C578D:
         from=<john.doe.2@example.com>, size=769 (queue active)
        Dec 1 04:02:59 internetsrv postfix/smtp[20925]: E325C578D:
         to=<john.doe.3@example.com>,
         relay=1.example.com.vp.pt[10.0.0.1], delay=2, status=sent
         (250 Requested mail action Ok.)
        Dec 1 06:58:22 internetsrv postfix/smtpd[21142]: connect
         from 2.example.com.fi[10.0.0.2]
        Dec 1 06:58:23 internetsrv postfix/smtpd[21142]:
         42BFE578D: client=2.example.com.fi[10.0.0.2]
        Dec 1 06:58:24 internetsrv postfix/cleanup[21143]:
         42BFE578D: message-id=<john.doe.4@example.com>
        Dec 1 06:58:24 internetsrv postfix/qmgr[20164]: 42BFE578D:
         from=<john.doe.5@example.com>, size=2473 (queue active)
        Dec 1 06:58:26 internetsrv postfix/smtp[21145]: 42BFE578D:
         to=<john.doe.6@example.com>,
         relay=1.example.com.vp.pt[10.0.0.1], delay=3, status=sent
         (250 Requested mail action Ok.)
        Dec 1 06:59:22 internetsrv postfix/smtpd[21142]:
         disconnect from 2.example.com.fi[10.0.0.2]
        Dec 1 07:08:28 internetsrv postfix/smtpd[21160]: connect
         from 2.example.com.fi[10.0.0.2]
        Dec 1 07:08:28 internetsrv postfix/smtpd[21160]:
         C7B39578D: client=2.example.com.fi[10.0.0.2]
        Dec 1 07:08:29 internetsrv postfix/cleanup[21161]:
         C7B39578D: message-id=<john.doe.7@example.com>
        Dec 1 07:08:29 internetsrv postfix/qmgr[20164]: C7B39578D:
         from=<john.doe.8@example.com>, size=2173 (queue active)
        Dec 1 07:08:32 internetsrv postfix/smtp[21163]: C7B39578D:
         to=<john.doe.9@example.com>,
         relay=3.example.com.vp.pt[10.0.0.3], delay=4, status=sent
         (250 Requested mail action Ok.)
        Dec 1 07:08:33 internetsrv postfix/smtpd[21160]:
         disconnect from 2.example.com.fi[10.0.0.2]
        Dec 1 07:18:42 internetsrv postfix/smtpd[21166]: connect
         from 2.example.com.fi[10.0.0.2]

       will get converted to

        1007175776 internetsrv 693A3578E <john.doe.1@example.com>
         john.doe.2 example.com localhost 127.0.0.1 617 3 0
         john.doe.2 example.com localhost 127.0.0.1 sent
         (forwarded_as_e325c578d)
        1007175779 internetsrv E325C578D <john.doe.1@example.com>
         john.doe.2 example.com localhost 127.0.0.1 769 2 0
         john.doe.3 example.com 1.example.com.vp.pt 10.0.0.1 sent
         (250_requested_mail_action_ok.)
        1007186303 internetsrv 42BFE578D <john.doe.4@example.com>
         john.doe.5 example.com 2.example.com.fi 10.0.0.2 2473 3 0
         john.doe.6 example.com 1.example.com.vp.pt 10.0.0.1 sent
         (250_requested_mail_action_ok.)
        1007186908 internetsrv C7B39578D <john.doe.7@example.com>
         john.doe.8 example.com 2.example.com.fi 10.0.0.2 2173 4 0
         john.doe.9 example.com 3.example.com.vp.pt 10.0.0.3 sent
         (250_requested_mail_action_ok.)

       postfix2dlf will be rarely used on its own, but is more likely called by lr_log2report:

        $ lr_log2report postfix < /var/log/mail.log > report

       .  If you'd really like to run this script standalone (e.g. for debugging) run it as

        $ LR_SERVICE=email LR_ID=`date +%s` /path/to/lire/convertors/postfix2dlf < /var/log/mail.log > mail.dlf

       .  Be sure to have /path/to/libexec/lire and /path/to/lire/convertors in your PATH, and be
       sure to have TMPDIR, LR_DBFILE and LR_DBDIR set.  You could manually source
       /path/to/etc/lire/profile_lean and /path/to/etc/lire/defaults to achieve this.

LOGFORMAT

       Postfix logs look like this:

   from local to remote
        postfix/pickup[81586]: 094BE204: uid=1001 from=<edwin>
        postfix/cleanup[81683]: 094BE204:
         message-id=<20000531080729.L39824@cgmd76206.c.nl>
        postfix/qmgr[13460]: 094BE204:
         from=<edwin@cgmd76206.c.nl>, size=1717 (queue active)
        postfix/smtp[81685]: 094BE204: to=<r.moeskops@c.nl>,
         relay=smtp.c.nl[212.83.68.146], delay=4, status=sent (250
         Message received:
         20000531060722.ZCOV13476.relay02@cgmd76206.c.nl)

   from local to local
        postfix/pickup[81849]: 473B9204: uid=1001 from=<edwin>
        postfix/cleanup[81916]: 473B9204:
         message-id=<200005310901.LAA56567@kludge.mpn.cp.p.com>
        postfix/qmgr[13460]: 473B9204:
         from=<edwin@cgmd76206.c.nl>, size=1997 (queue active)
        postfix/local[81918]: 473B9204: to=<edwin@cgmd76206.c.nl>,
         relay=local, delay=0, status=sent
         ("|exec /usr/local/bin/procmail -t")

   from remote to local
        postfix/smtpd[82056]: A17131C5:
         client=gw-nl1.o-it.com[193.79.128.34]
        postfix/cleanup[82057]: A17131C5:
         message-id=<023201bfcad1$16365ba0$775910ac@ehvbos.nl.oit.com>
        postfix/qmgr[13460]: A17131C5:
         from=<Jan.Stap@nl.o-it.com>, size=1692 (queue active)
        postfix/local[82059]: A17131C5:
         to=<majordomo-org@cgmd76206.c.nl>, relay=local, delay=1,
         status=sent ("|/usr/local/majordomo/wrapper majordomo")

   from remote to remote
        postfix/smtpd[58567]: connect from
         kweetal.t.nl[131.155.2.7]
        postfix/smtpd[58567]: 9A16E229:
         client=kweetal.t.nl[131.155.2.7]
        postfix/cleanup[58570]: 9A16E229:
         message-id=<200006041246.OAA23888@bw2.baub.bwk.t.nl>
        postfix/qmgr[236]: 9A16E229:
         from=<edwin@bw2.baub.bwk.t.nl>, size=774 (queue active)
        postfix/smtpd[58567]: disconnect from
         kweetal.t.nl[131.155.2.7]
        postfix/smtp[58574]: 9A16E229: to=<joostvb@x.nl>,
         relay=mx3.x.nl[194.109.6.48], delay=15, status=sent (250
         OAA23290 Message accepted for delivery)

PERFORMANCE

       We ran the postfix2dlf suite on a 24MB postfix logfile.  Results were:

        postfix2dlf_pre info read 195257 lines; output 177027 DLF lines; 3 errors
        postfix2dlf_pre info memory stats: vsize=5900K rss=4508K majflt=430
        postfix2dlf_pre info elapsed time in seconds real=626 user=449.16 system=8.7

        postfix2dlf_main info read 177027 lines; output 61671 DLF lines; 0 errors
        postfix2dlf_main info memory stats: vsize=5976K rss=4656K majflt=427
        postfix2dlf_main info elapsed time in seconds real=245 user=152.05 system=4.15

       .

EXAMPLES

       postfix2dlf will be rarely used on its own, but is more likely called by lr_log2report:

        $ lr_run lr_log2report postfix < /var/log/maillog

BUGS

       This script needs a lot of space in TMPDIR: it creates a tmpfile which is about the same
       in size as the raw log it's being fed.

       Occasionally, postfix reuses its queueids very fast.  We can't cope with this.

THANKS

       Brad Knowles, for supplying patches.  Emanuele "luca" for pointing out the lmtp delivery.

SEE ALSO

       postfix2dlf_main(1), postfix2dlf_pre(1) and the other email dlf convertors:
       argomail2dlf(1), exim2dlf(1), nms2dlf(1), qmail2dlf(1), sendmail2dlf(1); the caller
       lr_log2xml(1).

VERSION

       $Id: postfix2dlf.in,v 1.40 2006/07/23 13:16:34 vanbaal Exp $

COPYRIGHT

       Copyright (C) 2000, 2001, 2002 Stichting LogReport Foundation LogReport@LogReport.org

       This program is part of Lire.

       Lire is free software; you can redistribute it and/or modify it under the terms of the GNU
       General Public License as published by the Free Software Foundation; either version 2 of
       the License, or (at your option) any later version.

       This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
       without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
       See the GNU General Public License for more details.

       You should have received a copy of the GNU General Public License along with this program
       (see COPYING); if not, check with http://www.gnu.org/copyleft/gpl.html.

AUTHOR

       Joost van Baal, embrionic version by Edwin Groothuis.