NAME

       dnshistory  - processes various log file formats doing dns IP Address lookups. Store these
       pairs in a database for later retrieval.

SYNOPSIS

       dnshistory [OPTION]... [--file=FILE]

COPYRIGHT

       dnshistory is Copyright © 2004, 2005, 2006, 2007 Stephen McInerney

DESCRIPTION

       dnshistory currently processes Apache CLF  and  Combined  logs,  Squid  access  logs,  FTP
       xferlog files and iptables based logs.  The log format is auto-detected.

       dnshistory has five modes of operation:

       •      Do  Lookups.  The  default  mode. Given a web log file, dnshistory will perform DNS
              reverse lookups on each unique IP Address  and  store  the  results  in  a  history
              database.

       •      Do Translations. Given a raw web log file, dnshistory will make use of a previously
              created history database and send to STDOUT the same web  log  but  with  addresses
              replaced by the Fully Qualified Domain Name as previously looked up.

       •      Do Recombining. Given two web log files, one raw and one previously translated (eg.
              by using dnstran): Create a history database from the values in these separate  log
              files.

       •      Do Dump. Dump a given history database to STDOUT.

       •      Do Import. Import a previously dumped history database from a given file.

       •      Show  History. Given one or more IP Addresses on the command line, show the history
              of those addresses.

       The lookups make use of threads  for  maximum  speed,  and  use  the  standard  resolution
       libraries  on  a  system.  Thus  hosts  files, NIS, LDAP and other name resolution methods
       should work transparently.

       It is strongly recommended that for massive lookups a DNS server is  "nearby".  Preferably
       not a forwarding server.

       dnshistory can read .gz files. STDIN is assumed to not be gz encoded.

OPTIONS

       -L --dolookups
              The  default  mode.  Given a log file, either via STDIN or via '-f', do the lookups
              and store the results.

       -T --dotranslate
              Given a log file, either via STDIN or via '-f', lookup each  IP  Address  from  the
              history  database;  replace the IP Address with the FQDN and send the newly updated
              log line to STDOUT.

       -R --dorecombine=FILE
              Given a previously translated file (eg. via dnstran) via this option for the names,
              do  the lookups for a file given via STDIN or '-f' and store the results. This file
              can be gz encoded.

              The date/time of each stored entry is taken as being the actual time for the lookup
              stored in the recombine log file. This is probably incorrect, but "Good Enough".

       -D --dodump
              Dump the history database to STDOUT.

       -I --doimport=FILE
              Given  a previously dumped database, import that into a new database. Will fail and
              exit if the chosen database already exists.

       -S --showhistory
              Given one or more IP Addresses show their history. Address are the last item(s)  on
              the  command line.  Addresses with no as yet discovered FQDN will display 'NONAME'.
              The Date/Time displayed is formatted as YYYY-MM-DD:hh:mm:ss, vs the 'seconds  since
              epoch' for "--dodump"

       --logtype=LOGTYPE
              By  default  dnshistory  will  attempt  to autodetect what type of logfile is being
              processed. By using this option, the autodetection is overridden. The choices  are:
              auto, clf or www, squid, ftp or iptables.

       -c --cache=SIZE
              Set the size of the memory cache to use. Value is in Mb.  Default is 20Mb.

       -d --database=FILE
              Change the default database file to use to store stateful data.

       -f --file=FILE
              Web Log File to process. This file can be gz encoded.
              Will use STDIN if not set

       -h --help
              Help screen. Very brief.

       -l --maxlookups=NUMBER
              The  maximum  number of lookups to attempt. The default is 1. This has not shown to
              be at all useful in testing...

       -m --maxthreads=NUMBER
              How many name lookup threads to spawn off. The default is  100.  Setting  this  too
              high  can  do  evil  things  to  bandwidth  and  the  CPU  usage of any queried DNS
              server(s).

              If doing lots of DNS queries, setting this too high can have a very negative impact
              on the ability to successfully resolve anything.

       -t --timeout=VALUE
              The  time  in  seconds  before a stored DNS value is deemed "old". The default is 7
              days.

       -v --verbose
              Verboseness of a run. More v's will increase  the  level  of  verbosity,  up  to  a
              maximum of 5. All of the higher levels are only of value for debugging purposes.

       -V --version
              Display the version information and exit

       -w --wait
              Delay time between query retries within a single run

RESULTS

       At  verbose  level  1 (-v) some success/failure counters will be displayed. As well as any
       problematic log lines to STDERR.
       At verbose level 2 (-vv) lines that may  not  match  up  (eg.  Due  to  dnstran  modifying
       referrers or URL's) will be sent to STDERR.

EXAMPLES

       A  typical  run,  using  a  database  in  /tmp/ (/tmp/c.db), and a log file in the current
       directory (test.log). First, do the lookups:

           dnshistory -d /tmp/c.db -f test.log

       Then the translation run for input into, for example, a web log analyser:

           dnshistory -T -d /tmp/c.db -f test.log | webalizer ....

       Do three attempts on failed queries, with a 2 second delay between retries:

           dnshistory -l 3 -w 2 -d /tmp/c.db -f test.log

       Lookup and Display the history of three IP Addresses: 127.0.0.1,192.168.1.254,10.10.10.10

           dnshistory -S -d /tmp/c.db 127.0.0.1 192.168.1.254 10.10.10.10

       Import a previously dumped database via dnsdb.dump

           dnshistory -I dnsdb.dump -d /tmp/d.db

FILES

       /var/lib/dnshistory/dnshistory.db
           The default history database file.

BUGS

       •      Ignores IP Addresses located in the URL and Referrer fields.

AUTHOR

       Stephen McInerney <spm@stedee.id.au>