Provided by: tenshi_0.13-7_all bug


       tenshi - Log Monitoring and Reporting tool


       tenshi [ -c <conf file> ] [ -C ] [ -d <debug level> ] [ -f ] [ -h ] [ -p ] [ -P <pid file>


       tenshi is a log monitoring program, designed to watch one or  more  log  files  for  lines
       matching  user  defined  regular  expressions  and  report  on  the  matches.  The regular
       expressions are assigned to queues which have  an  alert  interval  and  a  list  of  mail

       Queues can be set to send a notification as soon as there is a log line assigned to it, or
       to send periodic reports.

       Additionally, uninteresting fields in the log lines (such as PID numbers)  can  be  masked
       with  the standard regular expression grouping operators ( ). This allows cleaner and more
       readable reports. All reports are separated by hostname and  all  messages  are  condensed
       when possible.

       The  program  reads  a  configuration  file  (tenshi.conf)  and  then  forks  a daemon for
       monitoring the specified log files.


       -c <conf file>
              Read configuration from file. The default file is /etc/tenshi/tenshi.conf .

       -C     Perform a syntax check of the configuration file. The program exits  after  parsing
              the configuration with either a return code of 0 or an error.

       -d <debug level>
              Enable  debug  messages.  Default  level is 1 if none is specified, level 2 enables
              SMTP connection debug messages. In this  mode  the  main  process  remains  in  the

       -f     Enable foreground mode. In this mode the main process operates normally but remains
              in the foreground, this is needed for some process supervisors.

       -p     Enable profiling mode. In this mode the main process remains in the foreground  and
              expects  log  lines  to be fed to standard in. When it receives an EOF it will stop
              processing. No alerts will be sent in this mode, it is used  solely  for  measuring
              tenshi's line processing speed. For example: time $(cat /var/log/messages|tenshi -p
              > /dev/null)

       -P <pid file>
              Define the file containing the PID of the process,  this  overrides  any  'pidfile'
              option present in the configuration file.


       All  directives are shown with the standard default value where applicable, if omitted the
       default value will be used.


       All configuration directives can be optionally split into  different  configuration  files
       and then read with the two following statements.

       include <configuration file>
              Parse the specified configuration file.

       includedir <directory>
              Parse all files inside <directory>. The files will be parsed in alphabetical order,
              keep in mind that regexps order matters so includedir should be used carefully, see
              REGEXP DEFINITIONS for details.


       These  options  will  be  read the first time tenshi reads its config file. They cannot be
       changed by re-reading the config file. If you change one of these options and  HUP  tenshi
       it will die. You have been warned.

       set uid tenshi
              Specify  the effective user ID of the process when in daemon mode. The user must be
              able to read the selected log files, the configuration file and write the specified
              pid  file.  Never  use  privileged users here since it's not usually necessary (log
              files perms can be set accordingly with most syslog implementations).

       set gid tenshi
              Specify the effective group ID of the process when in daemon mode.

       set pidfile /var/run/
              The file containing the PID of the process, useful for start/stop scripts.

       set logfile <log file path>
              A log file to monitor, this may be specified multiple times to watch more than  one
              log  file.  Depending  on  your  tail  implementation  you  might  need  to use the
              tail_multiple setting for multiple files to work. This mode can be used along  with
              fifo and listen settings.

       set tail /usr/bin/tail -q --follow=name --retry -n 0
              Specify  the path and arguments for the tail binary used for reading the log files.
              The invocation must be tuned against your current  'tail'  implementation.  Default
              values  are  configured  for  standard  GNU  coreutils  tail. The --follow=name and
              --retry  flags  should  deal  properly  with  log  rotation,  if  missing  on  your
              implementation  we  suggest that you use something like 'cp /dev/null logfile' as a
              safe way for clearing the log file upon rotation.

       set tail_multiple <on|off>
              Some tail implementations do not handle more than one log file. When this option is
              enabled  multiple  tail  commands  will be forked, instead of a single command with
              multiple arguments. This option is disabled by default.

       set fifo <fifo path>
              A FIFO file  to  monitor.  This  option  allows  you  to  use  a  syslog-ng  pipe()
              destination  (or  any other syslog implementation that allows FIFO usage). This may
              be specified multiple times to watch more than one fifo file. This option is  meant
              to  be  used  only  when  the  installed 'tail' binary doesn't behave properly with
              FIFOs, please test your tail implementation before using this.  This  mode  can  be
              used along with logfile and listen settings.

       set listen
              Enables  syslog  server  mode.  With  this option tenshi will bind to the specified
              address:port pair and  read  messages  acting  like  a  syslog  server.  We  always
              recommend  to  filter  the port accordingly and possibly use something like stunnel
              for encrypting the traffic. This mode can be  used  along  with  logfile  and  fifo


       These options are set each time the config file is read. tenshi reads its config file once
       on start-up and whenever it receives a HUP.

       set sleep 5
              The loop sleep time for the notification process. The value must be <= 60 seconds.

       set limit <number of lines>
              The maximum number of messages per hostname allowed in a report,  any  lines  after
              the  maximum will be omitted and a warning included. If this option is omitted then
              no limit is applied.

       set pager_limit <number of lines>
              The maximum number of messages per hostname allowed in pager friendly reports,  any
              lines after the maximum will be omitted. If this option is omitted then no limit is

       set logprefix <regexp>
              All valid syslog  messages  are  parsed  by  default,  while  non-syslog  ones  are
              discarded  unless  the  special  noprefix  queue  is set. This option allows one to
              define an additional valid prefix for watching other type of logs. If the regexp is
              matched  then  the  prefix  is removed from the log and the first grouped string is
              used for the hostname field. This may be specified multiple  times  to  watch  many
              different non-syslog logs.

       set mask ______
              The  mask  for  strings  enclosed  by  the  grouping  operators ( ). See the REGEXP
              DEFINITIONS section. 'set mask' on its own will set the mask to an empty string.

       set mailserver localhost
              The mail server to be contacted for sending out reports.

       set mailtimeout 10
              The timeout in seconds for mail server reply.

       set subject tenshi report
              The subject of report emails, the queue name is always automatically appended.

       set hidepid <on|off>
              This option turns on automatic stripping of 'foo[1234]:' style PID strings from the
              start  of  log  lines  i.e.  'foo[1234]:'  becomes 'foo:'. This allows you to write
              regexps without worrying about masking the PID. Bear in  mind  that  any  time  you
              change  this  option  you  will  need to re-write your regex rules or they will not
              work. This option is disabled by default.

       set filter <queue> <filter path> <arguments>
              When this option is enabled all reports matching the specified queue will be passed
              as  STDIN to the specified filter, the resulting output is sent via smtp instead of
              the original report. The full path of the filter application must be specified.

       set csv <cron_spec> <filter path> <arguments>
              This feature allows periodic reporting, using a five-field cron-style specification
              like  the set queue option, to the specified filter. The output is pre-formatted as
              CSV (Comma Separated Values) with hostname,log,hits format. This feature was  coded
              for  using  AfterGlow  (  as  a  filter and graphing tenshi
              output. Check the FAQ for sample usage.

       set sort_order <descending|ascending>
              The sorting order for reports. It can be either descending or ascending, the number
              of  messages  is  used  as a key for sorting the log messages. The default order is

       set resolve <on|off>
              This option turns on resolution of the fully qualified domain name for the hostname
              passed  along  with  log messages and, if found, reports it along with the original
              one. This only affects reports and not  pager  messages.  The  name  resolution  is
              cached  in  order  to avoid re-resolving addresses that have been seen already, you
              have to restart or HUP tenshi in order to flush the cache. This option is  disabled
              by default.

       set threshold <queue> <count> <regex>
              This  option can be used to discard lines from a report that have a count below the
              given threshold. If a line matches the regex in the given queue but has fewer  hits
              than  count, it is discarded and omitted from the report. Note that this matches on
              the content of the lines that will actually appear in the report,  in  contrast  to
              queue escalation which uses a count based on the regex that is matched.


       All  messages  are  assigned to queues. Every queue is processed periodically according to
       its notification interval. There are four default builtin queues, trash to which  unwanted
       messages can be assigned (think /dev/null), repeat which is used for smart repeat messages
       handling, group and group_host , see  REGEXP  DEFINITIONS  for  details.  There's  also  a
       special noprefix queue, read further for details about it.

       All  queues  are  automatically flushed before shutdown when a SIGTERM is received. Please
       see section SIGNALS for additional information.

       The syntax is the following:

       set queue <queue_name> <mail_from> [pager:]<mail_to> <cron_spec> [<subject>]

              The queue name. Can be any alphanumeric character string  except  for  the  builtin
              queues name.

              The mail sender for reports related to the queue.

              The  mail  recipient(s)  for  reports related to the queue. Multiple address can be
              specified, separated by commas. Using the pager: prefix enables  a  pager  friendly

              This  is  a  five-field  cron-style  specification  for  when the reports should be
              emailed. Ranges and skip values are supported as per the de  facto  crontab  syntax
              with  a few exceptions. Please see crontab man page for crontab syntax explanation.
              The supported day names are: Mon, Tue, Wed, Thu, Fri, Sat, Sun. Monday is 1, Sunday
              0  or  7.  Supported  month names are: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep,
              Oct, Nov, Dec. Day and Month names are not case sensitive. Additionally, 'now'  can
              be specified for immediate notifications.

              This  is  the  subject  for  to use for email reports regarding this queue. If this
              isn't specified then the default subject will be used.

       The special noprefix queue can  be  used  and  defined  like  any  other  queue  with  the
       difference that it will get all messages that don't match any configured prefix.

       set queue report tenshi@localhost sysadmin@localhost [0 9-17 * * *]
       set queue report tenshi@localhost sysadmin@localhost [30 18 * * *]
       set queue report tenshi@localhost sysadmin@localhost [*/10 * * * *]
       set   queue  critical  tenshi@localhost  sysadmin@localhost,noc@localhost  [now]  CRITICAL
       WARNING -
       set queue pager tenshi@localhost  pager:sysadmin_pager@localhost,pager:noc_pager@localhost
       [now] ALERT


       All  valid  syslog  messages  are  matched  against standard perl regexps, all regexps are
       defined with the following syntax:

       <queue_name>[,<queue_name>[:<escalation_number>]..] <regexp>

       The regexps are evaluated in order so  a  matched  message  is  not  checked  against  the
       subsequent  regexps.  Keep  this  in  mind  when  assembling  the configuration file. It's
       advisable to catch all messages by placing an all  matching  regexp  at  the  end  of  the
       configuration  file.  It's  also  good  for  performance  having trash rules not logically
       connected with other matching rules at the beginning of the section. Multiple  queues  can
       be  defined  with  a  comma  separated list, builtin queues cannot be used when using this

       If an escalation number is provided for a queue, the matched message will only  be  placed
       into  the  queue when <escalation_number> messages have matched the regexp. The queue will
       receive the message that matched the regexp at the time of escalation, with a count  equal
       to the escalation number. The count of messages matching the regexp will be reset when the
       left most queue mentioned in the queue list is mailed. The left most queue cannot have  an
       escalation  number  unless  it  is the only queue listed. When the number of messages that
       match the regexp reaches the greatest escalation number mentioned, escalation  will  begin
       again  into  the  escalation  queues, modulus the greatest escalation number. For example,
       using the queues `a,b:10,c:50', when 10 messages match the regexp, a message will go  into
       b,  when  50 match, one will go into c. At 60, another will go into b, and at 100, another
       into c, 110 to b, 150 to c, and so  on.  Escalation  numbers  must  be  positive  integers
       greater  than  zero  and must be listed in increasing order from left to right. All queues
       without escalation numbers must be listed  more  left  than  the  queues  with  escalation

       The  standard  grouping  operators ( ) can be used for string masking, literal "(" and ")"
       can be protected with the standard quotation operator "\". There's a lot of  documentation
       about regular expressions, a good start could be perl perlre and perlretut manual pages.
       You  can  also  use  the (?: ) operators to use groups without masking. This allows you to
       match, for example, output from several programs in a similar format.  There is an example
       of this below (the sudo/su line).

       The  builtin  queue  repeat  can  be used for special handling of "last message repeated x
       times" style log lines. When the assigned regexps are matched the line count for the  last
       line  received from the same host is incremented by the first grouped string. Keep in mind
       that it is possible for syslog lines to be received from remote hosts  out  of  order.  If
       this happens you should not use this feature because tenshi will mis-report line counts.

       The  builtin  queue  group  can  be  used to group sets of regex together to speed up line
       matching. If a line fails to match a regex assigned to the group queue  then  tenshi  will
       skip  all  the  regex up until the next group_end statement. Nested groups are allowed. An
       example of this is included below.

       The builtin group_host queue can be used for selective hostname matching. Like  the  group
       queue  it  is  also  terminated with the group_end statement. All regex definitions within
       that group will only apply if the hostname associated to the log entries matches the regex
       passed to the group_host definition.

       The  regexps  below  assume  hidepid is turned on. If you have it turned off then you will
       need to add in \[(.+)\] to the regex following the program name to get them to work.
       For example: mail ^sendmail: (.+): to=(.+),(.+)delay=(.+) becomes: mail ^sendmail\[(.+)\]:
       (.+): to=(.+),(.+)delay=(.+)


       trash ^xinetd

       repeat ^(?:last message repeated|above message repeats) (\d+) time

       group ^sendmail:
       mail ^sendmail: (.+): to=(.+),(.+)delay=(.+)
       mail ^sendmail: (.+): to=(.+),(.+)relay=(.+),(.+)stat=Sent

       group_host mailserver1
       mail1 ^sendmail
       mail1 ^sendmail:.+
       critical,mail1 ^sendmail:.+SYSERR.+

       mail ^ipop3d: Login user=(.+)

       critical,report ^sshd: Illegal user

       general,urgent:200,critical:1000 ^sshd: Illegal user

       root ^sshd\(pam_unix\): session opened for user root by root\(uid=0\)

       report ^sshd: Accepted rsa for (.+) from (.+) port (.+)

       trash ^sshd

       critical ^(?:sudo|su):

       critical,pager ^Oops

       misc .*


       tenshi  can  handle  different  signals  sent to the process, here's the list of supported

       TERM   flush all queues and then exit

       INT    flush all queues and then exit

       USR1   flush any queues which have reached their notification interval

       USR2   force all queues to be flushed, even if they have not  reached  their  notification

       HUP    force  all  queues  to be flushed, even if they have not reached their notification
              interval, re-read the config file and continue as normal.

       WARNING: If you change a STATIC OPTION in the config file and send tenshi a  HUP  it  will
       die. You will need to restart tenshi for changes to STATIC OPTIONs to take effect.


       See the included tenshi.conf.


       tenshi needs a working 'tail' implementation when not using FIFO mode.

       It  also  requires  Net::SMTP module for mailing reports, which should be included in your
       perl installation, and IO::BufferedSelect. If you miss any of them you can  grab  them  at
       CPAN ( or using the CPAN shell (`perl -e shell -MCPAN`).


       Double  quotation characters present in your logs might break csv output (depending on how
       you pipe/process it in the filter) since there's no escape code (yet).

       Please report any bugs you find at <>


       tenshi is distributed under the terms of the following ISC-style license:

       Permission to use, copy, modify, and distribute this software  for  any  purpose  with  or
       without  fee  is  hereby  granted,  provided  that  the  above  copyright  notice and this
       permission notice appear in all copies.



       The tenshi project page is


       tenshi  was  once known as wasabi but the name was changed as we were informed that wasabi
       is a registered a trademark relating to another piece of software.


       It   should   be   noted   that   tenshi   was   initially   a   perl   rewrite   of   oak

       Friedl, Jeffrey E. F. Mastering Regular Expressions, 2nd Edition. O'Reilly


       Copyright 2004-2011 Andrea Barisani <>