Provided by: watchdog_5.14-3ubuntu0.16.04.1_amd64 bug

NAME

       watchdog - a software watchdog daemon

SYNOPSIS

       watchdog  [-F|--foreground]  [-f|--force] [-c filename|--config-file filename] [-v|--verbose] [-s|--sync]
       [-b|--softboot] [-q|--no-action]

DESCRIPTION

       The Linux kernel can reset the system if serious problems are detected.   This  can  be  implemented  via
       special  watchdog  hardware,  or  via  a slightly less reliable software-only watchdog inside the kernel.
       Either way, there needs to be a daemon that tells the kernel the system is working fine.  If  the  daemon
       stops doing that, the system is reset.

       watchdog  is  such  a  daemon.  It  opens /dev/watchdog, and keeps writing to it often enough to keep the
       kernel from resetting, at least once per minute. Each write delays the reboot time another minute.  After
       a  minute  of inactivity the watchdog hardware will cause the reset. In the case of the software watchdog
       the ability to reboot will depend on the state of the machines and interrupts.

       The watchdog daemon can be stopped without causing  a  reboot  if  the  device  /dev/watchdog  is  closed
       correctly, unless your kernel is compiled with the CONFIG_WATCHDOG_NOWAYOUT option enabled.

TESTS

       The watchdog daemon does several tests to check the system status:

       •  Is the process table full?

       •  Is there enough free memory?

       •  Is there enough allocatable memory?

       •  Are some files accessible?

       •  Have some files changed within a given interval?

       •  Is the average work load too high?

       •  Has a file table overflow occurred?

       •  Is a process still running? The process is specified by a pid file.

       •  Do some IP addresses answer to ping?

       •  Do network interfaces receive traffic?

       •  Is the temperature too high? (Temperature data not always available.)

       •  Execute a user defined command to do arbitrary tests.

       •  Execute one or more test/repair commands found in /etc/watchdog.d.  These commands are called with the
          argument test or repair.

       If any of these checks fail watchdog will cause a shutdown. Should any of these  tests  except  the  user
       defined binary last longer than one minute the machine will be rebooted, too.

OPTIONS

       Available command line options are the following:

       -v, --verbose
              Set  verbose  mode.  Only  implemented  if  compiled  with SYSLOG feature. This mode will log each
              several infos in LOG_DAEMON with priority LOG_INFO.  This is useful if you  want  to  see  exactly
              what  happened  until  the  watchdog  rebooted  the  system. Currently it logs the temperature (if
              available), the load average, the change date of the files it checks and  how  often  it  went  to
              sleep.

       -s, --sync
              Try  to  synchronize  the  filesystem  every  time  the  process is awake. Note that the system is
              rebooted if for any reason the synchronizing lasts longer than a minute.

       -b, --softboot
              Soft-boot the system if an error occurs during the  main  loop,  e.g.  if  a  given  file  is  not
              accessible via the stat(2) call. Note that this does not apply to the opening of /dev/watchdog and
              /proc/loadavg, which are opened before the main loop starts.

       -F, --foreground
              Run in foreground mode, useful for running under systemd (for example).

       -f, --force
              Force the usage of the interval given or the maximal  load  average  given  in  the  config  file.
              Without this option these values are sanity checked.

       -c config-file, --config-file config-file
              Use config-file as the configuration file instead of the default /etc/watchdog.conf.

       -q, --no-action
              Do  not  reboot or halt the machine. This is for testing purposes. All checks are executed and the
              results are logged as usual, but no action is taken.   Also  your  hardware  card  or  the  kernel
              software watchdog driver is not enabled. Temperature checking is also disabled since this triggers
              the hardware watchdog on some cards.

FUNCTION

       After watchdog starts, it puts itself into the background and then tries  all  checks  specified  in  its
       configuration file in turn. Between each two tests it will write to the kernel device to prevent a reset.
       After finishing all tests watchdog goes to sleep for some time. The kernel drivers expects a write to the
       watchdog  device  every  minute. Otherwise the system will be reset.  watchdog will sleep for a configure
       interval that defaults to 1 second to make sure it triggers the device early enough.

       Under high system load watchdog might be swapped out of memory and may fail to make it back in  in  time.
       Under these circumstances the Linux kernel will reset the machine. To make sure you won't get unnecessary
       reboots make sure you have the variable realtime set to yes  in  the  configuration  file  watchdog.conf.
       This  adds real time support to watchdog: it will lock itself into memory and there should  be no problem
       even under the highest of loads.

       On system running out of memory the kernel will try  to  free  enough  memory  by  killing  process.  The
       watchdog daemon itself is exempted from this so-called out-of-memory killer.

       Also  you  can  specify  a  maximal allowed load average. Once this load average is reached the system is
       rebooted. You may specify maximal load averages for 1 minute, 5 minutes or 15 minutes. The default values
       is  to  disable  this  test.  Be  careful not to set this parameter too low. To set a value less then the
       predefined minimal value of 2, you have to use the -f option.

       You can also specify a minimal amount of virtual memory you want to have available as free.  As  soon  as
       more  virtual  memory  is  used  action  is  taken  by  watchdog.   Note, however, that watchdog does not
       distinguish between different types of memory usage. It just checks for free virtual memory.

       If you have a watchdog card with temperature sensor you can specify the maximal allowed temperature. Once
       this  temperature  is reached the system is halted. The default value is 120. There is no unit conversion
       so make sure you use the same unit as your hardware.  watchdog will issue warnings once  the  temperature
       increases 90%, 95% and 98% of this temperature.

       When using file mode watchdog will try to stat(2) the given files. Errors returned by stat will not cause
       a reboot. For a reboot the stat call has to last at least one minute.  This may happen  if  the  file  is
       located  on  an  NFS mounted filesystem. If your system relies on an NFS mounted filesystem you might try
       this option.  However, in such a case the sync option may not work if the NFS server is not answering.

       watchdog can read the pid from a pid file and see whether the process still exists.  If  not,  action  is
       taken by watchdog.  So you can for instance restart the server from your repair-binary.

       watchdog will try periodically to fork itself to see whether the process table is full. This process will
       leave a zombie process until watchdog wakes up again and catches it; this is harmless, don't worry  about
       it.

       In  ping  mode  watchdog tries to ping the given IP addresses. These addresses do not have to be a single
       machine. It is possible to ping to a broadcast address instead to see if at least one machine in a subnet
       is still living.

       Do  not  use  this  broadcast ping unless your MIS person a) knows about it and b) has given you explicit
       permission to use it!

       watchdog will send out three ping packages  and  wait  up  to  <interval>  seconds  for  the  reply  with
       <interval>  being  the  time  it  goes  to sleep between two times triggering the watchdog device. Thus a
       unreachable network will not cause a hard reset but a soft reboot.

       You can also test passively for an unreachable network by just monitoring a given interface for  traffic.
       If  no  traffic  arrives  the  network is considered unreachable causing a soft reboot or action from the
       repair binary.

       watchdog can run an external command for user-defined tests. A return code not equal  0  means  an  error
       occurred  and  watchdog  should  react.  If  the external command is killed by an uncaught signal this is
       considered an error by watchdog too.  The command may take longer than the time  slice  defined  for  the
       kernel  device  without a problem. However, error messages are generated into the syslog facility. If you
       have enabled softboot on error the machine will be rebooted if the binary doesn't exit in half  the  time
       watchdog sleeps between two tries triggering the kernel device.

       If  you specify a repair binary it will be started instead of shutting down the system. If this binary is
       not able to fix the problem watchdog will still cause a reboot afterwards.

       If the machine is halted an email is sent to notify a human that the machine is going down. Starting with
       version 4.4 watchdog will also notify the human in charge if the machine is rebooted.

SOFT REBOOT

       A  soft  reboot  (i.e.  controlled shutdown and reboot) is initiated for every error that is found. Since
       there might be no more processes available, watchdog does it all by himself. That means:

       1.  Kill all processes with SIGTERM.

       2.  After a short pause kill all remaining processes with SIGKILL.

       3.  Record a shutdown entry in wtmp.

       4.  Save the random seed from /dev/urandom.  If the device is non-existant or there is  no  filename  for
           saving this step is skipped.

       5.  Turn off accounting.

       6.  Turn off quota and swap.

       7.  Unmount all partitions except the root partition.

       8.  Remount the root partition read-only.

       9.  Shut down all network interfaces.

       10. Finally reboot.

CHECK BINARY

       If  the  return code of the check binary is not zero watchdog will assume an error and reboot the system.
       Be careful with this if you are using the real-time properties of watchdog since watchdog will  wait  for
       the  return  of  this binary before proceeding. An exit code smaller than 245 is interpreted as an system
       error code (see errno.h for details). Values of 245 or larger than are special to watchdog:

       255 (based on -1 as unsigned 8-bit number)
              Reboot the system. This is not exactly an error message but a command to watchdog.  If the  return
              code is this the watchdog will not try to run a shutdown script instead.

       254    Reset  the  system. This is not exactly an error message but a command to watchdog.  If the return
              code is this the watchdog will simply refuse to write the kernel device again.

       253    Maximum load average exceeded.

       252    The temperature inside is too high.

       251    /proc/loadavg contains no (or not enough) data.

       250    The given file was not changed in the given interval.

       249    /proc/meminfo contains invalid data.

       248    Child process was killed by a signal.

       247    Child process did not return in time.

       246    Free for personal watchdog-specific use (was -10 as an unsigned 8-bit number).

       245    Reserved for an unknown result, for example a slow  background  test  that  is  still  running  so
              neither a success nor an error.

REPAIR BINARY

       The  repair  binary  is started with one parameter: the error number that caused watchdog to initiate the
       boot process. After trying to repair the system  the  binary  should  exit  with  0  if  the  system  was
       successfully  repaired  and  thus  there  is  no  need  to boot anymore. A return value not equal 0 tells
       watchdog to reboot. The return code of the repair binary should be the error number of the error  causing
       watchdog  to  reboot.  Be careful with this if you are using the real-time properties since watchdog will
       wait for the return of this binary before proceeding.

TEST DIRECTORY

       Executables placed in the test directory are discovered by watchdog  on  startup  and  are  automatically
       executed.  They are bounded time-wise by the test-timeout directive in watchdog.conf.

       These  executables  are called with either "test" as the first argument (if a test is being performed) or
       "repair" as the first argument (if a  repair  for  a  previously-failed  "test"  operation  on  is  being
       performed).

       The  as  with  test  binaries  and  repair  binaries, expected exit codes for a successful test or repair
       operation is always zero.

       If an executable's test operation fails, the same executable is automatically called  with  the  "repair"
       argument as well as the return code of the previously-failed test operation.

       For example, if the following execution returns 42:

           /etc/watchdog.d/my-test test

       The watchdog daemon will attempt to repair the problem by calling:

           /etc/watchdog.d/my-test repair 42

       This  enables administrators and application developers to make intelligent test/repair commands.  If the
       "repair" operation is not required (or is not likely to succeed), it is important that the author of  the
       command return a non-zero value so the machine will still reboot as expected.

       Note  that  the  watchdog daemon may interpret and act upon any of the reserved return codes noted in the
       Check Binary section prior to calling a given command in "repair" mode.

BUGS

       None known so far.

AUTHORS

       The original code is an example written by Alan Cox <alan@lxorguk.ukuu.org.uk>, the author of the  kernel
       driver.   All   additions   were   written   by   Michael   Meskes   <meskes@debian.org>.  Johnie  Ingram
       <johnie@netgod.net> had the idea of testing the load average. He also took over the Debian specific work.
       Dave  Cinege  <dcinege@psychosis.com>  brought  up  some hardware watchdog issues and helped testing this
       stuff.

FILES

       /dev/watchdog
              The watchdog device.

       /var/run/watchdog.pid
              The pid file of the running watchdog.

SEE ALSO

       watchdog.conf(5)