supervisord Documentation Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. It shares some of the same goals of …
The server piece of supervisor is named supervisord. It is responsible for starting child programs at its own invocation, responding to commands from clients, restarting crashed or exited subprocesseses, logging its subprocess stdout and stderr output, and generating and handling "events" corresponding to points in subprocess lifetimes.
The server process uses a configuration file. This is typically
located in /etc/supervisord.conf. This configuration file is a
"Windows-INI" style config file. It is important to keep this file
secure via proper filesystem permissions because it may contain unencrypted
usernames and passwords.
This section makes reference to a BINDIR when explaining how to run the supervisord and supervisorctl commands. This is the "bindir" directory that your Python installation has been configured with. For example, for an installation of Python installed via ./configure --prefix=/usr/local/py; make; make install, BINDIR would be /usr/local/py/bin. Python interpreters on different platforms use a different BINDIR. Look at the output of setup.py install if you can't figure out where yours is.
Before supervisord will do anything useful for you, you'll need to add at least one program section to its configuration. The program section will define a program that is run and managed when you invoke the supervisord command. To add a program, you'll need to edit the supervisord.conf file.
One of the simplest possible programs to run is the UNIX cat program. A program section that will run cat when the supervisord process starts up is shown below.
[program:foo] command=/bin/cat
This stanza may be cut and pasted into the supervisord.conf file. This is the simplest possible program configuration, because it only names a command. Program configuration sections have many other configuration options which aren't shown here. See programx_section for more information.
To start supervisord, run $BINDIR/supervisord. The resulting process will daemonize itself and detach from the terminal. It keeps an operations log at $CWD/supervisor.log by default.
You may start the supervisord executable in the foreground by passing the -n flag on its command line. This is useful to debug startup problems.
WARNING:
To change the set of programs controlled by supervisord, edit the supervisord.conf file and kill -HUP or otherwise restart the supervisord process. This file has several example program definitions.
The supervisord command accepts a number of command-line options. Each of these command line options overrides any equivalent value in the configuration file.
The supervisord program may be sent signals which cause it to perform certain actions while it's running.
You can send any of these signals to the single supervisord process id. This process id can be found in the file represented by the pidfile parameter in the [supervisord] section of the configuration file (by default it's $CWD/supervisord.pid).
SIGTERM
SIGINT
SIGQUIT
SIGHUP
SIGUSR2
The developers have done their best to assure that use of a supervisord process running as root cannot lead to unintended privilege escalation. But caveat emptor. Supervisor is not as paranoid as something like DJ Bernstein's daemontools, inasmuch as supervisord allows for arbitrary path specifications in its configuration file to which data may be written. Allowing arbitrary path selections can create vulnerabilities from symlink attacks. Be careful when specifying paths in your configuration. Ensure that the supervisord configuration file cannot be read from or written to by unprivileged users and that all files installed by the supervisor package have "sane" file permission protection settings. Additionally, ensure that your PYTHONPATH is sane and that all Python standard library files have adequate file permission protections.
If you are using a distribution-packaged version of Supervisor, it should already be integrated into the service management infrastructure of your distribution.
There are user-contributed scripts for various operating systems at: https://github.com/Supervisor/initscripts
There are some answers at Serverfault in case you get stuck: How to automatically start supervisord on Linux (Ubuntu)
supervisord's primary purpose is to create and manage processes based on data in its configuration file. It does this by creating subprocesses. Each subprocess spawned by supervisor is managed for the entirety of its lifetime by supervisord (supervisord is the parent process of each process it creates). When a child dies, supervisor is notified of its death via the SIGCHLD signal, and it performs the appropriate operation.
Programs meant to be run under supervisor should not daemonize themselves. Instead, they should run in the foreground. They should not detach from the terminal from which they are started.
The easiest way to tell if a program will run in the foreground is to run the command that invokes the program from a shell prompt. If it gives you control of the terminal back, but continues running, it's daemonizing itself and that will almost certainly be the wrong way to run it under supervisor. You want to run a command that essentially requires you to press Ctrl-C to get control of the terminal back. If it gives you a shell prompt back after running it without needing to press Ctrl-C, it's not useful under supervisor. All programs have options to be run in the foreground but there's no "standard way" to do it; you'll need to read the documentation for each program.
Below are configuration file examples that are known to start common programs in "foreground" mode under Supervisor.
Here are some "real world" program configuration examples:
[program:apache2] command=/path/to/httpd -c "ErrorLog /dev/stdout" -DFOREGROUND redirect_stderr=true
[program:zeo] command=/path/to/runzeo priority=1 [program:zope1] command=/path/to/instance/home/bin/runzope priority=2 redirect_stderr=true [program:zope2] command=/path/to/another/instance/home/bin/runzope priority=2 redirect_stderr=true
[program:postgres] command=/path/to/postmaster ; we use the "fast" shutdown signal SIGINT stopsignal=INT redirect_stderr=true
[program:slapd] command=/path/to/slapd -f /path/to/slapd.conf -h ldap://0.0.0.0:8888 redirect_stderr=true
Other examples of shell scripts that could be used to start services under supervisord can be found at http://thedjbway.b0llix.net/services.html. These examples are actually for daemontools but the premise is the same for supervisor.
Another collection of recipes for starting various programs in the foreground is available from http://smarden.org/runit/runscripts.html.
Some processes (like mysqld) ignore signals sent to the actual process which is spawned by supervisord. Instead, a "special" thread/process is created by these kinds of programs which is responsible for handling signals. This is problematic because supervisord can only kill a process which it creates itself. If a process created by supervisord creates its own child processes, supervisord cannot kill them.
Fortunately, these types of programs typically write a "pidfile" which contains the "special" process' PID, and is meant to be read and used in order to kill the process. As a workaround for this case, a special pidproxy program can handle startup of these kinds of processes. The pidproxy program is a small shim that starts a process, and upon the receipt of a signal, sends the signal to the pid provided in a pidfile. A sample configuration program entry for a pidproxy-enabled program is provided below.
[program:mysql] command=/path/to/pidproxy /path/to/pidfile /path/to/mysqld_safe
The pidproxy program is put into your configuration's $BINDIR when supervisor is installed (it is a "console script").
Subprocesses will inherit the environment of the shell used to start the supervisord program. Several environment variables will be set by supervisord itself in the child's environment also, including SUPERVISOR_ENABLED (a flag indicating the process is under supervisor control), SUPERVISOR_PROCESS_NAME (the config-file-specified process name for this process) and SUPERVISOR_GROUP_NAME (the config-file-specified process group name for the child process).
These environment variables may be overridden within the [supervisord] section config option named environment (applies to all subprocesses) or within the per- [program:x] section environment config option (applies only to the subprocess specified within the [program:x] section). These "environment" settings are additive. In other words, each subprocess' environment will consist of:
... added-to/overridden-by ...
... added-to/overridden-by ...
... added-to/overridden-by ...
No shell is executed by supervisord when it runs a subprocess, so environment variables such as USER, PATH, HOME, SHELL, LOGNAME, etc. are not changed from their defaults or otherwise reassigned. This is particularly important to note when you are running a program from a supervisord run as root with a user= stanza in the configuration. Unlike cron, supervisord does not attempt to divine and override "fundamental" environment variables like USER, PATH, HOME, and LOGNAME when it performs a setuid to the user defined within the user= program config option. If you need to set environment variables for a particular program that might otherwise be set by a shell invocation for a particular user, you must do it explicitly within the environment= program config option. An example of setting these environment variables is as below.
[program:apache2] command=/home/chrism/bin/httpd -c "ErrorLog /dev/stdout" -DFOREGROUND user=chrism environment=HOME="/home/chrism",USER="chrism"
A process controlled by supervisord will be in one of the below states at any given time. You may see these state names in various user interface elements in clients.
STOPPED (0)
STARTING (10)
RUNNING (20)
BACKOFF (30)
STOPPING (40)
EXITED (100)
FATAL (200)
UNKNOWN (1000)
Each process run under supervisor progresses through these states as per the following directed graph.
A process is in the STOPPED state if it has been stopped adminstratively or if it has never been started.
When an autorestarting process is in the BACKOFF state, it will be automatically restarted by supervisord. It will switch between STARTING and BACKOFF states until it becomes evident that it cannot be started because the number of startretries has exceeded the maximum, at which point it will transition to the FATAL state. Each start retry will take progressively more time.
When a process is in the EXITED state, it will automatically restart:
A process automatically transitions from EXITED to RUNNING as a result of being configured to autorestart conditionally or unconditionally. The number of transitions between RUNNING and EXITED is not limited in any way: it is possible to create a configuration that endlessly restarts an exited process. This is a feature, not a bug.
An autorestarted process will never be automatically restarted if it ends up in the FATAL state (it must be manually restarted from this state).
A process transitions into the STOPPING state via an administrative stop request, and will then end up in the STOPPED state.
A process that cannot be stopped successfully will stay in the STOPPING state forever. This situation should never be reached during normal operations as it implies that the process did not respond to a final SIGKILL signal sent to it by supervisor, which is "impossible" under UNIX.
State transitions which always require user action to invoke are these:
FATAL -> STARTING
RUNNING -> STOPPING
State transitions which typically, but not always, require user action to invoke are these, with exceptions noted:
STOPPED -> STARTING (except at supervisord startup if process is configured to autostart)
EXITED -> STARTING (except if process is configured to autorestart)
All other state transitions are managed by supervisord automatically.
This man page was created by Orestis Ioannou <orestis@oioannou.com> using the official documentation.
2004-2015, Agendaless Consulting and Contributors
