Ubuntu Manpage: salt - Salt Documentation

Provided by: salt-common_2015.8.8+ds-1ubuntu0.1_all

NAME

       salt - Salt Documentation

INTRODUCTION TO SALT

We’re not just talking about NaCl..SS The 30 second summary

Salt is:

• a configuration management system, capable of maintaining remote nodes in defined states (for example,
ensuring that specific packages are installed and specific services are running)

• a distributed remote execution system used to execute commands and query data on remote nodes, either
individually or by arbitrary selection criteria

It was developed in order to bring the best solutions found in the world of remote execution together and
make them better, faster, and more malleable. Salt accomplishes this through its ability to handle large
loads of information, and not just dozens but hundreds and even thousands of individual servers quickly
through a simple and manageable interface.

Simplicity
Providing versatility between massive scale deployments and smaller systems may seem daunting, but Salt
is very simple to set up and maintain, regardless of the size of the project. The architecture of Salt is
designed to work with any number of servers, from a handful of local network systems to international
deployments across different data centers. The topology is a simple server/client model with the needed
functionality built into a single set of daemons. While the default configuration will work with little
to no modification, Salt can be fine tuned to meet specific needs.

Parallel execution
The core functions of Salt:

• enable commands to remote systems to be called in parallel rather than serially

• use a secure and encrypted protocol

• use the smallest and fastest network payloads possible

• provide a simple programming interface

Salt also introduces more granular controls to the realm of remote execution, allowing systems to be
targeted not just by hostname, but also by system properties.

Building on proven technology
Salt takes advantage of a number of technologies and techniques. The networking layer is built with the
excellent ZeroMQ networking library, so the Salt daemon includes a viable and transparent AMQ broker.
Salt uses public keys for authentication with the master daemon, then uses faster AES encryption for
payload communication; authentication and encryption are integral to Salt. Salt takes advantage of
communication via msgpack, enabling fast and light network traffic.

Python client interface
In order to allow for simple expansion, Salt execution routines can be written as plain Python modules.
The data collected from Salt executions can be sent back to the master server, or to any arbitrary
program. Salt can be called from a simple Python API, or from the command line, so that Salt can be used
to execute one-off commands as well as operate as an integral part of a larger application.

Fast, flexible, scalable
The result is a system that can execute commands at high speed on target server groups ranging from one
to very many servers. Salt is very fast, easy to set up, amazingly malleable and provides a single remote
execution architecture that can manage the diverse requirements of any number of servers. The Salt
infrastructure brings together the best of the remote execution world, amplifies its capabilities and
expands its range, resulting in a system that is as versatile as it is practical, suitable for any
network.

Open
Salt is developed under the Apache 2.0 license, and can be used for open and proprietary projects. Please
submit your expansions back to the Salt project so that we can all benefit together as Salt grows.
Please feel free to sprinkle Salt around your systems and let the deliciousness come forth.

Salt Community
Join the Salt!

There are many ways to participate in and communicate with the Salt community.

Salt has an active IRC channel and a mailing list.

Mailing List
Join the salt-users mailing list. It is the best place to ask questions about Salt and see whats going on
with Salt development! The Salt mailing list is hosted by Google Groups. It is open to new members.

https://groups.google.com/forum/#!forum/salt-users

There is also a low-traffic list used to announce new releases called salt-announce

https://groups.google.com/forum/#!forum/salt-announce

IRC
The #salt IRC channel is hosted on the popular Freenode network. You can use the Freenode webchat client
right from your browser.

Logs of the IRC channel activity are being collected courtesy of Moritz Lenz.

If you wish to discuss the development of Salt itself join us in #salt-devel.

Follow on GitHub
The Salt code is developed via GitHub. Follow Salt for constant updates on what is happening in Salt
development:

https://github.com/saltstack/salt

Blogs
SaltStack Inc. keeps a blog with recent news and advancements:

http://www.saltstack.com/blog/

Thomas Hatch also shares news and thoughts on Salt and related projects in his personal blog The Red45:

http://red45.wordpress.com/

Example Salt States
The official salt-states repository is: https://github.com/saltstack/salt-states

A few examples of salt states from the community:

• https://github.com/blast-hardcheese/blast-salt-states

• https://github.com/kevingranade/kevingranade-salt-state

• https://github.com/mattmcclean/salt-openstack/tree/master/salt

• https://github.com/rentalita/ubuntu-setup/

• https://github.com/brutasse/states

• https://github.com/bclermont/states

• https://github.com/pcrews/salt-data

Follow on ohloh
https://www.ohloh.net/p/salt

Other community links
• Salt Stack Inc.

• Subreddit

• Google+

• YouTube

• Facebook

• Twitter

• Wikipedia page

Hack the Source
If you want to get involved with the development of source code or the documentation efforts, please
review the hacking section!

INSTALLATION

SEE ALSO:
Installing Salt for development and contributing to the project.

Quick Install
On most distributions, you can set up a Salt Minion with the Salt Bootstrap.

Platform-specific Installation Instructions
These guides go into detail how to install Salt on a given platform.

Arch Linux
Installation
Salt (stable) is currently available via the Arch Linux Official repositories. There are currently -git
packages available in the Arch User repositories (AUR) as well.

Stable Release
Install Salt stable releases from the Arch Linux Official repositories as follows:

pacman -S salt-zmq

To install Salt stable releases using the RAET protocol, use the following:

pacman -S salt-raet

NOTE:
transports

Unlike other linux distributions, please be aware that Arch Linux's package manager pacman defaults to
RAET as the Salt transport. If you want to use ZeroMQ instead, make sure to enter the associated
number for the salt-zmq repository when prompted.

Tracking develop
To install the bleeding edge version of Salt (may include bugs!), use the -git package. Installing the
-git package as follows:

wget https://aur.archlinux.org/packages/sa/salt-git/salt-git.tar.gz
tar xf salt-git.tar.gz
cd salt-git/
makepkg -is

NOTE:
yaourt

If a tool such as Yaourt is used, the dependencies will be gathered and built automatically.

The command to install salt using the yaourt tool is:

yaourt salt-git

Post-installation tasks
systemd

Activate the Salt Master and/or Minion via systemctl as follows:

systemctl enable salt-master.service
systemctl enable salt-minion.service

Start the Master

Once you've completed all of these steps you're ready to start your Salt Master. You should be able to
start your Salt Master now using the command seen here:

systemctl start salt-master

Now go to the Configuring Salt page.

Debian GNU/Linux / Raspbian
Debian GNU/Linux distribution and some devariatives such as Raspbian already have included Salt packages
to their repositories. However, current stable release codenamed "Jessie" contains old outdated Salt
release. It is recommended to use SaltStack repository for Debian as described below.

Installation from official Debian and Raspbian repositories is described here.

Installation from the SaltStack Repository
2015.5 and later packages for Debian 8 ("Jessie") are available in the SaltStack repository.

NOTE:
SaltStack repository contains only packages suitable for i386 (32-bit Intel-compatible CPUs) and amd64
(64-bit) architectures. While Salt packages are built for all Debian ports (have all suffix in package
names), some of the dependencies are avaivable only for amd64 systems.

IMPORTANT:
The repository folder structure changed in the 2015.8.3 release, though the previous repository
structure that was documented in 2015.8.1 can continue to be used.

To install using the SaltStack repository:

1. Run the following command to import the SaltStack repository key:

wget -O - https://repo.saltstack.com/apt/debian/8/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -

2. Add the following line to /etc/apt/sources.list:

deb http://repo.saltstack.com/apt/debian/8/amd64/latest jessie main

3. Run sudo apt-get update.

4. Now go to the packages installation section.

Installation from the Community Repository
The SaltStack community maintains a Debian repository at debian.saltstack.com. Packages for Debian Old
Stable, Stable, and Unstable (Wheezy, Jessie, and Sid) for Salt 0.16 and later are published in this
repository.

NOTE:
Packages in this repository are community built, and it can take a little while until the latest
SaltStack release is available in this repository.

Jessie (Stable)
For Jessie, the following line is needed in either /etc/apt/sources.list or a file in
/etc/apt/sources.list.d:

deb http://debian.saltstack.com/debian jessie-saltstack main

Wheezy (Old Stable)
For Wheezy, the following line is needed in either /etc/apt/sources.list or a file in
/etc/apt/sources.list.d:

deb http://debian.saltstack.com/debian wheezy-saltstack main

Squeeze (Old Old Stable)
For Squeeze, you will need to enable the Debian backports repository as well as the debian.saltstack.com
repository. To do so, add the following to /etc/apt/sources.list or a file in /etc/apt/sources.list.d:

deb http://debian.saltstack.com/debian squeeze-saltstack main
deb http://backports.debian.org/debian-backports squeeze-backports main

Stretch (Testing)
For Stretch, the following line is needed in either /etc/apt/sources.list or a file in
/etc/apt/sources.list.d:

deb http://debian.saltstack.com/debian stretch-saltstack main

Sid (Unstable)
For Sid, the following line is needed in either /etc/apt/sources.list or a file in
/etc/apt/sources.list.d:

deb http://debian.saltstack.com/debian unstable main

Import the repository key
You will need to import the key used for signing.

wget -q -O- "http://debian.saltstack.com/debian-salt-team-joehealy.gpg.key" | apt-key add -

NOTE:
You can optionally verify the key integrity with sha512sum using the public key signature shown here.
E.g:

echo "b702969447140d5553e31e9701be13ca11cc0a7ed5fe2b30acb8491567560ee62f834772b5095d735dfcecb2384a5c1a20045f52861c417f50b68dd5ff4660e6 debian-salt-team-joehealy.gpg.key" | sha512sum -c

Update the package database
apt-get update

Installation from the Debian / Raspbian Official Repository
Stretch (Testing) and Sid (Unstable) distributions are already contain mostly up-to-date Salt packages
built by Debian Salt Team. You can install Salt components directly from Debian.

On Jessie (Stable) there is an option to install Salt minion from Stretch with python-tornado dependency
from jessie-backports repositories.

To install fresh release of Salt minion on Jessie:

1. Add jessie-backports and stretch repositories:

Debian:

echo 'deb http://httpredir.debian.org/debian jessie-backports main' >> /etc/apt/sources.list
echo 'deb http://httpredir.debian.org/debian stretch main' >> /etc/apt/sources.list

Raspbian:

echo 'deb http://archive.raspbian.org/raspbian/ stretch main' >> /etc/apt/sources.list

2. Make Jessie a default release:

echo 'APT::Default-Release "jessie";' > /etc/apt/apt.conf.d/10apt

3. Install Salt dependencies:

Debian:

apt-get update
apt-get install python-zmq python-tornado/jessie-backports salt-common/stretch

Raspbian:

apt-get update
apt-get install python-zmq python-tornado/stretch salt-common/stretch

4. Install Salt minion package from Stretch:

apt-get install salt-minion/stretch

Install Packages
Install the Salt master, minion or other packages from the repository with the apt-get command. These
examples each install one of Salt components, but more than one package name may be given at a time:

• apt-get install salt-api

• apt-get install salt-cloud

• apt-get install salt-master

• apt-get install salt-minion

• apt-get install salt-ssh

• apt-get install salt-syndic

Post-installation tasks
Now, go to the Configuring Salt page.

Fedora
Beginning with version 0.9.4, Salt has been available in the primary Fedora repositories and EPEL. It is
installable using yum. Fedora will have more up to date versions of Salt than other members of the Red
Hat family, which makes it a great place to help improve Salt!

WARNING: Fedora 19 comes with systemd 204. Systemd has known bugs fixed in later revisions that prevent
the salt-master from starting reliably or opening the network connections that it needs to. It's not
likely that a salt-master will start or run reliably on any distribution that uses systemd version 204 or
earlier. Running salt-minions should be OK.

Installation
Salt can be installed using yum and is available in the standard Fedora repositories.

Stable Release
Salt is packaged separately for the minion and the master. It is necessary only to install the
appropriate package for the role the machine will play. Typically, there will be one master and multiple
minions.

yum install salt-master
yum install salt-minion

Installing from updates-testing
When a new Salt release is packaged, it is first admitted into the updates-testing repository, before
being moved to the stable repo.

To install from updates-testing, use the enablerepo argument for yum:

yum --enablerepo=updates-testing install salt-master
yum --enablerepo=updates-testing install salt-minion

Installation Using pip
Since Salt is on PyPI, it can be installed using pip, though most users prefer to install using a package
manager.

Installing from pip has a few additional requirements:

• Install the group 'Development Tools', dnf groupinstall 'Development Tools'

• Install the 'zeromq-devel' package if it fails on linking against that afterwards as well.

A pip install does not make the init scripts or the /etc/salt directory, and you will need to provide
your own systemd service unit.

Installation from pip:

pip install salt

WARNING:
If installing from pip (or from source using setup.py install), be advised that the yum-utils package
is needed for Salt to manage packages. Also, if the Python dependencies are not already installed,
then you will need additional libraries/tools installed to build some of them. More information on
this can be found here.

Post-installation tasks
Master

To have the Master start automatically at boot time:

systemctl enable salt-master.service

To start the Master:

systemctl start salt-master.service

Minion

To have the Minion start automatically at boot time:

systemctl enable salt-minion.service

To start the Minion:

systemctl start salt-minion.service

Now go to the Configuring Salt page.

FreeBSD
Salt was added to the FreeBSD ports tree Dec 26th, 2011 by Christer Edwards <christer.edwards@gmail.com>.
It has been tested on FreeBSD 7.4, 8.2, 9.0, 9.1, 10.0 and later releases.

Installation
Salt is available in binary package form from both the FreeBSD pkgng repository or directly from
SaltStack. The instructions below outline installation via both methods:

FreeBSD repo
The FreeBSD pkgng repository is preconfigured on systems 10.x and above. No configuration is needed to
pull from these repositories.

pkg install py27-salt

These packages are usually available within a few days of upstream release.

SaltStack repo
SaltStack also hosts internal binary builds of the Salt package, available from
https://repo.saltstack.com/freebsd/. To make use of this repository, add the following file to your
system:

/usr/local/etc/pkg/repos/saltstack.conf:

saltstack: {
url: "https://repo.saltstack.com/freebsd/${ABI}/",
mirror_type: "http",
enabled: yes
priority: 10
}

You should now be able to install Salt from this new repository:

pkg install py27-salt

These packages are usually available earlier than upstream FreeBSD. Also available are release candidates
and development releases. Use these pre-release packages with caution.

Post-installation tasks
Master

Copy the sample configuration file:

cp /usr/local/etc/salt/master.sample /usr/local/etc/salt/master

rc.conf

Activate the Salt Master in /etc/rc.conf:

sysrc salt_master_enable="YES"

Start the Master

Start the Salt Master as follows:

service salt_master start

Minion

Copy the sample configuration file:

cp /usr/local/etc/salt/minion.sample /usr/local/etc/salt/minion

rc.conf

Activate the Salt Minion in /etc/rc.conf:

sysrc salt_minion_enable="YES"

Start the Minion

Start the Salt Minion as follows:

service salt_minion start

Now go to the Configuring Salt page.

Gentoo
Salt can be easily installed on Gentoo via Portage:

emerge app-admin/salt

Post-installation tasks
Now go to the Configuring Salt page.

OpenBSD
Salt was added to the OpenBSD ports tree on Aug 10th 2013. It has been tested on OpenBSD 5.5 onwards.

Salt is dependent on the following additional ports. These will be installed as dependencies of the
sysutils/salt port:

devel/py-futures
devel/py-progressbar
net/py-msgpack
net/py-zmq
security/py-crypto
security/py-M2Crypto
textproc/py-MarkupSafe
textproc/py-yaml
www/py-jinja2
www/py-requests
www/py-tornado

Installation
To install Salt from the OpenBSD pkg repo, use the command:

pkg_add salt

Post-installation tasks
Master

To have the Master start automatically at boot time:

rcctl enable salt_master

To start the Master:

rcctl start salt_master

Minion

To have the Minion start automatically at boot time:

rcctl enable salt_minion

To start the Minion:

rcctl start salt_minion

Now go to the Configuring Salt page.

OS X
Dependency Installation
It should be noted that Homebrew explicitly discourages the use of sudo:
Homebrew is designed to work without using sudo. You can decide to use it but we strongly recommend
not to do so. If you have used sudo and run into a bug then it is likely to be the cause. Please don’t
file a bug report unless you can reproduce it after reinstalling Homebrew from scratch without using
sudo

So when using Homebrew, if you want support from the Homebrew community, install this way:

brew install saltstack

When using MacPorts, install this way:

sudo port install salt

When only using the OS X system's pip, install this way:

sudo pip install salt

Salt-Master Customizations
To run salt-master on OS X, the root user maxfiles limit must be increased:

NOTE:
On OS X 10.10 (Yosemite) and higher, maxfiles should not be adjusted. The default limits are
sufficient in all but the most extreme scenarios. Overriding these values with the setting below will
cause system instability!

sudo launchctl limit maxfiles 4096 8192

And sudo add this configuration option to the /etc/salt/master file:

max_open_files: 8192

Now the salt-master should run without errors:

sudo salt-master --log-level=all

Post-installation tasks
Now go to the Configuring Salt page.

RHEL / CentOS / Scientific Linux / Amazon Linux / Oracle Linux
Salt should work properly with all mainstream derivatives of Red Hat Enterprise Linux, including CentOS,
Scientific Linux, Oracle Linux, and Amazon Linux. Report any bugs or issues on the issue tracker.

Installation from the SaltStack Repository
2015.5 and later packages for RHEL 5, 6, and 7 are available in the SaltStack repository.

IMPORTANT:
The repository folder structure changed in the 2015.8.3 release, though the previous repository
structure that was documented in 2015.8.1 can continue to be used.

To install using the SaltStack repository:

1. Run one of the following commands based on your version to import the SaltStack repository key:

Version 7:

rpm --import https://repo.saltstack.com/yum/redhat/7/x86_64/latest/SALTSTACK-GPG-KEY.pub

Version 6:

rpm --import https://repo.saltstack.com/yum/redhat/6/x86_64/latest/SALTSTACK-GPG-KEY.pub

Version 5:

wget https://repo.saltstack.com/yum/redhat/5/x86_64/latest/SALTSTACK-EL5-GPG-KEY.pub
rpm --import SALTSTACK-EL5-GPG-KEY.pub
rm -f SALTSTACK-EL5-GPG-KEY.pub

2. Save the following file to /etc/yum.repos.d/saltstack.repo:

Version 7 and 6:

[saltstack-repo]
name=SaltStack repo for RHEL/CentOS $releasever
baseurl=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest
enabled=1
gpgcheck=1
gpgkey=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest/SALTSTACK-GPG-KEY.pub

Version 5:

3. Run sudo yum clean expire-cache.

4. Run sudo yum update.

5. Install the salt-minion, salt-master, or other Salt components:

• yum install salt-master

• yum install salt-minion

• yum install salt-ssh

• yum install salt-syndic

• yum install salt-cloud

NOTE:
As of 2015.8.0, EPEL repository is no longer required for installing on RHEL systems. SaltStack
repository provides all needed dependencies.

WARNING:
If installing on Red Hat Enterprise Linux 7 with disabled (not subscribed on) 'RHEL Server Releases'
or 'RHEL Server Optional Channel' repositories, append CentOS 7 GPG key URL to SaltStack yum
repository configuration to install required base packages:

[saltstack-repo]
name=SaltStack repo for Red Hat Enterprise Linux $releasever
baseurl=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest
enabled=1
gpgcheck=1
gpgkey=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest/SALTSTACK-GPG-KEY.pub
https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest/base/RPM-GPG-KEY-CentOS-7

NOTE:
systemd and python-systemd are required by Salt, but are not installed by the Red Hat 7 @base
installation or by the Salt installation. These dependencies might need to be installed before Salt.

Installation from the Community Repository
Beginning with version 0.9.4, Salt has been available in EPEL. For RHEL/CentOS 5, Fedora COPR is a single
community repository that provides Salt packages due to the removal from EPEL5.

NOTE:
Packages in these repositories are built by community, and it can take a little while until the latest
stable SaltStack release become available.

RHEL/CentOS 6 and 7, Scientific Linux, etc.
WARNING:
Salt 2015.8 is currently not available in EPEL due to unsatisfied dependencies: python-crypto 2.6.1 or
higher, and python-tornado version 4.2.1 or higher. These packages are not currently available in EPEL
for Red Hat Enterprise Linux 6 and 7.

Enabling EPEL
If the EPEL repository is not installed on your system, you can download the RPM for RHEL/CentOS 6 or for
RHEL/CentOS 7 and install it using the following command:

rpm -Uvh epel-release-X-Y.rpm

Replace epel-release-X-Y.rpm with the appropriate filename.

Installing Stable Release
Salt is packaged separately for the minion and the master. It is necessary to install only the
appropriate package for the role the machine will play. Typically, there will be one master and multiple
minions.

• yum install salt-master

• yum install salt-minion

• yum install salt-ssh

• yum install salt-syndic

• yum install salt-cloud

Installing from epel-testing
When a new Salt release is packaged, it is first admitted into the epel-testing repository, before being
moved to the stable EPEL repository.

To install from epel-testing, use the enablerepo argument for yum:

yum --enablerepo=epel-testing install salt-minion

Installation Using pip
Since Salt is on PyPI, it can be installed using pip, though most users prefer to install using RPM
packages (which can be installed from EPEL).

Installing from pip has a few additional requirements:

• Install the group 'Development Tools', yum groupinstall 'Development Tools'

• Install the 'zeromq-devel' package if it fails on linking against that afterwards as well.

A pip install does not make the init scripts or the /etc/salt directory, and you will need to provide
your own systemd service unit.

Installation from pip:

pip install salt

ZeroMQ 4
We recommend using ZeroMQ 4 where available. SaltStack provides ZeroMQ 4.0.5 and pyzmq 14.5.0 in the
SaltStack Repository as well as a separate zeromq4 COPR repository.

If this repository is added before Salt is installed, then installing either salt-master or salt-minion
will automatically pull in ZeroMQ 4.0.5, and additional steps to upgrade ZeroMQ and pyzmq are
unnecessary.

WARNING:
RHEL/CentOS 5 Users Using COPR repos on RHEL/CentOS 5 requires that the python-hashlib package be
installed. Not having it present will result in checksum errors because YUM will not be able to
process the SHA256 checksums used by COPR.

NOTE:
For RHEL/CentOS 5 installations, if using the SaltStack repo or Fedora COPR to install Salt (as
described above), then it is not necessary to enable the zeromq4 COPR, because those repositories
already include ZeroMQ 4.

Package Management
Salt's interface to yum makes heavy use of the repoquery utility, from the yum-utils package. This
package will be installed as a dependency if salt is installed via EPEL. However, if salt has been
installed using pip, or a host is being managed using salt-ssh, then as of version 2014.7.0 yum-utils
will be installed automatically to satisfy this dependency.

Post-installation tasks
Master
To have the Master start automatically at boot time:

RHEL/CentOS 5 and 6

chkconfig salt-master on

RHEL/CentOS 7

systemctl enable salt-master.service

To start the Master:

RHEL/CentOS 5 and 6

service salt-master start

RHEL/CentOS 7

systemctl start salt-master.service

Minion
To have the Minion start automatically at boot time:

RHEL/CentOS 5 and 6

chkconfig salt-minion on

RHEL/CentOS 7

systemctl enable salt-minion.service

To start the Minion:

RHEL/CentOS 5 and 6

service salt-minion start

RHEL/CentOS 7

systemctl start salt-minion.service

Now go to the Configuring Salt page.

Solaris
Salt was added to the OpenCSW package repository in September of 2012 by Romeo Theriault <‐
romeot@hawaii.edu> at version 0.10.2 of Salt. It has mainly been tested on Solaris 10 (sparc), though it
is built for and has been tested minimally on Solaris 10 (x86), Solaris 9 (sparc/x86) and 11 (sparc/x86).
(Please let me know if you're using it on these platforms!) Most of the testing has also just focused on
the minion, though it has verified that the master starts up successfully on Solaris 10.

Comments and patches for better support on these platforms is very welcome.

As of version 0.10.4, Solaris is well supported under salt, with all of the following working well:

1. remote execution

2. grain detection

3. service control with SMF

4. 'pkg' states with 'pkgadd' and 'pkgutil' modules

5. cron modules/states

6. user and group modules/states

7. shadow password management modules/states

Salt is dependent on the following additional packages. These will automatically be installed as
dependencies of the py_salt package:

• py_yaml

• py_pyzmq

• py_jinja2

• py_msgpack_python

• py_m2crypto

• py_crypto

• python

Installation
To install Salt from the OpenCSW package repository you first need to install pkgutil assuming you don't
already have it installed:

On Solaris 10:

pkgadd -d http://get.opencsw.org/now

On Solaris 9:

wget http://mirror.opencsw.org/opencsw/pkgutil.pkg
pkgadd -d pkgutil.pkg all

Once pkgutil is installed you'll need to edit it's config file /etc/opt/csw/pkgutil.conf to point it at
the unstable catalog:

- #mirror=http://mirror.opencsw.org/opencsw/testing
+ mirror=http://mirror.opencsw.org/opencsw/unstable

OK, time to install salt.

# Update the catalog
root> /opt/csw/bin/pkgutil -U
# Install salt
root> /opt/csw/bin/pkgutil -i -y py_salt

Minion Configuration
Now that salt is installed you can find it's configuration files in /etc/opt/csw/salt/.

You'll want to edit the minion config file to set the name of your salt master server:

- #master: salt
+ master: your-salt-server

If you would like to use pkgutil as the default package provider for your Solaris minions, you can do so
using the providers option in the minion config file.

You can now start the salt minion like so:

On Solaris 10:

svcadm enable salt-minion

On Solaris 9:

/etc/init.d/salt-minion start

You should now be able to log onto the salt master and check to see if the salt-minion key is awaiting
acceptance:

salt-key -l un

Accept the key:

salt-key -a <your-salt-minion>

Run a simple test against the minion:

salt '<your-salt-minion>' test.ping

Troubleshooting
Logs are in /var/log/salt

Ubuntu
Installation from the SaltStack Repository
2015.5 and later packages for Ubuntu 14 (Trusty) and Ubuntu 12 (Precise) are available in the SaltStack
repository.

NOTE:
While Salt packages are built for all Ubuntu supported CPU architectures (i386 and amd64), some of the
dependencies avaivable from SaltStack corporate repository are only suitable for amd64 systems.

IMPORTANT:
The repository folder structure changed in the 2015.8.3 release, though the previous repository
structure that was documented in 2015.8.1 can continue to be used.

To install using the SaltStack repository:

1. Run the following command to import the SaltStack repository key:

Ubuntu 14:

wget -O - https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -

Ubuntu 12:

wget -O - https://repo.saltstack.com/apt/ubuntu/12.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add -

2. Add the following line to /etc/apt/sources.list:

Ubuntu 14:

deb http://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest trusty main

Ubuntu 12:

deb http://repo.saltstack.com/apt/ubuntu/12.04/amd64/latest precise main

3. Run sudo apt-get update.

4. Now go to the packages installation section.

Installation from the Community Repository
Packages for Ubuntu are also published in the saltstack PPA. If you have the add-apt-repository utility,
you can add the repository and import the key in one step:

sudo add-apt-repository ppa:saltstack/salt

In addition to the main repository, there are secondary repositories for each individual major release.
These repositories receive security and point releases but will not upgrade to any subsequent major
release. There are currently several available repos: salt16, salt17, salt2014-1, salt2014-7,
salt2015-5. For example to follow 2015.5.x releases:

sudo add-apt-repository ppa:saltstack/salt2015-5

add-apt-repository: command not found?

The add-apt-repository command is not always present on Ubuntu systems. This can be fixed by
installing python-software-properties:

sudo apt-get install python-software-properties

The following may be required as well:

sudo apt-get install software-properties-common

Note that since Ubuntu 12.10 (Raring Ringtail), add-apt-repository is found in the
software-properties-common package, and is part of the base install. Thus, add-apt-repository should
be able to be used out-of-the-box to add the PPA.

Alternately, manually add the repository and import the PPA key with these commands:

echo deb http://ppa.launchpad.net/saltstack/salt/ubuntu `lsb_release -sc` main | sudo tee /etc/apt/sources.list.d/saltstack.list
wget -q -O- "http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x4759FA960E27C0A6" | sudo apt-key add -

After adding the repository, update the package management database:

sudo apt-get update

• apt-get install salt-api

• apt-get install salt-cloud

• apt-get install salt-master

• apt-get install salt-minion

• apt-get install salt-ssh

• apt-get install salt-syndic

Post-installation tasks
Now go to the Configuring Salt page.

Windows
Salt has full support for running the Salt Minion on Windows.

There are no plans for the foreseeable future to develop a Salt Master on Windows. For now you must run
your Salt Master on a supported operating system to control your Salt Minions on Windows.

Many of the standard Salt modules have been ported to work on Windows and many of the Salt States
currently work on Windows, as well.

Windows Installer
Salt Minion Windows installers can be found here. The output of md5sum <salt minion exe> should match the
contents of the corresponding md5 file.

Latest stable build from the selected branch:

Earlier builds from supported branches

Archived builds from unsupported branches

NOTE:
The installation executable installs dependencies that the Salt minion requires.

The 64bit installer has been tested on Windows 7 64bit and Windows Server 2008R2 64bit. The 32bit
installer has been tested on Windows 2003 Server 32bit. Please file a bug report on our GitHub repo if
issues for other platforms are found.

The installer asks for 2 bits of information; the master hostname and the minion name. The installer will
update the minion config with these options and then start the minion.

The salt-minion service will appear in the Windows Service Manager and can be started and stopped there
or with the command line program sc like any other Windows service.

If the minion won't start, try installing the Microsoft Visual C++ 2008 x64 SP1 redistributable. Allow
all Windows updates to run salt-minion smoothly.

Silent Installer Options
The installer can be run silently by providing the /S option at the command line. The installer also
accepts the following options for configuring the Salt Minion silently:

• /master= A string value to set the IP address or host name of the master. Default value is 'salt'

• /minion-name= A string value to set the minion name. Default is 'hostname'

• /start-service= Either a 1 or 0. '1' will start the service, '0' will not. Default is to start the
service after installation.

Here's an example of using the silent installer:

Salt-Minion-2015.5.6-Setup-amd64.exe /S /master=yoursaltmaster /minion-name=yourminionname /start-service=0

Running the Salt Minion on Windows as an Unprivileged User
Notes: - These instructions were tested with Windows Server 2008 R2 - They are generalizable to any
version of Windows that supports a salt-minion

A. Create the Unprivileged User that the Salt Minion will Run As
1. Click Start > Control Panel > User Accounts.

2. Click Add or remove user accounts.

3. Click Create new account.

4. Enter salt-user (or a name of your preference) in the New account name field.

5. Select the Standard user radio button.

6. Click the Create Account button.

7. Click on the newly created user account.

8. Click the Create a password link.

9. In the New password and Confirm new password fields, provide a password (e.g
"SuperSecretMinionPassword4Me!").

10. In the Type a password hint field, provide appropriate text (e.g. "My Salt Password").

11. Click the Create password button.

12. Close the Change an Account window.

B. Add the New User to the Access Control List for the Salt Folder
1. In a File Explorer window, browse to the path where Salt is installed (the default path is C:\Salt).

2. Right-click on the Salt folder and select Properties.

3. Click on the Security tab.

4. Click the Edit button.

5. Click the Add button.

6. Type the name of your designated Salt user and click the OK button.

7. Check the box to Allow the Modify permission.

8. Click the OK button.

9. Click the OK button to close the Salt Properties window.

C. Update the Windows Service User for the salt-minion Service
1. Click Start > Administrative Tools > Services.

2. In the Services list, right-click on salt-minion and select Properties.

3. Click the Log On tab.

4. Click the This account radio button.

5. Provide the account credentials created in section A.

6. Click the OK button.

7. Click the OK button to the prompt confirming that the user has been granted the Log On As A Service
right.

8. Click the OK button to the prompt confirming that The new logon name will not take effect until you
stop and restart the service.

9. Right-Click on salt-minion and select Stop.

10. Right-Click on salt-minion and select Start.

Setting up a Windows build environment
This document will explain how to set up a development environment for salt on Windows. The development
environment allows you to work with the source code to customize or fix bugs. It will also allow you to
build your own installation.

The Easy Way
Prerequisite Software
To do this the easy way you only need to install Git for Windows.

Create the Build Environment
1. Clone the Salt-Windows-Dev repo from github.

Open a command line and type:

git clone https://github.com/saltstack/salt-windows-dev

2. Build the Python Environment

Go into the salt-windows-dev directory. Right-click the file named dev_env.ps1 and select Run with
PowerShell

If you get an error, you may need to change the execution policy.

Open a powershell window and type the following:

Set-ExecutionPolicy RemoteSigned

This will download and install Python with all the dependencies needed to develop and build salt.

3. Build the Salt Environment

Right-click on the file named dev_env_salt.ps1 and select Run with Powershell

This will clone salt into C:\Salt-Dev\salt and set it to the 2015.5 branch. You could optionally run
the command from a powershell window with a -Version switch to pull a different version. For example:

dev_env_salt.ps1 -Version '2014.7'

To view a list of available branches and tags, open a command prompt in your C:Salt-Devsalt directory
and type:

git branch -a
git tag -n

The Hard Way
Prerequisite Software
Install the following software:

1. Git for Windows

2. Nullsoft Installer

Download the Prerequisite zip file for your CPU architecture from the SaltStack download site:

• Salt32.zip

• Salt64.zip

These files contain all software required to build and develop salt. Unzip the contents of the file to
C:\Salt-Dev\temp.

Create the Build Environment
1. Build the Python Environment

• Install Python:

Browse to the C:\Salt-Dev\temp directory and find the Python installation file for your CPU
Architecture under the corresponding subfolder. Double-click the file to install python.

Make sure the following are in your PATH environment variable:

C:\Python27
C:\Python27\Scripts

• Install Pip

Open a command prompt and navigate to C:\Salt-Dev\temp Run the following command:

python get-pip.py

• Easy Install compiled binaries.

M2Crypto, PyCrypto, and PyWin32 need to be installed using Easy Install. Open a command prompt and
navigate to C:\Salt-Dev\temp\<cpuarch>. Run the following commands:

easy_install -Z <M2Crypto file name>
easy_install -Z <PyCrypto file name>
easy_install -Z <PyWin32 file name>

NOTE:
You can type the first part of the file name and then press the tab key to auto-complete the
name of the file.

• Pip Install Additional Prerequisites

All remaining prerequisites need to be pip installed. These prerequisites are as follow:

• MarkupSafe

• Jinja

• MsgPack

• PSUtil

• PyYAML

• PyZMQ

• WMI

• Requests

• Certifi

Open a command prompt and navigate to C:\Salt-Dev\temp. Run the following commands:

pip install <cpuarch>\<MarkupSafe file name>
pip install <Jinja file name>
pip install <cpuarch>\<MsgPack file name>
pip install <cpuarch>\<psutil file name>
pip install <cpuarch>\<PyYAML file name>
pip install <cpuarch>\<pyzmq file name>
pip install <WMI file name>
pip install <requests file name>
pip install <certifi file name>

2. Build the Salt Environment

• Clone Salt

Open a command prompt and navigate to C:\Salt-Dev. Run the following command to clone salt:

git clone https://github.com/saltstack/salt

• Checkout Branch

Checkout the branch or tag of salt you want to work on or build. Open a command prompt and navigate
to C:\Salt-Dev\salt. Get a list of available tags and branches by running the following commands:

git fetch --all

To view a list of available branches:
git branch -a

To view a list of availabel tags:
git tag -n

Checkout the branch or tag by typing the following command:

git checkout <branch/tag name>

• Clean the Environment

When switching between branches residual files can be left behind that will interfere with the
functionality of salt. Therefore, after you check out the branch you want to work on, type the
following commands to clean the salt environment:

Developing with Salt
There are two ways to develop with salt. You can run salt's setup.py each time you make a change to
source code or you can use the setup tools develop mode.

Configure the Minion
Both methods require that the minion configuration be in the C:\salt directory. Copy the conf and var
directories from C:\Salt-Dev\salt\pkg\ windows\buildenv to C:\salt. Now go into the C:\salt\conf
directory and edit the file name minion (no extension). You need to configure the master and id
parameters in this file. Edit the following lines:

master: <ip or name of your master>
id: <name of your minion>

Setup.py Method
Go into the C:\Salt-Dev\salt directory from a cmd prompt and type:

python setup.py install --force

This will install python into your python installation at C:\Python27. Everytime you make an edit to
your source code, you'll have to stop the minion, run the setup, and start the minion.

To start the salt-minion go into C:\Python27\Scripts from a cmd prompt and type:

salt-minion

For debug mode type:

salt-minion -l debug

To stop the minion press Ctrl+C.

Setup Tools Develop Mode (Preferred Method)
To use the Setup Tools Develop Mode go into C:\Salt-Dev\salt from a cmd prompt and type:

pip install -e .

This will install pointers to your source code that resides at C:\Salt-Dev\salt. When you edit your
source code you only have to restart the minion.

Build the windows installer
This is the method of building the installer as of version 2014.7.4.

Clean the Environment
Make sure you don't have any leftover salt files from previous versions of salt in your Python directory.

1. Remove all files that start with salt in the C:\Python27\Scripts directory

2. Remove all files and directorys that start with salt in the C:\Python27\Lib\site-packages directory

Install Salt
Install salt using salt's setup.py. From the C:\Salt-Dev\salt directory type the following command:

python setup.py install --force

Build the Installer
From cmd prompt go into the C:\Salt-Dev\salt\pkg\windows directory. Type the following command for the
branch or tag of salt you're building:

BuildSalt.bat <branch or tag>

This will copy python with salt installed to the buildenv\bin directory, make it portable, and then
create the windows installer . The .exe for the windows installer will be placed in the installer
directory.

Testing the Salt minion
1. Create the directory C:\salt (if it doesn't exist already)

2. Copy the example conf and var directories from pkg/windows/buildenv/ into C:\salt

3. Edit C:\salt\conf\minion

master: ipaddress or hostname of your salt-master

4. Start the salt-minion

cd C:\Python27\Scripts
python salt-minion

5. On the salt-master accept the new minion's key

sudo salt-key -A

This accepts all unaccepted keys. If you're concerned about security just accept the key for this
specific minion.

6. Test that your minion is responding

On the salt-master run:

sudo salt '*' test.ping

You should get the following response: {'your minion hostname': True}

Single command bootstrap script
On a 64 bit Windows host the following script makes an unattended install of salt, including all
dependencies:

Not up to date.

This script is not up to date. Please use the installer found above

# (All in one line.)

"PowerShell (New-Object System.Net.WebClient).DownloadFile('http://csa-net.dk/salt/bootstrap64.bat','C:\bootstrap.bat');(New-Object -com Shell.Application).ShellExecute('C:\bootstrap.bat');"

You can execute the above command remotely from a Linux host using winexe:

winexe -U "administrator" //fqdn "PowerShell (New-Object ......);"

For more info check http://csa-net.dk/salt

Packages management under Windows 2003
On windows Server 2003, you need to install optional component "wmi windows installer provider" to have
full list of installed packages. If you don't have this, salt-minion can't report some installed
software.

SUSE
With openSUSE 13.2, Salt 2014.1.11 is available in the primary repositories. The devel:language:python
repo will have more up to date versions of salt, all package development will be done there.

Installation
Salt can be installed using zypper and is available in the standard openSUSE repositories.

zypper install salt-master
zypper install salt-minion

Post-installation tasks openSUSE
Master

To have the Master start automatically at boot time:

systemctl enable salt-master.service

To start the Master:

systemctl start salt-master.service

Minion

To have the Minion start automatically at boot time:

systemctl enable salt-minion.service

To start the Minion:

systemctl start salt-minion.service

Post-installation tasks SLES
Master

To have the Master start automatically at boot time:

chkconfig salt-master on

To start the Master:

rcsalt-master start

Minion

To have the Minion start automatically at boot time:

chkconfig salt-minion on

To start the Minion:

rcsalt-minion start

Unstable Release
openSUSE
For openSUSE Factory run the following as root:

zypper addrepo http://download.opensuse.org/repositories/devel:languages:python/openSUSE_Factory/devel:languages:python.repo
zypper refresh
zypper install salt salt-minion salt-master

For openSUSE 13.2 run the following as root:

zypper addrepo http://download.opensuse.org/repositories/devel:languages:python/openSUSE_13.2/devel:languages:python.repo
zypper refresh
zypper install salt salt-minion salt-master

For openSUSE 13.1 run the following as root:

zypper addrepo http://download.opensuse.org/repositories/devel:languages:python/openSUSE_13.1/devel:languages:python.repo
zypper refresh
zypper install salt salt-minion salt-master

For bleeding edge python Factory run the following as root:

zypper addrepo http://download.opensuse.org/repositories/devel:languages:python/bleeding_edge_python_Factory/devel:languages:python.repo
zypper refresh
zypper install salt salt-minion salt-master

Suse Linux Enterprise
For SLE 12 run the following as root:

zypper addrepo http://download.opensuse.org/repositories/devel:languages:python/SLE_12/devel:languages:python.repo
zypper refresh
zypper install salt salt-minion salt-master

For SLE 11 SP3 run the following as root:

zypper addrepo http://download.opensuse.org/repositories/devel:languages:python/SLE_11_SP3/devel:languages:python.repo
zypper refresh
zypper install salt salt-minion salt-master

For SLE 11 SP2 run the following as root:

zypper addrepo http://download.opensuse.org/repositories/devel:languages:python/SLE_11_SP2/devel:languages:python.repo
zypper refresh
zypper install salt salt-minion salt-master

Now go to the Configuring Salt page.

Dependencies
Salt should run on any Unix-like platform so long as the dependencies are met.

• Python 2.6 >= 2.6 <3.0

• msgpack-python - High-performance message interchange format

• YAML - Python YAML bindings

• Jinja2 - parsing Salt States (configurable in the master settings)

• MarkupSafe - Implements a XML/HTML/XHTML Markup safe string for Python

• apache-libcloud - Python lib for interacting with many of the popular cloud service providers using a
unified API

• Requests - HTTP library

Depending on the chosen Salt transport, ZeroMQ or RAET, dependencies vary:

• ZeroMQ:

• ZeroMQ >= 3.2.0

• pyzmq >= 2.2.0 - ZeroMQ Python bindings

• PyCrypto - The Python cryptography toolkit

• RAET:

• libnacl - Python bindings to libsodium

• ioflo - The flo programming interface raet and salt-raet is built on

• RAET - The worlds most awesome UDP protocol

Salt defaults to the ZeroMQ transport, and the choice can be made at install time, for example:

python setup.py --salt-transport=raet install

This way, only the required dependencies are pulled by the setup script if need be.

If installing using pip, the --salt-transport install option can be provided like:

pip install --install-option="--salt-transport=raet" salt

NOTE:
Salt does not bundle dependencies that are typically distributed as part of the base OS. If you have
unmet dependencies and are using a custom or minimal installation, you might need to install some
additional packages from your OS vendor.

Optional Dependencies
• mako - an optional parser for Salt States (configurable in the master settings)

• gcc - dynamic Cython module compiling

Upgrading Salt
When upgrading Salt, the master(s) should always be upgraded first. Backward compatibility for minions
running newer versions of salt than their masters is not guaranteed.

Whenever possible, backward compatibility between new masters and old minions will be preserved.
Generally, the only exception to this policy is in case of a security vulnerability.

TUTORIALS

Introduction
Salt Masterless Quickstart
Running a masterless salt-minion lets you use Salt's configuration management for a single machine
without calling out to a Salt master on another machine.

Since the Salt minion contains such extensive functionality it can be useful to run it standalone. A
standalone minion can be used to do a number of things:

• Stand up a master server via States (Salting a Salt Master)

• Use salt-call commands on a system without connectivity to a master

• Masterless States, run states entirely from files local to the minion

It is also useful for testing out state trees before deploying to a production setup.

Bootstrap Salt Minion
The salt-bootstrap script makes bootstrapping a server with Salt simple for any OS with a Bourne shell:

curl -L https://bootstrap.saltstack.com -o install_salt.sh
sudo sh install_salt.sh

See the salt-bootstrap documentation for other one liners. When using Vagrant to test out salt, the
Vagrant salt provisioner will provision the VM for you.

Telling Salt to Run Masterless
To instruct the minion to not look for a master, the file_client configuration option needs to be set in
the minion configuration file. By default the file_client is set to remote so that the minion gathers
file server and pillar data from the salt master. When setting the file_client option to local the
minion is configured to not gather this data from the master.

file_client: local

Now the salt minion will not look for a master and will assume that the local system has all of the file
and pillar resources.

NOTE:
When running Salt in masterless mode, do not run the salt-minion daemon. Otherwise, it will attempt
to connect to a master and fail. The salt-call command stands on its own and does not need the
salt-minion daemon.

Create State Tree
Following the successful installation of a salt-minion, the next step is to create a state tree, which is
where the SLS files that comprise the possible states of the minion are stored.

The following example walks through the steps necessary to create a state tree that ensures that the
server has the Apache webserver installed.

NOTE:
For a complete explanation on Salt States, see the tutorial.

1. Create the top.sls file:

/srv/salt/top.sls:

base:
'*':
- webserver

2. Create the webserver state tree:

/srv/salt/webserver.sls:

apache: # ID declaration
pkg: # state declaration
- installed # function declaration

NOTE:
The apache package has different names on different platforms, for instance on Debian/Ubuntu it is
apache2, on Fedora/RHEL it is httpd and on Arch it is apache

The only thing left is to provision our minion using salt-call and the highstate command.

Salt-call
The salt-call command is used to run module functions locally on a minion instead of executing them from
the master. Normally the salt-call command checks into the master to retrieve file server and pillar
data, but when running standalone salt-call needs to be instructed to not check the master for this data:

salt-call --local state.highstate

The --local flag tells the salt-minion to look for the state tree in the local file system and not to
contact a Salt Master for instructions.

To provide verbose output, use -l debug:

salt-call --local state.highstate -l debug

The minion first examines the top.sls file and determines that it is a part of the group matched by *
glob and that the webserver SLS should be applied.

It then examines the webserver.sls file and finds the apache state, which installs the Apache package.

The minion should now have Apache installed, and the next step is to begin learning how to write more
complex states.

Basics
Standalone Minion
Since the Salt minion contains such extensive functionality it can be useful to run it standalone. A
standalone minion can be used to do a number of things:

• Use salt-call commands on a system without connectivity to a master

• Masterless States, run states entirely from files local to the minion

Telling Salt Call to Run Masterless
The salt-call command is used to run module functions locally on a minion instead of executing them from
the master. Normally the salt-call command checks into the master to retrieve file server and pillar
data, but when running standalone salt-call needs to be instructed to not check the master for this data.
To instruct the minion to not look for a master when running salt-call the file_client configuration
option needs to be set. By default the file_client is set to remote so that the minion knows that file
server and pillar data are to be gathered from the master. When setting the file_client option to local
the minion is configured to not gather this data from the master.

file_client: local

Now the salt-call command will not look for a master and will assume that the local system has all of the
file and pillar resources.

Running States Masterless
The state system can be easily run without a Salt master, with all needed files local to the minion. To
do this the minion configuration file needs to be set up to know how to return file_roots information
like the master. The file_roots setting defaults to /srv/salt for the base environment just like on the
master:

file_roots:
base:
- /srv/salt

Now set up the Salt State Tree, top file, and SLS modules in the same way that they would be set up on a
master. Now, with the file_client option set to local and an available state tree then calls to functions
in the state module will use the information in the file_roots on the minion instead of checking in with
the master.

Remember that when creating a state tree on a minion there are no syntax or path changes needed, SLS
modules written to be used from a master do not need to be modified in any way to work with a minion.

This makes it easy to "script" deployments with Salt states without having to set up a master, and allows
for these SLS modules to be easily moved into a Salt master as the deployment grows.

The declared state can now be executed with:

salt-call state.highstate

Or the salt-call command can be executed with the --local flag, this makes it unnecessary to change the
configuration file:

salt-call state.highstate --local

External Pillars
External pillars are supported when running in masterless mode.

Opening the Firewall up for Salt
The Salt master communicates with the minions using an AES-encrypted ZeroMQ connection. These
communications are done over TCP ports 4505 and 4506, which need to be accessible on the master only.
This document outlines suggested firewall rules for allowing these incoming connections to the master.

NOTE:
No firewall configuration needs to be done on Salt minions. These changes refer to the master only.

Fedora 18 and beyond / RHEL 7 / CentOS 7
Starting with Fedora 18 FirewallD is the tool that is used to dynamically manage the firewall rules on a
host. It has support for IPv4/6 settings and the separation of runtime and permanent configurations. To
interact with FirewallD use the command line client firewall-cmd.

firewall-cmd example:

firewall-cmd --permanent --zone=<zone> --add-port=4505-4506/tcp

Please choose the desired zone according to your setup. Don't forget to reload after you made your
changes.

firewall-cmd --reload

RHEL 6 / CentOS 6
The lokkit command packaged with some Linux distributions makes opening iptables firewall ports very
simple via the command line. Just be careful to not lock out access to the server by neglecting to open
the ssh port.

lokkit example:

lokkit -p 22:tcp -p 4505:tcp -p 4506:tcp

The system-config-firewall-tui command provides a text-based interface to modifying the firewall.

system-config-firewall-tui:

system-config-firewall-tui

openSUSE
Salt installs firewall rules in /etc/sysconfig/SuSEfirewall2.d/services/salt. Enable with:

SuSEfirewall2 open
SuSEfirewall2 start

If you have an older package of Salt where the above configuration file is not included, the
SuSEfirewall2 command makes opening iptables firewall ports very simple via the command line.

SuSEfirewall example:

SuSEfirewall2 open EXT TCP 4505
SuSEfirewall2 open EXT TCP 4506

The firewall module in YaST2 provides a text-based interface to modifying the firewall.

YaST2:

yast2 firewall

iptables
Different Linux distributions store their iptables (also known as netfilter) rules in different places,
which makes it difficult to standardize firewall documentation. Included are some of the more common
locations, but your mileage may vary.

Fedora / RHEL / CentOS:

/etc/sysconfig/iptables

Arch Linux:

/etc/iptables/iptables.rules

Debian

Follow these instructions: https://wiki.debian.org/iptables

Once you've found your firewall rules, you'll need to add the two lines below to allow traffic on
tcp/4505 and tcp/4506:

-A INPUT -m state --state new -m tcp -p tcp --dport 4505 -j ACCEPT
-A INPUT -m state --state new -m tcp -p tcp --dport 4506 -j ACCEPT

Ubuntu

Salt installs firewall rules in /etc/ufw/applications.d/salt.ufw. Enable with:

ufw allow salt

pf.conf
The BSD-family of operating systems uses packet filter (pf). The following example describes the
additions to pf.conf needed to access the Salt master.

pass in on $int_if proto tcp from any to $int_if port 4505
pass in on $int_if proto tcp from any to $int_if port 4506

Once these additions have been made to the pf.conf the rules will need to be reloaded. This can be done
using the pfctl command.

pfctl -vf /etc/pf.conf

Whitelist communication to Master
There are situations where you want to selectively allow Minion traffic from specific hosts or networks
into your Salt Master. The first scenario which comes to mind is to prevent unwanted traffic to your
Master out of security concerns, but another scenario is to handle Minion upgrades when there are
backwards incompatible changes between the installed Salt versions in your environment.

Here is an example Linux iptables ruleset to be set on the Master:

# Allow Minions from these networks
-I INPUT -s 10.1.2.0/24 -p tcp -m multiport --dports 4505,4506 -j ACCEPT
-I INPUT -s 10.1.3.0/24 -p tcp -m multiport --dports 4505,4506 -j ACCEPT
# Allow Salt to communicate with Master on the loopback interface
-A INPUT -i lo -p tcp -m multiport --dports 4505,4506 -j ACCEPT
# Reject everything else
-A INPUT -p tcp -m multiport --dports 4505,4506 -j REJECT

NOTE:
The important thing to note here is that the salt command needs to communicate with the listening
network socket of salt-master on the loopback interface. Without this you will see no outgoing Salt
traffic from the master, even for a simple salt '*' test.ping, because the salt client never reached
the salt-master to tell it to carry out the execution.

Using cron with Salt
The Salt Minion can initiate its own highstate using the salt-call command.

$ salt-call state.highstate

This will cause the minion to check in with the master and ensure it is in the correct 'state'.

Use cron to initiate a highstate
If you would like the Salt Minion to regularly check in with the master you can use the venerable cron to
run the salt-call command.

# PATH=/bin:/sbin:/usr/bin:/usr/sbin

00 00 * * * salt-call state.highstate

The above cron entry will run a highstate every day at midnight.

NOTE:
Be aware that you may need to ensure the PATH for cron includes any scripts or commands that need to
be executed.

Remote execution tutorial
Before continuing make sure you have a working Salt installation by following the installation and the
configuration instructions.

Stuck?

There are many ways to get help from the Salt community including our mailing list and our IRC
channel #salt.

Order your minions around
Now that you have a master and at least one minion communicating with each other you can perform commands
on the minion via the salt command. Salt calls are comprised of three main components:

salt '<target>' <function> [arguments]

TARGETING MINIONS

       Targeting minions is specifying which minions should run a command or execute a state by matching against
       hostnames, or system information, or defined groups, or even combinations thereof.

       For example the command salt web1 apache.signal restart to restart the Apache httpd server specifies  the
       machine web1 as the target and the command will only be run on that one minion.

       Similarly  when  using  States, the following top file specifies that only the web1 minion should execute
       the contents of webserver.sls:

          base:
            'web1':
              - webserver

       There are many ways to target individual minions or groups of minions in Salt:

   Matching the minion id
       Each minion needs a unique identifier. By default when a minion starts for the first time it chooses  its
       FQDN as that identifier. The minion id can be overridden via the minion's id configuration setting.

       TIP:
          minion id and minion keys

          The  minion  id is used to generate the minion's public/private keys and if it ever changes the master
          must then accept the new key as though the minion was a new host.

   Globbing
       The default matching that Salt utilizes is shell-style globbing around the minion id. This also works for
       states in the top file.

       NOTE:
          You  must  wrap  salt calls that use globbing in single-quotes to prevent the shell from expanding the
          globs before Salt is invoked.

       Match all minions:

          salt '*' test.ping

       Match all minions in the example.net domain or any of the example domains:

          salt '*.example.net' test.ping
          salt '*.example.*' test.ping

       Match  all  the  webN  minions  in  the  example.net   domain   (web1.example.net,   web2.example.net   …
       webN.example.net):

          salt 'web?.example.net' test.ping

       Match the web1 through web5 minions:

          salt 'web[1-5]' test.ping

       Match the web1 and web3 minions:

          salt 'web[1,3]' test.ping

       Match the web-x, web-y, and web-z minions:

          salt 'web-[x-z]' test.ping

       NOTE:
          For additional targeting methods please review the compound matchers documentation.

   Regular Expressions
       Minions can be matched using Perl-compatible regular expressions (which is globbing on steroids and a ton
       of caffeine).

       Match both web1-prod and web1-devel minions:

          salt -E 'web1-(prod|devel)' test.ping

       When using regular expressions in a State's top file, you must specify the matcher as the  first  option.
       The following example executes the contents of webserver.sls on the above-mentioned minions.

          base:
            'web1-(prod|devel)':
            - match: pcre
            - webserver

   Lists
       At the most basic level, you can specify a flat list of minion IDs:

          salt -L 'web1,web2,web3' test.ping

   Grains
       Salt  comes  with  an  interface  to  derive information about the underlying system.  This is called the
       grains interface, because it presents salt with grains of  information.  Grains  are  collected  for  the
       operating system, domain name, IP address, kernel, OS type, memory, and many other system properties.

       The  grains  interface  is  made  available  to Salt modules and components so that the right salt minion
       commands are automatically available on the right systems.

       Grain data is relatively static, though if system information changes (for example, if  network  settings
       are changed), or if a new value is assigned to a custom grain, grain data is refreshed.

       NOTE:
          Grains resolve to lowercase letters. For example, FOO, and foo target the same grain.

       IMPORTANT:
          See Is Targeting using Grain Data Secure? for important security information.

       Match all CentOS minions:

          salt -G 'os:CentOS' test.ping

       Match all minions with 64-bit CPUs, and return number of CPU cores for each matching minion:

          salt -G 'cpuarch:x86_64' grains.item num_cpus

       Additionally,  globs  can  be  used  in  grain matches, and grains that are nested in a dictionary can be
       matched by adding a colon for each level that is traversed. For example, the following will  match  hosts
       that have a grain called ec2_tags, which itself is a dict with a key named environment, which has a value
       that contains the word production:

          salt -G 'ec2_tags:environment:*production*'

   Listing Grains
       Available grains can be listed by using the 'grains.ls' module:

          salt '*' grains.ls

       Grains data can be listed by using the 'grains.items' module:

          salt '*' grains.items

   Grains in the Minion Config
       Grains can also be statically assigned within the minion configuration file.  Just add the option  grains
       and pass options to it:

          grains:
            roles:
              - webserver
              - memcache
            deployment: datacenter4
            cabinet: 13
            cab_u: 14-15

       Then  status  data specific to your servers can be retrieved via Salt, or used inside of the State system
       for matching. It also makes targeting, in the case of the example above, simply based  on  specific  data
       about your deployment.

   Grains in /etc/salt/grains
       If you do not want to place your custom static grains in the minion config file, you can also put them in
       /etc/salt/grains on the minion. They are configured in the same way as in the above example, only without
       a top-level grains: key:

          roles:
            - webserver
            - memcache
          deployment: datacenter4
          cabinet: 13
          cab_u: 14-15

   Matching Grains in the Top File
       With  correctly  configured  grains on the Minion, the top file used in Pillar or during Highstate can be
       made very efficient. For example, consider the following configuration:

          'node_type:web':
            - match: grain
            - webserver

          'node_type:postgres':
            - match: grain
            - database

          'node_type:redis':
            - match: grain
            - redis

          'node_type:lb':
            - match: grain
            - lb

       For this example to work, you would need to have defined the grain node_type for the minions you wish  to
       match.  This  simple  example is nice, but too much of the code is similar. To go one step further, Jinja
       templating can be used to simplify the top file.

          {% set the_node_type = salt['grains.get']('node_type', '') %}

          {% if the_node_type %}
            'node_type:{{ the_node_type }}':
              - match: grain
              - {{ the_node_type }}
          {% endif %}

       Using Jinja templating, only one match entry needs to be defined.

       NOTE:
          The example above uses the grains.get function to account for minions which do not have the  node_type
          grain set.

   Writing Grains
       The  grains  interface is derived by executing all of the "public" functions found in the modules located
       in the grains package or the custom grains directory. The functions in the modules  of  the  grains  must
       return  a  Python  dict,  where  the  keys in the dict are the names of the grains and the values are the
       values.

       Custom grains should be placed in a _grains directory located  under  the  file_roots  specified  by  the
       master  config  file.  The default path would be /srv/salt/_grains.  Custom grains will be distributed to
       the minions when state.highstate is run, or by executing the  saltutil.sync_grains  or  saltutil.sync_all
       functions.

       Grains  are  easy  to  write,  and  only  need  to  return a dictionary.  A common approach would be code
       something similar to the following:

          #!/usr/bin/env python
          def yourfunction():
               # initialize a grains dictionary
               grains = {}
               # Some code for logic that sets grains like
               grains['yourcustomgrain'] = True
               grains['anothergrain'] = 'somevalue'
               return grains

       Before adding a grain to Salt, consider what the grain is and remember that  grains  need  to  be  static
       data. If the data is something that is likely to change, consider using Pillar instead.

       WARNING:
          Custom  grains  will  not be available in the top file until after the first highstate. To make custom
          grains available on a minion's first highstate, it is recommended to use this example to  ensure  that
          the custom grains are synced when the minion starts.

   Loading Custom Grains
       If you have multiple functions specifying grains that are called from a main function, be sure to prepend
       grain function names with an underscore. This prevents Salt from including the  loaded  grains  from  the
       grain functions in the final grain data structure. For example, consider this custom grain file:

          #!/usr/bin/env python
          def _my_custom_grain():
              my_grain = {'foo': 'bar', 'hello': 'world'}
              return my_grain

          def main():
              # initialize a grains dictionary
              grains = {}
              grains['my_grains'] = _my_custom_grain()
              return grains

       The output of this example renders like so:

          # salt-call --local grains.items
          local:
              ----------
              <Snipped for brevity>
              my_grains:
                  ----------
                  foo:
                      bar
                  hello:
                      world

       However,  if  you  don't  prepend  the  my_custom_grain function with an underscore, the function will be
       rendered twice by Salt in the items output: once for the my_custom_grain call itself, and again  when  it
       is called in the main function:

          # salt-call --local grains.items
          local:
          ----------
              <Snipped for brevity>
              foo:
                  bar
              <Snipped for brevity>
              hello:
                  world
              <Snipped for brevity>
              my_grains:
                  ----------
                  foo:
                      bar
                  hello:
                      world

   Precedence
       Core  grains  can  be  overridden  by custom grains. As there are several ways of defining custom grains,
       there is an order of precedence which should be kept in mind when defining them. The order of  evaluation
       is as follows:

       1. Core grains.

       2. Custom grains in /etc/salt/grains.

       3. Custom grains in /etc/salt/minion.

       4. Custom grain modules in _grains directory, synced to minions.

       Each  successive  evaluation  overrides the previous ones, so any grains defined by custom grains modules
       synced to minions that have the same name as a core grain  will  override  that  core  grain.  Similarly,
       grains  from  /etc/salt/minion  override both core grains and custom grain modules, and grains in _grains
       will override any grains of the same name.

   Examples of Grains
       The core module in the grains package is where the main grains are loaded by the Salt minion and provides
       the principal example of how to write grains:

       https://github.com/saltstack/salt/blob/develop/salt/grains/core.py

   Syncing Grains
       Syncing  grains  can  be  done  a  number  of ways, they are automatically synced when state.highstate is
       called,  or  (as  noted  above)  the  grains  can  be  manually  synced  and  reloaded  by  calling   the
       saltutil.sync_grains or saltutil.sync_all functions.

   Targeting with Pillar
       Pillar  data  can  be  used when targeting minions. This allows for ultimate control and flexibility when
       targeting minions.

          salt -I 'somekey:specialvalue' test.ping

       Like with Grains, it is possible to use globbing as well as match nested  values  in  Pillar,  by  adding
       colons  for each level that is being traversed. The below example would match minions with a pillar named
       foo, which is a dict containing a key bar, with a value beginning with baz:

          salt -I 'foo:bar:baz*' test.ping

   Subnet/IP Address Matching
       Minions can easily be matched based on IP address, or by subnet (using CIDR notation).

          salt -S 192.168.40.20 test.ping
          salt -S 10.0.0.0/24 test.ping

       Ipcidr matching can also be used in compound matches

          salt -C 'S@10.0.0.0/24 and G@os:Debian' test.ping

       It is also possible to use in both pillar and state-matching

          '172.16.0.0/12':
             - match: ipcidr
             - internal

       NOTE:
          Only IPv4 matching is supported at this time.

   Compound matchers
       Compound matchers allow very granular minion targeting using any of Salt's matchers. The default  matcher
       is  a  glob  match,  just  as  with CLI and top file matching. To match using anything other than a glob,
       prefix the match string with the appropriate letter from the table below, followed by an @ sign.

               ┌───────┬───────────────────┬──────────────────────────────────────────┬────────────────┐
               │Letter │ Match Type        │ Example                                  │ Alt Delimiter? │
               ├───────┼───────────────────┼──────────────────────────────────────────┼────────────────┤
               │G      │ Grains glob       │ G@os:Ubuntu                              │ Yes            │
               ├───────┼───────────────────┼──────────────────────────────────────────┼────────────────┤
               │E      │ PCRE Minion ID    │ E@web\d+\.(dev|qa|prod)\.loc             │ No             │
               ├───────┼───────────────────┼──────────────────────────────────────────┼────────────────┤
               │P      │ Grains PCRE       │ P@os:(RedHat|Fedora|CentOS)              │ Yes            │
               ├───────┼───────────────────┼──────────────────────────────────────────┼────────────────┤
               │L      │ List of minions   │ L@minion1.example.com,minion3.domain.com │ No             │
               │       │                   │ or bl*.domain.com                        │                │
               ├───────┼───────────────────┼──────────────────────────────────────────┼────────────────┤
               │I      │ Pillar glob       │ I@pdata:foobar                           │ Yes            │
               ├───────┼───────────────────┼──────────────────────────────────────────┼────────────────┤
               │J      │ Pillar PCRE       │ J@pdata:^(foo|bar)$                      │ Yes            │
               ├───────┼───────────────────┼──────────────────────────────────────────┼────────────────┤
               │S      │ Subnet/IP address │ S@192.168.1.0/24 or S@192.168.1.100      │ No             │
               ├───────┼───────────────────┼──────────────────────────────────────────┼────────────────┤
               │R      │ Range cluster     │ R@%foo.bar                               │ No             │
               └───────┴───────────────────┴──────────────────────────────────────────┴────────────────┘

       Matchers can be joined using boolean and, or, and not operators.

       For example, the following string matches all Debian minions with a hostname that begins with webserv, as
       well as any minions that have a hostname which matches the regular expression web-dc1-srv.*:

          salt -C 'webserv* and G@os:Debian or E@web-dc1-srv.*' test.ping

       That same example expressed in a top file looks like the following:

          base:
            'webserv* and G@os:Debian or E@web-dc1-srv.*':
              - match: compound
              - webserver

       New in version 2015.8.0.

       Excluding a minion based on its ID is also possible:

          salt -C 'not web-dc1-srv' test.ping

       Versions  prior  to 2015.8.0 a leading not was not supported in compound matches. Instead, something like
       the following was required:

          salt -C '* and not G@kernel:Darwin' test.ping

       Excluding a minion based on its ID was also possible:

          salt -C '* and not web-dc1-srv' test.ping

   Precedence Matching
       Matchers can be grouped together with parentheses to explicitly declare precedence amongst groups.

          salt -C '( ms-1 or G@id:ms-3 ) and G@id:ms-3' test.ping

       NOTE:
          Be certain to note that spaces are required between the parentheses and targets. Failing to obey  this
          rule may result in incorrect targeting!

   Alternate Delimiters
       New in version 2015.8.0.

       Matchers that target based on a key value pair use a colon (:) as a delimiter. Matchers with a Yes in the
       Alt Delimiters column in the previous table support specifying an alternate delimiter character.

       This is done by specifying an alternate delimiter character between the leading matcher character and the
       @  pattern separator character. This avoids incorrect interpretation of the pattern in the case that : is
       part of the grain or pillar data structure traversal.

          salt -C 'J|@foo|bar|^foo:bar$ or J!@gitrepo!https://github.com:example/project.git' test.ping

   Node groups
       Nodegroups are declared using a compound target specification. The compound target documentation  can  be
       found here.

       The  nodegroups  master  config  file parameter is used to define nodegroups. Here's an example nodegroup
       configuration within /etc/salt/master:

          nodegroups:
            group1: 'L@foo.domain.com,bar.domain.com,baz.domain.com or bl*.domain.com'
            group2: 'G@os:Debian and foo.domain.com'
            group3: 'G@os:Debian and N@group1'
            group4:
              - 'G@foo:bar'
              - 'or'
              - 'G@foo:baz'

       NOTE:
          The L within group1 is matching a list of minions, while the G in group2 is matching specific  grains.
          See the compound matchers documentation for more details.

       New in version 2015.8.0.

       NOTE:
          Nodgroups  can  reference  other  nodegroups  as seen in group3.  Ensure that you do not have circular
          references.  Circular references will be detected and cause partial  expansion  with  a  logged  error
          message.

       New in version 2015.8.0.

       Compound  nodegroups  can  be  either  string  values or lists of string values.  When the nodegroup is A
       string value will be tokenized by splitting on whitespace.  This  may  be  a  problem  if  whitespace  is
       necessary  as part of a pattern.  When a nodegroup is a list of strings then tokenization will happen for
       each list element as a whole.

       To match a nodegroup on the CLI, use the -N command-line option:

          salt -N group1 test.ping

       NOTE:
          The N@ classifier cannot be used in compound mathes within the CLI or top file, it is only  recognized
          in the nodegroups master config file parameter.

       To match a nodegroup in your top file, make sure to put - match: nodegroup on the line directly following
       the nodegroup name.

          base:
            group1:
              - match: nodegroup
              - webserver

       NOTE:
          When adding or modifying nodegroups to a master configuration file, the master must be  restarted  for
          those changes to be fully recognized.

          A  limited  amount  of functionality, such as targeting with -N from the command-line may be available
          without a restart.

   Using Nodegroups in SLS files
       To use Nodegroups in Jinja logic for SLS files, the pillar_opts option in /etc/salt/master must be set to
       "True". This will pass the master's configuration as Pillar data to each minion.

       NOTE:
          If the master's configuration contains any sensitive data, this will be passed to each minion.  Do not
          enable this option if you have any configuration data that you do not want to get on your minions.

          Also, if you make changes to your nodegroups, you might need to run salt  '*'  saltutil.refresh_pillar
          after restarting the master.

       Once  pillar_opts  is  enabled, you can find the nodegroups under the "master" pillar.  To make sure that
       only the correct minions are targeted, you should use each matcher for  the  nodegroup  definition.   For
       example, to check if a minion is in the 'webserver' nodegroup:

          nodegroups:
            webserver: 'G@os:Debian and L@minion1,minion2'

          {% if grains.id in salt['pillar.get']('master:nodegroups:webserver', [])
          and grains.os in salt['pillar.get']('master:nodegroups:webserver', []) %}
          ...
          {% endif %}

       NOTE:
          If  you  do  not include all of the matchers used to define a nodegroup, Salt might incorrectly target
          minions that meet some of the nodegroup requirements, but not all of them.

   Batch Size
       The -b (or --batch-size) option allows commands to be executed on only a specified number of minions at a
       time. Both percentages and finite numbers are supported.

          salt '*' -b 10 test.ping

          salt -G 'os:RedHat' --batch-size 25% apache.signal restart

       This  will  only  run test.ping on 10 of the targeted minions at a time and then restart apache on 25% of
       the minions matching os:RedHat at a time and work through them all until the task is complete. This makes
       jobs  like rolling web server restarts behind a load balancer or doing maintenance on BSD firewalls using
       carp much easier with salt.

       The batch system maintains a window of running minions, so, if there are a total of 150 minions  targeted
       and  the  batch  size  is  10,  then  the command is sent to 10 minions, when one minion returns then the
       command is sent to one additional minion, so that the job is constantly running on 10 minions.

   SECO Range
       SECO range is a cluster-based metadata store developed and maintained by Yahoo!

       The Range project is hosted here:

       https://github.com/ytoolshed/range

       Learn more about range here:

       https://github.com/ytoolshed/range/wiki/

   Prerequisites
       To utilize range support in Salt, a range server is required. Setting up a range server  is  outside  the
       scope of this document. Apache modules are included in the range distribution.

       With  a  working  range server, cluster files must be defined. These files are written in YAML and define
       hosts contained inside a cluster. Full documentation on writing YAML range files is here:

       https://github.com/ytoolshed/range/wiki/%22yamlfile%22-module-file-spec

       Additionally, the Python seco range libraries must be installed on the salt master. One can  verify  that
       they have been installed correctly via the following command:

          python -c 'import seco.range'

       If no errors are returned, range is installed successfully on the salt master.

   Preparing Salt
       Range  support  must  be  enabled on the salt master by setting the hostname and port of the range server
       inside the master configuration file:

          range_server: my.range.server.com:80

       Following this, the master must be restarted for the change to have an effect.

   Targeting with Range
       Once a cluster has been defined, it can be targeted with a salt command by using the -R or --range flags.

       For example, given the following range YAML file being served from a range server:

          $ cat /etc/range/test.yaml
          CLUSTER: host1..100.test.com
          APPS:
            - frontend
            - backend
            - mysql

       One might target host1 through host100 in the test.com domain with Salt as follows:

          salt --range %test:CLUSTER test.ping

       The following salt command would target three hosts: frontend, backend, and mysql:

          salt --range %test:APPS test.ping

STORING STATIC DATA IN THE PILLAR

       Pillar is an interface for Salt designed to offer global values that can be distributed to  all  minions.
       Pillar data is managed in a similar way as the Salt State Tree.

       Pillar was added to Salt in version 0.9.8

       NOTE:
          Storing sensitive data

          Unlike  state  tree,  pillar  data  is only available for the targeted minion specified by the matcher
          type.  This makes it useful for storing sensitive data specific to a particular minion.

   Declaring the Master Pillar
       The Salt Master server maintains a pillar_roots setup that matches the structure of the  file_roots  used
       in  the Salt file server. Like the Salt file server the pillar_roots option in the master config is based
       on environments mapping to directories. The pillar data is then mapped to minions based on matchers in  a
       top  file  which is laid out in the same way as the state top file. Salt pillars can use the same matcher
       types as the standard top file.

       The configuration for the pillar_roots in the master config file is identical in behavior and function as
       file_roots:

          pillar_roots:
            base:
              - /srv/pillar

       This  example  configuration  declares  that  the  base  environment  will  be located in the /srv/pillar
       directory. It must not be in a subdirectory of the state tree.

       The top file used matches the name of the top file used for States, and has the same structure:

       /srv/pillar/top.sls

          base:
            '*':
              - packages

       In the above top file, it is declared that in the base environment, the glob matching  all  minions  will
       have  the  pillar  data  found in the packages pillar available to it. Assuming the pillar_roots value of
       /srv/pillar taken from above, the packages pillar would be located at /srv/pillar/packages.sls.

       Any number of matchers can be added to the base environment. For example, here is an expanded version  of
       the Pillar top file stated above:

       /srv/pillar/top.sls:

          base:
            '*':
              - packages
            'web*':
              - vim

       In this expanded top file, minions that match web* will have access to the /srv/pillar/pacakges.sls file,
       as well as the /srv/pillar/vim.sls file.

       Another example shows how to use other standard top matching types to deliver specific salt  pillar  data
       to minions with different properties.

       Here is an example using the grains matcher to target pillars to minions by their os grain:

          dev:
            'os:Debian':
              - match: grain
              - servers

       /srv/pillar/packages.sls

          {% if grains['os'] == 'RedHat' %}
          apache: httpd
          git: git
          {% elif grains['os'] == 'Debian' %}
          apache: apache2
          git: git-core
          {% endif %}

          company: Foo Industries

       IMPORTANT:
          See Is Targeting using Grain Data Secure? for important security information.

       The  above  pillar sets two key/value pairs. If a minion is running RedHat, then the apache key is set to
       httpd and the git key is set to the value of git. If the minion  is  running  Debian,  those  values  are
       changed  to  apache2 and git-core respectively. All minions that have this pillar targeting to them via a
       top file will have the key of company with a value of Foo Industries.

       Consequently this data can be used from within modules, renderers, State SLS  files,  and  more  via  the
       shared pillar dict:

          apache:
            pkg.installed:
              - name: {{ pillar['apache'] }}

          git:
            pkg.installed:
              - name: {{ pillar['git'] }}

       Finally, the above states can utilize the values provided to them via Pillar.  All pillar values targeted
       to a minion are available via the 'pillar' dictionary. As seen in the above example,  Jinja  substitution
       can then be utilized to access the keys and values in the Pillar dictionary.

       Note  that  you  cannot  just list key/value-information in top.sls. Instead, target a minion to a pillar
       file and then list the keys and values in the pillar. Here is an example top file that  illustrates  this
       point:

          base:
            '*':
               - common_pillar

       And the actual pillar file at '/srv/pillar/common_pillar.sls':

          foo: bar
          boo: baz

   Pillar namespace flattened
       The separate pillar files all share the same namespace. Given a top.sls of:

          base:
            '*':
              - packages
              - services

       a packages.sls file of:

          bind: bind9

       and a services.sls file of:

          bind: named

       Then a request for the bind pillar will only return named; the bind9 value is not available. It is better
       to structure your pillar files with more hierarchy. For example your package.sls file could look like:

          packages:
            bind: bind9

   Pillar Namespace Merges
       With some care, the pillar namespace can merge content from multiple pillar files under a single key,  so
       long as conflicts are avoided as described above.

       For example, if the above example were modified as follows, the values are merged below a single key:

          base:
            '*':
              - packages
              - services

       And a packages.sls file like:

          bind:
            package-name: bind9
            version: 9.9.5

       And a services.sls file like:

          bind:
            port: 53
            listen-on: any

       The resulting pillar will be as follows:

          $ salt-call pillar.get bind
          local:
              ----------
              listen-on:
                  any
              package-name:
                  bind9
              port:
                  53
              version:
                  9.9.5

       NOTE:
          Pillar  files  are  applied  in the order they are listed in the top file.  Therefore conflicting keys
          will be overwritten in a 'last one wins' manner!  For example, in the above scenario  conflicting  key
          values in services will overwrite those in packages because it's at the bottom of the list.

   Including Other Pillars
       New in version 0.16.0.

       Pillar  SLS  files may include other pillar files, similar to State files. Two syntaxes are available for
       this purpose. The simple form simply includes the additional pillar as if it were part of the same file:

          include:
            - users

       The full include form allows two additional options -- passing default values to  the  templating  engine
       for  the  included pillar file as well as an optional key under which to nest the results of the included
       pillar:

          include:
            - users:
                defaults:
                    sudo: ['bob', 'paul']
                key: users

       With this form, the included file (users.sls) will be nested within  the  'users'  key  of  the  compiled
       pillar. Additionally, the 'sudo' value will be available as a template variable to users.sls.

   Viewing Minion Pillar
       Once  the  pillar is set up the data can be viewed on the minion via the pillar module, the pillar module
       comes with functions, pillar.items and pillar.raw.  pillar.items will return a  freshly  reloaded  pillar
       and pillar.raw will return the current pillar without a refresh:

          salt '*' pillar.items

       NOTE:
          Prior to version 0.16.2, this function is named pillar.data. This function name is still supported for
          backwards compatibility.

   Pillar get Function
       New in version 0.14.0.

       The pillar.get function works much in the same way as the get method  in  a  python  dict,  but  with  an
       enhancement: nested dict components can be extracted using a : delimiter.

       If a structure like this is in pillar:

          foo:
            bar:
              baz: qux

       Extracting it from the raw pillar in an sls formula or file template is done this way:

          {{ pillar['foo']['bar']['baz'] }}

       Now,  with the new pillar.get function the data can be safely gathered and a default can be set, allowing
       the template to fall back if the value is not available:

          {{ salt['pillar.get']('foo:bar:baz', 'qux') }}

       This makes handling nested structures much easier.

       NOTE:
          pillar.get() vs salt['pillar.get']()

          It should be noted that within templating, the pillar variable is just a dictionary.  This means  that
          calling  pillar.get()  inside of a template will just use the default dictionary .get() function which
          does not include the extra : delimiter functionality.  It  must  be  called  using  the  above  syntax
          (salt['pillar.get']('foo:bar:baz', 'qux')) to get the salt function, instead of the default dictionary
          behavior.

   Refreshing Pillar Data
       When pillar data is changed on the master the minions need to refresh the data locally. This is done with
       the saltutil.refresh_pillar function.

          salt '*' saltutil.refresh_pillar

       This function triggers the minion to asynchronously refresh the pillar and will always return None.

   Set Pillar Data at the Command Line
       Pillar data can be set at the command line like the following example:

          salt '*' state.highstate pillar='{"cheese": "spam"}'

       This will create a dict with a key of 'cheese' and a value of 'spam'. A list can be created like this:

          salt '*' state.highstate pillar='["cheese", "milk", "bread"]'

       NOTE:
          Be  aware  that  when  sending  sensitive  data  via  pillar  on the command-line that the publication
          containing that data will be received by all minions and  will  not  be  restricted  to  the  targeted
          minions. This may represent a security concern in some cases.

   Master Config In Pillar
       For  convenience  the  data stored in the master configuration file can be made available in all minion's
       pillars. This makes global configuration of services and systems very easy but  may  not  be  desired  if
       sensitive data is stored in the master configuration. This option is disabled by default.

       To enable the master config from being added to the pillar set pillar_opts to True:

          pillar_opts: True

   Minion Config in Pillar
       Minion  configuration options can be set on pillars. Any option that you want to modify, should be in the
       first level of the pillars, in the same way you set the options in  the  config  file.  For  example,  to
       configure  the  MySQL  root  password to be used by MySQL Salt execution module, set the following pillar
       variable:

          mysql.pass: hardtoguesspassword

   Master Provided Pillar Error
       By default if there is an error rendering a pillar, the detailed error is hidden and replaced with:

          Rendering SLS 'my.sls' failed. Please see master log for details.

       The error is protected because it's possible to contain templating data  which  would  give  that  minion
       information it shouldn't know, like a password!

       To  have  the  master  provide  the  detailed  error  that  could  potentially  carry  protected data set
       pillar_safe_render_error to False:

          pillar_safe_render_error: False

REACTOR SYSTEM

       Salt version 0.11.0 introduced the reactor system. The premise behind the reactor  system  is  that  with
       Salt's  events  and the ability to execute commands, a logic engine could be put in place to allow events
       to trigger actions, or more accurately, reactions.

       This system binds sls files to event tags on the master. These sls  files  then  define  reactions.  This
       means  that  the  reactor  system  has two parts. First, the reactor option needs to be set in the master
       configuration file.  The reactor option allows for event tags to be associated with sls  reaction  files.
       Second, these reaction files use highdata (like the state system) to define reactions to be executed.

   Event System
       A  basic  understanding  of  the  event system is required to understand reactors.  The event system is a
       local ZeroMQ PUB interface which fires salt events. This event bus is an open  system  used  for  sending
       information notifying Salt and other systems about operations.

       The  event system fires events with a very specific criteria. Every event has a tag. Event tags allow for
       fast top level filtering of events. In addition to the tag, each event has a data  structure.  This  data
       structure is a dict, which contains information about the event.

   Mapping Events to Reactor SLS Files
       Reactor  SLS  files  and  event  tags  are  associated  in  the  master  config file.  By default this is
       /etc/salt/master, or /etc/salt/master.d/reactor.conf.

       New in version 2014.7.0: Added Reactor support for salt:// file paths.

       In the master config section 'reactor:' is a list of event tags to be matched and each event  tag  has  a
       list of reactor SLS files to be run.

          reactor:                            # Master config section "reactor"

            - 'salt/minion/*/start':          # Match tag "salt/minion/*/start"
              - /srv/reactor/start.sls        # Things to do when a minion starts
              - /srv/reactor/monitor.sls      # Other things to do

            - 'salt/cloud/*/destroyed':       # Globs can be used to matching tags
              - /srv/reactor/destroy/*.sls    # Globs can be used to match file names

            - 'myco/custom/event/tag':        # React to custom event tags
              - salt://reactor/mycustom.sls   # Put reactor files under file_roots

       Reactor  sls files are similar to state and pillar sls files.  They are by default yaml + Jinja templates
       and are passed familiar context variables.

       They differ because of the addition of the tag and data variables.

       • The tag variable is just the tag in the fired event.

       • The data variable is the event's data dict.

       Here is a simple reactor sls:

          {% if data['id'] == 'mysql1' %}
          highstate_run:
            local.state.highstate:
              - tgt: mysql1
          {% endif %}

       This simple reactor file uses Jinja to further refine the reaction to be made.  If the id  in  the  event
       data  is  mysql1  (in  other  words,  if the name of the minion is mysql1) then the following reaction is
       defined.  The same data structure and compiler used for the state system is used for the reactor  system.
       The  only  difference  is  that the data is matched up to the salt command API and the runner system.  In
       this example, a command is published to the mysql1 minion with a function of state.highstate.  Similarly,
       a runner can be called:

          {% if data['data']['orchestrate'] == 'refresh' %}
          orchestrate_run:
            runner.state.orchestrate
          {% endif %}

       This example will execute the state.orchestrate runner and initiate an orchestrate execution.

   Fire an event
       To fire an event from a minion call event.send

          salt-call event.send 'foo' '{orchestrate: refresh}'

       After   this   is   called,  any  reactor  sls  files  matching  event  tag  foo  will  execute  with  {{
       data['data']['orchestrate'] }} equal to 'refresh'.

       See salt.modules.event for more information.

   Knowing what event is being fired
       The best way to see exactly what events are fired and what data is available in each event is to use  the
       state.event runner.

       SEE ALSO:
          Common Salt Events

       Example usage:

          salt-run state.event pretty=True

       Example output:

          salt/job/20150213001905721678/new       {
              "_stamp": "2015-02-13T00:19:05.724583",
              "arg": [],
              "fun": "test.ping",
              "jid": "20150213001905721678",
              "minions": [
                  "jerry"
              ],
              "tgt": "*",
              "tgt_type": "glob",
              "user": "root"
          }
          salt/job/20150213001910749506/ret/jerry {
              "_stamp": "2015-02-13T00:19:11.136730",
              "cmd": "_return",
              "fun": "saltutil.find_job",
              "fun_args": [
                  "20150213001905721678"
              ],
              "id": "jerry",
              "jid": "20150213001910749506",
              "retcode": 0,
              "return": {},
              "success": true
          }

   Debugging the Reactor
       The  best  window into the Reactor is to run the master in the foreground with debug logging enabled. The
       output will include when the master sees the event, what the master does in response to that  event,  and
       it will also include the rendered SLS file (or any errors generated while rendering the SLS file).

       1. Stop the master.

       2. Start the master manually:

             salt-master -l debug

       3. Look for log entries in the form:

             [DEBUG   ] Gathering reactors for tag foo/bar
             [DEBUG   ] Compiling reactions for tag foo/bar
             [DEBUG   ] Rendered data from file: /path/to/the/reactor_file.sls:
             <... Rendered output appears here. ...>

          The  rendered  output  is  the  result  of  the  Jinja parsing and is a good way to view the result of
          referencing Jinja variables. If the result is empty then  Jinja  produced  an  empty  result  and  the
          Reactor will ignore it.

   Understanding the Structure of Reactor Formulas
       I.e., when to use `arg` and `kwarg` and when to specify the function arguments directly.

       While  the reactor system uses the same basic data structure as the state system, the functions that will
       be called using that data structure are different functions than are called via Salt's state system.  The
       Reactor  can  call  Runner modules using the runner prefix, Wheel modules using the wheel prefix, and can
       also cause minions to run Execution modules using the local prefix.

       Changed in version 2014.7.0: The cmd prefix was renamed to local for  consistency  with  other  parts  of
       Salt. A backward-compatible alias was added for cmd.

       The  Reactor  runs  on the master and calls functions that exist on the master. In the case of Runner and
       Wheel functions the Reactor can just call those functions directly since they exist on the master and are
       run on the master.

       In  the case of functions that exist on minions and are run on minions, the Reactor still needs to call a
       function on the master in order to send the necessary data to the minion so the minion can  execute  that
       function.

       The Reactor calls functions exposed in Salt's Python API documentation. and thus the structure of Reactor
       files very transparently reflects the function signatures of those functions.

   Calling Execution modules on Minions
       The Reactor sends commands down to minions in the exact same way Salt's CLI interface does.  It  calls  a
       function locally on the master that sends the name of the function as well as a list of any arguments and
       a dictionary of any keyword arguments that the minion should use to execute that function.

       Specifically, the Reactor calls the async version of this function. You can see that function  has  'arg'
       and 'kwarg' parameters which are both values that are sent down to the minion.

       Executing  remote  commands  maps  to  the  LocalClient interface which is used by the salt command. This
       interface more specifically maps to the cmd_async method inside of the LocalClient class. This means that
       the arguments passed are being passed to the cmd_async method, not the remote method. A field starts with
       local to use the LocalClient subsystem. The result is, to execute a remote  command,  a  reactor  formula
       would look like this:

          clean_tmp:
            local.cmd.run:
              - tgt: '*'
              - arg:
                - rm -rf /tmp/*

       The  arg  option  takes  a list of arguments as they would be presented on the command line, so the above
       declaration is the same as running this salt command:

          salt '*' cmd.run 'rm -rf /tmp/*'

       Use the expr_form argument to specify a matcher:

          clean_tmp:
            local.cmd.run:
              - tgt: 'os:Ubuntu'
              - expr_form: grain
              - arg:
                - rm -rf /tmp/*

          clean_tmp:
            local.cmd.run:
              - tgt: 'G@roles:hbase_master'
              - expr_form: compound
              - arg:
                - rm -rf /tmp/*

       Any other parameters in the LocalClient().cmd() method can be specified as well.

   Calling Runner modules and Wheel modules
       Calling Runner modules and Wheel modules from the Reactor uses a more direct syntax since the function is
       being executed locally instead of sending a command to a remote system to be executed there. There are no
       'arg' or 'kwarg' parameters (unless the Runner function or Wheel function accepts a parameter with either
       of those names.)

       For example:

          clear_the_grains_cache_for_all_minions:
            runner.cache.clear_grains

       If the runner takes arguments then they can be specified as well:

          spin_up_more_web_machines:
            runner.cloud.profile:
              - prof: centos_6
              - instances:
                - web11       # These VM names would be generated via Jinja in a
                - web12       # real-world example.

   Passing event data to Minions or Orchestrate as Pillar
       An  interesting  trick to pass data from the Reactor script to state.highstate or state.sls is to pass it
       as inline Pillar data since both functions take a keyword argument named pillar.

       The following example uses Salt's Reactor to listen for the event that is fired when the key  for  a  new
       minion is accepted on the master using salt-key.

       /etc/salt/master.d/reactor.conf:

          reactor:
            - 'salt/key':
              - /srv/salt/haproxy/react_new_minion.sls

       The  Reactor  then fires a state.sls command targeted to the HAProxy servers and passes the ID of the new
       minion from the event to the state file via inline Pillar.

       /srv/salt/haproxy/react_new_minion.sls:

          {% if data['act'] == 'accept' and data['id'].startswith('web') %}
          add_new_minion_to_pool:
            local.state.sls:
              - tgt: 'haproxy*'
              - arg:
                - haproxy.refresh_pool
              - kwarg:
                  pillar:
                    new_minion: {{ data['id'] }}
          {% endif %}

       The above command is equivalent to the following command at the CLI:

          salt 'haproxy*' state.sls haproxy.refresh_pool 'pillar={new_minion: minionid}'

       This works with Orchestrate files as well:

          call_some_orchestrate_file:
            runner.state.orchestrate:
              - mods: some_orchestrate_file
              - pillar:
                  stuff: things

       Which is equivalent to the following command at the CLI:

          salt-run state.orchestrate some_orchestrate_file pillar='{stuff: things}'

       Finally, that data is available in the state file using the normal Pillar lookup  syntax.  The  following
       example  is  grabbing web server names and IP addresses from Salt Mine. If this state is invoked from the
       Reactor then the custom Pillar value from above will be available and the new minion will be added to the
       pool but with the disabled flag so that HAProxy won't yet direct traffic to it.

       /srv/salt/haproxy/refresh_pool.sls:

          {% set new_minion = salt['pillar.get']('new_minion') %}

          listen web *:80
              balance source
              {% for server,ip in salt['mine.get']('web*', 'network.interfaces', ['eth0']).items() %}
              {% if server == new_minion %}
              server {{ server }} {{ ip }}:80 disabled
              {% else %}
              server {{ server }} {{ ip }}:80 check
              {% endif %}
              {% endfor %}

   A Complete Example
       In  this  example,  we're going to assume that we have a group of servers that will come online at random
       and need to have keys automatically accepted. We'll also  add  that  we  don't  want  all  servers  being
       automatically  accepted.  For  this example, we'll assume that all hosts that have an id that starts with
       'ink' will be automatically accepted and have state.highstate executed. On top of this,  we're  going  to
       add that a host coming up that was replaced (meaning a new key) will also be accepted.

       Our  master configuration will be rather simple. All minions that attempte to authenticate will match the
       tag of salt/auth. When it comes to the minion key being accepted, we get a more refined tag that includes
       the minion id, which we can use for matching.

       /etc/salt/master.d/reactor.conf:

          reactor:
            - 'salt/auth':
              - /srv/reactor/auth-pending.sls
            - 'salt/minion/ink*/start':
              - /srv/reactor/auth-complete.sls

       In  this sls file, we say that if the key was rejected we will delete the key on the master and then also
       tell the master to ssh in to the minion and tell it to restart the minion, since a  minion  process  will
       die if the key is rejected.

       We  also  say that if the key is pending and the id starts with ink we will accept the key. A minion that
       is waiting on a pending key will retry authentication every ten seconds by default.

       /srv/reactor/auth-pending.sls:

          {# Ink server faild to authenticate -- remove accepted key #}
          {% if not data['result'] and data['id'].startswith('ink') %}
          minion_remove:
            wheel.key.delete:
              - match: {{ data['id'] }}
          minion_rejoin:
            local.cmd.run:
              - tgt: salt-master.domain.tld
              - arg:
                - ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no "{{ data['id'] }}" 'sleep 10 && /etc/init.d/salt-minion restart'
          {% endif %}

          {# Ink server is sending new key -- accept this key #}
          {% if 'act' in data and data['act'] == 'pend' and data['id'].startswith('ink') %}
          minion_add:
            wheel.key.accept:
              - match: {{ data['id'] }}
          {% endif %}

       No if statements are needed here because we already limited this action to just Ink servers in the master
       configuration.

       /srv/reactor/auth-complete.sls:

          {# When an Ink server connects, run state.highstate. #}
          highstate_run:
            local.state.highstate:
              - tgt: {{ data['id'] }}
              - ret: smtp

       The above will also return the highstate result data using the smtp_return returner (use virtualname like
       when using from the command line with --return).  The returner needs to be configured on the  minion  for
       this to work.  See salt.returners.smtp_return documentation for that.

   Syncing Custom Types on Minion Start
       Salt  will sync all custom types (by running a saltutil.sync_all) on every highstate. However, there is a
       chicken-and-egg issue where, on the initial highstate, a minion will not  yet  have  these  custom  types
       synced when the top file is first compiled. This can be worked around with a simple reactor which watches
       for minion_start events, which each minion fires when it first starts up and connects to the master.

       On the master, create /srv/reactor/sync_grains.sls with the following contents:

          sync_grains:
            local.saltutil.sync_grains:
              - tgt: {{ data['id'] }}

       And in the master config file, add the following reactor configuration:

          reactor:
            - 'minion_start':
              - /srv/reactor/sync_grains.sls

       This will cause the master to instruct each minion to sync its custom grains when it starts, making these
       grains available when the initial highstate is executed.

       Other  types  can  be  synced  by  replacing local.saltutil.sync_grains with local.saltutil.sync_modules,
       local.saltutil.sync_all, or whatever else suits the intended use case.

THE SALT MINE

       The Salt Mine is used to collect arbitrary data from Minions and store it on the  Master.  This  data  is
       then made available to all Minions via the salt.modules.mine module.

       Mine  data  is  gathered  on  the  Minion  and sent back to the Master where only the most recent data is
       maintained (if long term data is required use returners or the external job cache).

   Mine vs Grains
       Mine data is designed to be much more up-to-date than grain data. Grains are refreshed on a very  limited
       basis  and are largely static data. Mines are designed to replace slow peer publishing calls when Minions
       need data from other Minions. Rather than having a Minion reach out to all the other Minions for a  piece
       of  data,  the Salt Mine, running on the Master, can collect it from all the Minions every mine-interval,
       resulting in almost fresh data at any given time, with much less overhead.

   Mine Functions
       To enable the Salt Mine the mine_functions option needs to be applied to a Minion.  This  option  can  be
       applied  via  the Minion's configuration file, or the Minion's Pillar. The mine_functions option dictates
       what functions are being executed and allows for arguments to be passed in. If no arguments  are  passed,
       an empty list must be added:

          mine_functions:
            test.ping: []
            network.ip_addrs:
              interface: eth0
              cidr: '10.0.0.0/8'

   Mine Functions Aliases
       Function  aliases  can  be used to provide friendly names, usage intentions or to allow multiple calls of
       the same function with different arguments.  There is a  different  syntax  for  passing  positional  and
       key-value arguments.  Mixing positional and key-value arguments is not supported.

       New in version 2014.7.0.

          mine_functions:
            network.ip_addrs: [eth0]
            networkplus.internal_ip_addrs: []
            internal_ip_addrs:
              mine_function: network.ip_addrs
              cidr: 192.168.0.0/16
            ip_list:
              - mine_function: grains.get
              - ip_interfaces

   Mine Interval
       The Salt Mine functions are executed when the Minion starts and at a given interval by the scheduler. The
       default interval is every 60 minutes and can be adjusted for the Minion via the mine_interval option:

          mine_interval: 60

   Mine in Salt-SSH
       As of the 2015.5.0 release of salt, salt-ssh supports mine.get.

       Because the Minions cannot provide their own mine_functions  configuration,  we  retrieve  the  args  for
       specified mine functions in one of three places, searched in the following order:

       1. Roster data

       2. Pillar

       3. Master config

       The mine_functions are formatted exactly the same as in normal salt, just stored in a different location.
       Here is an example of a flat roster containing mine_functions:

          test:
            host: 104.237.131.248
            user: root
            mine_functions:
              cmd.run: ['echo "hello!"']
              network.ip_addrs:
                interface: eth0

       NOTE:
          Because of the differences in the architecture of salt-ssh, mine.get calls are  somewhat  inefficient.
          Salt  must make a new salt-ssh call to each of the Minions in question to retrieve the requested data,
          much like a publish call. However, unlike publish, it must run the requested  function  as  a  wrapper
          function, so we can retrieve the function args from the pillar of the Minion in question. This results
          in a non-trivial delay in retrieving the requested data.

   Example
       One way to use data from Salt Mine is in a State. The values can be retrieved via Jinja and used  in  the
       SLS  file.  The following example is a partial HAProxy configuration file and pulls IP addresses from all
       Minions with the "web" grain to add them to the pool of load balanced servers.

       /srv/pillar/top.sls:

          base:
            'G@roles:web':
              - web

       /srv/pillar/web.sls:

          mine_functions:
            network.ip_addrs: [eth0]

       /etc/salt/minion.d/mine.conf:

          mine_interval: 5

       /srv/salt/haproxy.sls:

          haproxy_config:
            file.managed:
              - name: /etc/haproxy/config
              - source: salt://haproxy_config
              - template: jinja

       /srv/salt/haproxy_config:

          <...file contents snipped...>

          {% for server, addrs in salt['mine.get']('roles:web', 'network.ip_addrs', expr_form='pillar').items() %}
          server {{ server }} {{ addrs[0] }}:80 check
          {% endfor %}

          <...file contents snipped...>

EXTERNAL AUTHENTICATION SYSTEM

       Salt's External Authentication System (eAuth) allows for Salt to pass through  command  authorization  to
       any external authentication system, such as PAM or LDAP.

       NOTE:
          eAuth  using  the PAM external auth system requires salt-master to be run as root as this system needs
          root access to check authentication.

   Access Control System
       NOTE:
          When to Use client_acl and external_auth

          client_acl is useful for allowing local system users to run Salt commands  without  giving  them  root
          access.  If  you  can  log  into  the Salt master directly, then client_acl will allow you to use Salt
          without root privileges. If the local system is configured to authenticate against  a  remote  system,
          like LDAP or Active Directory, then client_acl will interact with the remote system transparently.

          external_auth is useful for salt-api or for making your own scripts that use Salt's Python API. It can
          be used at the CLI (with the -a flag) but it is more cumbersome as there are more steps involved.  The
          only  time  it is useful at the CLI is when the local system is not configured to authenticate against
          an external service but you still want Salt to authenticate against an external service.

       The external authentication system allows for specific users to be granted  access  to  execute  specific
       functions  on specific minions. Access is configured in the master configuration file and uses the access
       control system:

          external_auth:
            pam:
              thatch:
                - 'web*':
                  - test.*
                  - network.*
              steve:
                - .*

       The above configuration allows the user thatch to execute functions in the test and  network  modules  on
       the minions that match the web* target.  User steve is given unrestricted access to minion commands.

       Salt respects the current PAM configuration in place, and uses the 'login' service to authenticate.

       NOTE:
          The PAM module does not allow authenticating as root.

       To allow access to wheel modules or runner modules the following @ syntax must be used:

          external_auth:
            pam:
              thatch:
                - '@wheel'   # to allow access to all wheel modules
                - '@runner'  # to allow access to all runner modules
                - '@jobs'    # to allow access to the jobs runner and/or wheel module

       NOTE:
          The runner/wheel markup is different, since there are no minions to scope the acl to.

       NOTE:
          Globs will not match wheel or runners! They must be explicitly allowed with @wheel or @runner.

       The  external authentication system can then be used from the command-line by any user on the same system
       as the master with the -a option:

          $ salt -a pam web\* test.ping

       The system will ask the user for the credentials required by the authentication system and  then  publish
       the command.

       To apply permissions to a group of users in an external authentication system, append a % to the ID:

          external_auth:
            pam:
              admins%:
                - '*':
                  - 'pkg.*'

       WARNING:
          All  users  that have external authentication privileges are allowed to run saltutil.findjob. Be aware
          that this could inadvertently expose some data such as minion IDs.

   Tokens
       With external authentication alone, the authentication credentials will be required with  every  call  to
       Salt. This can be alleviated with Salt tokens.

       Tokens  are  short  term  authorizations  and  can  be  easily  created  by  just adding a -T option when
       authenticating:

          $ salt -T -a pam web\* test.ping

       Now a token will be created that has a expiration of 12 hours (by default).  This token is  stored  in  a
       file named salt_token in the active user's home directory.

       Once  the  token is created, it is sent with all subsequent communications.  User authentication does not
       need to be entered again until the token expires.

       Token expiration time can be set in the Salt master config file.

   LDAP and Active Directory
       NOTE:
          LDAP usage requires that you have installed python-ldap.

       Salt supports both user and group authentication for LDAP (and Active Directory  accessed  via  its  LDAP
       interface)

   OpenLDAP and similar systems
       LDAP configuration happens in the Salt master configuration file.

       Server configuration values and their defaults:

          # Server to auth against
          auth.ldap.server: localhost

          # Port to connect via
          auth.ldap.port: 389

          # Use TLS when connecting
          auth.ldap.tls: False

          # LDAP scope level, almost always 2
          auth.ldap.scope: 2

          # Server specified in URI format
          auth.ldap.uri: ''    # Overrides .ldap.server, .ldap.port, .ldap.tls above

          # Verify server's TLS certificate
          auth.ldap.no_verify: False

          # Bind to LDAP anonymously to determine group membership
          # Active Directory does not allow anonymous binds without special configuration
          auth.ldap.anonymous: False

          # FOR TESTING ONLY, this is a VERY insecure setting.
          # If this is True, the LDAP bind password will be ignored and
          # access will be determined by group membership alone with
          # the group memberships being retrieved via anonymous bind
          auth.ldap.auth_by_group_membership_only: False

          # Require authenticating user to be part of this Organizational Unit
          # This can be blank if your LDAP schema does not use this kind of OU
          auth.ldap.groupou: 'Groups'

          # Object Class for groups.  An LDAP search will be done to find all groups of this
          # class to which the authenticating user belongs.
          auth.ldap.groupclass: 'posixGroup'

          # Unique ID attribute name for the user
          auth.ldap.accountattributename: 'memberUid'

          # These are only for Active Directory
          auth.ldap.activedirectory: False
          auth.ldap.persontype: 'person'

       There  are  two  phases  to  LDAP  authentication.   First,  Salt  authenticates  to search for a users's
       Distinguished Name and group membership.  The user it authenticates as in this phase is often  a  special
       LDAP  system  user  with  read-only  access  to the LDAP directory.  After Salt searches the directory to
       determine the actual user's DN and groups, it re-authenticates as the user running the Salt commands.

       If you are already aware of the structure of your DNs and permissions in your LDAP  store  are  set  such
       that  users can look up their own group memberships, then the first and second users can be the same.  To
       tell Salt this is the case, omit the auth.ldap.bindpw parameter.  You can template the binddn like this:

          auth.ldap.basedn: dc=saltstack,dc=com
          auth.ldap.binddn: uid={{ username }},cn=users,cn=accounts,dc=saltstack,dc=com

       Salt will use the password entered on the salt command line in place of the bindpw.

       To use two separate users, specify the LDAP lookup user in the binddn directive,  and  include  a  bindpw
       like so

          auth.ldap.binddn: uid=ldaplookup,cn=sysaccounts,cn=etc,dc=saltstack,dc=com
          auth.ldap.bindpw: mypassword

       As  mentioned  before,  Salt uses a filter to find the DN associated with a user. Salt substitutes the {{
       username }} value for the username when querying LDAP

          auth.ldap.filter: uid={{ username }}

       For OpenLDAP, to determine group membership, one can specify an OU that  contains  group  data.  This  is
       prepended   to   the   basedn   to  create  a  search  path.   Then  the  results  are  filtered  against
       auth.ldap.groupclass, default posixGroup, and the account's 'name' attribute, memberUid by default.

          auth.ldap.groupou: Groups

   Active Directory
       Active Directory handles group membership differently, and does not  utilize  the  groupou  configuration
       variable.  AD needs the following options in the master config:

          auth.ldap.activedirectory: True
          auth.ldap.filter: sAMAccountName={{username}}
          auth.ldap.accountattributename: sAMAccountName
          auth.ldap.groupclass: group
          auth.ldap.persontype: person

       To  determine group membership in AD, the username and password that is entered when LDAP is requested as
       the eAuth mechanism on the command line is used to bind to AD's LDAP interface. If this  fails,  then  it
       doesn't  matter  what groups the user belongs to, he or she is denied access. Next, the distinguishedName
       of the user is looked up with the following LDAP search:

          (&(<value of auth.ldap.accountattributename>={{username}})
            (objectClass=<value of auth.ldap.persontype>)
          )

       This should return a distinguishedName that we  can  use  to  filter  for  group  membership.   Then  the
       following LDAP query is executed:

          (&(member=<distinguishedName from search above>)
            (objectClass=<value of auth.ldap.groupclass>)
          )

          external_auth:
            ldap:
              test_ldap_user:
                  - '*':
                      - test.ping

       To configure an LDAP group, append a % to the ID:

          external_auth:
            ldap:
              test_ldap_group%:
                - '*':
                  - test.echo

ACCESS CONTROL SYSTEM

New in version 0.10.4.

Salt maintains a standard system used to open granular control to non administrative users to execute
Salt commands. The access control system has been applied to all systems used to configure access to non
administrative control interfaces in Salt.These interfaces include, the peer system, the external auth
system and the client acl system.

The access control system mandated a standard configuration syntax used in all of the three
aforementioned systems. While this adds functionality to the configuration in 0.10.4, it does not negate
the old configuration.

Now specific functions can be opened up to specific minions from specific users in the case of external
auth and client ACLs, and for specific minions in the case of the peer system.

The access controls are manifested using matchers in these configurations:

client_acl:
fred:
- web\*:
- pkg.list_pkgs
- test.*
- apache.*

In the above example, fred is able to send commands only to minions which match the specified glob
target. This can be expanded to include other functions for other minions based on standard targets.

external_auth:
pam:
dave:
- test.ping
- mongo\*:
- network.*
- log\*:
- network.*
- pkg.*
- 'G@os:RedHat':
- kmod.*
steve:
- .*

The above allows for all minions to be hit by test.ping by dave, and adds a few functions that dave can
execute on other minions. It also allows steve unrestricted access to salt commands.

NOTE:
Functions are matched using regular expressions.

JOB MANAGEMENT

New in version 0.9.7.

Since Salt executes jobs running on many systems, Salt needs to be able to manage jobs running on many
systems.

The Minion proc System
Salt Minions maintain a proc directory in the Salt cachedir. The proc directory maintains files named
after the executed job ID. These files contain the information about the current running jobs on the
minion and allow for jobs to be looked up. This is located in the proc directory under the cachedir, with
a default configuration it is under /var/cache/salt/proc.

Functions in the saltutil Module
Salt 0.9.7 introduced a few new functions to the saltutil module for managing jobs. These functions are:

1. running Returns the data of all running jobs that are found in the proc directory.

2. find_job Returns specific data about a certain job based on job id.

3. signal_job Allows for a given jid to be sent a signal.

4. term_job Sends a termination signal (SIGTERM, 15) to the process controlling the specified job.

5. kill_job Sends a kill signal (SIGKILL, 9) to the process controlling the specified job.

These functions make up the core of the back end used to manage jobs at the minion level.

The jobs Runner
A convenience runner front end and reporting system has been added as well. The jobs runner contains
functions to make viewing data easier and cleaner.

The jobs runner contains a number of functions...

active
The active function runs saltutil.running on all minions and formats the return data about all running
jobs in a much more usable and compact format. The active function will also compare jobs that have
returned and jobs that are still running, making it easier to see what systems have completed a job and
what systems are still being waited on.

# salt-run jobs.active

lookup_jid
When jobs are executed the return data is sent back to the master and cached. By default it is cached
for 24 hours, but this can be configured via the keep_jobs option in the master configuration. Using the
lookup_jid runner will display the same return data that the initial job invocation with the salt command
would display.

# salt-run jobs.lookup_jid <job id number>

list_jobs
Before finding a historic job, it may be required to find the job id. list_jobs will parse the cached
execution data and display all of the job data for jobs that have already, or partially returned.

# salt-run jobs.list_jobs

Scheduling Jobs
In Salt versions greater than 0.12.0, the scheduling system allows incremental executions on minions or
the master. The schedule system exposes the execution of any execution function on minions or any runner
on the master.

Scheduling is enabled via the schedule option on either the master or minion config files, or via a
minion's pillar data. Schedules that are impletemented via pillar data, only need to refresh the minion's
pillar data, for example by using saltutil.refresh_pillar. Schedules implemented in the master or minion
config have to restart the application in order for the schedule to be implemented.

NOTE:
The scheduler executes different functions on the master and minions. When running on the master the
functions reference runner functions, when running on the minion the functions specify execution
functions.

A scheduled run has no output on the minion unless the config is set to info level or higher. Refer to
minion logging settings.

Specify maxrunning to ensure that there are no more than N copies of a particular routine running. Use
this for jobs that may be long-running and could step on each other or otherwise double execute. The
default for maxrunning is 1.

States are executed on the minion, as all states are. You can pass positional arguments and provide a
yaml dict of named arguments.

schedule:
job1:
function: state.sls
seconds: 3600
args:
- httpd
kwargs:
test: True

This will schedule the command: state.sls httpd test=True every 3600 seconds (every hour)

schedule:
job1:
function: state.sls
seconds: 3600
args:
- httpd
kwargs:
test: True
splay: 15

This will schedule the command: state.sls httpd test=True every 3600 seconds (every hour) splaying the
time between 0 and 15 seconds

schedule:
job1:
function: state.sls
seconds: 3600
args:
- httpd
kwargs:
test: True
splay:
start: 10
end: 15

This will schedule the command: state.sls httpd test=True every 3600 seconds (every hour) splaying the
time between 10 and 15 seconds

New in version 2014.7.0.

Frequency of jobs can also be specified using date strings supported by the python dateutil library. This
requires python-dateutil to be installed on the minion.

schedule:
job1:
function: state.sls
args:
- httpd
kwargs:
test: True
when: 5:00pm

This will schedule the command: state.sls httpd test=True at 5:00pm minion localtime.

schedule:
job1:
function: state.sls
args:
- httpd
kwargs:
test: True
when:
- Monday 5:00pm
- Tuesday 3:00pm
- Wednesday 5:00pm
- Thursday 3:00pm
- Friday 5:00pm

This will schedule the command: state.sls httpd test=True at 5pm on Monday, Wednesday, and Friday, and
3pm on Tuesday and Thursday.

schedule:
job1:
function: state.sls
seconds: 3600
args:
- httpd
kwargs:
test: True
range:
start: 8:00am
end: 5:00pm

This will schedule the command: state.sls httpd test=True every 3600 seconds (every hour) between the
hours of 8am and 5pm. The range parameter must be a dictionary with the date strings using the dateutil
format. This requires python-dateutil to be installed on the minion.

New in version 2014.7.0.

The scheduler also supports ensuring that there are no more than N copies of a particular routine
running. Use this for jobs that may be long-running and could step on each other or pile up in case of
infrastructure outage.

The default for maxrunning is 1.

schedule:
long_running_job:
function: big_file_transfer
jid_include: True

States
schedule:
log-loadavg:
function: cmd.run
seconds: 3660
args:
- 'logger -t salt < /proc/loadavg'
kwargs:
stateful: False
shell: /bin/sh

Highstates
To set up a highstate to run on a minion every 60 minutes set this in the minion config or pillar:

schedule:
highstate:
function: state.highstate
minutes: 60

Time intervals can be specified as seconds, minutes, hours, or days.

Runners
Runner executions can also be specified on the master within the master configuration file:

schedule:
run_my_orch:
function: state.orchestrate
hours: 6
splay: 600
args:
- orchestration.my_orch

The above configuration is analogous to running salt-run state.orch orchestration.my_orch every 6 hours.

Scheduler With Returner
The scheduler is also useful for tasks like gathering monitoring data about a minion, this schedule
option will gather status data and send it to a MySQL returner database:

schedule:
uptime:
function: status.uptime
seconds: 60
returner: mysql
meminfo:
function: status.meminfo
minutes: 5
returner: mysql

Since specifying the returner repeatedly can be tiresome, the schedule_returner option is available to
specify one or a list of global returners to be used by the minions when scheduling. In Salt versions
greater than 0.12.0, the scheduling system allows incremental executions on minions or the master. The
schedule system exposes the execution of any execution function on minions or any runner on the master.