Provided by: masakari-common_9.0.0-0ubuntu0.20.04.5_all 
NAME
masakari - masakari
Masakari is an OpenStack project designed to ensure high availability of instances and
compute processes running on hosts.
This documentation is intended to help explain the current scope of the Masakari project
and the architectural decisions made to support this scope. The documentation will
include the future architectural roadmap and the current development process and
policies.
MASAKARI API REFERENCES
The Masakari API is extensive. We provide a concept guide which gives some of the high
level details, as well as a more detailed API reference.
OPERATOR GUIDE
Architecture Overview
• Masakari architecture: An overview of how all the components in masakari work together.
Installation
A detailed install guide for masakari.
Masakari services
Masakari service overview
Masakari provides Virtual Machines High Availability(VMHA), and rescues KVM-based Virtual
Machines(VM) from a failure events described below:
•
VM process down - restart vm (use nova stop API, and nova start API).
Libvirt events will be also emitted by other failures.
•
Provisioning process down - restarts process, changes nova-compute service
status to maintenance mode (use nova service-disable).
•
nova-compute host failure - evacuate all the VMs from failure host to
reserved host (use nova evacuate API).
The below services enables deplores to integrate with the Masakari directly or through
custom plug-ins.
The Masakari service consists of the following components:
masakari-api
An OpenStack-native REST API that processes API requests by sending them to the
masakari-engine over Remote Procedure Call (RPC).
masakari-engine
Processes the notifications received from masakari-api by executing the recovery
workflow in asynchronous way.
Install and configure
This section describes how to install and configure Masakari services on the compute node.
This section assumes that you already have a working OpenStack environment with the
following components installed: Nova, Glance, Cinder, Neutron and Identity.
The installation and configuration vary by distribution.
Install and configure for Ubuntu
This section describes how to install and configure Masakari for Ubuntu 18.04 (bionic).
Prerequisites
Before you install and configure the masakari service, you must create databases, service
credentials, and API endpoints.
1. To create the masakari database, follow these steps:
• Use the database access client to connect to the database server as the root user:
# mysql
• Create the masakari database:
mysql> CREATE DATABASE masakari CHARACTER SET utf8;
• Grant access privileges to the databases:
mysql> GRANT ALL PRIVILEGES ON masakari.* TO 'username'@'localhost' \
IDENTIFIED BY 'MASAKARI_DBPASS';
mysql> GRANT ALL PRIVILEGES ON masakari.* TO 'username'@'%' \
IDENTIFIED BY 'MASAKARI_DBPASS';
Replace MASAKARI_DBPASS with a suitable password.
• Exit the database access client.
2. Source the admin credentials to gain access to admin-only CLI commands:
$ . admin-openrc
3. Create the Masakari service credentials:
• Create the masakari user with password as masakari:
$ openstack user create --password-prompt masakari
User Password:
Repeat User Password:
+---------------------+----------------------------------+
| Field | Value |
+---------------------+----------------------------------+
| domain_id | default |
| enabled | True |
| id | 8a7dbf5279404537b1c7b86c033620fe |
| name | masakari |
| options | {} |
| password_expires_at | None |
+---------------------+----------------------------------+
• Add the admin role to the masakari user:
$ openstack role add --project service --user masakari admin
• Create the masakari service entity:
$ openstack service create --name masakari \
--description "masakari high availability" instance-ha
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | masakari high availability |
| enabled | True |
| id | 060d59eac51b4594815603d75a00aba2 |
| name | masakari |
| type | instance-ha |
+-------------+----------------------------------+
4. Create the Masakari API service endpoints:
$ openstack endpoint create --region RegionOne \
masakari public http:// <CONTROLLER_IP>/instance-ha/v1/$\(tenant_id\)s
+--------------+-------------------------------------------------------+
| Field | Value |
+--------------+-------------------------------------------------------+
| enabled | True |
| id | 38f7af91666a47cfb97b4dc790b94424 |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 060d59eac51b4594815603d75a00aba2 |
| service_name | masakari |
| service_type | instance-ha |
| url | http://<CONTROLLER_IP>/instance-ha/v1/$(tenant_id)s |
+--------------+-------------------------------------------------------+
$ openstack endpoint create --region RegionOne \
masakari internal http:// <CONTROLLER_IP>/instance-ha/v1/$\(tenant_id\)s
+--------------+-------------------------------------------------------+
| Field | Value |
+--------------+-------------------------------------------------------+
| enabled | True |
| id | 38f7af91666a47cfb97b4dc790b94424 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 060d59eac51b4594815603d75a00aba2 |
| service_name | masakari |
| service_type | instance-ha |
| url | http://<CONTROLLER_IP>/instance-ha/v1/$(tenant_id)s |
+--------------+-------------------------------------------------------+
$ openstack endpoint create --region RegionOne \
masakari admin http://<CONTROLLER_IP>/instance-ha/v1/$\(tenant_id\)s
+--------------+-------------------------------------------------------+
| Field | Value |
+--------------+-------------------------------------------------------+
| enabled | True |
| id | 38f7af91666a47cfb97b4dc790b94424 |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 060d59eac51b4594815603d75a00aba2 |
| service_name | masakari |
| service_type | instance-ha |
| url | http://<CONTROLLER_IP>/instance-ha/v1/$(tenant_id)s |
+--------------+-------------------------------------------------------+
Install and configure Masakari
NOTE:
• You must install Masakari on the Controller Nodes only.
1. Clone masakari using:
# git clone https://opendev.org/openstack/masakari.git
2. Prepare the masakari configuration files:
1. Generate via tox:
Go to ‘opt/stack/masakari’ and execute the command below, this will generate
masakari.conf.sample, sample configuration file at /opt/stack/masakari/etc/masakari/
# tox -egenconfig
2. Download from:
# masakari.conf.sample
3. Rename masakari.conf.sample file to masakari.conf, and edit sections as shown below:
[default]
transport_url = rabbit://stackrabbit:admin@<CONTROLLER_IP>:5672/
graceful_shutdown_timeout = 5
os_privileged_user_tenant = service
os_privileged_user_password = admin
os_privileged_user_auth_url = http://<CONTROLLER_IP>/identity
os_privileged_user_name = nova
logging_exception_prefix = %(color)s%(asctime)s.%(msecs)03d TRACE %(name)s [01;35m%(instance)s[00m
logging_debug_format_suffix = [00;33mfrom (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d[00m
logging_default_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [[00;36m-%(color)s] [01;35m%(instance)s%(color)s%(message)s[00m
logging_context_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [[01;36m%(request_id)s [00;36m%(project_name)s %(user_name)s%(color)s] [01;35m%(instance)s%(color)s%(message)s[00m
use_syslog = False
debug = True
masakari_api_workers = 2
[database]
connection = mysql+pymysql://root:admin@1<CONTROLLER_IP>/masakari?charset=utf8
[keystone_authtoken]
memcached_servers = localhost:11211
cafile = /opt/stack/data/ca-bundle.pem
project_domain_name = Default
project_name = service
user_domain_name = Default
password = <MASAKARI_PASS>
username = masakari
auth_url = http://<CONTROLLER_IP>/identity
auth_type = password
[taskflow]
connection = mysql+pymysql://root:admin@<CONTROLLER_IP>/masakari?charset=utf8
NOTE:
Replace CONTROLLER_IP with the IP address of controller node.
Replace MASAKARI_PASS with the password you chose for the masakari user in the
Identity service.
4. Create masakari directory in /etc/:
Copy masakari.conf file to /etc/masakari/
# cp -p etc/masakari/masakari.conf.sample /etc/masakari/masakari.conf
3. To install masakari run setup.py from masakari:
# cd masakari
# sudo python setup.py install
4. Run below db command to sync database:
# masakari-manage db sync
Finalize installation
• Start masakari services:
# masakari-api
# masakari-engine
Verify operation
Verify Masakari installation.
1. Source the admin credentials to gain access to admin-only CLI commands:
$ . admin-openrc
2. List API endpoints in the Identity service to verify connectivity with the Identity
service:
NOTE:
Below endpoints list may differ depending on the installation of OpenStack
components.
$ openstack endpoint list
+-------------+----------------+--------------------------------------------------------+
| Name | Type | Endpoints |
+-------------+----------------+--------------------------------------------------------+
| nova_legacy | compute_legacy | RegionOne |
| | | public: http://controller/compute/v2/<tenant_id> |
| | | |
| nova | compute | RegionOne |
| | | public: http://controller/compute/v2.1 |
| | | |
| cinder | block-storage | RegionOne |
| | | public: http://controller/volume/v3/<tenant_id> |
| | | |
| glance | image | RegionOne |
| | | public: http://controller/image |
| | | |
| cinderv3 | volumev3 | RegionOne |
| | | public: http://controller/volume/v3/<tenant_id> |
| | | |
| masakari | instance-ha | RegionOne |
| | | internal: http://controller/instance-ha/v1/<tenant_id> |
| | | RegionOne |
| | | admin: http://controller/instance-ha/v1/<tenant_id> |
| | | RegionOne |
| | | public: http://controller/instance-ha/v1/<tenant_id> |
| | | |
| keystone | identity | RegionOne |
| | | public: http://controller/identity |
| | | RegionOne |
| | | admin: http://controller/identity |
| | | |
| cinderv2 | volumev2 | RegionOne |
| | | public: http://controller/volume/v2/<tenant_id> |
| | | |
| placement | placement | RegionOne |
| | | public: http://controller/placement |
| | | |
| neutron | network | RegionOne |
| | | public: http://controller:9696/ |
| | | |
+-------------+----------------+--------------------------------------------------------+
3. Run segment list command to verify masakari-api is running properly. This will return
empty segment list as you haven't yet configured Failover segments.
$ openstack segment list
NOTE:
Since Failover segments are not configured, there is no way to verify
masakari-engine is running properly as the notification cannot be sent from
masakari-api to masakari-engine.
Reference Material
• Configuration Guide: Information on configuration files.
• Custom Recovery Workflow Configuration Guide
• CLI Commands for Masakari: The complete command reference for Masakari.
• Versioned Notifications: This provides the list of existing versioned notifications with
sample payloads.
• Masakari team process
• Getting started with Masakari: This will help newcomers understand basics of Masakari
• Nova Docs: A collection of guides for Nova.
Masakari CLI Documentation
In this section you will find information on Masakari’s command line interface.
masakari-status
CLI interface for Masakari status commands
Synopsis
masakari-status <category> <command> [<args>]
Description
masakari-status is a tool that provides routines for checking the status of a Masakari
deployment.
Options
The standard pattern for executing a masakari-status command is:
masakari-status <category> <command> [<args>]
Run without arguments to see a list of available command categories:
masakari-status
Categories are:
• upgrade
Detailed descriptions are below:
You can also run with a category argument such as upgrade to see a list of all commands in
that category:
masakari-status upgrade
These sections describe the available categories and arguments for masakari-status.
Upgrade
masakari-status upgrade check
Performs a release-specific readiness check before restarting services with new
code. For example, missing or changed configuration options, incompatible object
states, or other conditions that could lead to failures while upgrading.
Return Codes
┌────────────┬──────────────────────────────────┐
│Return code │ Description │
├────────────┼──────────────────────────────────┤
│0 │ All upgrade readiness checks │
│ │ passed successfully and there is │
│ │ nothing to do. │
├────────────┼──────────────────────────────────┤
│1 │ At least one check encountered │
│ │ an issue and requires further │
│ │ investigation. This is │
│ │ considered a warning but the │
│ │ upgrade may be OK. │
├────────────┼──────────────────────────────────┤
│2 │ There was an upgrade status │
│ │ check failure that needs to be │
│ │ investigated. This should be │
│ │ considered something that stops │
│ │ an upgrade. │
├────────────┼──────────────────────────────────┤
│255 │ An unexpected error occurred. │
└────────────┴──────────────────────────────────┘
History of Checks
7.0.0 (Stein)
• Sample check to be filled in with checks as they are added in Stein.
masakari-manage
Control and manage masakari database
Synopsis
masakari-manage <category> <action> [<args>]
Description
masakari-manage controls DB by managing various admin-only aspects of masakari.
Options
The standard pattern for executing a masakari-manage command is:
masakari-manage <category> <command> [<args>]
Run without arguments to see a list of available command categories:
masakari-manage
You can also run with a category argument such as db to see a list of all commands in that
category:
masakari-manage db
These sections describe the available categories and arguments for masakari-manage.
Masakari Database
masakari-manage db version
Print the current main database version.
masakari-manage db sync [--version <version>]
Upgrade the main database schema up to the most recent version or --version if
specified.
masakari-manage db purge
Deleting rows older than 30 day(s) from table hosts, failover_segments and
notifications.
openstack masakari
To control and manage masakari operations, the extended command list available in
openstack command.
api-paste.ini
The masakari service stores its API configuration settings in the api-paste.ini file.
[composite:masakari_api]
use = call:masakari.api.urlmap:urlmap_factory
/: apiversions
/v1: masakari_api_v1
[composite:masakari_api_v1]
use = call:masakari.api.auth:pipeline_factory_v1
keystone = cors http_proxy_to_wsgi request_id faultwrap sizelimit authtoken keystonecontext osapi_masakari_app_v1
noauth2 = cors http_proxy_to_wsgi request_id faultwrap sizelimit noauth2 osapi_masakari_app_v1
# filters
[filter:cors]
paste.filter_factory = oslo_middleware.cors:filter_factory
oslo_config_project = masakari
[filter:http_proxy_to_wsgi]
paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory
[filter:request_id]
paste.filter_factory = oslo_middleware:RequestId.factory
[filter:faultwrap]
paste.filter_factory = masakari.api.openstack:FaultWrapper.factory
[filter:sizelimit]
paste.filter_factory = oslo_middleware:RequestBodySizeLimiter.factory
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
[filter:keystonecontext]
paste.filter_factory = masakari.api.auth:MasakariKeystoneContext.factory
[filter:noauth2]
paste.filter_factory = masakari.api.auth:NoAuthMiddleware.factory
# apps
[app:osapi_masakari_app_v1]
paste.app_factory = masakari.api.openstack.ha:APIRouterV1.factory
[pipeline:apiversions]
pipeline = faultwrap http_proxy_to_wsgi apiversionsapp
[app:apiversionsapp]
paste.app_factory = masakari.api.openstack.ha.versions:Versions.factory
Configuration Options
The following is an overview of all available configuration options in Masakari. For a
sample configuration file, refer to sample_config.
DEFAULT
auth_strategy
Type string
Default
keystone
Valid Values
keystone, noauth2
This determines the strategy to use for authentication: keystone or noauth2.
'noauth2' is designed for testing only, as it does no actual credential checking.
'noauth2' provides administrative credentials only if 'admin' is specified as the
username.
• Possible values:
Either 'keystone' (default) or 'noauth2'.
• Services that use this:
masakari-api
• Related options:
None
use_forwarded_for
Type boolean
Default
false
When True, the 'X-Forwarded-For' header is treated as the canonical remote address.
When False (the default), the 'remote_address' header is used.
You should only enable this if you have an HTML sanitizing proxy.
• Possible values:
True, False (default)
• Services that use this:
masakari-api
• Related options:
None
osapi_max_limit
Type integer
Default
1000
As a query can potentially return many thousands of items, you can limit the
maximum number of items in a single response by setting this option.
• Possible values:
Any positive integer. Default is 1000.
• Services that use this:
masakari-api
• Related options:
None
osapi_masakari_link_prefix
Type string
Default
<None>
This string is prepended to the normal URL that is returned in links to the
OpenStack Masakari API. If it is empty (the default), the URLs are returned
unchanged.
• Possible values:
Any string, including an empty string (the default).
• Services that use this:
masakari-api
• Related options:
None
tempdir
Type string
Default
<None>
Explicitly specify the temporary working directory.
monkey_patch
Type boolean
Default
false
Determine if monkey patching should be applied.
Related options:
• monkey_patch_modules: This must have values set for this option to have
any effect
monkey_patch_modules
Type list
Default
masakari.api:masakari.cmd
List of modules/decorators to monkey patch.
This option allows you to patch a decorator for all functions in specified modules.
Related options:
• monkey_patch: This must be set to True for this option to have any effect
masakari_topic
Type string
Default
ha_engine
This is the message queue topic that the masakari engine 'listens' on. It is used
when the masakari engine is started up to configure the queue, and whenever an RPC
call to the masakari engine is made.
• Possible values:
Any string, but there is almost never any reason to ever change this value
from its default of 'engine'.
• Services that use this:
masakari-engine
• Related options:
None
WARNING:
This option is deprecated for removal since 3.0.0. Its value may be silently
ignored in the future.
Reason Configurable RPC topic provides little value and it can easily break
Masakari if operator configures it to the same topic used by other
OpenStack services.
duplicate_notification_detection_interval
Type integer
Default
180
Minimum Value
0
Interval in seconds for identifying duplicate notifications. If the notification
received is identical to the previous ones whose status is either new or running
and if it's created_timestamp and the current timestamp is less than this config
option value, then the notification will be considered as duplicate and it will be
ignored.
wait_period_after_service_update
Type integer
Default
180
Number of seconds to wait after a service is enabled or disabled.
wait_period_after_evacuation
Type integer
Default
90
Wait until instance is evacuated
verify_interval
Type integer
Default
1
The monitoring interval for looping
wait_period_after_power_off
Type integer
Default
180
Number of seconds to wait for instance to shut down
wait_period_after_power_on
Type integer
Default
60
Number of seconds to wait for instance to start
process_unfinished_notifications_interval
Type integer
Default
120
Interval in seconds for processing notifications which are in error or new state.
retry_notification_new_status_interval
Type integer
Default
60
Mutable
This option can be changed without restarting.
Interval in seconds for identifying notifications which are in new state. If the
notification is in new state till this config option value after it's
generated_time, then it is considered that notification is ignored by the messaging
queue and will be processed by 'process_unfinished_notifications' periodic task.
check_expired_notifications_interval
Type integer
Default
600
Interval in seconds for checking running notifications.
notifications_expired_interval
Type integer
Default
86400
Interval in seconds for identifying running notifications expired.
host_failure_recovery_threads
Type integer
Default
3
Minimum Value
1
Number of threads to be used for evacuating and confirming instances during
execution of host_failure workflow.
notification_driver
Type string
Default
taskflow_driver
Defines which driver to use for executing notification workflows.
fatal_exception_format_errors
Type boolean
Default
false
Make exception message format errors fatal
nova_catalog_admin_info
Type string
Default
compute:nova:publicURL
Match this value when searching for nova in the service catalog. Format is:
separated values of the form: <service_type>:<service_name>:<endpoint_type>
os_region_name
Type string
Default
<None>
Region name of this node
nova_ca_certificates_file
Type string
Default
<None>
Location of ca certificates file to use for nova client requests.
nova_api_insecure
Type boolean
Default
false
Allow to perform insecure SSL requests to nova
os_privileged_user_name
Type string
Default
<None>
OpenStack privileged account username. Used for requests to other services (such as
Nova) that require an account with special rights.
os_privileged_user_password
Type string
Default
<None>
Password associated with the OpenStack privileged account.
os_privileged_user_tenant
Type string
Default
<None>
Tenant name associated with the OpenStack privileged account.
os_privileged_user_auth_url
Type URI
Default
<None>
Auth URL associated with the OpenStack privileged account.
os_user_domain_name
Type string
Default
default
User domain name associated with the OpenStack privileged account.
os_project_domain_name
Type string
Default
default
Project domain name associated with the OpenStack privileged account.
pybasedir
Type string
Default
/build/masakari-VZrMuh/masakari-9.0.0
Directory where the masakari python module is installed
bindir
Type string
Default
/usr/local/bin
Directory where masakari binaries are installed
state_path
Type string
Default
$pybasedir
Top-level directory for maintaining masakari's state
host
Type host address
Default
lgw01-amd64-035
Hostname, FQDN or IP address of this host. Must be valid within AMQP key.
Possible values:
• String with hostname, FQDN or IP address. Default is hostname of this host.
engine_manager
Type string
Default
masakari.engine.manager.MasakariManager
Full class name for the Manager for masakari engine
report_interval
Type integer
Default
10
Seconds between nodes reporting state to datastore
periodic_enable
Type boolean
Default
true
Enable periodic tasks
periodic_interval_max
Type integer
Default
300
Max interval time between periodic tasks execution in seconds.
periodic_fuzzy_delay
Type integer
Default
60
Range of seconds to randomly delay when starting the periodic task scheduler to
reduce stampeding. (Disable by setting to 0)
use_ssl
Type boolean
Default
false
Use APIs with SSL enabled
masakari_api_listen
Type host address
Default
0.0.0.0
The IP address on which the Masakari API will listen.
masakari_api_listen_port
Type integer
Default
15868
Minimum Value
1
Maximum Value
65535
The port on which the Masakari API will listen.
masakari_api_workers
Type integer
Default
<None>
Number of workers for Masakari API service. The default will be the number of CPUs
available.
service_down_time
Type integer
Default
60
Maximum time since last check-in for up service
backdoor_port
Type string
Default
<None>
Enable eventlet backdoor. Acceptable values are 0, <port>, and <start>:<end>,
where 0 results in listening on a random tcp port number; <port> results in
listening on the specified port number (and not enabling backdoor if that port is
in use); and <start>:<end> results in listening on the smallest unused port number
within the specified range of port numbers. The chosen port is displayed in the
service's log file.
backdoor_socket
Type string
Default
<None>
Enable eventlet backdoor, using the provided path as a unix socket that can receive
connections. This option is mutually exclusive with 'backdoor_port' in that only
one should be provided. If both are provided then the existence of this option
overrides the usage of that option. Inside the path {pid} will be replaced with the
PID of the current process.
log_options
Type boolean
Default
true
Enables or disables logging values of all registered options when starting a
service (at DEBUG level).
graceful_shutdown_timeout
Type integer
Default
60
Specify a timeout after which a gracefully shutdown server will exit. Zero value
means endless wait.
api_paste_config
Type string
Default
api-paste.ini
File name for the paste.deploy config for api service
wsgi_log_format
Type string
Default
%(client_ip)s "%(request_line)s" status: %(status_code)s len:
%(body_length)s time: %(wall_seconds).7f
A python format string that is used as the template to generate log lines. The
following values can beformatted into it: client_ip, date_time, request_line,
status_code, body_length, wall_seconds.
tcp_keepidle
Type integer
Default
600
Sets the value of TCP_KEEPIDLE in seconds for each server socket. Not supported on
OS X.
wsgi_default_pool_size
Type integer
Default
100
Size of the pool of greenthreads used by wsgi
max_header_line
Type integer
Default
16384
Maximum line size of message headers to be accepted. max_header_line may need to be
increased when using large tokens (typically those generated when keystone is
configured to use PKI tokens with big service catalogs).
wsgi_keep_alive
Type boolean
Default
true
If False, closes the client socket connection explicitly.
client_socket_timeout
Type integer
Default
900
Timeout for client connections' socket operations. If an incoming connection is
idle for this number of seconds it will be closed. A value of '0' means wait
forever.
rpc_conn_pool_size
Type integer
Default
30
Size of RPC connection pool.
Deprecated Variations
───────────────────────────────
Group Name
───────────────────────────────
DEFAULT rpc_conn_pool_size
┌────────┬────────────────────┐
│ │ │
conn_pool_min_size │ │ │
│ │ │
Type integer │ │ │
│ │ │
Default │ │ │
2 │ │ │
│ │ │
--
SEARCH
• search: Search the contents of this document.
• OpenStack wide search: Search the wider set of OpenStack documentation, including
forums.
COPYRIGHT
2021, OpenStack Foundation
Jun 11, 2021 MASAKARI(1)