Provided by: masakari-common_17.0.0-0ubuntu1_all 
NAME
masakari - masakari 17.0.0
Masakari is an OpenStack project designed to ensure high availability of instances and compute processes
running on hosts.
This documentation is intended to help explain the current scope of the Masakari project and the
architectural decisions made to support this scope. The documentation will include the future
architectural roadmap and the current development process and policies.
MASAKARI API REFERENCES
The Masakari API is extensive. We provide a concept guide which gives some of the high level details, as
well as a more detailed API reference.
OPERATOR GUIDE
Architecture Overview
• Masakari architecture: An overview of how all the components in masakari work together.
Installation
A detailed install guide for masakari.
Masakari services
Masakari service overview
Masakari provides Virtual Machines High Availability(VMHA), and rescues KVM-based Virtual Machines(VM)
from a failure events described below:
• VM process down - restart vm (use nova stop API, and nova start API). Libvirt events will be also
emitted by other failures.
• Provisioning process down - restarts process, changes nova-compute service status to maintenance mode
(use nova service-disable).
• nova-compute host failure - evacuate all the VMs from failure host according to the following recovery
methods (use nova evacuate API).
• auto - evacuate all the VMs with no destination node for nova scheduler.
• reserved_host - evacuate all the VMs with reserved hosts as the destination nodes for nova
scheduler.
• auto_priority - evacuate all the VMs by using auto recovery method firstly. If failed, then using
reserved_host recovery method.
• rh_priority - evacuate all the VMs by using reserved_host recovery method firstly. If failed,
then using auto recovery method.
The below services enables deplores to integrate with the Masakari directly or through custom plug-ins.
The Masakari service consists of the following components:
masakari-api
An OpenStack-native REST API that processes API requests by sending them to the masakari-engine
over Remote Procedure Call (RPC).
masakari-engine
Processes the notifications received from masakari-api by executing the recovery workflow in
asynchronous way.
Install and configure
This section describes how to install and configure Masakari services on the compute node.
This section assumes that you already have a working OpenStack environment with the following components
installed: Nova, Glance, Cinder, Neutron and Identity.
The installation and configuration vary by distribution.
Install and configure for Ubuntu
This section describes how to install and configure Masakari for Ubuntu 18.04 (bionic).
Prerequisites
Before you install and configure the masakari service, you must create databases, service credentials,
and API endpoints.
1. To create the masakari database, follow these steps:
• Use the database access client to connect to the database server as the root user:
# mysql
• Create the masakari database:
mysql> CREATE DATABASE masakari CHARACTER SET utf8;
• Grant access privileges to the databases:
mysql> GRANT ALL PRIVILEGES ON masakari.* TO 'username'@'localhost' \
IDENTIFIED BY 'MASAKARI_DBPASS';
mysql> GRANT ALL PRIVILEGES ON masakari.* TO 'username'@'%' \
IDENTIFIED BY 'MASAKARI_DBPASS';
Replace MASAKARI_DBPASS with a suitable password.
• Exit the database access client.
2. Source the admin credentials to gain access to admin-only CLI commands:
$ . admin-openrc
3. Create the Masakari service credentials:
• Create the masakari user with password as masakari:
$ openstack user create --password-prompt masakari
User Password:
Repeat User Password:
+---------------------+----------------------------------+
| Field | Value |
+---------------------+----------------------------------+
| domain_id | default |
| enabled | True |
| id | 8a7dbf5279404537b1c7b86c033620fe |
| name | masakari |
| options | {} |
| password_expires_at | None |
+---------------------+----------------------------------+
• Add the admin role to the masakari user:
$ openstack role add --project service --user masakari admin
• Create the masakari service entity:
$ openstack service create --name masakari \
--description "masakari high availability" instance-ha
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | masakari high availability |
| enabled | True |
| id | 060d59eac51b4594815603d75a00aba2 |
| name | masakari |
| type | instance-ha |
+-------------+----------------------------------+
4. Create the Masakari API service endpoints:
$ openstack endpoint create --region RegionOne \
masakari public http:// <CONTROLLER_IP>/instance-ha/v1/$\(tenant_id\)s
+--------------+-------------------------------------------------------+
| Field | Value |
+--------------+-------------------------------------------------------+
| enabled | True |
| id | 38f7af91666a47cfb97b4dc790b94424 |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 060d59eac51b4594815603d75a00aba2 |
| service_name | masakari |
| service_type | instance-ha |
| url | http://<CONTROLLER_IP>/instance-ha/v1/$(tenant_id)s |
+--------------+-------------------------------------------------------+
$ openstack endpoint create --region RegionOne \
masakari internal http:// <CONTROLLER_IP>/instance-ha/v1/$\(tenant_id\)s
+--------------+-------------------------------------------------------+
| Field | Value |
+--------------+-------------------------------------------------------+
| enabled | True |
| id | 38f7af91666a47cfb97b4dc790b94424 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 060d59eac51b4594815603d75a00aba2 |
| service_name | masakari |
| service_type | instance-ha |
| url | http://<CONTROLLER_IP>/instance-ha/v1/$(tenant_id)s |
+--------------+-------------------------------------------------------+
$ openstack endpoint create --region RegionOne \
masakari admin http://<CONTROLLER_IP>/instance-ha/v1/$\(tenant_id\)s
+--------------+-------------------------------------------------------+
| Field | Value |
+--------------+-------------------------------------------------------+
| enabled | True |
| id | 38f7af91666a47cfb97b4dc790b94424 |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 060d59eac51b4594815603d75a00aba2 |
| service_name | masakari |
| service_type | instance-ha |
| url | http://<CONTROLLER_IP>/instance-ha/v1/$(tenant_id)s |
+--------------+-------------------------------------------------------+
Install and configure Masakari
NOTE:
• You must install Masakari on the Controller Nodes only.
1. Clone masakari using:
# git clone https://opendev.org/openstack/masakari.git
2. Prepare the masakari configuration files:
1. Generate via tox:
Go to /opt/stack/masakari and execute the command below. This will generate masakari.conf.sample,
a sample configuration file, at /opt/stack/masakari/etc/masakari/:
# tox -egenconfig
2. Download from:
# masakari.conf.sample
3. Rename masakari.conf.sample file to masakari.conf, and edit sections as shown below:
[DEFAULT]
transport_url = rabbit://stackrabbit:admin@<CONTROLLER_IP>:5672/
graceful_shutdown_timeout = 5
os_privileged_user_tenant = service
os_privileged_user_password = admin
os_privileged_user_auth_url = http://<CONTROLLER_IP>/identity
os_privileged_user_name = nova
logging_exception_prefix = %(color)s%(asctime)s.%(msecs)03d TRACE %(name)s [01;35m%(instance)s[00m
logging_debug_format_suffix = [00;33mfrom (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d[00m
logging_default_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [[00;36m-%(color)s] [01;35m%(instance)s%(color)s%(message)s[00m
logging_context_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [[01;36m%(request_id)s [00;36m%(project_name)s %(user_name)s%(color)s] [01;35m%(instance)s%(color)s%(message)s[00m
use_syslog = False
debug = True
masakari_api_workers = 2
[database]
connection = mysql+pymysql://root:admin@1<CONTROLLER_IP>/masakari?charset=utf8
[keystone_authtoken]
memcached_servers = localhost:11211
cafile = /opt/stack/data/ca-bundle.pem
project_domain_name = Default
project_name = service
user_domain_name = Default
password = <MASAKARI_PASS>
username = masakari
auth_url = http://<CONTROLLER_IP>/identity
auth_type = password
[taskflow]
connection = mysql+pymysql://root:admin@<CONTROLLER_IP>/masakari?charset=utf8
NOTE:
Replace CONTROLLER_IP with the IP address of controller node.
Replace MASAKARI_PASS with the password you chose for the masakari user in the Identity service.
4. Create masakari directory in /etc/:
Copy masakari.conf file to /etc/masakari/
# cp -p etc/masakari/masakari.conf.sample /etc/masakari/masakari.conf
3. To install masakari run setup.py from masakari:
# cd masakari
# sudo python setup.py install
4. Run below db command to sync database:
# masakari-manage db sync
Finalize installation
• Start masakari services:
# masakari-api
# masakari-engine
Verify operation
Verify Masakari installation.
1. Source the admin credentials to gain access to admin-only CLI commands:
$ . admin-openrc
2. List API endpoints in the Identity service to verify connectivity with the Identity service:
NOTE:
Below endpoints list may differ depending on the installation of OpenStack components.
$ openstack endpoint list
+-------------+----------------+--------------------------------------------------------+
| Name | Type | Endpoints |
+-------------+----------------+--------------------------------------------------------+
| nova_legacy | compute_legacy | RegionOne |
| | | public: http://controller/compute/v2/<tenant_id> |
| | | |
| nova | compute | RegionOne |
| | | public: http://controller/compute/v2.1 |
| | | |
| cinder | block-storage | RegionOne |
| | | public: http://controller/volume/v3/<tenant_id> |
| | | |
| glance | image | RegionOne |
| | | public: http://controller/image |
| | | |
| cinderv3 | volumev3 | RegionOne |
| | | public: http://controller/volume/v3/<tenant_id> |
| | | |
| masakari | instance-ha | RegionOne |
| | | internal: http://controller/instance-ha/v1/<tenant_id> |
| | | RegionOne |
| | | admin: http://controller/instance-ha/v1/<tenant_id> |
| | | RegionOne |
| | | public: http://controller/instance-ha/v1/<tenant_id> |
| | | |
| keystone | identity | RegionOne |
| | | public: http://controller/identity |
| | | RegionOne |
| | | admin: http://controller/identity |
| | | |
| cinderv2 | volumev2 | RegionOne |
| | | public: http://controller/volume/v2/<tenant_id> |
| | | |
| placement | placement | RegionOne |
| | | public: http://controller/placement |
| | | |
| neutron | network | RegionOne |
| | | public: http://controller:9696/ |
| | | |
+-------------+----------------+--------------------------------------------------------+
3. Run segment list command to verify masakari-api is running properly. This will return empty segment
list as you haven’t yet configured Failover segments.
$ openstack segment list
NOTE:
Since Failover segments are not configured, there is no way to verify masakari-engine is running
properly as the notification cannot be sent from masakari-api to masakari-engine.
Reference Material
• Configuration Guide: Information on configuration files.
• Custom Recovery Workflow Configuration Guide
• CLI Commands for Masakari: The complete command reference for Masakari.
• Versioned Notifications: This provides the list of existing versioned notifications with sample
payloads. This will help newcomers understand basics of Masakari
• Nova Docs: A collection of guides for Nova.
Masakari CLI Documentation
In this section you will find information on Masakari’s command line interface.
masakari-status
CLI interface for Masakari status commands
Synopsis
masakari-status <category> <command> [<args>]
Description
masakari-status is a tool that provides routines for checking the status of a Masakari deployment.
Options
The standard pattern for executing a masakari-status command is:
masakari-status <category> <command> [<args>]
Run without arguments to see a list of available command categories:
masakari-status
Categories are:
• upgrade
Detailed descriptions are below:
You can also run with a category argument such as upgrade to see a list of all commands in that category:
masakari-status upgrade
These sections describe the available categories and arguments for masakari-status.
Upgrade
masakari-status upgrade check
Performs a release-specific readiness check before restarting services with new code. For example,
missing or changed configuration options, incompatible object states, or other conditions that
could lead to failures while upgrading.
Return Codes
┌─────────────┬───────────────────────────────────────┐
│ Return code │ Description │
├─────────────┼───────────────────────────────────────┤
│ 0 │ All upgrade readiness checks passed │
│ │ successfully and there is nothing to │
│ │ do. │
├─────────────┼───────────────────────────────────────┤
│ 1 │ At least one check encountered an │
│ │ issue and requires further │
│ │ investigation. This is considered a │
│ │ warning but the upgrade may be OK. │
├─────────────┼───────────────────────────────────────┤
│ 2 │ There was an upgrade status check │
│ │ failure that needs to be │
│ │ investigated. This should be │
│ │ considered something that stops an │
│ │ upgrade. │
├─────────────┼───────────────────────────────────────┤
│ 255 │ An unexpected error occurred. │
└─────────────┴───────────────────────────────────────┘
History of Checks
7.0.0 (Stein)
• Sample check to be filled in with checks as they are added in Stein.
masakari-manage
Control and manage masakari database
Synopsis
masakari-manage <category> <action> [<args>]
Description
masakari-manage controls DB by managing various admin-only aspects of masakari.
Options
The standard pattern for executing a masakari-manage command is:
masakari-manage <category> <command> [<args>]
Run without arguments to see a list of available command categories:
masakari-manage
You can also run with a category argument such as db to see a list of all commands in that category:
masakari-manage db
These sections describe the available categories and arguments for masakari-manage.
Masakari Database
masakari-manage db version
Print the current main database version.
masakari-manage db sync [--version <version>]
Upgrade the main database schema up to the most recent version or --version if specified.
masakari-manage db purge
Deleting rows older than 30 day(s) from table hosts, failover_segments and notifications.
openstack masakari
To control and manage masakari operations, the extended command list available in openstack command.
api-paste.ini
The masakari service stores its API configuration settings in the api-paste.ini file.
[composite:masakari_api]
use = call:masakari.api.urlmap:urlmap_factory
/: apiversions
/v1: masakari_api_v1
[composite:masakari_api_v1]
use = call:masakari.api.auth:pipeline_factory_v1
keystone = cors http_proxy_to_wsgi request_id faultwrap sizelimit authtoken keystonecontext osapi_masakari_app_v1
noauth2 = cors http_proxy_to_wsgi request_id faultwrap sizelimit noauth2 osapi_masakari_app_v1
# filters
[filter:cors]
paste.filter_factory = oslo_middleware.cors:filter_factory
oslo_config_project = masakari
[filter:http_proxy_to_wsgi]
paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory
[filter:request_id]
paste.filter_factory = oslo_middleware:RequestId.factory
[filter:faultwrap]
paste.filter_factory = masakari.api.openstack:FaultWrapper.factory
[filter:sizelimit]
paste.filter_factory = oslo_middleware:RequestBodySizeLimiter.factory
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
[filter:keystonecontext]
paste.filter_factory = masakari.api.auth:MasakariKeystoneContext.factory
[filter:noauth2]
paste.filter_factory = masakari.api.auth:NoAuthMiddleware.factory
# apps
[app:osapi_masakari_app_v1]
paste.app_factory = masakari.api.openstack.ha:APIRouterV1.factory
[pipeline:apiversions]
pipeline = faultwrap http_proxy_to_wsgi apiversionsapp
[app:apiversionsapp]
paste.app_factory = masakari.api.openstack.ha.versions:Versions.factory
Configuration Options
The following is an overview of all available configuration options in Masakari.
DEFAULT
auth_strategy
Type string
Default
keystone
Valid Values
keystone, noauth2
This determines the strategy to use for authentication: keystone or noauth2. ‘noauth2’ is
designed for testing only, as it does no actual credential checking. ‘noauth2’ provides
administrative credentials only if ‘admin’ is specified as the username.
• Possible values:
Either ‘keystone’ (default) or ‘noauth2’.
• Services that use this:
masakari-api
• Related options:
None
use_forwarded_for
Type boolean
Default
False
When True, the ‘X-Forwarded-For’ header is treated as the canonical remote address. When False
(the default), the ‘remote_address’ header is used.
You should only enable this if you have an HTML sanitizing proxy.
• Possible values:
True, False (default)
• Services that use this:
masakari-api
• Related options:
None
osapi_max_limit
Type integer
Default
1000
As a query can potentially return many thousands of items, you can limit the maximum number of
items in a single response by setting this option.
• Possible values:
Any positive integer. Default is 1000.
• Services that use this:
masakari-api
• Related options:
None
osapi_masakari_link_prefix
Type string
Default
<None>
This string is prepended to the normal URL that is returned in links to the OpenStack Masakari
API. If it is empty (the default), the URLs are returned unchanged.
• Possible values:
Any string, including an empty string (the default).
• Services that use this:
masakari-api
• Related options:
None
tempdir
Type string
Default
<None>
Explicitly specify the temporary working directory.
monkey_patch
Type boolean
Default
False
Determine if monkey patching should be applied.
Related options:
• monkey_patch_modules: This must have values set for this option to have
any effect
monkey_patch_modules
Type list
Default
['masakari.api:masakari.cmd']
List of modules/decorators to monkey patch.
This option allows you to patch a decorator for all functions in specified modules.
Related options:
• monkey_patch: This must be set to True for this option to have any effect
masakari_topic
Type string
Default
ha_engine
This is the message queue topic that the masakari engine ‘listens’ on. It is used when the
masakari engine is started up to configure the queue, and whenever an RPC call to the masakari
engine is made.
• Possible values:
Any string, but there is almost never any reason to ever change this value from its default
of ‘engine’.
• Services that use this:
masakari-engine
• Related options:
None
WARNING:
This option is deprecated for removal since 3.0.0. Its value may be silently ignored in the
future.
Reason Configurable RPC topic provides little value and it can easily break Masakari if
operator configures it to the same topic used by other OpenStack services.
duplicate_notification_detection_interval
Type integer
Default
180
Minimum Value
0
Interval in seconds for identifying duplicate notifications. If the notification received is
identical to the previous ones whose status is either new or running and if it’s created_timestamp
and the current timestamp is less than this config option value, then the notification will be
considered as duplicate and it will be ignored.
wait_period_after_service_update
Type integer
Default
180
Number of seconds to wait after a service is enabled or disabled.
wait_period_after_evacuation
Type integer
Default
90
Wait until instance is evacuated
verify_interval
Type integer
Default
1
The monitoring interval for looping
wait_period_after_power_off
Type integer
Default
180
Number of seconds to wait for instance to shut down
wait_period_after_power_on
Type integer
Default
60
Number of seconds to wait for instance to start
process_unfinished_notifications_interval
Type integer
Default
120
Interval in seconds for processing notifications which are in error or new state.
retry_notification_new_status_interval
Type integer
Default
60
Mutable
This option can be changed without restarting.
Interval in seconds for identifying notifications which are in new state. If the notification is
in new state till this config option value after it’s generated_time, then it is considered that
notification is ignored by the messaging queue and will be processed by
‘process_unfinished_notifications’ periodic task.
check_expired_notifications_interval
Type integer
Default
600
Interval in seconds for checking running notifications.
notifications_expired_interval
Type integer
Default
86400
Interval in seconds for identifying running notifications expired.
host_failure_recovery_threads
Type integer
Default
3
Minimum Value
1
Number of threads to be used for evacuating and confirming instances during execution of
host_failure workflow.
notification_driver
Type string
Default
taskflow_driver
Defines which driver to use for executing notification workflows.
fatal_exception_format_errors
Type boolean
Default
False
Make exception message format errors fatal
nova_catalog_admin_info
Type string
Default
compute:nova:publicURL
Match this value when searching for nova in the service catalog. Format is: separated values of
the form: <service_type>:<service_name>:<endpoint_type>
os_region_name
Type string
Default
<None>
Region name of this node
nova_ca_certificates_file
Type string
Default
<None>
Location of ca certificates file to use for nova client requests.
nova_api_insecure
Type boolean
Default
False
Allow to perform insecure SSL requests to nova
os_privileged_user_name
Type string
Default
<None>
OpenStack privileged account username. Used for requests to other services (such as Nova) that
require an account with special rights.
os_privileged_user_password
Type string
Default
<None>
Password associated with the OpenStack privileged account.
os_privileged_user_tenant
Type string
Default
<None>
Tenant name associated with the OpenStack privileged account.
os_privileged_user_auth_url
Type URI
Default
<None>
Auth URL associated with the OpenStack privileged account.
os_user_domain_name
Type string
Default
default
User domain name associated with the OpenStack privileged account.
os_project_domain_name
Type string
Default
default
Project domain name associated with the OpenStack privileged account.
os_system_scope
Type string
Default
<None>
Scope for system operations.
pybasedir
Type string
Default
/build/masakari-FF1dfn/masakari-17.0.0
Directory where the masakari python module is installed
bindir
Type string
Default
/usr/local/bin
Directory where masakari binaries are installed
state_path
Type string
Default
$pybasedir
Top-level directory for maintaining masakari’s state
host
Type host address
Default
lcy02-amd64-036
Hostname, FQDN or IP address of this host. Must be valid within AMQP key.
Possible values:
• String with hostname, FQDN or IP address. Default is hostname of this host.
engine_manager
Type string
Default
masakari.engine.manager.MasakariManager
Full class name for the Manager for masakari engine
report_interval
Type integer
Default
10
Seconds between nodes reporting state to datastore
periodic_enable
Type boolean
Default
True
Enable periodic tasks
periodic_interval_max
Type integer
Default
300
Max interval time between periodic tasks execution in seconds.
periodic_fuzzy_delay
Type integer
Default
60
Range of seconds to randomly delay when starting the periodic task scheduler to reduce stampeding.
(Disable by setting to 0)
use_ssl
Type boolean
Default
False
Use APIs with SSL enabled
masakari_api_listen
Type host address
Default
0.0.0.0
The IP address on which the Masakari API will listen.
masakari_api_listen_port
Type integer
Default
15868
Minimum Value
1
Maximum Value
65535
The port on which the Masakari API will listen.
masakari_api_workers
Type integer
Default
<None>
Number of workers for Masakari API service. The default will be the number of CPUs available.
service_down_time
Type integer
Default
60
Maximum time since last check-in for up service
debug
Type boolean
Default
False
Mutable
This option can be changed without restarting.
If set to true, the logging level will be set to DEBUG instead of the default INFO level.
log_config_append
Type string
Default
<None>
Mutable
This option can be changed without restarting.
The name of a logging configuration file. This file is appended to any existing logging
configuration files. For details about logging configuration files, see the Python logging module
documentation. Note that when logging configuration files are used then all logging configuration
is set in the configuration file and other logging configuration options are ignored (for example,
log-date-format).
Deprecated Variations
────────────────────────
Group Name
────────────────────────
DEFAULT log-config
────────────────────────
DEFAULT log_config
┌─────────┬────────────┐
│ │ │
log_date_format │ │ │
│ │ │
Type string │ │ │
│ │ │
Default │ │ │
%Y-%m-%d %H:%M:%S │ │ │
│ │ │
--
AUTHOR
unknown
COPYRIGHT
2024, OpenStack Foundation
17.0.0 Apr 05, 2024 MASAKARI(1)