xenial (1) hbal.1.gz

Provided by: ganeti-htools-2.15_2.15.2-3_amd64 bug

NAME

       hbal - Cluster balancer for Ganeti

SYNOPSIS

       hbal {backend options...} [algorithm options...] [reporting options...]

       hbal --version

       Backend options:

       { -m cluster | -L[ path ] [-X] | -t data-file | -I path }

       Algorithm options:

       [  --max-cpu  cpu-ratio  ]  [  --min-disk  disk-ratio  ]  [  -l  limit  ]  [  -e  score  ] [ -g delta ] [
       --min-gain-limit threshold ] [ -O name... ] [ --no-disk-moves ] [ --no-instance-moves ] [ -U util-file  ]
       [ --ignore-dynu ] [ --ignore-soft-errors ] [ --mond yes|no ] [ --mond-xen ] [ --exit-on-missing-mond-data
       ] [ --evac-mode ] [ --restricted-migration ] [ --select-instances inst... ] [ --exclude-instances inst...
       ]

       Reporting options:

       [ -C[ file ] ] [ -p[ fields ] ] [ --print-instances ] [ -S file ] [ -v... | -q ]

DESCRIPTION

       hbal  is  a  cluster  balancer that looks at the current state of the cluster (nodes with their total and
       free disk, memory, etc.) and instance placement and computes a series of  steps  designed  to  bring  the
       cluster into a better state.

       The  algorithm  used is designed to be stable (i.e.  it will give you the same results when restarting it
       from the middle of the solution) and reasonably fast.  It is not,  however,  designed  to  be  a  perfect
       algorithm:  it  is possible to make it go into a corner from which it can find no improvement, because it
       looks only one "step" ahead.

       The program accesses the cluster state via Rapi or Luxi.  It also requests data over the network from all
       MonDs with the --mond option.  Currently it uses only data produced by CPUload collector.

       By  default,  the  program  will show the solution incrementally as it is computed, in a somewhat cryptic
       format; for getting the actual Ganeti command list, use the -C option.

   ALGORITHM
       The program works in independent steps; at each step, we compute the best instance move that  lowers  the
       cluster score.

       The  possible  move type for an instance are combinations of failover/migrate and replace-disks such that
       we change one of the instance nodes, and the other one remains (but  possibly  with  changed  role,  e.g.
       from primary it becomes secondary).  The list is:

       • failover (f)

       • replace secondary (r)

       • replace primary, a composite move (f, r, f)

       • failover and replace secondary, also composite (f, r)

       • replace secondary and failover, also composite (r, f)

       We  don't  do  the only remaining possibility of replacing both nodes (r,f,r,f or the equivalent f,r,f,r)
       since these move needs an exhaustive search over both candidate  primary  and  secondary  nodes,  and  is
       O(n*n)  in  the  number of nodes.  Furthermore, it doesn't seems to give better scores but will result in
       more disk replacements.

   PLACEMENT RESTRICTIONS
       At each step, we prevent an instance move if it would cause:

       • a node to go into N+1 failure state

       • an instance to move onto an offline node (offline nodes are either read from the  cluster  or  declared
         with -O; drained nodes are considered offline)

       • an  exclusion-tag  based  conflict  (exclusion  tags  are  read from the cluster and/or defined via the
         --exclusion-tags option)

       • a max vcpu/pcpu ratio to be exceeded (configured via --max-cpu)

       • min disk free percentage to go below the configured limit (configured via --min-disk)

   CLUSTER SCORING
       As said before, the algorithm tries to minimise the cluster score at each step.  Currently this score  is
       computed as a weighted sum of the following components:

       • standard deviation of the percent of free memory

       • standard deviation of the percent of reserved memory

       • the sum of the percentages of reserved memory

       • standard deviation of the percent of free disk

       • count of nodes failing N+1 check

       • count  of instances living (either as primary or secondary) on offline nodes; in the sense of hbal (and
         the other htools) drained nodes are considered offline

       • count of instances living (as primary) on offline nodes; this differs from the above metric by  helping
         failover of such instances in 2-node clusters

       • standard deviation of the ratio of virtual-to-physical cpus (for primary instances of the node)

       • standard  deviation  of  the  fraction of the available spindles (in dedicated mode, spindles represent
         physical spindles; otherwise this oversubscribable measure for IO load, and the oversubscription factor
         is taken into account when computing the number of available spindles)

       • standard deviation of the dynamic load on the nodes, for cpus, memory, disk and network

       • standard deviation of the CPU load provided by MonD

       • the count of instances with primary and secondary in the same failure domain

       The  free  memory and free disk values help ensure that all nodes are somewhat balanced in their resource
       usage.  The reserved memory helps to ensure  that  nodes  are  somewhat  balanced  in  holding  secondary
       instances,  and  that  no  node  keeps too much memory reserved for N+1.  And finally, the N+1 percentage
       helps guide the algorithm towards eliminating N+1 failures, if possible.

       Except for the N+1 failures, offline instances counts, and failure domain violation counts,  we  use  the
       standard  deviation since when used with values within a fixed range (we use percents expressed as values
       between zero and one) it gives consistent results across all metrics (there are some small issues related
       to  different  means,  but  it works generally well).  The 'count' type values will have higher score and
       thus will matter more for balancing; thus these are better for hard constraints  (like  evacuating  nodes
       and fixing N+1 failures).  For example, the offline instances count (i.e.  the number of instances living
       on offline nodes) will cause the algorithm to actively move instances away  from  offline  nodes.   This,
       coupled with the restriction on placement given by offline nodes, will cause evacuation of such nodes.

       The  dynamic  load  values  need  to  be read from an external file (Ganeti doesn't supply them), and are
       computed for each node as: sum of primary instance cpu load, sum of primary instance memory load, sum  of
       primary  and  secondary instance disk load (as DRBD generates write load on secondary nodes too in normal
       case and in degraded scenarios also read load), and sum of primary instance network load.  An example  of
       how  to generate these values for input to hbal would be to track xm list for instances over a day and by
       computing the delta of the cpu values, and feed that via the -U option for all instances  (and  keep  the
       other  metrics  as one).  For the algorithm to work, all that is needed is that the values are consistent
       for a metric across all instances (e.g.  all instances use cpu% to report cpu usage,  and  not  something
       related to number of CPU seconds used if the CPUs are different), and that they are normalised to between
       zero and one.  Note that it's recommended to not have zero as the load  value  for  any  instance  metric
       since then secondary instances are not well balanced.

       The  CPUload  from  MonD's  data collector will be used only if all MonDs are running, otherwise it won't
       affect the cluster score.  Since we can't find the CPU load of each instance, we can assume that the  CPU
       load  of  an  instance  is  proportional to the number of its vcpus.  With this heuristic, instances from
       nodes with high CPU load will tend to move to nodes with less CPU load.

       On a perfectly balanced cluster (all nodes the same size, all instances the same size and  spread  across
       the  nodes equally), the values for all metrics would be zero, with the exception of the total percentage
       of reserved memory.  This doesn't happen too often in practice :)

   OFFLINE INSTANCES
       Since current Ganeti versions do not report the memory used by offline (down) instances, ignoring the run
       status  of  instances will cause wrong calculations.  For this reason, the algorithm subtracts the memory
       size of down instances from the free node memory of their primary node, in effect simulating the  startup
       of such instances.

   EXCLUSION TAGS
       The  exclusion tags mechanism is designed to prevent instances which run the same workload (e.g.  two DNS
       servers) to land on the same node, which would make the respective node a SPOF for the given service.

       It works by tagging instances with certain tags and then building exclusion maps based on  these.   Which
       tags  are actually used is configured either via the command line (option --exclusion-tags) or via adding
       them to the cluster tags:

       --exclusion-tags=a,b
              This will make all instance tags of the form a:*, b:* be considered for the exclusion map

       cluster tags htools:iextags:a, htools:iextags:b
              This will make instance tags a:*, b:* be considered for the exclusion map.   More  precisely,  the
              suffix of cluster tags starting with htools:iextags: will become the prefix of the exclusion tags.

       Both  the  above forms mean that two instances both having (e.g.) the tag a:foo or b:bar won't end on the
       same node.

   MIGRATION TAGS
       If Ganeti is deployed on a heterogeneous cluster, migration might not be possible between all nodes of  a
       node  group.   One  example  of  such a situation is upgrading the hypervisor node by node.  To make hbal
       aware of those restrictions, the following cluster tags are used.

       cluster tags htools:migration:a, htools:migration:b, etc
              This make make node tags of the form a:*, b:*, etc  be  considered  migration  restriction.   More
              precisely,  the  suffix  of cluster tags starting with htools:migration: will become the prefix of
              the migration tags.  Only those migrations will be taken into consideration  where  all  migration
              tags of the source node are also present on the target node.

       cluster tags htools:allowmigration:x::y for migration tags x and y
              This  asserts that a node taged y is able to receive instances in the same way as if they had an x
              tag.

       So in the simple case of a hypervisor upgrade, tagging all the nodes  that  have  been  upgraded  with  a
       migration  tag  suffices.   In  more  complicated  situations,  it  is always possible to use a different
       migration tag for each hypervisor used and explictly state the allowed migration directions by  means  of
       htools:allowmigration: tags.

   LOCATION TAGS
       Within  a  node group, certain nodes might be more likely to fail simultaneously due to a common cause of
       error (e.g., if they share the same power supply unit).  Ganeti can be made aware of thos  common  causes
       of failure by means of tags.

       cluster tags htools:nlocation:a, htools:nlocation:b, etc
              This  make  make  node  tags  of  the  form  a:*, b:*, etc be considered to have a common cause of
              failure.

       Instances with primary and secondary node having a common cause of failure are considered  badly  placed.
       While such placements are always allowed, they count heavily towards the cluster score.

OPTIONS

       The options that can be passed to the program are as follows:

       -C, --print-commands
              Print the command list at the end of the run.  Without this, the program will only show a shorter,
              but cryptic output.

              Note that the moves list will be split into independent steps,  called  "jobsets",  but  only  for
              visual  inspection,  not  for  actually  parallelisation.  It is not possible to parallelise these
              directly when executed via "gnt-instance" commands, since a compound command (e.g.   failover  and
              replace-disks) must be executed serially.  Parallel execution is only possible when using the Luxi
              backend and the -L option.

              The algorithm for splitting the moves into jobsets is by accumulating moves until the next move is
              touching  nodes already touched by the current moves; this means we can't execute in parallel (due
              to resource allocation in Ganeti) and thus we start a new jobset.

       -p, --print-nodes
              Prints the before and after node status, in a format designed to allow the user to understand  the
              node's most important parameters.  See the man page htools(1) for more details about this option.

       --print-instances
              Prints the before and after instance map.  This is less useful as the node status, but it can help
              in understanding instance moves.

       -O name
              This option (which can be given multiple times) will mark nodes as being offline.   This  means  a
              couple of things:

              • instances  won't  be placed on these nodes, not even temporarily; e.g.  the replace primary move
                is not available if the secondary node is offline, since this move requires a failover.

              • these nodes will not be included  in  the  score  calculation  (except  for  the  percentage  of
                instances on offline nodes)

              Note  that  algorithm  will  also mark as offline any nodes which are reported by RAPI as such, or
              that have "?" in file-based input in any numeric fields.

       -e score, --min-score=*score*
              This parameter denotes how much above the N+1 bound the cluster score can for us to be happy  with
              and alters the computation in two ways:

              • if the cluster has the initial score lower than this value, then we don't enter the algorithm at
                all, and exit with success

              • during the iterative process, if we reach a score lower than this value, we exit the algorithm

              The default value of the parameter is currently 1e-9 (chosen empirically).

       -g delta, --min-gain=*delta*
              Since the balancing algorithm can sometimes result in just very tiny improvements, that bring less
              gain that they cost in relocation time, this parameter (defaulting to 0.01) represents the minimum
              gain we require during a step, to continue balancing.

       --min-gain-limit=*threshold*
              The above min-gain option will only take effect if the cluster score is  already  below  threshold
              (defaults  to  0.1).   The  rationale  behind  this  setting is that at high cluster scores (badly
              balanced clusters), we don't want to abort the rebalance too quickly, as later gains  might  still
              be  significant.   However, under the threshold, the total gain is only the threshold value, so we
              can exit early.

       --no-disk-moves
              This parameter prevents hbal from using disk move (i.e.  "gnt-instance replace-disks") operations.
              This  will  result in a much quicker balancing, but of course the improvements are limited.  It is
              up to the user to decide when to use one or another.

       --no-instance-moves
              This parameter prevents hbal from using instance  moves  (i.e.   "gnt-instance  migrate/failover")
              operations.   This  will  only  use  the slow disk-replacement operations, and will also provide a
              worse balance, but can be useful if moving instances around is deemed unsafe or not preferred.

       --evac-mode
              This parameter restricts the list of instances  considered  for  moving  to  the  ones  living  on
              offline/drained nodes.  It can be used as a (bulk) replacement for Ganeti's own gnt-node evacuate,
              with the note that it doesn't guarantee full evacuation.

       --restricted-migration
              This parameter disallows any replace-primary moves (frf), as well  as  those  replace-and-failover
              moves  (rf)  where  the  primary  node  of the instance is not drained.  If used together with the
              --evac-mode option, the only migrations that hbal will  do  are  migrations  of  instances  off  a
              drained  node.  This can be useful if during a reinstall of the base operating system migration is
              only possible from the old OS to the new OS.  Note, however, that usually  the  use  of  migration
              tags is the better choice.

       --select-instances=*instances*
              This  parameter marks the given instances (as a comma-separated list) as the only ones being moved
              during the rebalance.

       --exclude-instances=*instances*
              This parameter marks the given instances (as a comma-separated list) from being moved  during  the
              rebalance.

       -U util-file
              This parameter specifies a file holding instance dynamic utilisation information that will be used
              to tweak the balancing algorithm to equalise load on the nodes  (as  opposed  to  static  resource
              usage).   The file is in the format "instance_name cpu_util mem_util disk_util net_util" where the
              "_util" parameters are interpreted as numbers  and  the  instance  name  must  match  exactly  the
              instance as read from Ganeti.  In case of unknown instance names, the program will abort.

              If not given, the default values are one for all metrics and thus dynamic utilisation has only one
              effect on the algorithm: the equalisation of the secondary instances across  nodes  (this  is  the
              only  metric  that is not tracked by another, dedicated value, and thus the disk load of instances
              will cause secondary instance equalisation).  Note that value of one will also influence  slightly
              the  primary  instance count, but that is already tracked via other metrics and thus the influence
              of the dynamic utilisation will be practically insignificant.

       --ignore-dynu
              If given, all dynamic utilisation information will be ignored by assuming it to be 0.  This option
              will take precedence over any data passed by the -U option or by the MonDs with the --mond and the
              --mond-data option.

       --ignore-soft-errors
              If given, all checks for soft errors will be ommitted when considering balancing moves.   In  this
              way,  progress  can  be  made  in  a  cluster where all nodes are in a policy-wise bad state, like
              exceeding oversubscription ratios on CPU or spindles.

       -S filename, --save-cluster=*filename*
              If given, the state of the cluster before the balancing is  saved  to  the  given  file  plus  the
              extension "original" (i.e.  filename.original), and the state at the end of the balancing is saved
              to the given file plus the extension "balanced" (i.e.  filename.balanced).  This allows re-feeding
              the cluster state to either hbal itself or for example hspace via the -t option.

       -t datafile, --text-data=*datafile*
              Backend  specification:  the  name  of  the  file  holding  node  and instance information (if not
              collecting via RAPI or LUXI).  This or one of the other backends must be selected.  The option  is
              described in the man page htools(1).

       --mond=*yes|no*
              If  given  the  program will query all MonDs to fetch data from the supported data collectors over
              the network.

       --mond-xen
              If given, also query Xen-specific collectors from  MonD,  provided  that  monitoring  daemons  are
              queried at all.

       --exit-on-missing-mond-data
              If given, abort if the data obtainable from querying MonDs is incomplete.  The default behavior is
              to continue with a best guess based on the static information.

       --mond-data datafile
              The name of the file holding the data provided  by  MonD,  to  override  quering  MonDs  over  the
              network.  This is mostly used for debugging.  The file must be in JSON format and present an array
              of JSON objects , one for every node, with two members.  The first member named node is  the  name
              of the node and the second member named reports is an array of report objects.  The report objects
              must be in the same format as produced by the monitoring agent.

       -m cluster
              Backend specification: collect data directly from the cluster given as an argument via RAPI.   The
              option is described in the man page htools(1).

       -L [path]
              Backend  specification: collect data directly from the master daemon, which is to be contacted via
              LUXI (an internal Ganeti protocol).  The option is described in the man page htools(1).

       -X     When using the Luxi backend, hbal can also execute the given commands.  The execution method is to
              execute  the individual jobsets (see the -C option for details) in separate stages, aborting if at
              any time a jobset doesn't have all jobs successful.  Each step in the balancing solution  will  be
              translated  into  exactly one Ganeti job (having between one and three OpCodes), and all the steps
              in a jobset will be executed in parallel.  The jobsets themselves are executed serially.

              The execution of the job series can be interrupted, see below for signal handling.

       -l N, --max-length=*N*
              Restrict the solution to this length.  This can be used for example to automate the  execution  of
              the balancing.

       --max-cpu=*cpu-ratio*
              The  maximum  virtual  to  physical cpu ratio, as a floating point number greater than or equal to
              one.  For example, specifying cpu-ratio as 2.5 means that, for a 4-cpu machine, a  maximum  of  10
              virtual  cpus  should be allowed to be in use for primary instances.  A value of exactly one means
              there will be no over-subscription of CPU (except for the CPU time used by the node  itself),  and
              values  below  one  do  not  make sense, as that means other resources (e.g.  disk) won't be fully
              utilised due to CPU restrictions.

       --min-disk=*disk-ratio*
              The minimum amount of free disk space  remaining,  as  a  floating  point  number.   For  example,
              specifying disk-ratio as 0.25 means that at least one quarter of disk space should be left free on
              nodes.

       -G uuid, --group=*uuid*
              On an multi-group cluster, select this group for processing.  Otherwise hbal will abort, since  it
              cannot balance multiple groups at the same time.

       -v, --verbose
              Increase  the  output verbosity.  Each usage of this option will increase the verbosity (currently
              more than 2 doesn't make sense) from the default of one.

       -q, --quiet
              Decrease the output verbosity.  Each usage of this option will decrease the verbosity  (less  than
              zero doesn't make sense) from the default of one.

       -V, --version
              Just show the program version and exit.

SIGNAL HANDLING

       When  executing jobs via LUXI (using the -X option), normally hbal will execute all jobs until either one
       errors out or all the jobs finish successfully.

       Since balancing can take a long time, it is possible to stop hbal early in two ways:

       • by sending a SIGINT (^C), hbal will register the termination request, and will wait until the currently
         submitted  jobs  finish,  at which point it will exit (with exit code 0 if all jobs finished correctly,
         otherwise with exit code 1 as usual)

       • by sending a SIGTERM, hbal will immediately exit (with exit code 2); it is the  responsibility  of  the
         user to follow up with Ganeti and check the result of the currently-executing jobs

       Note  that  in  any  situation, it's perfectly safe to kill hbal, either via the above signals or via any
       other signal (e.g.  SIGQUIT, SIGKILL), since the jobs themselves are processed  by  Ganeti  whereas  hbal
       (after  submission) only watches their progression.  In this case, the user will have to query Ganeti for
       job results.

EXIT STATUS

       The exit status of the command will be zero, unless for some reason the  algorithm  failed  (e.g.   wrong
       node  or  instance data), invalid command line options, or (in case of job execution) one of the jobs has
       failed.

       Once job execution via Luxi has started (-X), if the balancing was interrupted early (via SIGINT, or  via
       --max-length)  but  all  jobs  executed  successfully, then the exit status is zero; a non-zero exit code
       means that the cluster state should be investigated, since a job failed or we couldn't compute its status
       and this can also point to a problem on the Ganeti side.

BUGS

       The  program  does  not check all its input data for consistency, and sometime aborts with cryptic errors
       messages with invalid data.

       The algorithm is not perfect.

EXAMPLE

       Note that these examples are not for the latest version (they don't have full node data).

   Default output
       With the default options, the program shows each individual  step  and  the  improvements  it  brings  in
       cluster score:

              $ hbal
              Loaded 20 nodes, 80 instances
              Cluster is not N+1 happy, continuing but no guarantee that the cluster will end N+1 happy.
              Initial score: 0.52329131
              Trying to minimize the CV...
                  1. instance14  node1:node10  => node16:node10 0.42109120 a=f r:node16 f
                  2. instance54  node4:node15  => node16:node15 0.31904594 a=f r:node16 f
                  3. instance4   node5:node2   => node2:node16  0.26611015 a=f r:node16
                  4. instance48  node18:node20 => node2:node18  0.21361717 a=r:node2 f
                  5. instance93  node19:node18 => node16:node19 0.16166425 a=r:node16 f
                  6. instance89  node3:node20  => node2:node3   0.11005629 a=r:node2 f
                  7. instance5   node6:node2   => node16:node6  0.05841589 a=r:node16 f
                  8. instance94  node7:node20  => node20:node16 0.00658759 a=f r:node16
                  9. instance44  node20:node2  => node2:node15  0.00438740 a=f r:node15
                 10. instance62  node14:node18 => node14:node16 0.00390087 a=r:node16
                 11. instance13  node11:node14 => node11:node16 0.00361787 a=r:node16
                 12. instance19  node10:node11 => node10:node7  0.00336636 a=r:node7
                 13. instance43  node12:node13 => node12:node1  0.00305681 a=r:node1
                 14. instance1   node1:node2   => node1:node4   0.00263124 a=r:node4
                 15. instance58  node19:node20 => node19:node17 0.00252594 a=r:node17
              Cluster score improved from 0.52329131 to 0.00252594

       In the above output, we can see:

       • the input data (here from files) shows a cluster with 20 nodes and 80 instances

       • the cluster is not initially N+1 compliant

       • the initial score is 0.52329131

       The  step  list  follows,  showing  the  instance,  its  initial primary/secondary nodes, the new primary
       secondary, the cluster list, and the actions taken in this step (with 'f' denoting  failover/migrate  and
       'r' denoting replace secondary).

       Finally, the program shows the improvement in cluster score.

       A more detailed output is obtained via the -C and -p options:

              $ hbal
              Loaded 20 nodes, 80 instances
              Cluster is not N+1 happy, continuing but no guarantee that the cluster will end N+1 happy.
              Initial cluster status:
              N1 Name   t_mem f_mem r_mem t_dsk f_dsk pri sec  p_fmem  p_fdsk
               * node1  32762  1280  6000  1861  1026   5   3 0.03907 0.55179
                 node2  32762 31280 12000  1861  1026   0   8 0.95476 0.55179
               * node3  32762  1280  6000  1861  1026   5   3 0.03907 0.55179
               * node4  32762  1280  6000  1861  1026   5   3 0.03907 0.55179
               * node5  32762  1280  6000  1861   978   5   5 0.03907 0.52573
               * node6  32762  1280  6000  1861  1026   5   3 0.03907 0.55179
               * node7  32762  1280  6000  1861  1026   5   3 0.03907 0.55179
                 node8  32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node9  32762  7280  6000  1861  1026   4   4 0.22221 0.55179
               * node10 32762  7280 12000  1861  1026   4   4 0.22221 0.55179
                 node11 32762  7280  6000  1861   922   4   5 0.22221 0.49577
                 node12 32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node13 32762  7280  6000  1861   922   4   5 0.22221 0.49577
                 node14 32762  7280  6000  1861   922   4   5 0.22221 0.49577
               * node15 32762  7280 12000  1861  1131   4   3 0.22221 0.60782
                 node16 32762 31280     0  1861  1860   0   0 0.95476 1.00000
                 node17 32762  7280  6000  1861  1106   5   3 0.22221 0.59479
               * node18 32762  1280  6000  1396   561   5   3 0.03907 0.40239
               * node19 32762  1280  6000  1861  1026   5   3 0.03907 0.55179
                 node20 32762 13280 12000  1861   689   3   9 0.40535 0.37068

              Initial score: 0.52329131
              Trying to minimize the CV...
                  1. instance14  node1:node10  => node16:node10 0.42109120 a=f r:node16 f
                  2. instance54  node4:node15  => node16:node15 0.31904594 a=f r:node16 f
                  3. instance4   node5:node2   => node2:node16  0.26611015 a=f r:node16
                  4. instance48  node18:node20 => node2:node18  0.21361717 a=r:node2 f
                  5. instance93  node19:node18 => node16:node19 0.16166425 a=r:node16 f
                  6. instance89  node3:node20  => node2:node3   0.11005629 a=r:node2 f
                  7. instance5   node6:node2   => node16:node6  0.05841589 a=r:node16 f
                  8. instance94  node7:node20  => node20:node16 0.00658759 a=f r:node16
                  9. instance44  node20:node2  => node2:node15  0.00438740 a=f r:node15
                 10. instance62  node14:node18 => node14:node16 0.00390087 a=r:node16
                 11. instance13  node11:node14 => node11:node16 0.00361787 a=r:node16
                 12. instance19  node10:node11 => node10:node7  0.00336636 a=r:node7
                 13. instance43  node12:node13 => node12:node1  0.00305681 a=r:node1
                 14. instance1   node1:node2   => node1:node4   0.00263124 a=r:node4
                 15. instance58  node19:node20 => node19:node17 0.00252594 a=r:node17
              Cluster score improved from 0.52329131 to 0.00252594

              Commands to run to reach the above solution:
                echo step 1
                echo gnt-instance migrate instance14
                echo gnt-instance replace-disks -n node16 instance14
                echo gnt-instance migrate instance14
                echo step 2
                echo gnt-instance migrate instance54
                echo gnt-instance replace-disks -n node16 instance54
                echo gnt-instance migrate instance54
                echo step 3
                echo gnt-instance migrate instance4
                echo gnt-instance replace-disks -n node16 instance4
                echo step 4
                echo gnt-instance replace-disks -n node2 instance48
                echo gnt-instance migrate instance48
                echo step 5
                echo gnt-instance replace-disks -n node16 instance93
                echo gnt-instance migrate instance93
                echo step 6
                echo gnt-instance replace-disks -n node2 instance89
                echo gnt-instance migrate instance89
                echo step 7
                echo gnt-instance replace-disks -n node16 instance5
                echo gnt-instance migrate instance5
                echo step 8
                echo gnt-instance migrate instance94
                echo gnt-instance replace-disks -n node16 instance94
                echo step 9
                echo gnt-instance migrate instance44
                echo gnt-instance replace-disks -n node15 instance44
                echo step 10
                echo gnt-instance replace-disks -n node16 instance62
                echo step 11
                echo gnt-instance replace-disks -n node16 instance13
                echo step 12
                echo gnt-instance replace-disks -n node7 instance19
                echo step 13
                echo gnt-instance replace-disks -n node1 instance43
                echo step 14
                echo gnt-instance replace-disks -n node4 instance1
                echo step 15
                echo gnt-instance replace-disks -n node17 instance58

              Final cluster status:
              N1 Name   t_mem f_mem r_mem t_dsk f_dsk pri sec  p_fmem  p_fdsk
                 node1  32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node2  32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node3  32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node4  32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node5  32762  7280  6000  1861  1078   4   5 0.22221 0.57947
                 node6  32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node7  32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node8  32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node9  32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node10 32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node11 32762  7280  6000  1861  1022   4   4 0.22221 0.54951
                 node12 32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node13 32762  7280  6000  1861  1022   4   4 0.22221 0.54951
                 node14 32762  7280  6000  1861  1022   4   4 0.22221 0.54951
                 node15 32762  7280  6000  1861  1031   4   4 0.22221 0.55408
                 node16 32762  7280  6000  1861  1060   4   4 0.22221 0.57007
                 node17 32762  7280  6000  1861  1006   5   4 0.22221 0.54105
                 node18 32762  7280  6000  1396   761   4   2 0.22221 0.54570
                 node19 32762  7280  6000  1861  1026   4   4 0.22221 0.55179
                 node20 32762 13280  6000  1861  1089   3   5 0.40535 0.58565

       Here  we  see, beside the step list, the initial and final cluster status, with the final one showing all
       nodes being N+1 compliant, and the command list to reach the final solution.  In the initial listing,  we
       see which nodes are not N+1 compliant.

       The  algorithm is stable as long as each step above is fully completed, e.g.  in step 8, both the migrate
       and the replace-disks are done.  Otherwise, if only the migrate is done, the input data is changed  in  a
       way that the program will output a different solution list (but hopefully will end in the same state).

REPORTING BUGS

       Report  bugs  to  project  website (http://code.google.com/p/ganeti/) or contact the developers using the
       Ganeti mailing list (ganeti@googlegroups.com).

SEE ALSO

       Ganeti overview and  specifications:  ganeti(7)  (general  overview),  ganeti-os-interface(7)  (guest  OS
       definitions), ganeti-extstorage-interface(7) (external storage providers).

       Ganeti  commands:  gnt-cluster(8) (cluster-wide commands), gnt-job(8) (job-related commands), gnt-node(8)
       (node-related  commands),  gnt-instance(8)  (instance   commands),   gnt-os(8)   (guest   OS   commands),
       gnt-storage(8)   (storage   commands),   gnt-group(8)  (node  group  commands),  gnt-backup(8)  (instance
       import/export commands), gnt-debug(8) (debug commands).

       Ganeti daemons: ganeti-watcher(8) (automatic instance restarter), ganeti-cleaner(8) (job queue  cleaner),
       ganeti-noded(8) (node daemon), ganeti-rapi(8) (remote API daemon).

       Ganeti  htools: htools(1) (generic binary), hbal(1) (cluster balancer), hspace(1) (capacity calculation),
       hail(1) (IAllocator plugin), hscan(1) (data gatherer from remote clusters), hinfo(1) (cluster information
       printer), mon-collector(7) (data collectors interface).

       Copyright (C) 2006-2015 Google Inc.  All rights reserved.

       Redistribution  and  use in source and binary forms, with or without modification, are permitted provided
       that the following conditions are met:

       1.  Redistributions of source code must retain the above copyright notice, this list  of  conditions  and
       the following disclaimer.

       2.  Redistributions in binary form must reproduce the above copyright notice, this list of conditions and
       the following disclaimer in the documentation and/or other materials provided with the distribution.

       THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY  EXPRESS  OR  IMPLIED
       WARRANTIES,  INCLUDING,  BUT  NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
       PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE  FOR
       ANY  DIRECT,  INDIRECT,  INCIDENTAL,  SPECIAL,  EXEMPLARY,  OR  CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
       LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF  USE,  DATA,  OR  PROFITS;  OR  BUSINESS
       INTERRUPTION)  HOWEVER  CAUSED  AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
       TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE  OF  THIS  SOFTWARE,  EVEN  IF
       ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.