Provided by: drbd-utils_8.9.6-1_i386 bug

NAME

       drbdsetup - Configure the DRBD kernel module

SYNOPSIS

       drbdsetup command {argument...} [option...]

DESCRIPTION

       The drbdsetup utility serves to configure the DRBD kernel module and to
       show its current configuration. Users usually interact with the drbdadm
       utility, which provides a more high-level interface to DRBD than
       drbdsetup. (See drbdadm's --dry-run option to see how drbdadm uses
       drbdsetup.)

       Some option arguments have a default scale which applies when a plain
       number is specified (for example Kilo, or 1024 times the numeric
       value). Such default scales can be overridden by using a suffix (for
       example, M for Mega). The common suffixes K = 2^10 = 1024, M = 1024 K,
       and G = 1024 M are supported.

COMMANDS

       drbdsetup attach minor lower_dev meta_data_dev meta_data_index,
       drbdsetup disk-options minor
           The attach command attaches a lower-level device to an existing
           replicated device. The disk-options command changes the disk
           options of an attached lower-level device. In either case, the
           replicated device must have been created with drbdsetup new-minor.

           Both commands refer to the replicated device by its minor number.
           lower_dev is the name of the lower-level device.  meta_data_dev is
           the name of the device containing the metadata, and may be the same
           as lower_dev.  meta_data_index is either a numeric metadata index,
           or the keyword internal for internal metadata, or the keyword
           flexible for variable-size external metadata. Available options:

           --al-extents extents
               DRBD automatically maintains a "hot" or "active" disk area
               likely to be written to again soon based on the recent write
               activity. The "active" disk area can be written to immediately,
               while "inactive" disk areas must be "activated" first, which
               requires a meta-data write. We also refer to this active disk
               area as the "activity log".

               The activity log saves meta-data writes, but the whole log must
               be resynced upon recovery of a failed node. The size of the
               activity log is a major factor of how long a resync will take
               and how fast a replicated disk will become consistent after a
               crash.

               The activity log consists of a number of 4-Megabyte segments;
               the al-extents parameter determines how many of those segments
               can be active at the same time. The default value for
               al-extents is 1237, with a minimum of 7 and a maximum of 65536.

               Note that the effective maximum may be smaller, depending on
               how you created the device meta data, see also drbdmeta(8) The
               effective maximum is 919 * (available on-disk activity-log
               ring-buffer area/4kB -1), the default 32kB ring-buffer effects
               a maximum of 6433 (covers more than 25 GiB of data) We
               recommend to keep this well within the amount your backend
               storage and replication link are able to resync inside of about
               5 minutes.

           --al-updates {yes | no}
               With this parameter, the activity log can be turned off
               entirely (see the al-extents parameter). This will speed up
               writes because fewer meta-data writes will be necessary, but
               the entire device needs to be resynchronized opon recovery of a
               failed primary node. The default value for al-updates is yes.

           --disk-barrier,
           --disk-flushes,
           --disk-drain
               DRBD has three methods of handling the ordering of dependent
               write requests:

               disk-barrier
                   Use disk barriers to make sure that requests are written to
                   disk in the right order. Barriers ensure that all requests
                   submitted before a barrier make it to the disk before any
                   requests submitted after the barrier. This is implemented
                   using 'tagged command queuing' on SCSI devices and 'native
                   command queuing' on SATA devices. Only some devices and
                   device stacks support this method. The device mapper (LVM)
                   only supports barriers in some configurations.

                   Note that on systems which do not support disk barriers,
                   enabling this option can lead to data loss or corruption.
                   Until DRBD 8.4.1, disk-barrier was turned on if the I/O
                   stack below DRBD did support barriers. Kernels since
                   linux-2.6.36 (or 2.6.32 RHEL6) no longer allow to detect if
                   barriers are supported. Since drbd-8.4.2, this option is
                   off by default and needs to be enabled explicitly.

               disk-flushes
                   Use disk flushes between dependent write requests, also
                   referred to as 'force unit access' by drive vendors. This
                   forces all data to disk. This option is enabled by default.

               disk-drain
                   Wait for the request queue to "drain" (that is, wait for
                   the requests to finish) before submitting a dependent write
                   request. This method requires that requests are stable on
                   disk when they finish. Before DRBD 8.0.9, this was the only
                   method implemented. This option is enabled by default. Do
                   not disable in production environments.

               From these three methods, drbd will use the first that is
               enabled and supported by the backing storage device. If all
               three of these options are turned off, DRBD will submit write
               requests without bothering about dependencies. Depending on the
               I/O stack, write requests can be reordered, and they can be
               submitted in a different order on different cluster nodes. This
               can result in data loss or corruption. Therefore, turning off
               all three methods of controlling write ordering is strongly
               discouraged.

               A general guideline for configuring write ordering is to use
               disk barriers or disk flushes when using ordinary disks (or an
               ordinary disk array) with a volatile write cache. On storage
               without cache or with a battery backed write cache, disk
               draining can be a reasonable choice.

           --disk-timeout
               If the lower-level device on which a DRBD device stores its
               data does not finish an I/O request within the defined
               disk-timeout, DRBD treats this as a failure. The lower-level
               device is detached, and the device's disk state advances to
               Diskless. If DRBD is connected to one or more peers, the failed
               request is passed on to one of them.

               This option is dangerous and may lead to kernel panic!

               "Aborting" requests, or force-detaching the disk, is intended
               for completely blocked/hung local backing devices which do no
               longer complete requests at all, not even do error completions.
               In this situation, usually a hard-reset and failover is the
               only way out.

               By "aborting", basically faking a local error-completion, we
               allow for a more graceful swichover by cleanly migrating
               services. Still the affected node has to be rebooted "soon".

               By completing these requests, we allow the upper layers to
               re-use the associated data pages.

               If later the local backing device "recovers", and now DMAs some
               data from disk into the original request pages, in the best
               case it will just put random data into unused pages; but
               typically it will corrupt meanwhile completely unrelated data,
               causing all sorts of damage.

               Which means delayed successful completion, especially for READ
               requests, is a reason to panic(). We assume that a delayed
               *error* completion is OK, though we still will complain noisily
               about it.

               The default value of disk-timeout is 0, which stands for an
               infinite timeout. Timeouts are specified in units of 0.1
               seconds. This option is available since DRBD 8.3.12.

           --md-flushes
               Enable disk flushes and disk barriers on the meta-data device.
               This option is enabled by default. See the disk-flushes
               parameter.

           --on-io-error handler
               Configure how DRBD reacts to I/O errors on a lower-level
               device. The following policies are defined:

               pass_on
                   Change the disk status to Inconsistent, mark the failed
                   block as inconsistent in the bitmap, and retry the I/O
                   operation on a remote cluster node.

               call-local-io-error
                   Call the local-io-error handler (see the handlers section).

               detach
                   Detach the lower-level device and continue in diskless
                   mode.

           --read-balancing policy
               Distribute read requests among cluster nodes as defined by
               policy. The supported policies are prefer-local (the default),
               prefer-remote, round-robin, least-pending,
               when-congested-remote, 32K-striping, 64K-striping,
               128K-striping, 256K-striping, 512K-striping and 1M-striping.

               This option is available since DRBD 8.4.1.

           resync-after minor
               Define that a device should only resynchronize after the
               specified other device. By default, no order between devices is
               defined, and all devices will resynchronize in parallel.
               Depending on the configuration of the lower-level devices, and
               the available network and disk bandwidth, this can slow down
               the overall resync process. This option can be used to form a
               chain or tree of dependencies among devices.

           --size size
               Specify the size of the lower-level device explicitly instead
               of determining it automatically. The device size must be
               determined once and is remembered for the lifetime of the
               device. In order to determine it automatically, all the
               lower-level devices on all nodes must be attached, and all
               nodes must be connected. If the size is specified explicitly,
               this is not necessary. The size value is assumed to be in units
               of sectors (512 bytes) by default.

           --discard-zeroes-if-aligned {yes | no}
               There are several aspects to discard/trim/unmap support on
               linux block devices. Even if discard is supported in general,
               it may fail silently, or may partially ignore discard requests.
               Devices also announce whether reading from unmapped blocks
               returns defined data (usually zeroes), or undefined data
               (possibly old data, possibly garbage).

               If on different nodes, DRBD is backed by devices with differing
               discard characteristics, discards may lead to data divergence
               (old data or garbage left over on one backend, zeroes due to
               unmapped areas on the other backend). Online verify would now
               potentially report tons of spurious differences. While probably
               harmless for most use cases (fstrim on a file system), DRBD
               cannot have that.

               To play safe, we have to disable discard support, if our local
               backend (on a Primary) does not support
               "discard_zeroes_data=true". We also have to translate discards
               to explicit zero-out on the receiving side, unless the
               receiving side (Secondary) supports "discard_zeroes_data=true",
               thereby allocating areas what were supposed to be unmapped.

               There are some devices (notably the LVM/DM thin provisioning)
               that are capable of discard, but announce
               discard_zeroes_data=false. In the case of DM-thin, discards
               aligned to the chunk size will be unmapped, and reading from
               unmapped sectors will return zeroes. However, unaligned partial
               head or tail areas of discard requests will be silently
               ignored.

               If we now add a helper to explicitly zero-out these unaligned
               partial areas, while passing on the discard of the aligned full
               chunks, we effectively achieve discard_zeroes_data=true on such
               devices.

               Setting discard-zeroes-if-aligned to yes will allow DRBD to use
               discards, and to announce discard_zeroes_data=true, even on
               backends that announce discard_zeroes_data=false.

               Setting discard-zeroes-if-aligned to no will cause DRBD to
               always fall-back to zero-out on the receiving side, and to not
               even announce discard capabilities on the Primary, if the
               respective backend announces discard_zeroes_data=false.

               We used to ignore the discard_zeroes_data setting completely.
               To not break established and expected behaviour, and suddenly
               cause fstrim on thin-provisioned LVs to run out-of-space
               instead of freeing up space, the default value is yes.

               This option is available since 8.4.7.

           --rs-discard-granularity byte
               When rs-discard-granularity is set to a non zero, positive
               value then DRBD tries to do a resync operation in requests of
               this size. In case such a block contains only zero bytes on the
               sync source node, the sync target node will issue a
               discard/trim/unmap command for the area.

               The value is constrained by the discard granularity of the
               backing block device. In case rs-discard-granularity is not a
               multiplier of the discard granularity of the backing block
               device DRBD rounds it up. The feature only gets active if the
               backing block device reads back zeroes after a discard command.

               The default value of is 0. This option is available since
               8.4.7.

       drbdsetup peer-device-options resource peer_node_id volume
           These are options that affect the peer's device.

           --c-delay-target delay_target,
           --c-fill-target fill_target,
           --c-max-rate max_rate,
           --c-plan-ahead plan_time
               Dynamically control the resync speed. This mechanism is enabled
               by setting the c-plan-ahead parameter to a positive value. The
               goal is to either fill the buffers along the data path with a
               defined amount of data if c-fill-target is defined, or to have
               a defined delay along the path if c-delay-target is defined.
               The maximum bandwidth is limited by the c-max-rate parameter.

               The c-plan-ahead parameter defines how fast drbd adapts to
               changes in the resync speed. It should be set to five times the
               network round-trip time or more. Common values for
               c-fill-target for "normal" data paths range from 4K to 100K. If
               drbd-proxy is used, it is advised to use c-delay-target instead
               of c-fill-target. The c-delay-target parameter is used if the
               c-fill-target parameter is undefined or set to 0. The
               c-delay-target parameter should be set to five times the
               network round-trip time or more. The c-max-rate option should
               be set to either the bandwidth available between the DRBD-hosts
               and the machines hosting DRBD-proxy, or to the available disk
               bandwidth.

               The default values of these parameters are: c-plan-ahead = 20
               (in units of 0.1 seconds), c-fill-target = 0 (in units of
               sectors), c-delay-target = 1 (in units of 0.1 seconds), and
               c-max-rate = 102400 (in units of KiB/s).

               Dynamic resync speed control is available since DRBD 8.3.9.

           --c-min-rate min_rate
               A node which is primary and sync-source has to schedule
               application I/O requests and resync I/O requests. The
               c-min-rate parameter limits how much bandwidth is available for
               resync I/O; the remaining bandwidth is used for application
               I/O.

               A c-min-rate value of 0 means that there is no limit on the
               resync I/O bandwidth. This can slow down application I/O
               significantly. Use a value of 1 (1 KiB/s) for the lowest
               possible resync rate.

               The default value of c-min-rate is 4096, in units of KiB/s.

           --resync-rate rate
               Define how much bandwidth DRBD may use for resynchronizing.
               DRBD allows "normal" application I/O even during a resync. If
               the resync takes up too much bandwidth, application I/O can
               become very slow. This parameter allows to avoid that. Please
               note this is option only works when the dynamic resync
               controller is disabled.

       drbdsetup check-resize minor
           Remember the current size of the lower-level device of the
           specified replicated device. Used by drbdadm. The size information
           is stored in file /var/lib/drbd/drbd-minor-minor.lkbd.

       drbdsetup new-peer resource peer_node_id,
       drbdsetup net-options resource peer_node_id
           The new-peer command creates a connection within a resource. The
           resource must have been created with drbdsetup new-resource. The
           net-options command changes the network options of an existing
           connection. Before a connection can be activated with the connect
           command, at least one path need to added with the new-path command.
           Available options:

           --after-sb-0pri policy
               Define how to react if a split-brain scenario is detected and
               none of the two nodes is in primary role. (We detect
               split-brain scenarios when two nodes connect; split-brain
               decisions are always between two nodes.) The defined policies
               are:

               disconnect
                   No automatic resynchronization; simply disconnect.

               discard-younger-primary,
               discard-older-primary
                   Resynchronize from the node which became primary first
                   (discard-younger-primary) or last (discard-older-primary).
                   If both nodes became primary independently, the
                   discard-least-changes policy is used.

               discard-zero-changes
                   If only one of the nodes wrote data since the split brain
                   situation was detected, resynchronize from this node to the
                   other. If both nodes wrote data, disconnect.

               discard-least-changes
                   Resynchronize from the node with more modified blocks.

               discard-node-nodename
                   Always resynchronize to the named node.

           --after-sb-1pri policy
               Define how to react if a split-brain scenario is detected, with
               one node in primary role and one node in secondary role. (We
               detect split-brain scenarios when two nodes connect, so
               split-brain decisions are always among two nodes.) The defined
               policies are:

               disconnect
                   No automatic resynchronization, simply disconnect.

               consensus
                   Discard the data on the secondary node if the after-sb-0pri
                   algorithm would also discard the data on the secondary
                   node. Otherwise, disconnect.

               violently-as0p
                   Always take the decision of the after-sb-0pri algorithm,
                   even if it causes an erratic change of the primary's view
                   of the data. This is only useful if a single-node file
                   system (i.e., not OCFS2 or GFS) with the
                   allow-two-primaries flag is used. This option can cause the
                   primary node to crash, and should not be used.

               discard-secondary
                   Discard the data on the secondary node.

               call-pri-lost-after-sb
                   Always take the decision of the after-sb-0pri algorithm. If
                   the decision is to discard the data on the primary node,
                   call the pri-lost-after-sb handler on the primary node.

           --after-sb-2pri policy
               Define how to react if a split-brain scenario is detected and
               both nodes are in primary role. (We detect split-brain
               scenarios when two nodes connect, so split-brain decisions are
               always among two nodes.) The defined policies are:

               disconnect
                   No automatic resynchronization, simply disconnect.

               violently-as0p
                   See the violently-as0p policy for after-sb-1pri.

               call-pri-lost-after-sb
                   Call the pri-lost-after-sb helper program on one of the
                   machines unless that machine can demote to secondary. The
                   helper program is expected to reboot the machine, which
                   brings the node into a secondary role. Which machine runs
                   the helper program is determined by the after-sb-0pri
                   strategy.

           --allow-two-primaries
               The most common way to configure DRBD devices is to allow only
               one node to be primary (and thus writable) at a time.

               In some scenarios it is preferable to allow two nodes to be
               primary at once; a mechanism outside of DRBD then must make
               sure that writes to the shared, replicated device happen in a
               coordinated way. This can be done with a shared-storage cluster
               file system like OCFS2 and GFS, or with virtual machine images
               and a virtual machine manager that can migrate virtual machines
               between physical machines.

               The allow-two-primaries parameter tells DRBD to allow two nodes
               to be primary at the same time. Never enable this option when
               using a non-distributed file system; otherwise, data corruption
               and node crashes will result!

           --always-asbp
               Normally the automatic after-split-brain policies are only used
               if current states of the UUIDs do not indicate the presence of
               a third node.

               With this option you request that the automatic
               after-split-brain policies are used as long as the data sets of
               the nodes are somehow related. This might cause a full sync, if
               the UUIDs indicate the presence of a third node. (Or double
               faults led to strange UUID sets.)

           --connect-int time
               As soon as a connection between two nodes is configured with
               drbdsetup connect, DRBD immediately tries to establish the
               connection. If this fails, DRBD waits for connect-int seconds
               and then repeats. The default value of connect-int is 10
               seconds.

           --cram-hmac-alg hash-algorithm
               Configure the hash-based message authentication code (HMAC) or
               secure hash algorithm to use for peer authentication. The
               kernel supports a number of different algorithms, some of which
               may be loadable as kernel modules. See the shash algorithms
               listed in /proc/crypto. By default, cram-hmac-alg is unset.
               Peer authentication also requires a shared-secret to be
               configured.

           --csums-alg hash-algorithm
               Normally, when two nodes resynchronize, the sync target
               requests a piece of out-of-sync data from the sync source, and
               the sync source sends the data. With many usage patterns, a
               significant number of those blocks will actually be identical.

               When a csums-alg algorithm is specified, when requesting a
               piece of out-of-sync data, the sync target also sends along a
               hash of the data it currently has. The sync source compares
               this hash with its own version of the data. It sends the sync
               target the new data if the hashes differ, and tells it that the
               data are the same otherwise. This reduces the network bandwidth
               required, at the cost of higher cpu utilization and possibly
               increased I/O on the sync target.

               The csums-alg can be set to one of the secure hash algorithms
               supported by the kernel; see the shash algorithms listed in
               /proc/crypto. By default, csums-alg is unset.

           --csums-after-crash-only
               Enabling this option (and csums-alg, above) makes it possible
               to use the checksum based resync only for the first resync
               after primary crash, but not for later "network hickups".

               In most cases, block that are marked as need-to-be-resynced are
               in fact changed, so calculating checksums, and both reading and
               writing the blocks on the resync target is all effective
               overhead.

               The advantage of checksum based resync is mostly after primary
               crash recovery, where the recovery marked larger areas (those
               covered by the activity log) as need-to-be-resynced, just in
               case. Introduced in 8.4.5.

           --data-integrity-alg  alg
               DRBD normally relies on the data integrity checks built into
               the TCP/IP protocol, but if a data integrity algorithm is
               configured, it will additionally use this algorithm to make
               sure that the data received over the network match what the
               sender has sent. If a data integrity error is detected, DRBD
               will close the network connection and reconnect, which will
               trigger a resync.

               The data-integrity-alg can be set to one of the secure hash
               algorithms supported by the kernel; see the shash algorithms
               listed in /proc/crypto. By default, this mechanism is turned
               off.

               Because of the CPU overhead involved, we recommend not to use
               this option in production environments. Also see the notes on
               data integrity below.

           --fencing fencing_policy
               Fencing is a preventive measure to avoid situations where both
               nodes are primary and disconnected. This is also known as a
               split-brain situation. DRBD supports the following fencing
               policies:

               dont-care
                   No fencing actions are taken. This is the default policy.

               resource-only
                   If a node becomes a disconnected primary, it tries to fence
                   the peer. This is done by calling the fence-peer handler.
                   The handler is supposed to reach the peer over an
                   alternative communication path and call 'drbdadm outdate
                   minor' there.

               resource-and-stonith
                   If a node becomes a disconnected primary, it freezes all
                   its IO operations and calls its fence-peer handler. The
                   fence-peer handler is supposed to reach the peer over an
                   alternative communication path and call 'drbdadm outdate
                   minor' there. In case it cannot do that, it should stonith
                   the peer. IO is resumed as soon as the situation is
                   resolved. In case the fence-peer handler fails, I/O can be
                   resumed manually with 'drbdadm resume-io'.

           --ko-count number
               If a secondary node fails to complete a write request in
               ko-count times the timeout parameter, it is excluded from the
               cluster. The primary node then sets the connection to this
               secondary node to Standalone. To disable this feature, you
               should explicitly set it to 0; defaults may change between
               versions.

           --max-buffers number
               Limits the memory usage per DRBD minor device on the receiving
               side, or for internal buffers during resync or online-verify.
               Unit is PAGE_SIZE, which is 4 KiB on most systems. The minimum
               possible setting is hard coded to 32 (=128 KiB). These buffers
               are used to hold data blocks while they are written to/read
               from disk. To avoid possible distributed deadlocks on
               congestion, this setting is used as a throttle threshold rather
               than a hard limit. Once more than max-buffers pages are in use,
               further allocation from this pool is throttled. You want to
               increase max-buffers if you cannot saturate the IO backend on
               the receiving side.

           --max-epoch-size number
               Define the maximum number of write requests DRBD may issue
               before issuing a write barrier. The default value is 2048, with
               a minimum of 1 and a maximum of 20000. Setting this parameter
               to a value below 10 is likely to decrease performance.

           --on-congestion policy,
           --congestion-fill threshold,
           --congestion-extents threshold
               By default, DRBD blocks when the TCP send queue is full. This
               prevents applications from generating further write requests
               until more buffer space becomes available again.

               When DRBD is used together with DRBD-proxy, it can be better to
               use the pull-ahead on-congestion policy, which can switch DRBD
               into ahead/behind mode before the send queue is full. DRBD then
               records the differences between itself and the peer in its
               bitmap, but it no longer replicates them to the peer. When
               enough buffer space becomes available again, the node
               resynchronizes with the peer and switches back to normal
               replication.

               This has the advantage of not blocking application I/O even
               when the queues fill up, and the disadvantage that peer nodes
               can fall behind much further. Also, while resynchronizing, peer
               nodes will become inconsistent.

               The available congestion policies are block (the default) and
               pull-ahead. The congestion-fill parameter defines how much data
               is allowed to be "in flight" in this connection. The default
               value is 0, which disables this mechanism of congestion
               control, with a maximum of 10 GiBytes. The congestion-extents
               parameter defines how many bitmap extents may be active before
               switching into ahead/behind mode, with the same default and
               limits as the al-extents parameter. The congestion-extents
               parameter is effective only when set to a value smaller than
               al-extents.

               Ahead/behind mode is available since DRBD 8.3.10.

           --ping-int interval
               When the TCP/IP connection to a peer is idle for more than
               ping-int seconds, DRBD will send a keep-alive packet to make
               sure that a failed peer or network connection is detected
               reasonably soon. The default value is 10 seconds, with a
               minimum of 1 and a maximum of 120 seconds. The unit is seconds.

           --ping-timeout timeout
               Define the timeout for replies to keep-alive packets. If the
               peer does not reply within ping-timeout, DRBD will close and
               try to reestablish the connection. The default value is 0.5
               seconds, with a minimum of 0.1 seconds and a maximum of 3
               seconds. The unit is tenths of a second.

           --socket-check-timeout timeout
               In setups involving a DRBD-proxy and connections that
               experience a lot of buffer-bloat it might be necessary to set
               ping-timeout to an unusual high value. By default DRBD uses the
               same value to wait if a newly established TCP-connection is
               stable. Since the DRBD-proxy is usually located in the same
               data center such a long wait time may hinder DRBD's connect
               process.

               In such setups socket-check-timeout should be set to at least
               to the round trip time between DRBD and DRBD-proxy. I.e. in
               most cases to 1.

               The default unit is tenths of a second, the default value is 0
               (which causes DRBD to use the value of ping-timeout instead).
               Introduced in 8.4.5.

           --protocol name
               Use the specified protocol on this connection. The supported
               protocols are:

               A
                   Writes to the DRBD device complete as soon as they have
                   reached the local disk and the TCP/IP send buffer.

               B
                   Writes to the DRBD device complete as soon as they have
                   reached the local disk, and all peers have acknowledged the
                   receipt of the write requests.

               C
                   Writes to the DRBD device complete as soon as they have
                   reached the local and all remote disks.

           --rcvbuf-size size
               Configure the size of the TCP/IP receive buffer. A value of 0
               (the default) causes the buffer size to adjust dynamically.
               This parameter usually does not need to be set, but it can be
               set to a value up to 10 MiB. The default unit is bytes.

           --rr-conflict policy
               This option helps to solve the cases when the outcome of the
               resync decision is incompatible with the current role
               assignment in the cluster. The defined policies are:

               disconnect
                   No automatic resynchronization, simply disconnect.

               violently
                   Resync to the primary node is allowed, violating the
                   assumption that data on a block device are stable for one
                   of the nodes.  Do not use this option, it is dangerous.

               call-pri-lost
                   Call the pri-lost handler on one of the machines. The
                   handler is expected to reboot the machine, which puts it
                   into secondary role.

           --shared-secret secret
               Configure the shared secret used for peer authentication. The
               secret is a string of up to 64 characters. Peer authentication
               also requires the cram-hmac-alg parameter to be set.

           --sndbuf-size size
               Configure the size of the TCP/IP send buffer. Since DRBD 8.0.13
               / 8.2.7, a value of 0 (the default) causes the buffer size to
               adjust dynamically. Values below 32 KiB are harmful to the
               throughput on this connection. Large buffer sizes can be useful
               especially when protocol A is used over high-latency networks;
               the maximum value supported is 10 MiB.

           --tcp-cork
               By default, DRBD uses the TCP_CORK socket option to prevent the
               kernel from sending partial messages; this results in fewer and
               bigger packets on the network. Some network stacks can perform
               worse with this optimization. On these, the tcp-cork parameter
               can be used to turn this optimization off.

           --timeout time
               Define the timeout for replies over the network: if a peer node
               does not send an expected reply within the specified timeout,
               it is considered dead and the TCP/IP connection is closed. The
               timeout value must be lower than connect-int and lower than
               ping-int. The default is 6 seconds; the value is specified in
               tenths of a second.

           --unplug-watermark number
               Mainline kernels before version 2.6.39-rc1 use an explicit plug
               / unplug mechanism to control when a block device starts
               processing queued requests. On those kernels, the
               unplug-watermark parameter defines how many requests must be
               queued until a secondary node starts processing them. Some
               storage controllers perform best when unplug-watermark is set
               to the same value as max-buffers; others are more efficient
               with smaller values. The default value for unplug-watermark is
               128, with a minimum of 16 and a maximum of 131072.

               More recent kernels handle plugging and unplugging implicitly;
               on those kernels, this parameter has no effect. Note that some
               distributions have backported this feature to older kernel
               versions.

           --use-rle
               Each replicated device on a cluster node has a separate bitmap
               for each of its peer devices. The bitmaps are used for tracking
               the differences between the local and peer device: depending on
               the cluster state, a disk range can be marked as different from
               the peer in the device's bitmap, in the peer device's bitmap,
               or in both bitmaps. When two cluster nodes connect, they
               exchange each other's bitmaps, and they each compute the union
               of the local and peer bitmap to determine the overall
               differences.

               Bitmaps of very large devices are also relatively large, but
               they usually compress very well using run-length encoding. This
               can save time and bandwidth for the bitmap transfers.

               The use-rle parameter determines if run-length encoding should
               be used. It is on by default since DRBD 8.4.0.

           --verify-alg hash-algorithm
               Online verification (drbdadm verify) computes and compares
               checksums of disk blocks (i.e., hash values) in order to detect
               if they differ. The verify-alg parameter determines which
               algorithm to use for these checksums. It must be set to one of
               the secure hash algorithms supported by the kernel before
               online verify can be used; see the shash algorithms listed in
               /proc/crypto.

               We recommend to schedule online verifications regularly during
               low-load periods, for example once a month. Also see the notes
               on data integrity below.

       drbdsetup new-path resource peer_node_id local-addr remote-addr
           The new-path command creates a path within a connection. The
           connection must have been created with drbdsetup new-peer.
           Local_addr and remote_addr refer to the local and remote protocol,
           network address, and port in the format
           [address-family:]address[:port]. The address families ipv4, ipv6,
           ssocks (Dolphin Interconnect Solutions' "super sockets"), sdp
           (Infiniband Sockets Direct Protocol), and sci are supported (sci is
           an alias for ssocks). If no address family is specified, ipv4 is
           assumed. For all address families except ipv6, the address uses
           IPv4 address notation (for example, 1.2.3.4). For ipv6, the address
           is enclosed in brackets and uses IPv6 address notation (for
           example, [fd01:2345:6789:abcd::1]). The port defaults to 7788.

       drbdsetup connect resource peer_node_id
           The connect command activates a connection. That means that the
           DRBD driver will bind and listen on all local addresses of the
           connection-'s paths. It will begin to try to establish one or more
           paths of the connection. Available options:

           --tentative
               Only determine if a connection to the peer can be established
               and if a resync is necessary (and in which direction) without
               actually establishing the connection or starting the resync.
               Check the system log to see what DRBD would do without the
               --tentative option.

           --discard-my-data
               Discard the local data and resynchronize with the peer that has
               the most up-to-data data. Use this option to manually recover
               from a split-brain situation.

       drbdsetup del-peer resource peer_node_id
           The del-peer command removes a connection from a resource.

       drbdsetup del-path resource peer_node_id local-addr remote-addr
           The del-path command removes a path from a connection. Please not
           that it fails if the path is necessary to keep a connected
           connection in tact. In order to remove all paths, disconnect the
           connection first.

       drbdsetup cstate resource peer_node_id
           Show the current state of a connection. The connection is
           identified by the node-id of the peer; see the drbdsetup connect
           command.

       drbdsetup del-minor minor
           Remove a replicated device. No lower-level device may be attached;
           see drbdsetup detach.

       drbdsetup del-resource resource
           Remove a resource. All volumes and connections must be removed
           first (drbdsetup del-minor, drbdsetup disconnect). Alternatively,
           drbdsetup down can be used to remove a resource together with all
           its volumes and connections.

       drbdsetup detach minor
           Detach the lower-level device of a replicated device. Available
           options:

           --force
               Force the detach and return immediately. This puts the
               lower-level device into failed state until all pending I/O has
               completed, and then detaches the device. Any I/O not yet
               submitted to the lower-level device (for example, because I/O
               on the device was suspended) is assumed to have failed.

       drbdsetup disconnect resource peer_node_id
           Remove a connection to a peer host. The connection is identified by
           the node-id of the peer; see the drbdsetup connect command.

       drbdsetup down {resource | all}
           Take a resource down by removing all volumes, connections, and the
           resource itself.

       drbdsetup dstate minor
           Show the current disk state of a lower-level device.

       drbdsetup events2 {resource | all}
           Show the current state of all configured DRBD objects, followed by
           all changes to the state.

           The output format is meant to be human as well as machine readable.
           The line starts with a word that indicates the kind of event:
           exists for an existing object; create, destroy, and change if an
           object is created, destroyed, or changed; or call or response if an
           event handler is called or it returns. The second word indicates
           the object the event applies to: resource, device, connection,
           peer-device, helper, or a dash (-) to indicate that the current
           state has been dumped completely.

           The remaining words identify the object and describe the state that
           he object is in. Available options:

           --now
               Terminate after reporting the current state. The default is to
               continuously listen and report state changes.

           --statistics
               Include statistics in the output.

       drbdsetup get-gi resource peer_node_id volume
           Show the data generation identifiers for a device on a particular
           connection. The device is identified by its volume number. The
           connection is identified by its endpoints; see the drbdsetup
           connect command.

           The output consists of the current UUID, bitmap UUID, and the first
           two history UUIDS, folowed by a set of flags. The current UUID and
           history UUIDs are device specific; the bitmap UUID and flags are
           peer device specific. This command only shows the first two history
           UUIDs. Internally, DRBD maintains one history UUID for each
           possible peer device.

       drbdsetup invalidate minor
           Replace the local data of a device with that of a peer. All the
           local data will be marked out-of-sync, and a resync with the
           specified peer device will be initialted.

       drbdsetup invalidate-remote resource peer_node_id volume
           Replace a peer device's data of a resource with the local data. The
           peer device's data will be marked out-of-sync, and a resync from
           the local node to the specified peer will be initiated.

       drbdsetup new-current-uuid minor
           Generate a new current UUID and rotates all other UUID values. This
           has at least two use cases, namely to skip the initial sync, and to
           reduce network bandwidth when starting in a single node
           configuration and then later (re-)integrating a remote site.

           Available option:

           --clear-bitmap
               Clears the sync bitmap in addition to generating a new current
               UUID.

           This can be used to skip the initial sync, if you want to start
           from scratch. This use-case does only work on "Just Created" meta
           data. Necessary steps:

            1. On both nodes, initialize meta data and configure the device.

               drbdadm create-md --force res

            2. They need to do the initial handshake, so they know their
               sizes.

               drbdadm up res

            3. They are now Connected Secondary/Secondary
               Inconsistent/Inconsistent. Generate a new current-uuid and
               clear the dirty bitmap.

               drbdadm --clear-bitmap new-current-uuid res

            4. They are now Connected Secondary/Secondary UpToDate/UpToDate.
               Make one side primary and create a file system.

               drbdadm primary res

               mkfs -t fs-type $(drbdadm sh-dev res)

           One obvious side-effect is that the replica is full of old garbage
           (unless you made them identical using other means), so any
           online-verify is expected to find any number of out-of-sync blocks.

           You must not use this on pre-existing data!  Even though it may
           appear to work at first glance, once you switch to the other node,
           your data is toast, as it never got replicated. So do not leave out
           the mkfs (or equivalent).

           This can also be used to shorten the initial resync of a cluster
           where the second node is added after the first node is gone into
           production, by means of disk shipping. This use-case works on
           disconnected devices only, the device may be in primary or
           secondary role.

           The necessary steps on the current active server are:

            1. drbdsetup new-current-uuid --clear-bitmap minor

            2. Take the copy of the current active server. E.g. by pulling a
               disk out of the RAID1 controller, or by copying with dd. You
               need to copy the actual data, and the meta data.

            3. drbdsetup new-current-uuid minor

           Now add the disk to the new secondary node, and join it to the
           cluster. You will get a resync of that parts that were changed
           since the first call to drbdsetup in step 1.

       drbdsetup new-minor resource minor volume
           Create a new replicated device within a resource. The command
           creates a block device inode for the replicated device (by default,
           /dev/drbdminor). The volume number identifies the device within the
           resource.

       drbdsetup new-resource resource node_id,
       drbdsetup resource-options resource
           The new-resource command creates a new resource. The
           resource-options command changes the resource options of an
           existing resource. Available options:

           --auto-promote bool-value
               A resource must be promoted to primary role before any of its
               devices can be mounted or opened for writing.

               Before DRBD 9, this could only be done explicitly ("drbdadm
               primary"). Since DRBD 9, the auto-promote parameter allows to
               automatically promote a resource to primary role when one of
               its devices is mounted or opened for writing. As soon as all
               devices are unmounted or closed with no more remaining users,
               the role of the resource changes back to secondary.

               Automatic promotion only succeeds if the cluster state allows
               it (that is, if an explicit drbdadm primary command would
               succeed). Otherwise, mounting or opening the device fails as it
               already did before DRBD 9: the mount(2) system call fails with
               errno set to EROFS (Read-only file system); the open(2) system
               call fails with errno set to EMEDIUMTYPE (wrong medium type).

               Irrespective of the auto-promote parameter, if a device is
               promoted explicitly (drbdadm primary), it also needs to be
               demoted explicitly (drbdadm secondary).

               The auto-promote parameter is available since DRBD 9.0.0, and
               defaults to yes.

           --cpu-mask cpu-mask
               Set the cpu affinity mask for DRBD kernel threads. The cpu mask
               is specified as a hexadecimal number. The default value is 0,
               which lets the scheduler decide which kernel threads run on
               which CPUs. CPU numbers in cpu-mask which do not exist in the
               system are ignored.

           --on-no-data-accessible policy
               Determine how to deal with I/O requests when the requested data
               is not available locally or remotely (for example, when all
               disks have failed). The defined policies are:

               io-error
                   System calls fail with errno set to EIO.

               suspend-io
                   The resource suspends I/O. I/O can be resumed by
                   (re)attaching the lower-level device, by connecting to a
                   peer which has access to the data, or by forcing DRBD to
                   resume I/O with drbdadm resume-io res. When no data is
                   available, forcing I/O to resume will result in the same
                   behavior as the io-error policy.

               This setting is available since DRBD 8.3.9; the default policy
               is io-error.

           --peer-ack-window value
               On each node and for each device, DRBD maintains a bitmap of
               the differences between the local and remote data for each peer
               device. For example, in a three-node setup (nodes A, B, C) each
               with a single device, every node maintains one bitmap for each
               of its peers.

               When nodes receive write requests, they know how to update the
               bitmaps for the writing node, but not how to update the bitmaps
               between themselves. In this example, when a write request
               propagates from node A to B and C, nodes B and C know that they
               have the same data as node A, but not whether or not they both
               have the same data.

               As a remedy, the writing node occasionally sends peer-ack
               packets to its peers which tell them which state they are in
               relative to each other.

               The peer-ack-window parameter specifies how much data a primary
               node may send before sending a peer-ack packet. A low value
               causes increased network traffic; a high value causes less
               network traffic but higher memory consumption on secondary
               nodes and higher resync times between the secondary nodes after
               primary node failures. (Note: peer-ack packets may be sent due
               to other reasons as well, e.g. membership changes or expiry of
               the peer-ack-delay timer.)

               The default value for peer-ack-window is 2 MiB, the default
               unit is sectors. This option is available since 9.0.0.

           --peer-ack-delay expiry-time
               If after the last finished write request no new write request
               gets issued for expiry-time, then a peer-ack packet is sent. If
               a new write request is issued before the timer expires, the
               timer gets reset to expiry-time. (Note: peer-ack packets may be
               sent due to other reasons as well, e.g. membership changes or
               the peer-ack-window option.)

               This parameter may influence resync behavior on remote nodes.
               Peer nodes need to wait until they receive an peer-ack for
               releasing a lock on an AL-extent. Resync operations between
               peers may need to wait for for these locks.

               The default value for peer-ack-delay is 100 milliseconds, the
               default unit is milliseconds. This option is available since
               9.0.0.

       drbdsetup outdate minor
           Mark the data on a lower-level device as outdated. This is used for
           fencing, and prevents the resource the device is part of from
           becoming primary in the future. See the --fencing disk option.

       drbdsetup pause-sync resource peer_node_id volume
           Stop resynchronizing between a local and a peer device by setting
           the local pause flag. The resync can only resume if the pause flags
           on both sides of a connection are cleared.

       drbdsetup primary resource
           Change the role of a node in a resource to primary. This allows the
           replicated devices in this resource to be mounted or opened for
           writing. Available options:

           --overwrite-data-of-peer
               This option is an alias for the --force option.

           --force
               Force the resource to become primary even if some devices are
               not guaranteed to have up-to-date data. This option is used to
               turn one of the nodes in a newly created cluster into the
               primary node, or when manually recovering from a disaster.

               Note that this can lead to split-brain scenarios. Also, when
               forcefully turning an inconsistent device into an up-to-date
               device, it is highly recommended to use any integrity checks
               available (such as a filesystem check) to make sure that the
               device can at least be used without crashing the system.

           Note that DRBD usually only allows one node in a cluster to be in
           primary role at any time; this allows DRBD to coordinate access to
           the devices in a resource across nodes. The --allow-two-primaries
           network option changes this; in that case, a mechanism outside of
           DRBD needs to coordinate device access.

       drbdsetup resize minor
           Reexamine the size of the lower-level devices of a replicated
           device on all nodes. This command is called after the lower-level
           devices on all nodes have been grown to adjust the size of the
           replicated device. Available options:

           --assume-peer-has-space
               Resize the device even if some of the peer devices are not
               connected at the moment. DRBD will try to resize the peer
               devices when they next connect. It will refuse to connect to a
               peer device which is too small.

           --assume-clean
               Do not resynchronize the added disk space; instead, assume that
               it is identical on all nodes. This option can be used when the
               disk space is uninitialized and differences do not matter, or
               when it is known to be identical on all nodes. See the
               drbdsetup verify command.

           --size val
               This option can be used to online shrink the usable size of a
               drbd device. It's the users responsibility to make sure that a
               file system on the device is not truncated by that operation.

           --al-stripes val --al-stripes val
               These options may be used to change the layout of the activity
               log online. In case of internal meta data this may invovle
               shrinking the user visible size at the same time (unsing the
               --size) or increasing the avalable space on the backing
               devices.

       drbdsetup resume-io minor
           Resume I/O on a replicated device. See the --fencing net option.

       drbdsetup resume-sync resource peer_node_id volume
           Allow resynchronization to resume by clearing the local sync pause
           flag.

       drbdsetup role resource
           Show the current role of a resource.

       drbdsetup secondary resource
           Change the role of a node in a resource to secondary. This command
           fails if the replicated device is in use.

       drbdsetup show {resource | all}
           Show the current configuration of a resource, or of all resources.
           Available options:

           --show-defaults
               Show all configuration parameters, even the ones with default
               values. Normally, parameters with default values are not shown.

       drbdsetup show-gi resource peer_node_id volume
           Show the data generation identifiers for a device on a particular
           connection. In addition, explain the output. The output otherwise
           is the same as in the drbdsetup get-gi command.

       drbdsetup state
           This is an alias for drbdsetup role. Deprecated.

       drbdsetup status {resource | all}
           Show the status of a resource, or of all resources. The output
           consists of one paragraph for each configured resource. Each
           paragraph contains one line for each resource, followed by one line
           for each device, and one line for each connection. The device and
           connection lines are indented. The connection lines are followed by
           one line for each peer device; these lines are indented against the
           connection line.

           Long lines are wrapped around at terminal width, and indented to
           indicate how the lines belongs together. Available options:

           --verbose
               Include more information in the output even when it is likely
               redundant or irrelevant.

           --statistics
               Include data transfer statistics in the output.

           --color={always | auto | never}
               Colorize the output. With --color=auto, drbdsetup emits color
               codes only when standard output is connected to a terminal.

           For example, the non-verbose output for a resource with only one
           connection and only one volume could look like this:

               drbd0 role:Primary
                 disk:UpToDate
                 host2.example.com role:Secondary
                   disk:UpToDate

           With the --verbose option, the same resource could be reported as:

               drbd0 node-id:1 role:Primary suspended:no
                 volume:0 minor:1 disk:UpToDate blocked:no
                 host2.example.com local:ipv4:192.168.123.4:7788
                     peer:ipv4:192.168.123.2:7788 node-id:0 connection:WFReportParams
                     role:Secondary congested:no
                   volume:0 replication:Connected disk:UpToDate resync-suspended:no

       drbdsetup suspend-io minor
           Suspend I/O on a replicated device. It is not usually necessary to
           use this command.

       drbdsetup verify resource peer_node_id volume
           Start online verification, change which part of the device will be
           verified, or stop online verification. The command requires the
           specified peer to be connected.

           Online verification compares each disk block on the local and peer
           node. Blocks which differ between the nodes are marked as
           out-of-sync, but they are not automatically brought back into sync.
           To bring them into sync, the resource must be disconnected and
           reconnected. Progress can be monitored in the output of drbdsetup
           status --statistics. Available options:

           --start position
               Define where online verification should start. This parameter
               is ignored if online verification is already in progress. If
               the start parameter is not specified, online verification will
               continue where it was interrupted (if the connection to the
               peer was lost while verifying), after the previous stop sector
               (if the previous online verification has finished), or at the
               beginning of the device (if the end of the device was reached,
               or online verify has not run before).

               The position on disk is specified in disk sectors (512 bytes)
               by default.

           --stop position
               Define where online verification should stop. If online
               verification is already in progress, the stop position of the
               active online verification process is changed. Use this to stop
               online verification.

               The position on disk is specified in disk sectors (512 bytes)
               by default.

           Also see the notes on data integrity in the drbd.conf(5) manual
           page.

       drbdsetup wait-connect-volume resource peer_node_id volume,
       drbdsetup wait-connect-connection resource peer_node_id,
       drbdsetup wait-connect-resource resource,
       drbdsetup wait-sync-volume resource peer_node_id volume,
       drbdsetup wait-sync-connection resource peer_node_id,
       drbdsetup wait-sync-resource resource
           The wait-connect-* commands waits until a device on a peer is
           visible. The wait-sync-* commands waits until a device on a peer is
           up to date. Available options for both commands:

           --degr-wfc-timeout timeout
               Define how long to wait until all peers are connected in case
               the cluster consisted of a single node only when the system
               went down. This parameter is usually set to a value smaller
               than wfc-timeout. The assumption here is that peers which were
               unreachable before a reboot are less likely to be be reachable
               after the reboot, so waiting is less likely to help.

               The timeout is specified in seconds. The default value is 0,
               which stands for an infinite timeout. Also see the wfc-timeout
               parameter.

           --outdated-wfc-timeout timeout
               Define how long to wait until all peers are connected if all
               peers were outdated when the system went down. This parameter
               is usually set to a value smaller than wfc-timeout. The
               assumption here is that an outdated peer cannot have become
               primary in the meantime, so we don't need to wait for it as
               long as for a node which was alive before.

               The timeout is specified in seconds. The default value is 0,
               which stands for an infinite timeout. Also see the wfc-timeout
               parameter.

           --wait-after-sb
               This parameter causes DRBD to continue waiting in the init
               script even when a split-brain situation has been detected, and
               the nodes therefore refuse to connect to each other.

           --wfc-timeout timeout
               Define how long the init script waits until all peers are
               connected. This can be useful in combination with a cluster
               manager which cannot manage DRBD resources: when the cluster
               manager starts, the DRBD resources will already be up and
               running. With a more capable cluster manager such as Pacemaker,
               it makes more sense to let the cluster manager control DRBD
               resources. The timeout is specified in seconds. The default
               value is 0, which stands for an infinite timeout. Also see the
               degr-wfc-timeout parameter.

       drbdsetup forget-peer resource peer_node_id
           The forget-peer command removes all traces of a peer node from the
           meta-data. It frees a bitmap slot in the meta-data and make it
           avalable for futher bitmap slot allocation in case a so-far never
           seen node connects.

           The connection must be taken down before this command may be used.
           In case the peer re-connects at a later point a bit-map based
           resync will be turned into a full-sync.

EXAMPLES

       Please see the DRBD User's Guide[1] for examples.

VERSION

       This document was revised for version 9.0.0 of the DRBD distribution.

AUTHOR

       Written by Philipp Reisner <philipp.reisner@linbit.com> and Lars
       Ellenberg <lars.ellenberg@linbit.com>.

REPORTING BUGS

       Report bugs to <drbd-user@lists.linbit.com>.

COPYRIGHT

       Copyright 2001-2012 LINBIT Information Technologies, Philipp Reisner,
       Lars Ellenberg. This is free software; see the source for copying
       conditions. There is NO warranty; not even for MERCHANTABILITY or
       FITNESS FOR A PARTICULAR PURPOSE.

SEE ALSO

       drbd.conf(5), drbd(8), drbddisk(8), drbdadm(8), DRBD User's Guide[1],
       DRBD Web Site[2]

NOTES

        1. DRBD User's Guide
           http://www.drbd.org/users-guide/

        2. DRBD Web Site
           http://www.drbd.org/