Provided by: ibacm_1.0.9-3_amd64 bug

NAME

       ibacm - address and route resolution services for InfiniBand.

SYNOPSIS

       ibacm [-D] [-P] [-A addr_file] [-O option_file]

DESCRIPTION

       The  IB  ACM  implements  and  provides  a  framework  for name, address, and route (path)
       resolution  services  over  InfiniBand.   It  is  intended  to  address  connection  setup
       scalability  issues  running  MPI  applications  on  large  clusters.  The IB ACM provides
       information needed to establish a connection, but does not implement the CM protocol.

       A primary user of the ibacm service is the librdmacm library.  This  enables  applications
       to  make  use  of  the  ibacm service without code changes or needing to be aware that the
       service is in use.  librdmacm versions 1.0.12 - 1.0.15 can invoke  IB  ACM  services  when
       built  using  the  --with-ib_acm  option.   Version  1.0.16  and  newer  of librdmacm will
       automatically use the IB ACM if it is installed.  The IB ACM services  tie  in  under  the
       rdma_resolve_addr,   rdma_resolve_route,   and  rdma_getaddrinfo  routines.   For  maximum
       benefit, the rdma_getaddrinfo routine should be used, however existing applications should
       still  see  significant connection scaling benefits using the calls available in librdmacm
       1.0.11 and previous releases.

       The IB ACM is focused on being scalable and efficient.  The current implementation  limits
       network  traffic,  SA  interactions,  and  centralized  services.   ACM  supports multiple
       resolution protocols in order to handle different fabric topologies.

       The  IB  ACM  package  is  comprised  of  two  components:  the  ibacm   service   and   a
       test/configuration utility - ib_acme.  Both are userspace components and are available for
       Linux and Windows.  Additional details are given below.

OPTIONS

       -D     run in daemon mode (default)

       -P     run as standard process

       -A addr_file
              address configuration file

       -O option_file
              option configuration file

QUICK START GUIDE

       1. Prerequisites: libibverbs and libibumad must be installed.   The  IB  stack  should  be
       running  with  IPoIB  configured.   These  steps  assume  that the user has administrative
       privileges.

       2. Install the IB ACM package.  This installs ibacm, ib_acme, and init.d scripts.

       3. Run 'ibacm' as administrator to start the ibacm daemon.

       4. Optionally, run 'ib_acme -d <dest_ip> -v' to verify that the ibacm service is running.

       5. Install librdmacm, using the build option --with-ib_acm if needed.  This  build  option
       is  not  needed  with librdmacm 1.0.17 or newer.  The librdmacm will automatically use the
       ibacm service.  On failures, the librdmacm will fall back to normal resolution.

       6. You can use ib_acme -P to gather performance statistics from the local ibacm daemon  to
       see if the service is working correctly.

NOTES

       ib_acme:

       The  ib_acme program serves a dual role.  It acts as a utility to test ibacm operation and
       help verify if the ibacm service and selected protocol  is  usable  for  a  given  cluster
       configuration.    Additionally,  it  automatically  generates ibacm configuration files to
       assist with or eliminate manual setup.

       ibacm configuration files:

       The ibacm service relies on two configuration files.

       The ibacm_addr.cfg file contains name and address mappings  for  each  IB  <device,  port,
       pkey>  endpoint.   Although  the names in the ibacm_addr.cfg file can be anything, ib_acme
       maps the host name and IP addresses to the IB endpoints.  If the address  file  cannot  be
       found, the ibacm service will attempt to create one using default values.

       The ibacm_opts.cfg file provides a set of configurable options for the ibacm service, such
       as timeout, number of retries, logging level, etc.  ib_acme generates  the  ibacm_opts.cfg
       file  using static information.  If an option file cannot be found, ibacm will use default
       values.

       ibacm:

       The ibacm service is responsible for resolving names  and  addresses  to  InfiniBand  path
       information and caching such data.  It should execute with administrative privileges.

       The  ibacm  implements  a  client  interface  over TCP sockets, which is abstracted by the
       librdmacm library.  One or more back-end protocols  are  used  by  the  ibacm  service  to
       satisfy user requests.  Although the ibacm supports standard SA path record queries on the
       back-end, it also supports a resolution protocol based on multicast traffic.   The  latter
       is  not  usable  on  all fabric topologies, specifically ones that may not have reversible
       paths or fabrics using torus routing.  Users should use the ib_acme utility to verify that
       multicast protocol is usable before running other applications.

       Conceptually,  the  ibacm  service  implements  an  ARP  like  protocol and either uses IB
       multicast records to construct path record data or queries the SA directly,  depending  on
       the  selected  route  protocol.   By  default,  the ibacm services uses and caches SA path
       record queries.

       Specifically, all IB endpoints join a number of multicast groups.  Multicast groups differ
       based  on  rates, mtu, sl, etc., and are prioritized.  All participating endpoints must be
       able to communicate on the lowest priority multicast group.  The ibacm assigns one or more
       names/addresses to each IB endpoint using the ibacm_addr.cfg file.  Clients provide source
       and destination names or addresses as input to the service, and  receive  as  output  path
       record data.

       The  service maps a client's source name/address to a local IB endpoint.  If a client does
       not provide a source address, then  the  ibacm  service  will  select  one  based  on  the
       destination  and  local  routing  tables.   If  the destination name/address is not cached
       locally, it sends a multicast request out on the lowest priority multicast  group  on  the
       local  endpoint.   The request carries a list of multicast groups that the sender can use.
       The recipient of the request selects the highest priority multicast group that it can  use
       as  well  and returns that information directly to the sender.  The request data is cached
       by all endpoints that receive the multicast request message.   The  source  endpoint  also
       caches  the response and uses the multicast group that was selected to construct or obtain
       path record data, which is returned to the client.

       The current implementation of the IB ACM has several additional restrictions:

       - The ibacm is limited in its handling of dynamic changes.   ibacm  must  be  stopped  and
       restarted if a cluster is reconfigured.

       -  Cached  data  does  not  timed  out  and is only updated if a new resolution request is
       received from a different QPN than a cached request.

       - Support for IPv6 has not been verified.

       - The number of addresses that can be assigned to a single endpoint is limited to 4.

       - The number of multicast groups that an endpoint can support is limited to 2.

       The ibacm contains several  internal  caches.   These  include  caches  for  GID  and  LID
       destination  addresses.   These  caches  can  be  optionally preloaded. ibacm supports the
       OpenSM dump_pr plugin "full" PathRecord format which is used to preload these caches.  The
       file  format  is  specified in the ibacm_opts.cfg file via the route_preload setting which
       should be set to full_opensm_v1 for this file format.  Default format is none  which  does
       not  preload  these  caches.  See dump_pr.notes.txt in dump_pr for more information on the
       full_opensm_v1 file format and how to configure OpenSM to generate this file.

       Additionally, the  name,  IPv4,  and  IPv6  caches  can  be  be  preloaded  by  using  the
       addr_preload  option.  The default is none which does not preload these caches. To preload
       these caches, set this option to acm_hosts and configure the addr_data_file appropriately.

SEE ALSO

       ibacm(7), ib_acme(1), rdma_cm(7)