Provided by: libfabric-dev_1.6.2-3ubuntu0.1_amd64 bug

NAME

       fi_cq - Completion queue operations

       fi_cq_open / fi_close : Open/close a completion queue

       fi_control : Control CQ operation or attributes.

       fi_cq_read / fi_cq_readfrom / fi_cq_readerr : Read a completion from a completion queue

       fi_cq_sread / fi_cq_sreadfrom : A synchronous (blocking) read that waits until a specified
       condition has been met before reading a completion from a completion queue.

       fi_cq_signal : Unblock any thread waiting in fi_cq_sread or fi_cq_sreadfrom.

       fi_cq_strerror : Converts provider specific error information into a printable string

SYNOPSIS

              #include <rdma/fi_domain.h>

              int fi_cq_open(struct fid_domain *domain, struct fi_cq_attr *attr,
                  struct fid_cq **cq, void *context);

              int fi_close(struct fid *cq);

              int fi_control(struct fid *cq, int command, void *arg);

              ssize_t fi_cq_read(struct fid_cq *cq, void *buf, size_t count);

              ssize_t fi_cq_readfrom(struct fid_cq *cq, void *buf, size_t count,
                  fi_addr_t *src_addr);

              ssize_t fi_cq_readerr(struct fid_cq *cq, struct fi_cq_err_entry *buf,
                  uint64_t flags);

              ssize_t fi_cq_sread(struct fid_cq *cq, void *buf, size_t count,
                  const void *cond, int timeout);

              ssize_t fi_cq_sreadfrom(struct fid_cq *cq, void *buf, size_t count,
                  fi_addr_t *src_addr, const void *cond, int timeout);

              int fi_cq_signal(struct fid_cq *cq);

              const char * fi_cq_strerror(struct fid_cq *cq, int prov_errno,
                    const void *err_data, char *buf, size_t len);

ARGUMENTS

       domain : Open resource domain

       cq : Completion queue

       attr : Completion queue attributes

       context : User specified context associated with the completion queue.

       buf : For read calls, the data buffer to write  completions  into.   For  write  calls,  a
       completion  to  insert  into the completion queue.  For fi_cq_strerror, an optional buffer
       that receives printable error information.

       count : Number of CQ entries.

       len : Length of data buffer

       src_addr : Source address of a completed receive operation

       flags : Additional flags to apply to the operation

       command : Command of control operation to perform on CQ.

       arg : Optional control argument

       cond : Condition that must be met before a completion is generated

       timeout : Time in milliseconds to wait.  A negative value indicates infinite timeout.

       prov_errno : Provider specific error value

       err_data : Provider specific error data related to a completion

DESCRIPTION

       Completion queues are used to report events associated  with  data  transfers.   They  are
       associated  with  message  sends and receives, RMA, atomic, tagged messages, and triggered
       events.  Reported events are usually associated with a fabric endpoint, but may also refer
       to memory regions used as the target of an RMA or atomic operation.

   fi_cq_open
       fi_cq_open  allocates  a new completion queue.  Unlike event queues, completion queues are
       associated with a resource domain and may be offloaded entirely in provider hardware.

       The properties and behavior of a completion queue are defined by struct fi_cq_attr.

              struct fi_cq_attr {
                  size_t               size;      /* # entries for CQ */
                  uint64_t             flags;     /* operation flags */
                  enum fi_cq_format    format;    /* completion format */
                  enum fi_wait_obj     wait_obj;  /* requested wait object */
                  int                  signaling_vector; /* interrupt affinity */
                  enum fi_cq_wait_cond wait_cond; /* wait condition format */
                  struct fid_wait     *wait_set;  /* optional wait set */
              };

       size : Specifies the minimum size of a completion queue.  A value of 0 indicates that  the
       provider may choose a default value.

       flags : Flags that control the configuration of the CQ.

       • FI_AFFINITY : Indicates that the signaling_vector field (see below) is valid.

       format  :  Completion  queues allow the application to select the amount of detail that it
       must store and report.  The format attribute allows  the  application  to  select  one  of
       several completion formats, indicating the structure of the data that the completion queue
       should return when read.  Supported formats and the structures that correspond to each are
       listed  below.   The  meaning  of the CQ entry fields are defined in the Completion Fields
       section.

       • FI_CQ_FORMAT_UNSPEC : If an unspecified format is requested, then  the  CQ  will  use  a
         provider selected default format.

       • FI_CQ_FORMAT_CONTEXT : Provides only user specified context that was associated with the
         completion.

         struct fi_cq_entry {
             void     *op_context; /* operation context */
         };

       • FI_CQ_FORMAT_MSG : Provides minimal  data  for  processing  completions,  with  expanded
         support for reporting information about received messages.

         struct fi_cq_msg_entry {
             void     *op_context; /* operation context */
             uint64_t flags;       /* completion flags */
             size_t   len;         /* size of received data */
         };

       • FI_CQ_FORMAT_DATA  :  Provides  data associated with a completion.  Includes support for
         received message length, remote CQ data, and multi-receive buffers.

         struct fi_cq_data_entry {
             void     *op_context; /* operation context */
             uint64_t flags;       /* completion flags */
             size_t   len;         /* size of received data */
             void     *buf;        /* receive data buffer */
             uint64_t data;        /* completion data */
         };

       • FI_CQ_FORMAT_TAGGED : Expands completion data to include support for the tagged  message
         interfaces.

         struct fi_cq_tagged_entry {
             void     *op_context; /* operation context */
             uint64_t flags;       /* completion flags */
             size_t   len;         /* size of received data */
             void     *buf;        /* receive data buffer */
             uint64_t data;        /* completion data */
             uint64_t tag;         /* received tag */
         };

       wait_obj  :  CQ's  may  be  associated  with  a  specific wait object.  Wait objects allow
       applications to block until the wait object is signaled, indicating that a  completion  is
       available  to  be  read.   Users may use fi_control to retrieve the underlying wait object
       associated with a CQ, in order to use it in other system calls.  The following values  may
       be  used  to  specify  the  type  of  wait  object  associated  with  a  CQ: FI_WAIT_NONE,
       FI_WAIT_UNSPEC,  FI_WAIT_SET,  FI_WAIT_FD,  and  FI_WAIT_MUTEX_COND.    The   default   is
       FI_WAIT_NONE.

       • FI_WAIT_NONE  :  Used to indicate that the user will not block (wait) for completions on
         the CQ.  When FI_WAIT_NONE is specified, the application may  not  call  fi_cq_sread  or
         fi_cq_sreadfrom.

       • FI_WAIT_UNSPEC : Specifies that the user will only wait on the CQ using fabric interface
         calls, such as fi_cq_sread or fi_cq_sreadfrom.  In this case,  the  underlying  provider
         may  select  the most appropriate or highest performing wait object available, including
         custom wait mechanisms.  Applications that select FI_WAIT_UNSPEC are not  guaranteed  to
         retrieve the underlying wait object.

       • FI_WAIT_SET  :  Indicates that the completion queue should use a wait set object to wait
         for completions.  If specified, the wait_set field must reference an existing  wait  set
         object.

       • FI_WAIT_FD  :  Indicates that the CQ should use a file descriptor as its wait mechanism.
         A file descriptor wait object must be  usable  in  select,  poll,  and  epoll  routines.
         However, a provider may signal an FD wait object by marking it as readable, writable, or
         with an error.

       • FI_WAIT_MUTEX_COND : Specifies that the CQ should use a pthread mutex and cond  variable
         as a wait object.

       • FI_WAIT_CRITSEC_COND  :  Windows  specific.  Specifies that the CQ should use a critical
         section and condition variable as a wait object.

       signaling_vector : If the FI_AFFINITY flag is set, this indicates the logical  cpu  number
       (0..max  cpu - 1) that interrupts associated with the CQ should target.  This field should
       be treated as a hint to the provider and may be ignored if the provider does  not  support
       interrupt affinity.

       wait_cond  :  By  default,  when a completion is inserted into a CQ that supports blocking
       reads (fi_cq_sread/fi_cq_sreadfrom), the corresponding wait object is signaled.  Users may
       specify  a  condition  that  must  first  be met before the wait is satisfied.  This field
       indicates how the provider should interpret the cond field, which describes the  condition
       needed to signal the wait object.

       A wait condition should be treated as an optimization.  Providers are not required to meet
       the requirements of the condition before signaling the wait object.   Applications  should
       not rely on the condition necessarily being true when a blocking read call returns.

       If  wait_cond  is set to FI_CQ_COND_NONE, then no additional conditions are applied to the
       signaling of the CQ wait object, and the insertion of any new entry will trigger the  wait
       condition.   If  wait_cond  is  set  to  FI_CQ_COND_THRESHOLD,  then  the  cond  field  is
       interpreted as a size_t threshold value.  The threshold indicates the  number  of  entries
       that are to be queued before at the CQ before the wait is satisfied.

       This field is ignored if wait_obj is set to FI_WAIT_NONE.

       wait_set  :  If  wait_obj is FI_WAIT_SET, this field references a wait object to which the
       completion queue should attach.  When an event is inserted into the completion queue,  the
       corresponding wait set will be signaled if all necessary conditions are met.  The use of a
       wait_set enables an optimized method of waiting  for  events  across  multiple  event  and
       completion queues.  This field is ignored if wait_obj is not FI_WAIT_SET.

   fi_close
       The  fi_close  call  releases  all  resources  associated  with  a  completion queue.  Any
       completions which remain on the CQ when it is closed are lost.

       When closing the CQ, there must be no opened  endpoints,  transmit  contexts,  or  receive
       contexts  associated  with  the  CQ.   If  resources are still associated with the CQ when
       attempting to close, the call will return -FI_EBUSY.

   fi_control
       The fi_control call is used to access provider or implementation specific details  of  the
       completion  queue.  Access to the CQ should be serialized across all calls when fi_control
       is invoked, as it may redirect the implementation of CQ operations.  The following control
       commands are usable with a CQ.

       FI_GETWAIT  (void **) : This command allows the user to retrieve the low-level wait object
       associated with the CQ.  The format of the wait-object is specified  during  CQ  creation,
       through  the  CQ  attributes.   The  fi_control arg parameter should be an address where a
       pointer to the returned wait object will be written.  See  fi_eq.3  for  addition  details
       using fi_control with FI_GETWAIT.

   fi_cq_read
       The fi_cq_read operation performs a non-blocking read of completion data from the CQ.  The
       format of the completion event is  determined  using  the  fi_cq_format  option  that  was
       specified  when  the  CQ was opened.  Multiple completions may be retrieved from a CQ in a
       single call.  The maximum number of entries to return is limited to  the  specified  count
       parameter,  with the number of entries successfully read from the CQ returned by the call.
       (See return values section below.)

       CQs are optimized to report operations  which  have  completed  successfully.   Operations
       which  fail  are  reported  'out  of  band'.   Such  operations  are  retrieved  using the
       fi_cq_readerr function.  When an operation that has completed with an unexpected error  is
       encountered,  it  is  placed  into  a temporary error queue.  Attempting to read from a CQ
       while an item is in the error queue results in fi_cq_read failing with a  return  code  of
       -FI_EAVAIL.    Applications   may   use  this  return  code  to  determine  when  to  call
       fi_cq_readerr.

   fi_cq_readfrom
       The fi_cq_readfrom call behaves identical to fi_cq_read, with the exception that it allows
       the  CQ  to  return  source address information to the user for any received data.  Source
       address data is only available for those endpoints configured with  FI_SOURCE  capability.
       If  fi_cq_readfrom  is  called  on  an  endpoint  for  which source addressing data is not
       available, the source address will be  set  to  FI_ADDR_NOTAVAIL.   The  number  of  input
       src_addr entries must be the same as the count parameter.

       Returned  source  addressing  data  is  converted  from  the  native  address  used by the
       underlying fabric into an fi_addr_t, which may be used in transmit operations.  Typically,
       returning  fi_addr_t  requires that the source address be inserted into the address vector
       associated with the receiving endpoint.  For endpoints allocated using  the  FI_SOURCE_ERR
       capability,  if  the  source  address  has  not  been  inserted  into  the address vector,
       fi_cq_readfrom will return -FI_EAVAIL.  The  completion  will  then  be  reported  through
       fi_cq_readerr with error code -FI_EADDRNOTAVAIL.  See fi_cq_readerr for details.

       If  FI_SOURCE  is specified without FI_SOURCE_ERR, source addresses which cannot be mapped
       to a local fi_addr_t will be reported as FI_ADDR_NOTAVAIL.  The behavior is  dependent  on
       the  type  of  address  vector in use.  For AVs of type FI_AV_MAP, source addresses may be
       mapped directly to an fi_addr_t value, even if the source address were not  inserted  into
       the  AV.   This  allows  the  provider  to  optimize the reporting of the source fi_addr_t
       without the overhead of verifying whether the address is  in  the  AV.   If  full  address
       validation is necessary, FI_SOURCE_ERR must be used.

   fi_cq_sread / fi_cq_sreadfrom
       The  fi_cq_sread  and  fi_cq_sreadfrom  calls  are  the  blocking equivalent operations to
       fi_cq_read and fi_cq_readfrom.  Their behavior is similar to the non-blocking calls,  with
       the  exception that the calls will not return until either a completion has been read from
       the CQ or an error or timeout occurs.

       It is invalid for applications to call these functions if the CQ has been configured  with
       a wait object of FI_WAIT_NONE or FI_WAIT_SET.

   fi_cq_readerr
       The  read  error function, fi_cq_readerr, retrieves information regarding any asynchronous
       operation which has completed with an unexpected error.  fi_cq_readerr is  a  non-blocking
       call, returning immediately whether an error completion was found or not.

       Error  information  is reported to the user through struct fi_cq_err_entry.  The format of
       this structure is defined below.

              struct fi_cq_err_entry {
                  void     *op_context; /* operation context */
                  uint64_t flags;       /* completion flags */
                  size_t   len;         /* size of received data */
                  void     *buf;        /* receive data buffer */
                  uint64_t data;        /* completion data */
                  uint64_t tag;         /* message tag */
                  size_t   olen;        /* overflow length */
                  int      err;         /* positive error code */
                  int      prov_errno;  /* provider error code */
                  void    *err_data;    /*  error data */
                  size_t   err_data_size; /* size of err_data */
              };

       The general reason for the error is provided through the  err  field.   Provider  specific
       error information may also be available through the prov_errno and err_data fields.  Users
       may call fi_cq_strerror to convert provider specific error information  into  a  printable
       string for debugging purposes.  See field details below for more information on the use of
       err_data and err_data_size.

       Notable completion error codes are given below.

       FI_EADDRNOTAVAIL : This error code is used by CQs configured with FI_SOURCE_ERR to  report
       completions  for  which  a matching fi_addr_t source address could not be found.  An error
       code of FI_EADDRNOTAVAIL indicates that the data transfer was  successfully  received  and
       processed,  with  the  fi_cq_err_entry fields containing information about the completion.
       The err_data field will be set to the source address data.  The source address will be  in
       the  same format as specified through the fi_info addr_format field for the opened domain.
       This may be passed directly into an fi_av_insert call to add the  source  address  to  the
       address vector.

   fi_cq_signal
       The  fi_cq_signal  call will unblock any thread waiting in fi_cq_sread or fi_cq_sreadfrom.
       This may be used to wake-up a  thread  that  is  blocked  waiting  to  read  a  completion
       operation.   The  fi_cq_signal operation is only available if the CQ was configured with a
       wait object.

COMPLETION FIELDS

       The CQ entry data structures share many of the same fields.  The meanings of these  fields
       are the same for all CQ entry structure formats.

       op_context  :  The  operation  context is the application specified context value that was
       provided  with  an  asynchronous  operation.   The  op_context  field  is  valid  for  all
       completions that are associated with an asynchronous operation.

       For  completion events that are not associated with a posted operation, this field will be
       set to NULL.  This includes completions generated at the target in response to  RMA  write
       operations  that  carry  CQ data (FI_REMOTE_WRITE | FI_REMOTE_CQ_DATA flags set), when the
       FI_RX_CQ_DATA mode bit is not required.

       flags : This specifies flags associated with  the  completed  operation.   The  Completion
       Flags section below lists valid flag values.  Flags are set for all relevant completions.

       len  :  This  len  field  only  applies  to  completed  receive operations (e.g.  fi_recv,
       fi_trecv, etc.).  It indicates the size of received message data -- i.e.   how  many  data
       bytes   were   placed   into   the   associated   receive   buffer   by   a  corresponding
       fi_send/fi_tsend/et al call.  If an endpoint has been configured  with  the  FI_MSG_PREFIX
       mode, the len also reflects the size of the prefix buffer.

       buf  : The buf field is only valid for completed receive operations, and only applies when
       the receive buffer was posted with the FI_MULTI_RECV flag.  In this case,  buf  points  to
       the starting location where the receive data was placed.

       data  :  The data field is only valid if the FI_REMOTE_CQ_DATA completion flag is set, and
       only applies to receive completions.  If FI_REMOTE_CQ_DATA is set, this field will contain
       the  completion  data  provided  by  the  peer  as  part  of  their transmit request.  The
       completion data will be given in host byte order.

       tag : A tag applies only to received messages that  occur  using  the  tagged  interfaces.
       This  field contains the tag that was included with the received message.  The tag will be
       in host byte order.

       olen : The olen field applies to received  messages.   It  is  used  to  indicate  that  a
       received  message has overrun the available buffer space and has been truncated.  The olen
       specifies the amount of data that did not fit into the available receive  buffer  and  was
       discarded.

       err  :  This  err  code  is a positive fabric errno associated with a completion.  The err
       value indicates the general reason for an error, if one occurred.  See  fi_errno.3  for  a
       list of possible error codes.

       prov_errno  : On an error, prov_errno may contain a provider specific error code.  The use
       of this field and its meaning is provider specific.  It  is  intended  to  be  used  as  a
       debugging  aid.   See fi_cq_strerror for additional details on converting this error value
       into a human readable string.

       err_data : On an error,  err_data  may  reference  a  provider  specific  amount  of  data
       associated with an error.  The use of this field and its meaning is provider specific.  It
       is intended to be used as a debugging aid.  See fi_cq_strerror for additional  details  on
       converting this error data into a human readable string.

       err_data_size  :  On  input,  err_data_size  indicates  the size of the err_data buffer in
       bytes.  On output, err_data_size will be set to the number of bytes copied to the err_data
       buffer.  The err_data information is typically used with fi_cq_strerror to provide details
       about the type of error that occurred.

       For compatibility purposes, if err_data_size is 0 on input, or the fabric was opened  with
       release  < 1.5, err_data will be set to a data buffer owned by the provider.  The contents
       of the buffer will remain valid until a subsequent read call against the CQ.  Applications
       must  serialize  access  to  the  CQ  when  processing  errors  to  ensure that the buffer
       referenced by err_data does not change.

COMPLETION FLAGS

       Completion flags provide  additional  details  regarding  the  completed  operation.   The
       following completion flags are defined.

       FI_SEND  :  Indicates  that  the  completion  was  for a send operation.  This flag may be
       combined with an FI_MSG or FI_TAGGED flag.

       FI_RECV : Indicates that the completion was for a receive operation.   This  flag  may  be
       combined with an FI_MSG or FI_TAGGED flag.

       FI_RMA  :  Indicates  that  an RMA operation completed.  This flag may be combined with an
       FI_READ, FI_WRITE, FI_REMOTE_READ, or FI_REMOTE_WRITE flag.

       FI_ATOMIC : Indicates that an atomic operation completed.  This flag may be combined  with
       an FI_READ, FI_WRITE, FI_REMOTE_READ, or FI_REMOTE_WRITE flag.

       FI_MSG  :  Indicates  that a message-based operation completed.  This flag may be combined
       with an FI_SEND or FI_RECV flag.

       FI_TAGGED : Indicates that a  tagged  message  operation  completed.   This  flag  may  be
       combined with an FI_SEND or FI_RECV flag.

       FI_MULTICAST  : Indicates that a multicast operation completed.  This flag may be combined
       with FI_MSG and relevant flags.  This flag is only guaranteed to  be  valid  for  received
       messages if the endpoint has been configured with FI_SOURCE.

       FI_READ  :  Indicates that a locally initiated RMA or atomic read operation has completed.
       This flag may be combined with an FI_RMA or FI_ATOMIC flag.

       FI_WRITE : Indicates that a locally initiated RMA or atomic write operation has completed.
       This flag may be combined with an FI_RMA or FI_ATOMIC flag.

       FI_REMOTE_READ  :  Indicates  that  a  remotely initiated RMA or atomic read operation has
       completed.  This flag may be combined with an FI_RMA or FI_ATOMIC flag.

       FI_REMOTE_WRITE : Indicates that a remotely initiated RMA or atomic  write  operation  has
       completed.  This flag may be combined with an FI_RMA or FI_ATOMIC flag.

       FI_REMOTE_CQ_DATA  :  This  indicates  that  remote  CQ  data  is available as part of the
       completion.

       FI_MULTI_RECV  :  This  flag  applies  to  receive  buffers  that  were  posted  with  the
       FI_MULTI_RECV  flag  set.  This completion flag indicates that the original receive buffer
       referenced by the  completion  has  been  consumed  and  was  released  by  the  provider.
       Providers  may  set  this  flag  on the last message that is received into the multi- recv
       buffer, or may generate a separate completion that indicates  that  the  buffer  has  been
       released.

       Applications  can  distinguish  between  these two cases by examining the completion entry
       flags field.  If additional flags, such as FI_RECV, are set, the completion is  associated
       with  a  received  message.  In this case, the buf field will reference the location where
       the received message  was  placed  into  the  multi-recv  buffer.   Other  fields  in  the
       completion entry will be determined based on the received message.  If other flag bits are
       zero, the provider is reporting that the multi-recv buffer  has  been  released,  and  the
       completion entry is not associated with a received message.

NOTES

       A  completion  queue  must  be bound to at least one enabled endpoint before any operation
       such as fi_cq_read, fi_cq_readfrom, fi_cq_sread, fi_cq_sreadfrom etc.  can  be  called  on
       it.

       Completion  flags  may  be  suppressed  if the FI_NOTIFY_FLAGS_ONLY mode bit has been set.
       When enabled, only the following flags are guaranteed to be set in  completion  data  when
       they  are  valid: FI_REMOTE_READ and FI_REMOTE_WRITE (when FI_RMA_EVENT capability bit has
       been set), FI_REMOTE_CQ_DATA, and FI_MULTI_RECV.

       If a completion queue has been overrun, it will be placed into an 'overrun'  state.   Read
       operations  will  continue  to  return any valid, non-corrupted completions, if available.
       After all valid completions have been retrieved, any attempt to read the CQ will result in
       it  returning  an FI_EOVERRUN error event.  Overrun completion queues are considered fatal
       and may not be used to report additional completions once the overrun occurs.

RETURN VALUES

       fi_cq_open  /  fi_cq_signal  :  Returns  0  on  success.   On  error,  a  negative   value
       corresponding to fabric errno is returned.

       fi_cq_read  /  fi_cq_readfrom  / fi_cq_readerr fi_cq_sread / fi_cq_sreadfrom : On success,
       returns the number of completion events retrieved from the completion queue.  On error,  a
       negative value corresponding to fabric errno is returned.  If no completions are available
       to return from the CQ, -FI_EAGAIN will be returned.

       fi_cq_strerror : Returns a character string interpretation of the provider specific  error
       returned with a completion.

       Fabric errno values are defined in rdma/fi_errno.h.

SEE ALSO

       fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_eq(3), fi_cntr(3), fi_poll(3)

AUTHORS

       OpenFabrics.