Provided by: libibverbs-dev_50.0-2build2_amd64 bug

NAME

       mlx5dv_wr_set_dc_addr - Attach a DC info to the last work request

       mlx5dv_wr_raw_wqe - Build a raw work request

       mlx5dv_wr_memcpy - Build a DMA memcpy work request

SYNOPSIS

              #include <infiniband/mlx5dv.h>

              static inline void mlx5dv_wr_set_dc_addr(struct mlx5dv_qp_ex *mqp,
                                                       struct ibv_ah *ah,
                                                       uint32_t remote_dctn,
                                                       uint64_t remote_dc_key);

              static inline void mlx5dv_wr_set_dc_addr_stream(struct mlx5dv_qp_ex *mqp,
                                      struct ibv_ah *ah,
                                      uint32_t remote_dctn,
                                      uint64_t remote_dc_key,
                                      uint16_t stream_id);

              struct mlx5dv_mr_interleaved {
                  uint64_t        addr;
                  uint32_t        bytes_count;
                  uint32_t        bytes_skip;
                  uint32_t        lkey;
              };

              static inline void mlx5dv_wr_mr_interleaved(struct mlx5dv_qp_ex *mqp,
                                      struct mlx5dv_mkey *mkey,
                                      uint32_t access_flags, /* use enum ibv_access_flags */
                                      uint32_t repeat_count,
                                      uint16_t num_interleaved,
                                      struct mlx5dv_mr_interleaved *data);

              static inline void mlx5dv_wr_mr_list(struct mlx5dv_qp_ex *mqp,
                                    struct mlx5dv_mkey *mkey,
                                    uint32_t access_flags, /* use enum ibv_access_flags */
                                    uint16_t num_sges,
                                    struct ibv_sge *sge);

              static inline int mlx5dv_wr_raw_wqe(struct mlx5dv_qp_ex *mqp, const void *wqe);

              static inline void mlx5dv_wr_memcpy(struct mlx5dv_qp_ex *mqp_ex,
                                  uint32_t dest_lkey, uint64_t dest_addr,
                                  uint32_t src_lkey, uint64_t src_addr,
                                  size_t length)

DESCRIPTION

       The  MLX5DV  work  request  APIs  (mlx5dv_wr_*)  is  an extension for IBV work request API
       (ibv_wr_*) with mlx5 specific features for send work request.  This may be  used  together
       with or without ibv_wr_* calls.

USAGE

       To  use  these  APIs  a QP must be created using mlx5dv_create_qp() with send_ops_flags of
       struct ibv_qp_init_attr_ex set.

       If the QP does not support all the requested work request  types  then  QP  creation  will
       fail.

       The    mlx5dv_qp_ex    is   extracted   from   the   IBV_QP   by   ibv_qp_to_qp_ex()   and
       mlx5dv_qp_ex_from_ibv_qp_ex().  This should be used to apply the mlx5 specific features on
       the posted WR.

       A  work  request  creation  requires  to  use  the  ibv_qp_ex  as described in the man for
       ibv_wr_post and mlx5dv_qp with its available builders and setters.

   QP Specific builders
       RC QPs mlx5dv_wr_mr_interleaved()

              registers an  interleaved  memory  layout  by  using  an  indirect  mkey  and  some
              interleaved  data.   The  layout  of  the  memory  pointed  by  the  mkey after its
              registration will be the data representation for the num_interleaved entries.  This
              single layout representation is repeated by repeat_count.

              The  data  as described by struct mlx5dv_mr_interleaved will hold real data defined
              by bytes_count and then a padding of bytes_skip.  Post a  successful  registration,
              RDMA operations can use this mkey.  The hardware will scatter the data according to
              the pattern.  The mkey should be used in a zero-based mode.  The addr field in  its
              ibv_sge  is  an offset in the total data.  To create this mkey mlx5dv_create_mkey()
              should be used.

              Current  implementation  requires  the  IBV_SEND_INLINE  option   to   be   on   in
              ibv_qp_ex->wr_flags field.  To be able to have more than 3 num_interleaved entries,
              the QP should be created with a larger WQE size that may fit it.   This  should  be
              done using the max_inline_data attribute of struct ibv_qp_cap upon its creation.

              As  one  entry will be consumed for strided header, the mkey should be created with
              one more entry than the required num_interleaved.

              In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the reported WC opcode will
              be  MLX5DV_WC_UMR.   Unregister  the  mkey  to  enable another pattern registration
              should be done via ibv_post_send with IBV_WR_LOCAL_INV opcode.
       mlx5dv_wr_mr_list()

              registers a memory layout based on list of  ibv_sge.   The  layout  of  the  memory
              pointed by the mkey after its registration will be based on the list of sge counted
              by num_sges.  Post a successful registration RDMA operations can use this mkey, the
              hardware  will  scatter the data according to the pattern.  The mkey should be used
              in a zero-based mode, the addr field in its ibv_sge is an offset in the total data.

              Current  implementation  requires  the  IBV_SEND_INLINE  option   to   be   on   in
              ibv_qp_ex->wr_flags  field.  To be able to have more than 4 num_sge entries, the QP
              should be created with a larger WQE size that may fit  it.   This  should  be  done
              using the max_inline_data attribute of struct ibv_qp_cap upon its creation.

              In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the reported WC opcode will
              be MLX5DV_WC_UMR.  Unregister the mkey to enable other pattern registration  should
              be done via ibv_post_send with IBV_WR_LOCAL_INV opcode.

       RC or DCI QPs
              mlx5dv_wr_memcpy()

              Builds  a  DMA  memcpy  work request to copy data of length length from src_addr to
              dest_addr.  The copy operation will be done using the DMA MMO functionality of  the
              device to copy data on PCI bus.

              The MLX5DV_QP_EX_WITH_MEMCPY flag in mlx5dv_qp_init_attr.send_ops_flags needs to be
              set during QP creation.  If the device or QP doesn’t support it  then  QP  creation
              will  fail.   The maximum memcpy length that is supported by the device is reported
              in      mlx5dv_context->max_wr_memcpy_length.       A      zero      value       in
              mlx5dv_context->max_wr_memcpy_length   means  the  device  doesn’t  support  memcpy
              operations.

              IBV_SEND_FENCE indicator should be used  on  a  following  send  request  which  is
              dependent on dest_addr of the memcpy operation.

              In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the reported WC opcode will
              be MLX5DV_WC_MEMCPY.

   Raw WQE builders
       mlx5dv_wr_raw_wqe()
              It is used to build a custom work request (WQE) and post it on a  normal  QP.   The
              caller  needs  to  set  all  details  of  the  WQE (except the “ctrl.wqe_index” and
              “ctrl.signature” fields,  which  is  the  driver’s  responsibility  to  set).   The
              MLX5DV_QP_EX_WITH_RAW_WQE flag in mlx5_qp_attr.send_ops_flags needs to be set.

              The wr_flags are ignored as it’s the caller’s responsibility to set flags in WQE.

              No  matter  what  the  send  opcode is, the work completion opcode for a raw WQE is
              IBV_WC_DRIVER2.

   QP Specific setters
       DCI QPs
              mlx5dv_wr_set_dc_addr()  must  be  called  to  set  the  DCI  WR  properties.   The
              destination  address  of  the  work  is  specified  by ah, the remote DCT number is
              specified by remote_dctn and the DC key is specified by remote_dc_key.  This setter
              is   available   when  the  QP  transport  is  DCI  and  send_ops_flags  in  struct
              ibv_qp_init_attr_ex is set.  The available builders and setters for DCI QP are  the
              same  as  RC QP.  DCI QP created with MLX5DV_QP_INIT_ATTR_MASK_DCI_STREAMS can call
              mlx5dv_wr_set_dc_addr_stream() to define the stream_id of the operation to allow HW
              to   choose   one   of   the   multiple   concurrent   DCI   resources.   Calls  to
              mlx5dv_wr_set_dc_addr() are equivalent to using stream_id=0

EXAMPLE

              /* create DC QP type and specify the required send opcodes */
              attr_ex.qp_type = IBV_QPT_DRIVER;
              attr_ex.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS;
              attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_WRITE;

              attr_dv.comp_mask |= MLX5DV_QP_INIT_ATTR_MASK_DC;
              attr_dv.dc_init_attr.dc_type = MLX5DV_DCTYPE_DCI;

              ibv_qp *qp = mlx5dv_create_qp(ctx, attr_ex, attr_dv);
              ibv_qp_ex *qpx = ibv_qp_to_qp_ex(qp);
              mlx5dv_qp_ex *mqpx = mlx5dv_qp_ex_from_ibv_qp_ex(qpx);

              ibv_wr_start(qpx);

              /* Use ibv_qp_ex object to set WR generic attributes */
              qpx->wr_id = my_wr_id_1;
              qpx->wr_flags = IBV_SEND_SIGNALED;
              ibv_wr_rdma_write(qpx, rkey, remote_addr_1);
              ibv_wr_set_sge(qpx, lkey, local_addr_1, length_1);

              /* Use mlx5 DC setter using mlx5dv_qp_ex object */
              mlx5dv_wr_set_wr_dc_addr(mqpx, ah, remote_dctn, remote_dc_key);

              ret = ibv_wr_complete(qpx);

SEE ALSO

       ibv_post_send(3), ibv_create_qp_ex(3), ibv_wr_post(3), mlx5dv_create_mkey(3).

AUTHOR

       Guy Levi <guyle@mellanox.com>

       Mark Zhang <markzhang@nvidia.com>