UCT Tag matching operations

Data Structures
struct	uct_tag_context
	Posted tag context. More...

Typedefs
typedef ucs_status_t(*	uct_tag_unexp_eager_cb_t) (void *arg, void *data, size_t length, unsigned flags, uct_tag_t stag, uint64_t imm, void **context)
	Callback to process unexpected eager tagged message.
typedef ucs_status_t(*	uct_tag_unexp_rndv_cb_t) (void *arg, unsigned flags, uint64_t stag, const void *header, unsigned header_length, uint64_t remote_addr, size_t length, const void *rkey_buf)
	Callback to process unexpected rendezvous tagged message.

Functions
ucs_status_t	uct_ep_tag_eager_short (uct_ep_h ep, uct_tag_t tag, const void *data, size_t length)
	Short eager tagged-send operation.
ssize_t	uct_ep_tag_eager_bcopy (uct_ep_h ep, uct_tag_t tag, uint64_t imm, uct_pack_callback_t pack_cb, void *arg, unsigned flags)
	Bcopy eager tagged-send operation.
ucs_status_t	uct_ep_tag_eager_zcopy (uct_ep_h ep, uct_tag_t tag, uint64_t imm, const uct_iov_t *iov, size_t iovcnt, unsigned flags, uct_completion_t *comp)
	Zcopy eager tagged-send operation.
ucs_status_ptr_t	uct_ep_tag_rndv_zcopy (uct_ep_h ep, uct_tag_t tag, const void *header, unsigned header_length, const uct_iov_t *iov, size_t iovcnt, unsigned flags, uct_completion_t *comp)
	Rendezvous tagged-send operation.
ucs_status_t	uct_ep_tag_rndv_cancel (uct_ep_h ep, void *op)
	Cancel outstanding rendezvous operation.
ucs_status_t	uct_ep_tag_rndv_request (uct_ep_h ep, uct_tag_t tag, const void *header, unsigned header_length, unsigned flags)
	Send software rendezvous request.
ucs_status_t	uct_iface_tag_recv_zcopy (uct_iface_h iface, uct_tag_t tag, uct_tag_t tag_mask, const uct_iov_t *iov, size_t iovcnt, uct_tag_context_t *ctx)
	Post a tag to a transport interface.
ucs_status_t	uct_iface_tag_recv_cancel (uct_iface_h iface, uct_tag_context_t *ctx, int force)
	Cancel a posted tag.

Detailed Description

Defines tag matching operations.

Typedef Documentation

uct_tag_unexp_eager_cb_t

typedef ucs_status_t(* uct_tag_unexp_eager_cb_t) (void *arg, void *data, size_t length, unsigned flags, uct_tag_t stag, uint64_t imm, void **context)

This callback is invoked when tagged message sent by eager protocol has arrived and no corresponding tag has been posted.

Note

The callback is always invoked from the context (thread, process) that called uct_iface_progress().

It is allowed to call other communication routines from the callback.

Parameters

[in]	arg	User-defined argument
[in]	data	Points to the received unexpected data.
[in]	length	Length of data.
[in]	flags	Mask with uct_cb_param_flags flags. If it contains UCT_CB_PARAM_FLAG_DESC value, this means data is part of a descriptor which must be released later using uct_iface_release_desc by the user if the callback returns UCS_INPROGRESS.
[in]	stag	Tag from sender.
[in]	imm	Immediate data from sender.
[in,out]	context	Storage for a per-message user-defined context. In this context, the message is defined by the sender side as a single call to uct_ep_tag_eager_short/bcopy/zcopy. On the transport level the message can be fragmented and delivered to the target over multiple fragments. The fragments will preserve the original order of the message. Each fragment will result in invocation of the above callback. The user can use UCT_CB_PARAM_FLAG_FIRST to identify the first fragment, allocate the context object and use the context as a token that is set by the user and passed to subsequent callbacks of the same message. The user is responsible for allocation and release of the context.

Note

No need to allocate the context in the case of a single fragment message (i.e. flags contains UCT_CB_PARAM_FLAG_FIRST, but does not contain UCT_CB_PARAM_FLAG_MORE).

Return values

UCS_OK	- data descriptor was consumed, and can be released by the caller.
UCS_INPROGRESS	- data descriptor is owned by the callee, and will be released later.

uct_tag_unexp_rndv_cb_t

typedef ucs_status_t(* uct_tag_unexp_rndv_cb_t) (void *arg, unsigned flags, uint64_t stag, const void *header, unsigned header_length, uint64_t remote_addr, size_t length, const void *rkey_buf)

This callback is invoked when rendezvous send notification has arrived and no corresponding tag has been posted.

Note

The callback is always invoked from the context (thread, process) that called uct_iface_progress().

It is allowed to call other communication routines from the callback.

Parameters

[in]	arg	User-defined argument
[in]	flags	Mask with uct_cb_param_flags
[in]	stag	Tag from sender.
[in]	header	User defined header.
[in]	header_length	User defined header length in bytes.
[in]	remote_addr	Sender's buffer virtual address.
[in]	length	Sender's buffer length.
[in]	rkey_buf	Sender's buffer packed remote key. It can be passed to uct_rkey_unpack() to create uct_rkey_t.

Warning

If the user became the owner of the desc (by returning UCS_INPROGRESS) the descriptor must be released later by uct_iface_release_desc by the user.

Return values

UCS_OK	- descriptor was consumed, and can be released by the caller.
UCS_INPROGRESS	- descriptor is owned by the callee, and would be released later.

Function Documentation

uct_ep_tag_eager_short()

ucs_status_t uct_ep_tag_eager_short	(	uct_ep_h	ep,
		uct_tag_t	tag,
		const void *	data,
		size_t	length )

This routine sends a message using short eager protocol. Eager protocol means that the whole data is sent to the peer immediately without any preceding notification. The data is provided as buffer and its length,and must not be larger than the corresponding max_short value in uct_iface_attr. The immediate value delivered to the receiver is implicitly equal to 0. If it's required to pass nonzero imm value, uct_ep_tag_eager_bcopy should be used.

Parameters

[in]	ep	Destination endpoint handle.
[in]	tag	Tag to use for the eager message.
[in]	data	Data to send.
[in]	length	Data length.

Returns

UCS_OK - operation completed successfully.

UCS_ERR_NO_RESOURCE - could not start the operation due to lack of send resources.

uct_ep_tag_eager_bcopy()

ssize_t uct_ep_tag_eager_bcopy	(	uct_ep_h	ep,
		uct_tag_t	tag,
		uint64_t	imm,
		uct_pack_callback_t	pack_cb,
		void *	arg,
		unsigned	flags )

This routine sends a message using bcopy eager protocol. Eager protocol means that the whole data is sent to the peer immediately without any preceding notification. Custom data callback is used to copy the data to the network buffers.

Note

The resulted data length must not be larger than the corresponding max_bcopy value in uct_iface_attr.

Parameters

[in]	ep	Destination endpoint handle.
[in]	tag	Tag to use for the eager message.
[in]	imm	Immediate value which will be available to the receiver.
[in]	pack_cb	User callback to pack the data.
[in]	arg	Custom argument to pack_cb.
[in]	flags	Tag message flags, see uct_msg_flags.

Returns

>=0 - The size of the data packed by pack_cb.

otherwise - Error code.

uct_ep_tag_eager_zcopy()

ucs_status_t uct_ep_tag_eager_zcopy	(	uct_ep_h	ep,
		uct_tag_t	tag,
		uint64_t	imm,
		const uct_iov_t *	iov,
		size_t	iovcnt,
		unsigned	flags,
		uct_completion_t *	comp )

This routine sends a message using zcopy eager protocol. Eager protocol means that the whole data is sent to the peer immediately without any preceding notification. The input data (which has to be previously registered) in iov array of uct_iov_t structures sent to remote side ("gather output"). Buffers in iov are processed in array order, so the function complete iov[0] before proceeding to iov[1], and so on.

Note

The resulted data length must not be larger than the corresponding max_zcopy value in uct_iface_attr.

Parameters

[in]	ep	Destination endpoint handle.
[in]	tag	Tag to use for the eager message.
[in]	imm	Immediate value which will be available to the receiver.
[in]	iov	Points to an array of uct_iov_t structures. A particular structure pointer must be a valid address. A NULL terminated array is not required.
[in]	iovcnt	Size of the iov array. If iovcnt is zero, the data is considered empty. Note that iovcnt is limited by the corresponding max_iov value in uct_iface_attr.
[in]	flags	Tag message flags, see uct_msg_flags.
[in]	comp	Completion callback which will be called when the data is reliably received by the peer, and the buffer can be reused or invalidated.

Returns

UCS_OK - operation completed successfully.

UCS_ERR_NO_RESOURCE - could not start the operation due to lack of send resources.

UCS_INPROGRESS - operation started, and comp will be used to notify when it's completed.

uct_ep_tag_rndv_zcopy()

ucs_status_ptr_t uct_ep_tag_rndv_zcopy	(	uct_ep_h	ep,
		uct_tag_t	tag,
		const void *	header,
		unsigned	header_length,
		const uct_iov_t *	iov,
		size_t	iovcnt,
		unsigned	flags,
		uct_completion_t *	comp )

This routine sends a message using rendezvous protocol. Rendezvous protocol means that only a small notification is sent at first, and the data itself is transferred later (when there is a match) to avoid extra memory copy.

Note

The header will be available to the receiver in case of unexpected rendezvous operation only, i.e. the peer has not posted tag for this message yet (by means of uct_iface_tag_recv_zcopy), when it is arrived.

Parameters

[in]	ep	Destination endpoint handle.
[in]	tag	Tag to use for the eager message.
[in]	header	User defined header.
[in]	header_length	User defined header length in bytes. Note that it is limited by the corresponding max_hdr value in uct_iface_attr.
[in]	iov	Points to an array of uct_iov_t structures. A particular structure pointer must be valid address. A NULL terminated array is not required.
[in]	iovcnt	Size of the iov array. If iovcnt is zero, the data is considered empty. Note that iovcnt is limited by the corresponding max_iov value in uct_iface_attr.
[in]	flags	Tag message flags, see uct_msg_flags.
[in]	comp	Completion callback which will be called when the data is reliably received by the peer, and the buffer can be reused or invalidated.

Returns

>=0 - The operation is in progress and the return value is a handle which can be used to cancel the outstanding rendezvous operation.

otherwise - Error code.

uct_ep_tag_rndv_cancel()

ucs_status_t uct_ep_tag_rndv_cancel	(	uct_ep_h	ep,
		void *	op )

This routine signals the underlying transport disregard the outstanding operation without calling completion callback provided in uct_ep_tag_rndv_zcopy.

Note

The operation handle should be valid at the time the routine is invoked. I.e. it should be a handle of the real operation which is not completed yet.

Parameters

[in]	ep	Destination endpoint handle.
[in]	op	Rendezvous operation handle, as returned from uct_ep_tag_rndv_zcopy.

Returns

UCS_OK - The operation has been canceled.

uct_ep_tag_rndv_request()

ucs_status_t uct_ep_tag_rndv_request	(	uct_ep_h	ep,
		uct_tag_t	tag,
		const void *	header,
		unsigned	header_length,
		unsigned	flags )

This routine sends a rendezvous request only, which indicates that the data transfer should be completed in software.

Parameters

[in]	ep	Destination endpoint handle.
[in]	tag	Tag to use for matching.
[in]	header	User defined header
[in]	header_length	User defined header length in bytes. Note that it is limited by the corresponding max_hdr value in uct_iface_attr.
[in]	flags	Tag message flags, see uct_msg_flags.

Returns

UCS_OK - operation completed successfully.

UCS_ERR_NO_RESOURCE - could not start the operation due to lack of send resources.

uct_iface_tag_recv_zcopy()

ucs_status_t uct_iface_tag_recv_zcopy	(	uct_iface_h	iface,
		uct_tag_t	tag,
		uct_tag_t	tag_mask,
		const uct_iov_t *	iov,
		size_t	iovcnt,
		uct_tag_context_t *	ctx )

This routine posts a tag to be matched on a transport interface. When a message with the corresponding tag arrives it is stored in the user buffer (described by iov and iovcnt) directly. The operation completion is reported using callbacks on the ctx structure.

Parameters

[in]	iface	Interface to post the tag on.
[in]	tag	Tag to expect.
[in]	tag_mask	Mask which specifies what bits of the tag to compare.
[in]	iov	Points to an array of uct_iov_t structures. The iov pointer must be a valid address of an array of uct_iov_t structures. A particular structure pointer must be a valid address. A NULL terminated array is not required.
[in]	iovcnt	Size of the iov data uct_iov_t structures array. If iovcnt is zero, the data is considered empty. iovcnt is limited by uct_iface_attr::cap::tag::max_iov.
[in,out]	ctx	Context associated with this particular tag, "priv" field in this structure is used to track the state internally.

Returns

UCS_OK - The tag is posted to the transport.

UCS_ERR_NO_RESOURCE - Could not start the operation due to lack of resources.

UCS_ERR_EXCEEDS_LIMIT - No more room for tags in the transport.

uct_iface_tag_recv_cancel()

ucs_status_t uct_iface_tag_recv_cancel	(	uct_iface_h	iface,
		uct_tag_context_t *	ctx,
		int	force )

This routine cancels a tag, which was previously posted by uct_iface_tag_recv_zcopy. The tag would be either matched or canceled, in a bounded time, regardless of the peer actions. The original completion callback of the tag would be called with the status if force is not set.

Parameters

[in]	iface	Interface to cancel the tag on.
[in]	ctx	Tag context which was used for posting the tag. If force is 0, ctx->completed_cb will be called with either UCS_OK which means the tag was matched and data received despite the cancel request, or UCS_ERR_CANCELED which means the tag was successfully canceled before it was matched.
[in]	force	Whether to report completions to ctx->completed_cb. If nonzero, the cancel is assumed to be successful, and the callback is not called.

Returns

UCS_OK - The tag is canceled in the transport.