UCP Memory routines

Data Structures
struct	ucp_mem_map_params
	Tuning parameters for the UCP memory mapping. More...
struct	ucp_mem_advise_params
	Tuning parameters for the UCP memory advice. More...
struct	ucp_mem_attr
	Attributes of the UCP Memory handle, filled by ucp_mem_query function. More...

Typedefs
typedef struct ucp_mem_map_params	ucp_mem_map_params_t
	Tuning parameters for the UCP memory mapping. More...
typedef enum ucp_mem_advice	ucp_mem_advice_t
	list of UCP memory use advice. More...
typedef struct ucp_mem_advise_params	ucp_mem_advise_params_t
	Tuning parameters for the UCP memory advice. More...
typedef struct ucp_rkey *	ucp_rkey_h
	UCP Remote memory handle. More...
typedef struct ucp_mem *	ucp_mem_h
	UCP Memory handle. More...
typedef struct ucp_mem_attr	ucp_mem_attr_t
	Attributes of the UCP Memory handle, filled by ucp_mem_query function. More...

Enumerations
enum	ucp_mem_map_params_field { UCP_MEM_MAP_PARAM_FIELD_ADDRESS = UCS_BIT(0), UCP_MEM_MAP_PARAM_FIELD_LENGTH = UCS_BIT(1), UCP_MEM_MAP_PARAM_FIELD_FLAGS = UCS_BIT(2) }
	UCP memory mapping parameters field mask. More...
enum	ucp_mem_advise_params_field { UCP_MEM_ADVISE_PARAM_FIELD_ADDRESS = UCS_BIT(0), UCP_MEM_ADVISE_PARAM_FIELD_LENGTH = UCS_BIT(1), UCP_MEM_ADVISE_PARAM_FIELD_ADVICE = UCS_BIT(2) }
	UCP memory advice parameters field mask. More...
enum	{ UCP_MEM_MAP_NONBLOCK = UCS_BIT(0), UCP_MEM_MAP_ALLOCATE = UCS_BIT(1), UCP_MEM_MAP_FIXED = UCS_BIT(2) }
	UCP memory mapping flags. More...
enum	ucp_mem_advice { UCP_MADV_NORMAL = 0, UCP_MADV_WILLNEED }
	list of UCP memory use advice. More...
enum	ucp_mem_attr_field { UCP_MEM_ATTR_FIELD_ADDRESS = UCS_BIT(0), UCP_MEM_ATTR_FIELD_LENGTH = UCS_BIT(1) }
	UCP Memory handle attributes field mask. More...

Functions
ucs_status_t	ucp_mem_map (ucp_context_h context, const ucp_mem_map_params_t *params, ucp_mem_h *memh_p)
	Map or allocate memory for zero-copy operations. More...
ucs_status_t	ucp_mem_unmap (ucp_context_h context, ucp_mem_h memh)
	Unmap memory segment. More...
ucs_status_t	ucp_mem_query (const ucp_mem_h memh, ucp_mem_attr_t *attr)
	query mapped memory segment More...
ucs_status_t	ucp_mem_advise (ucp_context_h context, ucp_mem_h memh, ucp_mem_advise_params_t *params)
	give advice about the use of memory More...
ucs_status_t	ucp_rkey_pack (ucp_context_h context, ucp_mem_h memh, void **rkey_buffer_p, size_t *size_p)
	Pack memory region remote access key. More...
void	ucp_rkey_buffer_release (void *rkey_buffer)
	Release packed remote key buffer. More...
ucs_status_t	ucp_ep_rkey_unpack (ucp_ep_h ep, const void *rkey_buffer, ucp_rkey_h *rkey_p)
	Create remote access key from packed buffer. More...
ucs_status_t	ucp_rkey_ptr (ucp_rkey_h rkey, uint64_t raddr, void **addr_p)
	Get a local pointer to remote memory. More...
void	ucp_rkey_destroy (ucp_rkey_h rkey)
	Destroy the remote key. More...

Detailed Description

UCP Memory routines

---

Data Structure Documentation

struct ucp_mem_map_params

The structure defines the parameters that are used for the UCP memory mapping tuning during the ucp_mem_map routine.

Data Fields
uint64_t	field_mask	Mask of valid fields in this structure, using bits from ucp_mem_map_params_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.
void *	address	If the address is not NULL, the routine maps (registers) the memory segment pointed to by this address. If the pointer is NULL, the library allocates mapped (registered) memory segment and returns its address in this argument. Therefore, this value is optional. If it's not set (along with its corresponding bit in the field_mask - UCP_MEM_MAP_PARAM_FIELD_ADDRESS), the ucp_mem_map routine will consider address as set to NULL and will allocate memory.
size_t	length	Length (in bytes) to allocate or map (register). This field is mandatory for filling (along with its corresponding bit in the field_mask - UCP_MEM_MAP_PARAM_FIELD_LENGTH). The ucp_mem_map routine will return with an error if the length isn't specified.
unsigned	flags	Allocation flags, e.g. UCP_MEM_MAP_NONBLOCK. This value is optional. If it's not set (along with its corresponding bit in the field_mask - UCP_MEM_MAP_PARAM_FIELD_FLAGS), the ucp_mem_map routine will consider the flags as set to zero.

struct ucp_mem_advise_params

This structure defines the parameters that are used for the UCP memory advice tuning during the ucp_mem_advise routine.

Data Fields
uint64_t	field_mask	Mask of valid fields in this structure, using bits from ucp_mem_advise_params_field. All fields are mandatory. Provides ABI compatibility with respect to adding new fields.
void *	address	Memory base address.
size_t	length	Length (in bytes) to allocate or map (register).
ucp_mem_advice_t	advice	Memory use advice ucp_mem_advice

struct ucp_mem_attr

Data Fields
uint64_t	field_mask	Mask of valid fields in this structure, using bits from ucp_mem_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.
void *	address	Address of the memory segment.
size_t	length	Size of the memory segment.

Typedef Documentation

typedef struct ucp_mem_map_params ucp_mem_map_params_t

The structure defines the parameters that are used for the UCP memory mapping tuning during the ucp_mem_map routine.

typedef enum ucp_mem_advice ucp_mem_advice_t

The enumeration list describes memory advice supported by ucp_mem_advise() function.

typedef struct ucp_mem_advise_params ucp_mem_advise_params_t

This structure defines the parameters that are used for the UCP memory advice tuning during the ucp_mem_advise routine.

typedef struct ucp_rkey* ucp_rkey_h

Remote memory handle is an opaque object representing remote memory access information. Typically, the handle includes a memory access key and other network hardware specific information, which are input to remote memory access operations, such as PUT, GET, and ATOMIC. The object is communicated to remote peers to enable an access to the memory region.

typedef struct ucp_mem* ucp_mem_h

Memory handle is an opaque object representing a memory region allocated through UCP library, which is optimized for remote memory access operations (zero-copy operations). The memory handle is a self-contained object, which includes the information required to access the memory region locally, while remote key is used to access it remotely. The memory could be registered to one or multiple network resources that are supported by UCP, such as InfiniBand, Gemini, and others.

typedef struct ucp_mem_attr ucp_mem_attr_t

Enumeration Type Documentation

enum ucp_mem_map_params_field

The enumeration allows specifying which fields in ucp_mem_map_params_t are present. It is used for the enablement of backward compatibility support.

Enumerator
UCP_MEM_MAP_PARAM_FIELD_ADDRESS	Address of the memory that will be used in the ucp_mem_map routine.
UCP_MEM_MAP_PARAM_FIELD_LENGTH	The size of memory that will be allocated or registered in the ucp_mem_map routine.
UCP_MEM_MAP_PARAM_FIELD_FLAGS	Allocation flags.

enum ucp_mem_advise_params_field

The enumeration allows specifying which fields in ucp_mem_advise_params_t are present. It is used for the enablement of backward compatibility support.

Enumerator
UCP_MEM_ADVISE_PARAM_FIELD_ADDRESS	Address of the memory
UCP_MEM_ADVISE_PARAM_FIELD_LENGTH	The size of memory
UCP_MEM_ADVISE_PARAM_FIELD_ADVICE	Advice on memory usage

anonymous enum

The enumeration list describes the memory mapping flags supported by ucp_mem_map() function.

Enumerator
UCP_MEM_MAP_NONBLOCK	Complete the mapping faster, possibly by not populating the pages in the mapping up-front, and mapping them later when they are accessed by communication routines.
UCP_MEM_MAP_ALLOCATE	Identify requirement for allocation, if passed address is not a null-pointer then it will be used as a hint or direct address for allocation.
UCP_MEM_MAP_FIXED	Don't interpret address as a hint: place the mapping at exactly that address. The address must be a multiple of the page size.

enum ucp_mem_advice

The enumeration list describes memory advice supported by ucp_mem_advise() function.

Enumerator
UCP_MADV_NORMAL	No special treatment
UCP_MADV_WILLNEED	can be used on the memory mapped with UCP_MEM_MAP_NONBLOCK to speed up memory mapping and to avoid page faults when the memory is accessed for the first time.

enum ucp_mem_attr_field

The enumeration allows specifying which fields in ucp_mem_attr_t are present. It is used for the enablement of backward compatibility support.

Enumerator
UCP_MEM_ATTR_FIELD_ADDRESS	Virtual address
UCP_MEM_ATTR_FIELD_LENGTH	The size of memory region

Function Documentation

ucs_status_t ucp_mem_map	(	ucp_context_h	context,
		const ucp_mem_map_params_t *	params,
		ucp_mem_h *	memh_p
	)

This routine maps or/and allocates a user-specified memory segment with UCP application context and the network resources associated with it. If the application specifies NULL as an address for the memory segment, the routine allocates a mapped memory segment and returns its address in the address_p argument. The network stack associated with an application context can typically send and receive data from the mapped memory without CPU intervention; some devices and associated network stacks require the memory to be mapped to send and receive data. The memory handle includes all information required to access the memory locally using UCP routines, while remote registration handle provides an information that is necessary for remote memory access.

Note

Another well know terminology for the "map" operation that is typically used in the context of networking is memory "registration" or "pinning". The UCP library registers the memory the available hardware so it can be assessed directly by the hardware.

Memory mapping assumptions:

• A given memory segment can be mapped by several different communication stacks, if these are compatible.
• The memh_p handle returned may be used with any sub-region of the mapped memory.
• If a large segment is registered, and then segmented for subsequent use by a user, then the user is responsible for segmentation and subsequent management.

Matrix of behavior

parameter/flag	NONBLOCK	ALLOCATE	FIXED	address	result
value	0/1 - the value only affects the register/map phase	0	0	0	error if length > 0
		1	0	0	alloc+register
		0	1	0	error
		0	0	defined	register
		1	1	0	error
		1	0	defined	alloc+register,hint
		0	1	defined	error
		1	1	defined	alloc+register,fixed

Note

• register means that the memory will be registered in corresponding transports for RMA/AMO operations. This case intends that the memory was allocated by user before.
• alloc+register means that the memory will be allocated in the memory provided by the system and registered in corresponding transports for RMA/AMO operations.
• alloc+register,hint means that the memory will be allocated with using ucp_mem_map_params::address as a hint and registered in corresponding transports for RMA/AMO operations.
• alloc+register,fixed means that the memory will be allocated and registered in corresponding transports for RMA/AMO operations.
• error is an erroneous combination of the parameters.

Parameters

[in]	context	Application context to map (register) and allocate the memory on.
[in]	params	User defined ucp_mem_map_params_t configurations for the UCP memory handle.
[out]	memh_p	UCP handle for the allocated segment.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_mem_unmap	(	ucp_context_h	context,
		ucp_mem_h	memh
	)

This routine unmaps a user specified memory segment, that was previously mapped using the ucp_mem_map() routine. The unmap routine will also release the resources associated with the memory handle. When the function returns, the ucp_mem_h and associated remote key will be invalid and cannot be used with any UCP routine.

Note

Another well know terminology for the "unmap" operation that is typically used in the context of networking is memory "de-registration". The UCP library de-registers the memory the available hardware so it can be returned back to the operation system.

Error cases:

• Once memory is unmapped a network access to the region may cause a failure.

Parameters

[in]	context	Application context which was used to allocate/map the memory.
[in]	memh	Handle to memory region.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_mem_query	(	const ucp_mem_h	memh,
		ucp_mem_attr_t *	attr
	)

This routine returns address and length of memory segment mapped with ucp_mem_map() routine.

Parameters

[in]	memh	Handle to memory region.
[out]	attr	Filled with attributes of the UCP memory handle.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_mem_advise	(	ucp_context_h	context,
		ucp_mem_h	memh,
		ucp_mem_advise_params_t *	params
	)

This routine advises the UCP about how to handle memory range beginning at address and size of length bytes. This call does not influence the semantics of the application, but may influence its performance. The UCP may ignore the advice.

Parameters

[in]	context	Application context which was used to allocate/map the memory.
[in]	memh	Handle to memory region.
[in]	params	Memory base address and length. The advice field is used to pass memory use advice as defined in the ucp_mem_advice list The memory range must belong to the memh

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_rkey_pack	(	ucp_context_h	context,
		ucp_mem_h	memh,
		void **	rkey_buffer_p,
		size_t *	size_p
	)

This routine allocates memory buffer and packs into the buffer a remote access key (RKEY) object. RKEY is an opaque object that provides the information that is necessary for remote memory access. This routine packs the RKEY object in a portable format such that the object can be unpacked on any platform supported by the UCP library. In order to release the memory buffer allocated by this routine the application is responsible for calling the ucp_rkey_buffer_release() routine.

Note

• RKEYs for InfiniBand and Cray Aries networks typically includes InifiniBand and Aries key.
• In order to enable remote direct memory access to the memory associated with the memory handle the application is responsible for sharing the RKEY with the peers that will initiate the access.

Parameters

[in]	context	Application context which was used to allocate/map the memory.
[in]	memh	Handle to memory region.
[out]	rkey_buffer_p	Memory buffer allocated by the library. The buffer contains packed RKEY.
[out]	size_p	Size (in bytes) of the packed RKEY.

Returns

Error code as defined by ucs_status_t

void ucp_rkey_buffer_release

(

void *

rkey_buffer

)

This routine releases the buffer that was allocated using ucp_rkey_pack().

Warning

• Once memory is released an access to the memory may cause a failure.
• If the input memory address was not allocated using ucp_rkey_pack() routine the behaviour of this routine is undefined.

Parameters

[in]

rkey_buffer

Buffer to release.

ucs_status_t ucp_ep_rkey_unpack	(	ucp_ep_h	ep,
		const void *	rkey_buffer,
		ucp_rkey_h *	rkey_p
	)

This routine unpacks the remote key (RKEY) object into the local memory such that it can be accessed and used by UCP routines. The RKEY object has to be packed using the ucp_rkey_pack() routine. Application code should not make any changes to the content of the RKEY buffer.

Note

The application is responsible for releasing the RKEY object when it is no longer needed by calling the ucp_rkey_destroy() routine.

Parameters

[in]	ep	Endpoint to access using the remote key.
[in]	rkey_buffer	Packed rkey.
[out]	rkey_p	Remote key handle.

Returns

Error code as defined by ucs_status_t

ucs_status_t ucp_rkey_ptr	(	ucp_rkey_h	rkey,
		uint64_t	raddr,
		void **	addr_p
	)

This routine returns a local pointer to the remote memory described by the rkey.

Note

This routine can return a valid pointer only for the endpoints that are reachable via shared memory.

Parameters

[in]	rkey	A remote key handle.
[in]	raddr	A remote memory address within the memory area described by the rkey.
[out]	addr_p	A pointer that can be used for direct access to the remote memory.

Returns

Error code as defined by ucs_status_t if the remote memory cannot be accessed directly or the remote memory address is not valid.

void ucp_rkey_destroy

(

ucp_rkey_h

rkey

)

This routine destroys the RKEY object and the memory that was allocated using the ucp_ep_rkey_unpack() routine. This routine also releases any resources that are associated with the RKEY object.

Warning

• Once the RKEY object is released an access to the memory will cause an undefined failure.
• If the RKEY object was not created using ucp_ep_rkey_unpack() routine the behaviour of this routine is undefined.

Parameters

[in]

rkey

Remote key to destroy.