UCX  1.8
Unified Communication X

Data Structures

struct  ucp_mem_map_params
 Tuning parameters for the UCP memory mapping. More...
 
struct  ucp_mem_advise_params
 Tuning parameters for the UCP memory advice. More...
 
struct  ucp_mem_attr
 Attributes of the UCP Memory handle, filled by ucp_mem_query function. More...
 

Typedefs

typedef struct ucp_mem_map_params ucp_mem_map_params_t
 Tuning parameters for the UCP memory mapping. More...
 
typedef enum ucp_mem_advice ucp_mem_advice_t
 list of UCP memory use advice. More...
 
typedef struct ucp_mem_advise_params ucp_mem_advise_params_t
 Tuning parameters for the UCP memory advice. More...
 
typedef struct ucp_rkey * ucp_rkey_h
 UCP Remote memory handle. More...
 
typedef struct ucp_mem * ucp_mem_h
 UCP Memory handle. More...
 
typedef struct ucp_mem_attr ucp_mem_attr_t
 Attributes of the UCP Memory handle, filled by ucp_mem_query function. More...
 

Enumerations

enum  ucp_mem_map_params_field { UCP_MEM_MAP_PARAM_FIELD_ADDRESS = UCS_BIT(0), UCP_MEM_MAP_PARAM_FIELD_LENGTH = UCS_BIT(1), UCP_MEM_MAP_PARAM_FIELD_FLAGS = UCS_BIT(2) }
 UCP memory mapping parameters field mask. More...
 
enum  ucp_mem_advise_params_field { UCP_MEM_ADVISE_PARAM_FIELD_ADDRESS = UCS_BIT(0), UCP_MEM_ADVISE_PARAM_FIELD_LENGTH = UCS_BIT(1), UCP_MEM_ADVISE_PARAM_FIELD_ADVICE = UCS_BIT(2) }
 UCP memory advice parameters field mask. More...
 
enum  { UCP_MEM_MAP_NONBLOCK = UCS_BIT(0), UCP_MEM_MAP_ALLOCATE = UCS_BIT(1), UCP_MEM_MAP_FIXED = UCS_BIT(2) }
 UCP memory mapping flags. More...
 
enum  ucp_mem_advice { UCP_MADV_NORMAL = 0, UCP_MADV_WILLNEED }
 list of UCP memory use advice. More...
 
enum  ucp_mem_attr_field { UCP_MEM_ATTR_FIELD_ADDRESS = UCS_BIT(0), UCP_MEM_ATTR_FIELD_LENGTH = UCS_BIT(1) }
 UCP Memory handle attributes field mask. More...
 

Functions

ucs_status_t ucp_mem_map (ucp_context_h context, const ucp_mem_map_params_t *params, ucp_mem_h *memh_p)
 Map or allocate memory for zero-copy operations. More...
 
ucs_status_t ucp_mem_unmap (ucp_context_h context, ucp_mem_h memh)
 Unmap memory segment. More...
 
ucs_status_t ucp_mem_query (const ucp_mem_h memh, ucp_mem_attr_t *attr)
 query mapped memory segment More...
 
void ucp_mem_print_info (const char *mem_size, ucp_context_h context, FILE *stream)
 Print memory mapping information. More...
 
ucs_status_t ucp_mem_advise (ucp_context_h context, ucp_mem_h memh, ucp_mem_advise_params_t *params)
 give advice about the use of memory More...
 
ucs_status_t ucp_rkey_pack (ucp_context_h context, ucp_mem_h memh, void **rkey_buffer_p, size_t *size_p)
 Pack memory region remote access key. More...
 
void ucp_rkey_buffer_release (void *rkey_buffer)
 Release packed remote key buffer. More...
 
ucs_status_t ucp_ep_rkey_unpack (ucp_ep_h ep, const void *rkey_buffer, ucp_rkey_h *rkey_p)
 Create remote access key from packed buffer. More...
 
ucs_status_t ucp_rkey_ptr (ucp_rkey_h rkey, uint64_t raddr, void **addr_p)
 Get a local pointer to remote memory. More...
 
void ucp_rkey_destroy (ucp_rkey_h rkey)
 Destroy the remote key. More...
 

Detailed Description

UCP Memory routines


Data Structure Documentation

◆ ucp_mem_map_params

struct ucp_mem_map_params

The structure defines the parameters that are used for the UCP memory mapping tuning during the ucp_mem_map routine.

Data Fields
uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_mem_map_params_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

void * address

If the address is not NULL, the routine maps (registers) the memory segment pointed to by this address. If the pointer is NULL, the library allocates mapped (registered) memory segment and returns its address in this argument. Therefore, this value is optional. If it's not set (along with its corresponding bit in the field_mask - UCP_MEM_MAP_PARAM_FIELD_ADDRESS), the ucp_mem_map routine will consider address as set to NULL and will allocate memory.

size_t length

Length (in bytes) to allocate or map (register). This field is mandatory for filling (along with its corresponding bit in the field_mask - UCP_MEM_MAP_PARAM_FIELD_LENGTH). The ucp_mem_map routine will return with an error if the length isn't specified.

unsigned flags

Allocation flags, e.g. UCP_MEM_MAP_NONBLOCK. This value is optional. If it's not set (along with its corresponding bit in the field_mask - UCP_MEM_MAP_PARAM_FIELD_FLAGS), the ucp_mem_map routine will consider the flags as set to zero.

◆ ucp_mem_advise_params

struct ucp_mem_advise_params

This structure defines the parameters that are used for the UCP memory advice tuning during the ucp_mem_advise routine.

Data Fields
uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_mem_advise_params_field. All fields are mandatory. Provides ABI compatibility with respect to adding new fields.

void * address

Memory base address.

size_t length

Length (in bytes) to allocate or map (register).

ucp_mem_advice_t advice

Memory use advice ucp_mem_advice

◆ ucp_mem_attr

struct ucp_mem_attr
Data Fields
uint64_t field_mask

Mask of valid fields in this structure, using bits from ucp_mem_attr_field. Fields not specified in this mask will be ignored. Provides ABI compatibility with respect to adding new fields.

void * address

Address of the memory segment.

size_t length

Size of the memory segment.

Typedef Documentation

◆ ucp_mem_map_params_t

The structure defines the parameters that are used for the UCP memory mapping tuning during the ucp_mem_map routine.

◆ ucp_mem_advice_t

The enumeration list describes memory advice supported by ucp_mem_advise() function.

◆ ucp_mem_advise_params_t

This structure defines the parameters that are used for the UCP memory advice tuning during the ucp_mem_advise routine.

◆ ucp_rkey_h

typedef struct ucp_rkey* ucp_rkey_h

Remote memory handle is an opaque object representing remote memory access information. Typically, the handle includes a memory access key and other network hardware specific information, which are input to remote memory access operations, such as PUT, GET, and ATOMIC. The object is communicated to remote peers to enable an access to the memory region.

◆ ucp_mem_h

typedef struct ucp_mem* ucp_mem_h

Memory handle is an opaque object representing a memory region allocated through UCP library, which is optimized for remote memory access operations (zero-copy operations). The memory handle is a self-contained object, which includes the information required to access the memory region locally, while remote key is used to access it remotely. The memory could be registered to one or multiple network resources that are supported by UCP, such as InfiniBand, Gemini, and others.

◆ ucp_mem_attr_t

typedef struct ucp_mem_attr ucp_mem_attr_t

Enumeration Type Documentation

◆ ucp_mem_map_params_field

The enumeration allows specifying which fields in ucp_mem_map_params_t are present. It is used to enable backward compatibility support.

Enumerator
UCP_MEM_MAP_PARAM_FIELD_ADDRESS 

Address of the memory that will be used in the ucp_mem_map routine.

UCP_MEM_MAP_PARAM_FIELD_LENGTH 

The size of memory that will be allocated or registered in the ucp_mem_map routine.

UCP_MEM_MAP_PARAM_FIELD_FLAGS 

Allocation flags.

◆ ucp_mem_advise_params_field

The enumeration allows specifying which fields in ucp_mem_advise_params_t are present. It is used to enable backward compatibility support.

Enumerator
UCP_MEM_ADVISE_PARAM_FIELD_ADDRESS 

Address of the memory

UCP_MEM_ADVISE_PARAM_FIELD_LENGTH 

The size of memory

UCP_MEM_ADVISE_PARAM_FIELD_ADVICE 

Advice on memory usage

◆ anonymous enum

anonymous enum

The enumeration list describes the memory mapping flags supported by ucp_mem_map() function.

Enumerator
UCP_MEM_MAP_NONBLOCK 

Complete the mapping faster, possibly by not populating the pages in the mapping up-front, and mapping them later when they are accessed by communication routines.

UCP_MEM_MAP_ALLOCATE 

Identify requirement for allocation, if passed address is not a null-pointer then it will be used as a hint or direct address for allocation.

UCP_MEM_MAP_FIXED 

Don't interpret address as a hint: place the mapping at exactly that address. The address must be a multiple of the page size.

◆ ucp_mem_advice

The enumeration list describes memory advice supported by ucp_mem_advise() function.

Enumerator
UCP_MADV_NORMAL 

No special treatment

UCP_MADV_WILLNEED 

can be used on the memory mapped with UCP_MEM_MAP_NONBLOCK to speed up memory mapping and to avoid page faults when the memory is accessed for the first time.

◆ ucp_mem_attr_field

The enumeration allows specifying which fields in ucp_mem_attr_t are present. It is used to enable backward compatibility support.

Enumerator
UCP_MEM_ATTR_FIELD_ADDRESS 

Virtual address

UCP_MEM_ATTR_FIELD_LENGTH 

The size of memory region

Function Documentation

◆ ucp_mem_map()

ucs_status_t ucp_mem_map ( ucp_context_h  context,
const ucp_mem_map_params_t params,
ucp_mem_h memh_p 
)

This routine maps or/and allocates a user-specified memory segment with UCP application context and the network resources associated with it. If the application specifies NULL as an address for the memory segment, the routine allocates a mapped memory segment and returns its address in the address_p argument. The network stack associated with an application context can typically send and receive data from the mapped memory without CPU intervention; some devices and associated network stacks require the memory to be mapped to send and receive data. The memory handle includes all information required to access the memory locally using UCP routines, while remote registration handle provides an information that is necessary for remote memory access.

Note
Another well know terminology for the "map" operation that is typically used in the context of networking is memory "registration" or "pinning". The UCP library registers the memory the available hardware so it can be assessed directly by the hardware.

Memory mapping assumptions:

  • A given memory segment can be mapped by several different communication stacks, if these are compatible.
  • The memh_p handle returned may be used with any sub-region of the mapped memory.
  • If a large segment is registered, and then segmented for subsequent use by a user, then the user is responsible for segmentation and subsequent management.
Matrix of behavior
parameter/flag NONBLOCK ALLOCATE FIXED address result
value 0/1 - the value
only affects the
register/map
phase
0 0 0 error if length > 0
1 0 0 alloc+register
0 1 0 error
0 0 defined register
1 1 0 error
1 0 defined alloc+register,hint
0 1 defined error
1 1 defined alloc+register,fixed
Note
  • register means that the memory will be registered in corresponding transports for RMA/AMO operations. This case intends that the memory was allocated by user before.
  • alloc+register means that the memory will be allocated in the memory provided by the system and registered in corresponding transports for RMA/AMO operations.
  • alloc+register,hint means that the memory will be allocated with using ucp_mem_map_params::address as a hint and registered in corresponding transports for RMA/AMO operations.
  • alloc+register,fixed means that the memory will be allocated and registered in corresponding transports for RMA/AMO operations.
  • error is an erroneous combination of the parameters.
Parameters
[in]contextApplication context to map (register) and allocate the memory on.
[in]paramsUser defined ucp_mem_map_params_t configurations for the UCP memory handle.
[out]memh_pUCP handle for the allocated segment.
Returns
Error code as defined by ucs_status_t

◆ ucp_mem_unmap()

ucs_status_t ucp_mem_unmap ( ucp_context_h  context,
ucp_mem_h  memh 
)

This routine unmaps a user specified memory segment, that was previously mapped using the ucp_mem_map() routine. The unmap routine will also release the resources associated with the memory handle. When the function returns, the ucp_mem_h and associated remote key will be invalid and cannot be used with any UCP routine.

Note
Another well know terminology for the "unmap" operation that is typically used in the context of networking is memory "de-registration". The UCP library de-registers the memory the available hardware so it can be returned back to the operation system.

Error cases:

  • Once memory is unmapped a network access to the region may cause a failure.
Parameters
[in]contextApplication context which was used to allocate/map the memory.
[in]memhHandle to memory region.
Returns
Error code as defined by ucs_status_t

◆ ucp_mem_query()

ucs_status_t ucp_mem_query ( const ucp_mem_h  memh,
ucp_mem_attr_t attr 
)

This routine returns address and length of memory segment mapped with ucp_mem_map() routine.

Parameters
[in]memhHandle to memory region.
[out]attrFilled with attributes of the UCP memory handle.
Returns
Error code as defined by ucs_status_t

◆ ucp_mem_print_info()

void ucp_mem_print_info ( const char *  mem_size,
ucp_context_h  context,
FILE *  stream 
)

This routine maps memory and prints information about the created memory handle: including the mapped memory length, the allocation method, and other useful information associated with the memory handle.

Parameters
[in]mem_sizeSize of the memory to map.
[in]contextThe context on which the memory is mapped.
[in]streamOutput stream on which to print the information.

◆ ucp_mem_advise()

ucs_status_t ucp_mem_advise ( ucp_context_h  context,
ucp_mem_h  memh,
ucp_mem_advise_params_t params 
)

This routine advises the UCP about how to handle memory range beginning at address and size of length bytes. This call does not influence the semantics of the application, but may influence its performance. The UCP may ignore the advice.

Parameters
[in]contextApplication context which was used to allocate/map the memory.
[in]memhHandle to memory region.
[in]paramsMemory base address and length. The advice field is used to pass memory use advice as defined in the ucp_mem_advice list The memory range must belong to the memh
Returns
Error code as defined by ucs_status_t

◆ ucp_rkey_pack()

ucs_status_t ucp_rkey_pack ( ucp_context_h  context,
ucp_mem_h  memh,
void **  rkey_buffer_p,
size_t *  size_p 
)

This routine allocates memory buffer and packs into the buffer a remote access key (RKEY) object. RKEY is an opaque object that provides the information that is necessary for remote memory access. This routine packs the RKEY object in a portable format such that the object can be unpacked on any platform supported by the UCP library. In order to release the memory buffer allocated by this routine the application is responsible for calling the ucp_rkey_buffer_release() routine.

Note
  • RKEYs for InfiniBand and Cray Aries networks typically includes InifiniBand and Aries key.
  • In order to enable remote direct memory access to the memory associated with the memory handle the application is responsible for sharing the RKEY with the peers that will initiate the access.
Parameters
[in]contextApplication context which was used to allocate/map the memory.
[in]memhHandle to memory region.
[out]rkey_buffer_pMemory buffer allocated by the library. The buffer contains packed RKEY.
[out]size_pSize (in bytes) of the packed RKEY.
Returns
Error code as defined by ucs_status_t

◆ ucp_rkey_buffer_release()

void ucp_rkey_buffer_release ( void *  rkey_buffer)

This routine releases the buffer that was allocated using ucp_rkey_pack().

Warning
  • Once memory is released an access to the memory may cause a failure.
  • If the input memory address was not allocated using ucp_rkey_pack() routine the behaviour of this routine is undefined.
Parameters
[in]rkey_bufferBuffer to release.

◆ ucp_ep_rkey_unpack()

ucs_status_t ucp_ep_rkey_unpack ( ucp_ep_h  ep,
const void *  rkey_buffer,
ucp_rkey_h rkey_p 
)

This routine unpacks the remote key (RKEY) object into the local memory such that it can be accessed and used by UCP routines. The RKEY object has to be packed using the ucp_rkey_pack() routine. Application code should not make any changes to the content of the RKEY buffer.

Note
The application is responsible for releasing the RKEY object when it is no longer needed, by calling the ucp_rkey_destroy() routine.
The remote key object can be used for communications only on the endpoint on which it was unpacked.
Parameters
[in]epEndpoint to access using the remote key.
[in]rkey_bufferPacked rkey.
[out]rkey_pRemote key handle.
Returns
Error code as defined by ucs_status_t

◆ ucp_rkey_ptr()

ucs_status_t ucp_rkey_ptr ( ucp_rkey_h  rkey,
uint64_t  raddr,
void **  addr_p 
)

This routine returns a local pointer to the remote memory described by the rkey.

Note
This routine can return a valid pointer only for the endpoints that are reachable via shared memory.
Parameters
[in]rkeyA remote key handle.
[in]raddrA remote memory address within the memory area described by the rkey.
[out]addr_pA pointer that can be used for direct access to the remote memory.
Returns
Error code as defined by ucs_status_t if the remote memory cannot be accessed directly or the remote memory address is not valid.

◆ ucp_rkey_destroy()

void ucp_rkey_destroy ( ucp_rkey_h  rkey)

This routine destroys the RKEY object and the memory that was allocated using the ucp_ep_rkey_unpack() routine. This routine also releases any resources that are associated with the RKEY object.

Warning
  • Once the RKEY object is released an access to the memory will cause an undefined failure.
  • If the RKEY object was not created using ucp_ep_rkey_unpack() routine the behavior of this routine is undefined.
  • The RKEY object must be destroyed after all outstanding operations which are using it are flushed, and before the endpoint on which it was unpacked is destroyed.
Parameters
[in]rkeyRemote key to destroy.