UCX  1.8
Unified Communication X

Data Structures

struct  uct_md_attr
 Memory domain attributes. More...
 
struct  uct_md_attr.cap
 
struct  uct_allocated_memory
 Describes a memory allocated by UCT. More...
 
struct  uct_rkey_bundle
 Remote key with its type. More...
 

Typedefs

typedef struct uct_allocated_memory uct_allocated_memory_t
 Describes a memory allocated by UCT. More...
 
typedef struct uct_rkey_bundle uct_rkey_bundle_t
 Remote key with its type. More...
 

Enumerations

enum  uct_sockaddr_accessibility_t { UCT_SOCKADDR_ACC_LOCAL, UCT_SOCKADDR_ACC_REMOTE }
 Socket address accessibility type. More...
 
enum  {
  UCT_MD_FLAG_ALLOC = UCS_BIT(0), UCT_MD_FLAG_REG = UCS_BIT(1), UCT_MD_FLAG_NEED_MEMH = UCS_BIT(2), UCT_MD_FLAG_NEED_RKEY = UCS_BIT(3),
  UCT_MD_FLAG_ADVISE = UCS_BIT(4), UCT_MD_FLAG_FIXED = UCS_BIT(5), UCT_MD_FLAG_RKEY_PTR = UCS_BIT(6), UCT_MD_FLAG_SOCKADDR = UCS_BIT(7)
}
 Memory domain capability flags. More...
 
enum  uct_md_mem_flags {
  UCT_MD_MEM_FLAG_NONBLOCK = UCS_BIT(0), UCT_MD_MEM_FLAG_FIXED = UCS_BIT(1), UCT_MD_MEM_FLAG_LOCK = UCS_BIT(2), UCT_MD_MEM_FLAG_HIDE_ERRORS = UCS_BIT(3),
  UCT_MD_MEM_ACCESS_REMOTE_PUT = UCS_BIT(5), UCT_MD_MEM_ACCESS_REMOTE_GET = UCS_BIT(6), UCT_MD_MEM_ACCESS_REMOTE_ATOMIC = UCS_BIT(7), UCT_MD_MEM_ACCESS_ALL,
  UCT_MD_MEM_ACCESS_RMA
}
 Memory allocation/registration flags. More...
 
enum  uct_mem_advice_t { UCT_MADV_NORMAL = 0, UCT_MADV_WILLNEED }
 list of UCT memory use advice More...
 

Functions

ucs_status_t uct_md_query (uct_md_h md, uct_md_attr_t *md_attr)
 Query for memory domain attributes. More...
 
ucs_status_t uct_md_mem_alloc (uct_md_h md, size_t *length_p, void **address_p, unsigned flags, const char *name, uct_mem_h *memh_p)
 Allocate memory for zero-copy sends and remote access. More...
 
ucs_status_t uct_md_mem_free (uct_md_h md, uct_mem_h memh)
 Release memory allocated by uct_md_mem_alloc. More...
 
ucs_status_t uct_md_mem_advise (uct_md_h md, uct_mem_h memh, void *addr, size_t length, uct_mem_advice_t advice)
 Give advice about the use of memory. More...
 
ucs_status_t uct_md_mem_reg (uct_md_h md, void *address, size_t length, unsigned flags, uct_mem_h *memh_p)
 Register memory for zero-copy sends and remote access. More...
 
ucs_status_t uct_md_mem_dereg (uct_md_h md, uct_mem_h memh)
 Undo the operation of uct_md_mem_reg(). More...
 
ucs_status_t uct_md_detect_memory_type (uct_md_h md, const void *addr, size_t length, ucs_memory_type_t *mem_type_p)
 Detect memory type. More...
 
ucs_status_t uct_mem_alloc (void *addr, size_t min_length, unsigned flags, uct_alloc_method_t *methods, unsigned num_methods, uct_md_h *mds, unsigned num_mds, const char *name, uct_allocated_memory_t *mem)
 Allocate memory for zero-copy communications and remote access. More...
 
ucs_status_t uct_mem_free (const uct_allocated_memory_t *mem)
 Release allocated memory. More...
 
ucs_status_t uct_md_config_read (uct_component_h component, const char *env_prefix, const char *filename, uct_md_config_t **config_p)
 Read the configuration for a memory domain. More...
 
int uct_md_is_sockaddr_accessible (uct_md_h md, const ucs_sock_addr_t *sockaddr, uct_sockaddr_accessibility_t mode)
 Check if remote sock address is accessible from the memory domain. More...
 
ucs_status_t uct_md_mkey_pack (uct_md_h md, uct_mem_h memh, void *rkey_buffer)
 Pack a remote key. More...
 
ucs_status_t uct_rkey_unpack (uct_component_h component, const void *rkey_buffer, uct_rkey_bundle_t *rkey_ob)
 Unpack a remote key. More...
 
ucs_status_t uct_rkey_ptr (uct_component_h component, uct_rkey_bundle_t *rkey_ob, uint64_t remote_addr, void **addr_p)
 Get a local pointer to remote memory. More...
 
ucs_status_t uct_rkey_release (uct_component_h component, const uct_rkey_bundle_t *rkey_ob)
 Release a remote key. More...
 

Detailed Description

The Memory Domain abstracts resources required for network communication, which typically includes memory, transport mechanisms, compute and network resources. It is an isolation mechanism that can be employed by the applications for isolating resources between multiple programming models. The attributes of the Memory Domain are defined by the structure uct_md_attr(). The communication and memory operations are defined in the context of Memory Domain.


Data Structure Documentation

◆ uct_md_attr

struct uct_md_attr

This structure defines the attributes of a Memory Domain which includes maximum memory that can be allocated, credentials required for accessing the memory, and CPU mask indicating the proximity of CPUs.

Examples
uct_hello_world.c.
Data Fields
struct uct_md_attr cap
uct_linear_growth_t reg_cost

Memory registration cost estimation (time,seconds) as a linear function of the buffer size.

char component_name[UCT_COMPONENT_NAME_MAX]

Component name

size_t rkey_packed_size

Size of buffer needed for packed rkey

ucs_cpu_set_t local_cpus

Mask of CPUs near the resource

◆ uct_md_attr.cap

struct uct_md_attr.cap
Data Fields
size_t max_alloc

Maximal allocation size

size_t max_reg

Maximal registration size

uint64_t flags

UCT_MD_FLAG_xx

uint64_t reg_mem_types

Bitmap of memory types that Memory Domain can be registered with

uint64_t detect_mem_types

Bitmap of memory types that Memory Domain can detect if address belongs to it

ucs_memory_type_t access_mem_type

Memory type MD can access

◆ uct_allocated_memory

struct uct_allocated_memory

This structure describes the memory block which includes the address, size, and Memory Domain used for allocation. This structure is passed to interface and the memory is allocated by memory allocation functions uct_mem_alloc.

Data Fields
void * address

Address of allocated memory

size_t length

Real size of allocated memory

uct_alloc_method_t method

Method used to allocate the memory

ucs_memory_type_t mem_type

type of allocated memory

uct_md_h md

if method==MD: MD used to allocate the memory

uct_mem_h memh

if method==MD: MD memory handle

◆ uct_rkey_bundle

struct uct_rkey_bundle

This structure describes the credentials (typically key) and information required to access the remote memory by the communication interfaces.

Data Fields
uct_rkey_t rkey

Remote key descriptor, passed to RMA functions

void * handle

Handle, used internally for releasing the key

void * type

Remote key type

Typedef Documentation

◆ uct_allocated_memory_t

This structure describes the memory block which includes the address, size, and Memory Domain used for allocation. This structure is passed to interface and the memory is allocated by memory allocation functions uct_mem_alloc.

◆ uct_rkey_bundle_t

This structure describes the credentials (typically key) and information required to access the remote memory by the communication interfaces.

Enumeration Type Documentation

◆ uct_sockaddr_accessibility_t

Enumerator
UCT_SOCKADDR_ACC_LOCAL 

Check if local address exists. Address should belong to a local network interface

UCT_SOCKADDR_ACC_REMOTE 

Check if remote address can be reached. Address is routable from one of the local network interfaces

◆ anonymous enum

anonymous enum
Enumerator
UCT_MD_FLAG_ALLOC 

MD supports memory allocation

UCT_MD_FLAG_REG 

MD supports memory registration

UCT_MD_FLAG_NEED_MEMH 

The transport needs a valid local memory handle for zero-copy operations

UCT_MD_FLAG_NEED_RKEY 

The transport needs a valid remote memory key for remote memory operations

UCT_MD_FLAG_ADVISE 

MD supports memory advice

UCT_MD_FLAG_FIXED 

MD supports memory allocation with fixed address

UCT_MD_FLAG_RKEY_PTR 

MD supports direct access to remote memory via a pointer that is returned by uct_rkey_ptr

UCT_MD_FLAG_SOCKADDR 

MD support for client-server connection establishment via sockaddr

◆ uct_md_mem_flags

Enumerator
UCT_MD_MEM_FLAG_NONBLOCK 

Hint to perform non-blocking allocation/registration: page mapping may be deferred until it is accessed by the CPU or a transport.

UCT_MD_MEM_FLAG_FIXED 

Place the mapping at exactly defined address

UCT_MD_MEM_FLAG_LOCK 

Registered memory should be locked. May incur extra cost for registration, but memory access is usually faster.

UCT_MD_MEM_FLAG_HIDE_ERRORS 

Hide errors on memory registration. In some cases registration failure is not an error (e. g. for merged memory regions).

UCT_MD_MEM_ACCESS_REMOTE_PUT 

enable remote put access

UCT_MD_MEM_ACCESS_REMOTE_GET 

enable remote get access

UCT_MD_MEM_ACCESS_REMOTE_ATOMIC 

enable remote atomic access

UCT_MD_MEM_ACCESS_ALL 

enable local and remote access for all operations

UCT_MD_MEM_ACCESS_RMA 

enable local and remote access for put and get operations

◆ uct_mem_advice_t

Enumerator
UCT_MADV_NORMAL 

No special treatment

UCT_MADV_WILLNEED 

can be used on the memory mapped with UCT_MD_MEM_FLAG_NONBLOCK to speed up memory mapping and to avoid page faults when the memory is accessed for the first time.

Function Documentation

◆ uct_md_query()

ucs_status_t uct_md_query ( uct_md_h  md,
uct_md_attr_t md_attr 
)
Parameters
[in]mdMemory domain to query.
[out]md_attrFilled with memory domain attributes.
Examples
uct_hello_world.c.

◆ uct_md_mem_alloc()

ucs_status_t uct_md_mem_alloc ( uct_md_h  md,
size_t *  length_p,
void **  address_p,
unsigned  flags,
const char *  name,
uct_mem_h memh_p 
)

Allocate memory on the memory domain. In order to use this function, MD must support UCT_MD_FLAG_ALLOC flag.

Parameters
[in]mdMemory domain to allocate memory on.
[in,out]length_pPoints to the size of memory to allocate. Upon successful return, filled with the actual size that was allocated, which may be larger than the one requested. Must be >0.
[in,out]address_pThe address
[in]flagsMemory allocation flags, see uct_md_mem_flags.
[in]nameName of the allocated region, used to track memory usage for debugging and profiling.
[out]memh_pFilled with handle for allocated region.

◆ uct_md_mem_free()

ucs_status_t uct_md_mem_free ( uct_md_h  md,
uct_mem_h  memh 
)
Parameters
[in]mdMemory domain memory was allocated on.
[in]memhMemory handle, as returned from uct_md_mem_alloc.

◆ uct_md_mem_advise()

ucs_status_t uct_md_mem_advise ( uct_md_h  md,
uct_mem_h  memh,
void *  addr,
size_t  length,
uct_mem_advice_t  advice 
)

This routine advises the UCT about how to handle memory range beginning at address and size of length bytes. This call does not influence the semantics of the application, but may influence its performance. The advice may be ignored.

Parameters
[in]mdMemory domain memory was allocated or registered on.
[in]memhMemory handle, as returned from uct_md_mem_alloc
[in]addrMemory base address. Memory range must belong to the memh
[in]lengthLength of memory to advise. Must be >0.
[in]adviceMemory use advice as defined in the uct_mem_advice_t list

◆ uct_md_mem_reg()

ucs_status_t uct_md_mem_reg ( uct_md_h  md,
void *  address,
size_t  length,
unsigned  flags,
uct_mem_h memh_p 
)

Register memory on the memory domain. In order to use this function, MD must support UCT_MD_FLAG_REG flag.

Parameters
[in]mdMemory domain to register memory on.
[out]addressMemory to register.
[in]lengthSize of memory to register. Must be >0.
[in]flagsMemory allocation flags, see uct_md_mem_flags.
[out]memh_pFilled with handle for allocated region.
Examples
uct_hello_world.c.

◆ uct_md_mem_dereg()

ucs_status_t uct_md_mem_dereg ( uct_md_h  md,
uct_mem_h  memh 
)
Parameters
[in]mdMemory domain which was used to register the memory.
[in]memhLocal access key to memory region.
Examples
uct_hello_world.c.

◆ uct_md_detect_memory_type()

ucs_status_t uct_md_detect_memory_type ( uct_md_h  md,
const void *  addr,
size_t  length,
ucs_memory_type_t *  mem_type_p 
)
Parameters
[in]mdMemory domain to detect memory type
[in]addrMemory address to detect.
[in]lengthSize of memory
[out]mem_type_pFilled with memory type of the address range if function succeeds
Returns
UCS_OK If memory type is successfully detected UCS_ERR_INVALID_ADDR If failed to detect memory type

◆ uct_mem_alloc()

ucs_status_t uct_mem_alloc ( void *  addr,
size_t  min_length,
unsigned  flags,
uct_alloc_method_t methods,
unsigned  num_methods,
uct_md_h mds,
unsigned  num_mds,
const char *  name,
uct_allocated_memory_t mem 
)

Allocate potentially registered memory. Every one of the provided allocation methods will be used, in turn, to perform the allocation, until one succeeds. Whenever the MD method is encountered, every one of the provided MDs will be used, in turn, to allocate the memory, until one succeeds, or they are exhausted. In this case the next allocation method from the initial list will be attempted.

Parameters
[in]addrIf addr is NULL, the underlying allocation routine will choose the address at which to create the mapping. If addr is non-NULL but UCT_MD_MEM_FLAG_FIXED is not set, the address will be interpreted as a hint as to where to establish the mapping. If addr is non-NULL and UCT_MD_MEM_FLAG_FIXED is set, then the specified address is interpreted as a requirement. In this case, if the mapping to the exact address cannot be made, the allocation request fails.
[in]min_lengthMinimal size to allocate. The actual size may be larger, for example because of alignment restrictions.
[in]flagsMemory allocation flags, see uct_md_mem_flags.
[in]methodsArray of memory allocation methods to attempt.
[in]num_methodsLength of 'methods' array.
[in]mdsArray of memory domains to attempt to allocate the memory with, for MD allocation method.
[in]num_mdsLength of 'mds' array. May be empty, in such case 'mds' may be NULL, and MD allocation method will be skipped.
[in]nameName of the allocation. Used for memory statistics.
[out]memIn case of success, filled with information about the allocated memory. uct_allocated_memory_t.

◆ uct_mem_free()

ucs_status_t uct_mem_free ( const uct_allocated_memory_t mem)

Release the memory allocated by uct_mem_alloc.

Parameters
[in]memDescription of allocated memory, as returned from uct_mem_alloc.

◆ uct_md_config_read()

ucs_status_t uct_md_config_read ( uct_component_h  component,
const char *  env_prefix,
const char *  filename,
uct_md_config_t **  config_p 
)
Parameters
[in]componentRead the configuration of this component.
[in]env_prefixIf non-NULL, search for environment variables starting with this UCT_<prefix>_. Otherwise, search for environment variables starting with just UCT_.
[in]filenameIf non-NULL, read configuration from this file. If the file does not exist, it will be ignored.
[out]config_pFilled with a pointer to the configuration.
Returns
Error code.
Examples
uct_hello_world.c.

◆ uct_md_is_sockaddr_accessible()

int uct_md_is_sockaddr_accessible ( uct_md_h  md,
const ucs_sock_addr_t sockaddr,
uct_sockaddr_accessibility_t  mode 
)

This function checks if a remote sock address can be accessed from a local memory domain. Accessibility can be checked in local or remote mode.

Parameters
[in]mdMemory domain to check accessibility from. This memory domain must support the UCT_MD_FLAG_SOCKADDR flag.
[in]sockaddrSocket address to check accessibility to.
[in]modeMode for checking accessibility, as defined in uct_sockaddr_accessibility_t. Indicates if accessibility is tested on the server side - for binding to the given sockaddr, or on the client side - for connecting to the given remote peer's sockaddr.
Returns
Nonzero if accessible, 0 if inaccessible.

◆ uct_md_mkey_pack()

ucs_status_t uct_md_mkey_pack ( uct_md_h  md,
uct_mem_h  memh,
void *  rkey_buffer 
)
Parameters
[in]mdHandle to memory domain.
[in]memhLocal key, whose remote key should be packed.
[out]rkey_bufferFilled with packed remote key.
Returns
Error code.

◆ uct_rkey_unpack()

ucs_status_t uct_rkey_unpack ( uct_component_h  component,
const void *  rkey_buffer,
uct_rkey_bundle_t rkey_ob 
)
Parameters
[in]componentComponent on which to unpack the remote key.
[in]rkey_bufferPacked remote key buffer.
[out]rkey_obFilled with the unpacked remote key and its type.
Note
The remote key must be unpacked with the same component that was used to pack it. For example, if a remote device address on the remote memory domain which was used to pack the key is reachable by a transport on a local component, then that component is eligible to unpack the key. If the remote key buffer cannot be unpacked with the given component, UCS_ERR_INVALID_PARAM will be returned.
Returns
Error code.

◆ uct_rkey_ptr()

ucs_status_t uct_rkey_ptr ( uct_component_h  component,
uct_rkey_bundle_t rkey_ob,
uint64_t  remote_addr,
void **  addr_p 
)

This routine returns a local pointer to the remote memory described by the rkey bundle. The MD must support UCT_MD_FLAG_RKEY_PTR flag.

Parameters
[in]componentComponent on which to obtain the pointer to the remote key.
[in]rkey_obA remote key bundle as returned by the uct_rkey_unpack function.
[in]remote_addrA remote address within the memory area described by the rkey_ob.
[out]addr_pA pointer that can be used for direct access to the remote memory.
Note
The component used to obtain a local pointer to the remote memory must be the same component that was used to pack the remote key. See notes section for uct_rkey_unpack.
Returns
Error code if the remote memory cannot be accessed directly or the remote address is not valid.

◆ uct_rkey_release()

ucs_status_t uct_rkey_release ( uct_component_h  component,
const uct_rkey_bundle_t rkey_ob 
)
Parameters
[in]componentComponent which was used to unpack the remote key.
[in]rkey_obRemote key to release.