UCT Memory Domain

Data Structures
struct	uct_md_attr
	Memory domain attributes. More...
struct	uct_md_attr.cap
struct	uct_allocated_memory
	Describes a memory allocated by UCT. More...
struct	uct_rkey_bundle
	Remote key with its type. More...

Typedefs
typedef struct uct_allocated_memory	uct_allocated_memory_t
	Describes a memory allocated by UCT. More...
typedef struct uct_rkey_bundle	uct_rkey_bundle_t
	Remote key with its type. More...

Enumerations
enum	uct_sockaddr_accessibility_t { UCT_SOCKADDR_ACC_LOCAL, UCT_SOCKADDR_ACC_REMOTE }
	Socket address accessibility type. More...
enum	{   UCT_MD_FLAG_ALLOC = UCS_BIT(0), UCT_MD_FLAG_REG = UCS_BIT(1), UCT_MD_FLAG_NEED_MEMH = UCS_BIT(2), UCT_MD_FLAG_NEED_RKEY = UCS_BIT(3),   UCT_MD_FLAG_ADVISE = UCS_BIT(4), UCT_MD_FLAG_FIXED = UCS_BIT(5), UCT_MD_FLAG_RKEY_PTR = UCS_BIT(6), UCT_MD_FLAG_SOCKADDR = UCS_BIT(7) }
	Memory domain capability flags. More...
enum	uct_md_mem_flags {   UCT_MD_MEM_FLAG_NONBLOCK = UCS_BIT(0), UCT_MD_MEM_FLAG_FIXED = UCS_BIT(1), UCT_MD_MEM_FLAG_LOCK = UCS_BIT(2), UCT_MD_MEM_FLAG_HIDE_ERRORS = UCS_BIT(3),   UCT_MD_MEM_ACCESS_REMOTE_PUT = UCS_BIT(5), UCT_MD_MEM_ACCESS_REMOTE_GET = UCS_BIT(6), UCT_MD_MEM_ACCESS_REMOTE_ATOMIC = UCS_BIT(7), UCT_MD_MEM_ACCESS_ALL,   UCT_MD_MEM_ACCESS_RMA }
	Memory allocation/registration flags. More...
enum	uct_mem_advice_t { UCT_MADV_NORMAL = 0, UCT_MADV_WILLNEED }
	list of UCT memory use advice More...

Functions
ucs_status_t	uct_md_query (uct_md_h md, uct_md_attr_t *md_attr)
	Query for memory domain attributes. More...
ucs_status_t	uct_md_mem_alloc (uct_md_h md, size_t *length_p, void **address_p, unsigned flags, const char *name, uct_mem_h *memh_p)
	Allocate memory for zero-copy sends and remote access. More...
ucs_status_t	uct_md_mem_free (uct_md_h md, uct_mem_h memh)
	Release memory allocated by uct_md_mem_alloc. More...
ucs_status_t	uct_md_mem_advise (uct_md_h md, uct_mem_h memh, void *addr, size_t length, uct_mem_advice_t advice)
	Give advice about the use of memory. More...
ucs_status_t	uct_md_mem_reg (uct_md_h md, void *address, size_t length, unsigned flags, uct_mem_h *memh_p)
	Register memory for zero-copy sends and remote access. More...
ucs_status_t	uct_md_mem_dereg (uct_md_h md, uct_mem_h memh)
	Undo the operation of uct_md_mem_reg(). More...
ucs_status_t	uct_md_detect_memory_type (uct_md_h md, const void *addr, size_t length, ucs_memory_type_t *mem_type_p)
	Detect memory type. More...
ucs_status_t	uct_mem_alloc (void *addr, size_t min_length, unsigned flags, uct_alloc_method_t *methods, unsigned num_methods, uct_md_h *mds, unsigned num_mds, const char *name, uct_allocated_memory_t *mem)
	Allocate memory for zero-copy communications and remote access. More...
ucs_status_t	uct_mem_free (const uct_allocated_memory_t *mem)
	Release allocated memory. More...
ucs_status_t	uct_md_config_read (uct_component_h component, const char *env_prefix, const char *filename, uct_md_config_t **config_p)
	Read the configuration for a memory domain. More...
int	uct_md_is_sockaddr_accessible (uct_md_h md, const ucs_sock_addr_t *sockaddr, uct_sockaddr_accessibility_t mode)
	Check if remote sock address is accessible from the memory domain. More...
ucs_status_t	uct_md_mkey_pack (uct_md_h md, uct_mem_h memh, void *rkey_buffer)
	Pack a remote key. More...
ucs_status_t	uct_rkey_unpack (uct_component_h component, const void *rkey_buffer, uct_rkey_bundle_t *rkey_ob)
	Unpack a remote key. More...
ucs_status_t	uct_rkey_ptr (uct_component_h component, uct_rkey_bundle_t *rkey_ob, uint64_t remote_addr, void **addr_p)
	Get a local pointer to remote memory. More...
ucs_status_t	uct_rkey_release (uct_component_h component, const uct_rkey_bundle_t *rkey_ob)
	Release a remote key. More...

Detailed Description

The Memory Domain abstracts resources required for network communication, which typically includes memory, transport mechanisms, compute and network resources. It is an isolation mechanism that can be employed by the applications for isolating resources between multiple programming models. The attributes of the Memory Domain are defined by the structure uct_md_attr(). The communication and memory operations are defined in the context of Memory Domain.

---

Data Structure Documentation

uct_md_attr

struct uct_md_attr

This structure defines the attributes of a Memory Domain which includes maximum memory that can be allocated, credentials required for accessing the memory, and CPU mask indicating the proximity of CPUs.

Examples

uct_hello_world.c.

Data Fields
struct uct_md_attr	cap
uct_linear_growth_t	reg_cost	Memory registration cost estimation (time,seconds) as a linear function of the buffer size.
char	component_name[UCT_COMPONENT_NAME_MAX]	Component name
size_t	rkey_packed_size	Size of buffer needed for packed rkey
ucs_cpu_set_t	local_cpus	Mask of CPUs near the resource

uct_md_attr.cap

struct uct_md_attr.cap

Data Fields
size_t	max_alloc	Maximal allocation size
size_t	max_reg	Maximal registration size
uint64_t	flags	UCT_MD_FLAG_xx
uint64_t	reg_mem_types	Bitmap of memory types that Memory Domain can be registered with
uint64_t	detect_mem_types	Bitmap of memory types that Memory Domain can detect if address belongs to it
ucs_memory_type_t	access_mem_type	Memory type MD can access

uct_allocated_memory

struct uct_allocated_memory

This structure describes the memory block which includes the address, size, and Memory Domain used for allocation. This structure is passed to interface and the memory is allocated by memory allocation functions uct_mem_alloc.

Data Fields
void *	address	Address of allocated memory
size_t	length	Real size of allocated memory
uct_alloc_method_t	method	Method used to allocate the memory
ucs_memory_type_t	mem_type	type of allocated memory
uct_md_h	md	if method==MD: MD used to allocate the memory
uct_mem_h	memh	if method==MD: MD memory handle

uct_rkey_bundle

struct uct_rkey_bundle

This structure describes the credentials (typically key) and information required to access the remote memory by the communication interfaces.

Data Fields
uct_rkey_t	rkey	Remote key descriptor, passed to RMA functions
void *	handle	Handle, used internally for releasing the key
void *	type	Remote key type

Typedef Documentation

uct_allocated_memory_t

typedef struct uct_allocated_memory uct_allocated_memory_t

This structure describes the memory block which includes the address, size, and Memory Domain used for allocation. This structure is passed to interface and the memory is allocated by memory allocation functions uct_mem_alloc.

uct_rkey_bundle_t

typedef struct uct_rkey_bundle uct_rkey_bundle_t

This structure describes the credentials (typically key) and information required to access the remote memory by the communication interfaces.

Enumeration Type Documentation

uct_sockaddr_accessibility_t

enum uct_sockaddr_accessibility_t

Enumerator
UCT_SOCKADDR_ACC_LOCAL	Check if local address exists. Address should belong to a local network interface
UCT_SOCKADDR_ACC_REMOTE	Check if remote address can be reached. Address is routable from one of the local network interfaces

anonymous enum

anonymous enum

Enumerator
UCT_MD_FLAG_ALLOC	MD supports memory allocation
UCT_MD_FLAG_REG	MD supports memory registration
UCT_MD_FLAG_NEED_MEMH	The transport needs a valid local memory handle for zero-copy operations
UCT_MD_FLAG_NEED_RKEY	The transport needs a valid remote memory key for remote memory operations
UCT_MD_FLAG_ADVISE	MD supports memory advice
UCT_MD_FLAG_FIXED	MD supports memory allocation with fixed address
UCT_MD_FLAG_RKEY_PTR	MD supports direct access to remote memory via a pointer that is returned by uct_rkey_ptr
UCT_MD_FLAG_SOCKADDR	MD support for client-server connection establishment via sockaddr

uct_md_mem_flags

enum uct_md_mem_flags

Enumerator
UCT_MD_MEM_FLAG_NONBLOCK	Hint to perform non-blocking allocation/registration: page mapping may be deferred until it is accessed by the CPU or a transport.
UCT_MD_MEM_FLAG_FIXED	Place the mapping at exactly defined address
UCT_MD_MEM_FLAG_LOCK	Registered memory should be locked. May incur extra cost for registration, but memory access is usually faster.
UCT_MD_MEM_FLAG_HIDE_ERRORS	Hide errors on memory registration. In some cases registration failure is not an error (e. g. for merged memory regions).
UCT_MD_MEM_ACCESS_REMOTE_PUT	enable remote put access
UCT_MD_MEM_ACCESS_REMOTE_GET	enable remote get access
UCT_MD_MEM_ACCESS_REMOTE_ATOMIC	enable remote atomic access
UCT_MD_MEM_ACCESS_ALL	enable local and remote access for all operations
UCT_MD_MEM_ACCESS_RMA	enable local and remote access for put and get operations

uct_mem_advice_t

enum uct_mem_advice_t

Enumerator
UCT_MADV_NORMAL	No special treatment
UCT_MADV_WILLNEED	can be used on the memory mapped with UCT_MD_MEM_FLAG_NONBLOCK to speed up memory mapping and to avoid page faults when the memory is accessed for the first time.

Function Documentation

uct_md_query()

ucs_status_t uct_md_query	(	uct_md_h	md,
		uct_md_attr_t *	md_attr
	)

Parameters

[in]	md	Memory domain to query.
[out]	md_attr	Filled with memory domain attributes.

Examples

uct_hello_world.c.

uct_md_mem_alloc()

ucs_status_t uct_md_mem_alloc	(	uct_md_h	md,
		size_t *	length_p,
		void **	address_p,
		unsigned	flags,
		const char *	name,
		uct_mem_h *	memh_p
	)

Allocate memory on the memory domain. In order to use this function, MD must support UCT_MD_FLAG_ALLOC flag.

Parameters

[in]	md	Memory domain to allocate memory on.
[in,out]	length_p	Points to the size of memory to allocate. Upon successful return, filled with the actual size that was allocated, which may be larger than the one requested. Must be >0.
[in,out]	address_p	The address
[in]	flags	Memory allocation flags, see uct_md_mem_flags.
[in]	name	Name of the allocated region, used to track memory usage for debugging and profiling.
[out]	memh_p	Filled with handle for allocated region.

uct_md_mem_free()

ucs_status_t uct_md_mem_free	(	uct_md_h	md,
		uct_mem_h	memh
	)

Parameters

[in]	md	Memory domain memory was allocated on.
[in]	memh	Memory handle, as returned from uct_md_mem_alloc.

uct_md_mem_advise()

ucs_status_t uct_md_mem_advise	(	uct_md_h	md,
		uct_mem_h	memh,
		void *	addr,
		size_t	length,
		uct_mem_advice_t	advice
	)

This routine advises the UCT about how to handle memory range beginning at address and size of length bytes. This call does not influence the semantics of the application, but may influence its performance. The advice may be ignored.

Parameters

[in]	md	Memory domain memory was allocated or registered on.
[in]	memh	Memory handle, as returned from uct_md_mem_alloc
[in]	addr	Memory base address. Memory range must belong to the memh
[in]	length	Length of memory to advise. Must be >0.
[in]	advice	Memory use advice as defined in the uct_mem_advice_t list

uct_md_mem_reg()

ucs_status_t uct_md_mem_reg	(	uct_md_h	md,
		void *	address,
		size_t	length,
		unsigned	flags,
		uct_mem_h *	memh_p
	)

Register memory on the memory domain. In order to use this function, MD must support UCT_MD_FLAG_REG flag.

Parameters

[in]	md	Memory domain to register memory on.
[out]	address	Memory to register.
[in]	length	Size of memory to register. Must be >0.
[in]	flags	Memory allocation flags, see uct_md_mem_flags.
[out]	memh_p	Filled with handle for allocated region.

Examples

uct_hello_world.c.

uct_md_mem_dereg()

ucs_status_t uct_md_mem_dereg	(	uct_md_h	md,
		uct_mem_h	memh
	)

Parameters

[in]	md	Memory domain which was used to register the memory.
[in]	memh	Local access key to memory region.

Examples

uct_hello_world.c.

uct_md_detect_memory_type()

ucs_status_t uct_md_detect_memory_type	(	uct_md_h	md,
		const void *	addr,
		size_t	length,
		ucs_memory_type_t *	mem_type_p
	)

Parameters

[in]	md	Memory domain to detect memory type
[in]	addr	Memory address to detect.
[in]	length	Size of memory
[out]	mem_type_p	Filled with memory type of the address range if function succeeds

Returns

UCS_OK If memory type is successfully detected UCS_ERR_INVALID_ADDR If failed to detect memory type

uct_mem_alloc()

ucs_status_t uct_mem_alloc	(	void *	addr,
		size_t	min_length,
		unsigned	flags,
		uct_alloc_method_t *	methods,
		unsigned	num_methods,
		uct_md_h *	mds,
		unsigned	num_mds,
		const char *	name,
		uct_allocated_memory_t *	mem
	)

Allocate potentially registered memory. Every one of the provided allocation methods will be used, in turn, to perform the allocation, until one succeeds. Whenever the MD method is encountered, every one of the provided MDs will be used, in turn, to allocate the memory, until one succeeds, or they are exhausted. In this case the next allocation method from the initial list will be attempted.

Parameters

[in]	addr	If addr is NULL, the underlying allocation routine will choose the address at which to create the mapping. If addr is non-NULL but UCT_MD_MEM_FLAG_FIXED is not set, the address will be interpreted as a hint as to where to establish the mapping. If addr is non-NULL and UCT_MD_MEM_FLAG_FIXED is set, then the specified address is interpreted as a requirement. In this case, if the mapping to the exact address cannot be made, the allocation request fails.
[in]	min_length	Minimal size to allocate. The actual size may be larger, for example because of alignment restrictions.
[in]	flags	Memory allocation flags, see uct_md_mem_flags.
[in]	methods	Array of memory allocation methods to attempt.
[in]	num_methods	Length of 'methods' array.
[in]	mds	Array of memory domains to attempt to allocate the memory with, for MD allocation method.
[in]	num_mds	Length of 'mds' array. May be empty, in such case 'mds' may be NULL, and MD allocation method will be skipped.
[in]	name	Name of the allocation. Used for memory statistics.
[out]	mem	In case of success, filled with information about the allocated memory. uct_allocated_memory_t.

uct_mem_free()

ucs_status_t uct_mem_free

(

const uct_allocated_memory_t *

mem

)

Release the memory allocated by uct_mem_alloc.

Parameters

[in]

mem

Description of allocated memory, as returned from uct_mem_alloc.

uct_md_config_read()

ucs_status_t uct_md_config_read	(	uct_component_h	component,
		const char *	env_prefix,
		const char *	filename,
		uct_md_config_t **	config_p
	)

Parameters

[in]	component	Read the configuration of this component.
[in]	env_prefix	If non-NULL, search for environment variables starting with this UCT_<prefix>_. Otherwise, search for environment variables starting with just UCT_.
[in]	filename	If non-NULL, read configuration from this file. If the file does not exist, it will be ignored.
[out]	config_p	Filled with a pointer to the configuration.

Returns

Error code.

Examples

uct_hello_world.c.

uct_md_is_sockaddr_accessible()

int uct_md_is_sockaddr_accessible	(	uct_md_h	md,
		const ucs_sock_addr_t *	sockaddr,
		uct_sockaddr_accessibility_t	mode
	)

This function checks if a remote sock address can be accessed from a local memory domain. Accessibility can be checked in local or remote mode.

Parameters

[in]	md	Memory domain to check accessibility from. This memory domain must support the UCT_MD_FLAG_SOCKADDR flag.
[in]	sockaddr	Socket address to check accessibility to.
[in]	mode	Mode for checking accessibility, as defined in uct_sockaddr_accessibility_t. Indicates if accessibility is tested on the server side - for binding to the given sockaddr, or on the client side - for connecting to the given remote peer's sockaddr.

Returns

Nonzero if accessible, 0 if inaccessible.

uct_md_mkey_pack()

ucs_status_t uct_md_mkey_pack	(	uct_md_h	md,
		uct_mem_h	memh,
		void *	rkey_buffer
	)

Parameters

[in]	md	Handle to memory domain.
[in]	memh	Local key, whose remote key should be packed.
[out]	rkey_buffer	Filled with packed remote key.

Returns

Error code.

uct_rkey_unpack()

ucs_status_t uct_rkey_unpack	(	uct_component_h	component,
		const void *	rkey_buffer,
		uct_rkey_bundle_t *	rkey_ob
	)

Parameters

[in]	component	Component on which to unpack the remote key.
[in]	rkey_buffer	Packed remote key buffer.
[out]	rkey_ob	Filled with the unpacked remote key and its type.

Note

The remote key must be unpacked with the same component that was used to pack it. For example, if a remote device address on the remote memory domain which was used to pack the key is reachable by a transport on a local component, then that component is eligible to unpack the key. If the remote key buffer cannot be unpacked with the given component, UCS_ERR_INVALID_PARAM will be returned.

Returns

Error code.

uct_rkey_ptr()

ucs_status_t uct_rkey_ptr	(	uct_component_h	component,
		uct_rkey_bundle_t *	rkey_ob,
		uint64_t	remote_addr,
		void **	addr_p
	)

This routine returns a local pointer to the remote memory described by the rkey bundle. The MD must support UCT_MD_FLAG_RKEY_PTR flag.

Parameters

[in]	component	Component on which to obtain the pointer to the remote key.
[in]	rkey_ob	A remote key bundle as returned by the uct_rkey_unpack function.
[in]	remote_addr	A remote address within the memory area described by the rkey_ob.
[out]	addr_p	A pointer that can be used for direct access to the remote memory.

Note

The component used to obtain a local pointer to the remote memory must be the same component that was used to pack the remote key. See notes section for uct_rkey_unpack.

Returns

Error code if the remote memory cannot be accessed directly or the remote address is not valid.

uct_rkey_release()

ucs_status_t uct_rkey_release	(	uct_component_h	component,
		const uct_rkey_bundle_t *	rkey_ob
	)

Parameters

[in]	component	Component which was used to unpack the remote key.
[in]	rkey_ob	Remote key to release.