fabric

package: libfabric-devel

title: fabric(7)

text:
# fabric(7)

## NAME
fabric - Fabric Interface Library

## SYNOPSIS

```
#include <rdma/fabric.h>

```

Libfabric is a high-performance fabric software library designed to provide low-latency interfaces to fabric hardware. For an in-depth discussion of the motivation and design see fi_guide(7).

## OVERVIEW
Libfabric provides `process direct I/O' to application software communicating across fabric software and hardware. Process direct I/O, historically referred to as RDMA, allows an application to directly access network resources without operating system interventions. Data transfers can occur directly to and from application memory.

There are two components to the libfabric software:

Fabric Providers

Conceptually, a fabric provider may be viewed as a local hardware NIC driver, though a provider is not limited by this definition. The first component of libfabric is a general purpose framework that is capable of handling different types of fabric hardware. All fabric hardware devices and their software drivers are required to support this framework. Devices and the drivers that plug into the libfabric framework are referred to as fabric providers, or simply providers. Provider details may be found in fi_provider(7).

Fabric Interfaces

The second component is a set of communication operations. Libfabric defines several sets of communication functions that providers can support. It is not required that providers implement all the interfaces that are defined; however, providers clearly indicate which interfaces they do support.

## FABRIC INTERFACES
The fabric interfaces are designed such that they are cohesive and not simply a union of disjoint interfaces. The interfaces are logically divided into two groups: control interfaces and communication operations. The control interfaces are a common set of operations that provide access to local communication resources, such as address vectors and event queues. The communication operations expose particular models of communication and fabric functionality, such as message queues, remote memory access, and atomic operations. Communication operations are associated with fabric endpoints.

Applications will typically use the control interfaces to discover local capabilities and allocate necessary resources. They will then allocate and configure a communication endpoint to send and receive data, or perform other types of data transfers, with remote endpoints.

## CONTROL INTERFACES
The control interfaces APIs provide applications access to network resources. This involves listing all the interfaces available, obtaining the capabilities of the interfaces and opening a provider.

fi_getinfo - Fabric Information

The fi_getinfo call is the base call used to discover and request fabric services offered by the system. Applications can use this call to indicate the type of communication that they desire. The results from fi_getinfo, fi_info, are used to reserve and configure fabric resources.

fi_getinfo returns a list of fi_info structures. Each structure references a single fabric provider, indicating the interfaces that the provider supports, along with a named set of resources. A fabric provider may include multiple fi_info structures in the returned list.

fi_fabric - Fabric Domain

A fabric domain represents a collection of hardware and software resources that access a single physical or virtual network. All network ports on a system that can communicate with each other through the fabric belong to the same fabric domain. A fabric domain shares network addresses and can span multiple providers. libfabric supports systems connected to multiple fabrics.

fi_domain - Access Domains

An access domain represents a single logical connection into a fabric. It may map to a single physical or virtual NIC or a port. An access domain defines the boundary across which fabric resources may be associated. Each access domain belongs to a single fabric domain.

fi_endpoint - Fabric Endpoint

A fabric endpoint is a communication portal. An endpoint may be either active or passive. Passive endpoints are used to listen for connection requests. Active endpoints can perform data transfers. Endpoints are configured with specific communication capabilities and data transfer interfaces.

fi_eq - Event Queue

Event queues, are used to collect and report the completion of asynchronous operations and events. Event queues report events that are not directly associated with data transfer operations.

fi_cq - Completion Queue

Completion queues are high-performance event queues used to report the completion of data transfer operations.

fi_cntr - Event Counters

Event counters are used to report the number of completed asynchronous operations. Event counters are considered light-weight, in that a completion simply increments a counter, rather than placing an entry into an event queue.

fi_mr - Memory Region

Memory regions describe application local memory buffers. In order for fabric resources to access application memory, the application must first grant permission to the fabric provider by constructing a memory region. Memory regions are required for specific types of data transfer operations, such as RMA transfers (see below).

fi_av - Address Vector

Address vectors are used to map higher level addresses, such as IP addresses, which may be more natural for an application to use, into fabric specific addresses. The use of address vectors allows providers to reduce the amount of memory required to maintain large address look-up tables, and eliminate expensive address resolution and look-up methods during data transfer operations.

## DATA TRANSFER INTERFACES
Fabric endpoints are associated with multiple data transfer interfaces. Each interface set is designed to support a specific style of communication, with an endpoint allowing the different interfaces to be used in conjunction. The following data transfer interfaces are defined by libfabric.

fi_msg - Message Queue

Message queues expose a simple, message-based FIFO queue interface to the application. Message data transfers allow applications to send and receive data with message boundaries being maintained.

fi_tagged - Tagged Message Queues

Tagged message lists expose send/receive data transfer operations built on the concept of tagged messaging. The tagged message queue is conceptually similar to standard message queues, but with the addition of 64-bit tags for each message. Sent messages are matched with receive buffers that are tagged with a similar value.

fi_rma - Remote Memory Access

RMA transfers are one-sided operations that read or write data directly to a remote memory region. Other than defining the appropriate memory region, RMA operations do not require interaction at the target side for the data transfer to complete.

fi_atomic - Atomic

Atomic operations can perform one of several operations on a remote memory region. Atomic operations include well-known functionality, such as atomic-add and compare-and-swap, plus several other pre-defined calls. Unlike other data transfer interfaces, atomic operations are aware of the data formatting at the target memory region.

## LOGGING INTERFACE
Logging can be controlled using the FI_LOG_LEVEL, FI_LOG_LOCATION, FI_LOG_LOCATION_MODE, FI_LOG_PROV, and FI_LOG_SUBSYS environment variables.

FI_LOG_LEVEL

FI_LOG_LEVEL controls the amount of logging data that is output. The following log levels are defined.

- Warn

Warn is the least verbose setting and is intended for reporting errors or warnings.

- Trace

Trace is more verbose and is meant to include non-detailed output helpful to tracing program execution.

- Info

Info is high traffic and meant for detailed output.

- Debug

Debug is high traffic and is likely to impact application performance. Debug output is only available if the library has been compiled with debugging enabled.

FI_LOG_LOCATION

FI_LOG_LOCATION controls the location of the log output. Possible options are stdout, stderr, as well as a path to a file or a directory. The default location is stderr. If set to a file path, then the specified file will be created and used. The file pointed to by the setting must not exist prior to launching libfabric. If set to a directory path, then a new file ofi_<pid>.log in the specified directory will be used. Log files use the file mode specified via FI_LOG_LOCATION_MODE.

FI_LOG_LOCATION_MODE

FI_LOG_LOCATION_MODE controls the mode of the log file if FI_LOG_LOCATION is set to a file or directory. The default is 0600.

FI_LOG_PROV

The FI_LOG_PROV environment variable enables or disables logging from specific providers. Providers can be enabled by listing them in a comma separated fashion. If the list begins with the `^' symbol, then the list will be negated. By default all providers are enabled.

Example: To enable logging from the psm3 and sockets provider: FI_LOG_PROV=“psm3,sockets”

Example: To enable logging from providers other than psm3: FI_LOG_PROV=“^psm3”

FI_LOG_SUBSYS

The FI_LOG_SUBSYS environment variable enables or disables logging at the subsystem level. The syntax for enabling or disabling subsystems is similar to that used for FI_LOG_PROV. The following subsystems are defined.

- core

Provides output related to the core framework and its management of providers.

- fabric

Provides output specific to interactions associated with the fabric object.

- domain

Provides output specific to interactions associated with the domain object.

- ep_ctrl

Provides output specific to endpoint non-data transfer operations, such as CM operations.

- ep_data

Provides output specific to endpoint data transfer operations.

- av

Provides output specific to address vector operations.

- cq

Provides output specific to completion queue operations.

- eq

Provides output specific to event queue operations.

- mr

Provides output specific to memory registration.

## PROVIDER INSTALLATION AND SELECTION
The libfabric build scripts will install all providers that are supported by the installation system. Providers that are missing build prerequisites will be disabled. Installed providers will dynamically check for necessary hardware on library initialization and respond appropriately to application queries.

Users can enable or disable available providers through build configuration options. See `configure –help' for details. In general, a specific provider can be controlled using the configure option `–enable-'. For example, `–enable-udp' (or `–enable-udp=yes') will add the udp provider to the build. To disable the provider, `–enable-udp=no' can be used. To build the provider as a stand-alone dynamically loadable library (i.e. DL provider), `–enable-udp=dl' can be used.

Providers can also be enable or disabled at run time using the FI_PROVIDER environment variable. The FI_PROVIDER variable is set to a comma separated list of providers to include. If the list begins with the `^' symbol, then the list will be negated.

Example: To enable the udp and tcp providers only, set: FI_PROVIDER="udp,tcp"

When libfabric is installed, DL providers are put under the default provider path, which is determined by how libfabric is built and installed. Usually the default provider path is <libfabric-install-dir>/lib/libfabric or <libfabric-install-dir>/lib64/libfabric. By default, libfabric tries to find DL providers in the following order:

1.

Use `dlopen' to load provider libraries named lib<prov_name>-fi.so for all providers enabled at build time. The search path of `ld.so' is used to locate the files. This step is skipped if libfabric is configured with the option `–enable-restricted-dl'.

2.

Try to load every file under the default provider path as a DL provider.

The FI_PROVIDER_PATH variable can be used to change the location to search for DL providers and how to resolve conflicts if multiple providers with the same name are found. Setting FI_PROVIDER_PATH to any value, even if empty, would cause step 1 be skipped, and may change the search directory used in step 2.

In the simplest form, the FI_PROVIDER_PATH variable is set to a colon separated list of directories. These directories replace the default provider path used in step 2. For example:

```
FI_PROVIDER_PATH=/opt/libfabric:/opt/libfabric2

```

By default, if multiple providers (including the built-in providers) with the same name are found, the first one with the highest version is active and all the others are hidden. This can be changed by setting the FI_PROVIDER_PATH variable to start with `@', which force the first one to be active regardless of the version. For example:

```
FI_PROVIDER_PATH=@/opt/libfabric:/opt/libfabric2

```

The FI_PROVIDER_PATH variable can also specify preferred providers by supplying full paths to libraries instead of directories to search under. A preferred provider takes precedence over other providers with the same name. The specification of a preferred provider must be prefixed with `+'. For example:

```
FI_PROVIDER_PATH=+/opt/libfabric2/libtcp-fi.so:/opt/libfabric:+/opt/libfabric2/libudp-fi.so

```

If FI_PROVIDER_PATH is set, but no directory is supplied, the default provider path is used. Some examples:

```
FI_PROVIDER_PATH=
FI_PROVIDER_PATH=@
FI_PROVIDER_PATH=+/opt/libfabric/libtcp-fi.so
FI_PROVIDER_PATH=@+/opt/libfabric/libtcp-fi.so

```

The fi_info utility, which is included as part of the libfabric package, can be used to retrieve information about which providers are available in the system. Additionally, it can retrieve a list of all environment variables that may be used to configure libfabric and each provider. See fi_info(1) for more details.

## ENVIRONMENT VARIABLE CONTROLS
Core features of libfabric and its providers may be configured by an administrator through the use of environment variables. Man pages will usually describe the most commonly accessed variables, such as those mentioned above. However, libfabric defines interfaces for publishing and obtaining environment variables. These are targeted for providers, but allow applications and users to obtain the full list of variables that may be set, along with a brief description of their use.

A full list of variables available may be obtained by running the fi_info application, with the -e or –env command line option.

## NOTES

### System Calls
Because libfabric is designed to provide applications direct access to fabric hardware, there are limits on how libfabric resources may be used in conjunction with system calls. These limitations are notable for developers who may be familiar programming to the sockets interface. Although limits are provider specific, the following restrictions apply to many providers and should be adhered to by applications desiring portability across providers.

fork

Fabric resources are not guaranteed to be available by child processes. This includes objects, such as endpoints and completion queues, as well as application controlled data buffers which have been assigned to the network. For example, data buffers that have been registered with a fabric domain may not be available in a child process because of copy on write restrictions.

### CUDA deadlock
In some cases, calls to cudaMemcpy() within libfabric may result in a deadlock. This typically occurs when a CUDA kernel blocks until a cudaMemcpy on the host completes. Applications which can cause such behavior can restrict Libfabric’s ability to invoke CUDA API operations with the endpoint option FI_OPT_CUDA_API_PERMITTED. See fi_endpoint(3) for more details.

Another mechanism which can be used to avoid deadlock is Nvidia’s GDRCopy. Using GDRCopy requires an external library and kernel module available at https://github.com/NVIDIA/gdrcopy. Libfabric must be configured with GDRCopy support using the --with-gdrcopy option, and be run with FI_HMEM_CUDA_USE_GDRCOPY=1. This may not be supported by all providers.

## ABI CHANGES
libfabric releases maintain compatibility with older releases, so that compiled applications can continue to work as-is, and previously written applications will compile against newer versions of the library without needing source code changes. The changes below describe ABI updates that have occurred and which libfabric release corresponds to the changes.

Note that because most functions called by applications actually call static inline functions, which in turn reference function pointers in order to call directly into providers, libfabric only exports a handful of functions directly. ABI changes are limited to those functions, most notably the fi_getinfo call and its returned attribute structures.

The ABI version is independent from the libfabric release version.

### ABI 1.0
The initial libfabric release (1.0.0) also corresponds to ABI version 1.0. The 1.0 ABI was unchanged for libfabric major.minor versions 1.0, 1.1, 1.2, 1.3, and 1.4.

### ABI 1.1
A number of external data structures were appended starting with libfabric version 1.5. These changes included adding the fields to the following data structures. The 1.1 ABI was exported by libfabric versions 1.5 and 1.6.

fi_fabric_attr

Added api_version

fi_domain_attr

Added cntr_cnt, mr_iov_limit, caps, mode, auth_key, auth_key_size, max_err_data, and mr_cnt fields. The mr_mode field was also changed from an enum to an integer flag field.

fi_ep_attr

Added auth_key_size and auth_key fields.

### ABI 1.2
The 1.2 ABI version was exported by libfabric versions 1.7 and 1.8, and expanded the following structure.

fi_info

The fi_info structure was expanded to reference a new fabric object, fid_nic. When available, the fid_nic references a new set of attributes related to network hardware details.

### ABI 1.3
The 1.3 ABI version was exported by libfabric versions 1.9, 1.10, and 1.11. Added new fields to the following attributes:

fi_domain_attr

Added tclass

fi_tx_attr

Added tclass

### ABI 1.4
The 1.4 ABI version was exported by libfabric 1.12. Added fi_tostr_r, a thread-safe (re-entrant) version of fi_tostr.

### ABI 1.5
ABI version starting with libfabric 1.13. Added new fi_open API call.

### ABI 1.6
ABI version starting with libfabric 1.14. Added fi_log_ready for providers.

### ABI 1.7
ABI version starting with libfabric 1.20. Added new fields to the following attributes:

fi_domain_attr

Added max_ep_auth_key

### ABI 1.8
ABI version starting with libfabric 2.0. Added new fi_fabric2 API call. Added new fields to the following attributes:

fi_domain_attr

Added max_group_id

### ABI 1.9
ABI version starting with libfabric 2.5. Added new fields to the following attributes:

fi_domain_attr

Added max_cntr_value, max_err_cntr_value

## SEE ALSO
fi_info(1), fi_provider(7), fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_av(3), fi_eq(3), fi_cq(3), fi_cntr(3), fi_mr(3)

## AUTHORS
OpenFabrics.


title: fi_arch(7)

text:
# fi_arch(7)

```
     /

    / (9 CONNECTED)

```

/Event

/

```
Connections require the use of both passive and active endpoints.
In order to establish a connection, an application must first create a
passive endpoint and associate it with an event queue. The event queue
will be used to report the connection management events. The application
then calls listen on the passive endpoint. A single passive endpoint can
be used to form multiple connections.
The connecting peer allocates an active endpoint, which is also
associated with an event queue. Connect is called on the active
endpoint, which results in sending a connection request (CONNREQ)
message to the passive endpoint. The CONNREQ event is inserted into
the passive endpoint’s event queue, where the listening application can
process it.
Upon processing the CONNREQ, the listening application will allocate
an active endpoint to use with the connection. The active endpoint is
bound with an event queue. Although the diagram shows the use of a
separate event queue, the active endpoint may use the same event queue
as used by the passive endpoint. Accept is called on the active endpoint
to finish forming the connection. It should be noted that the OFI accept
call is different than the accept call used by sockets. The differences
result from OFI supporting process direct I/O.
libfabric does not define the connection establishment protocol, but
does support a traditional three-way handshake used by many technologies.
After calling accept, a response is sent to the connecting active endpoint.
That response generates a CONNECTED event on the remote event queue. If a
three-way handshake is used, the remote endpoint will generate an
acknowledgment message that will generate a CONNECTED event for the accepting
endpoint. Regardless of the connection protocol, both the active and passive
sides of the connection will receive a CONNECTED event that signals that the
connection has been established.
## Connectionless Communications
Connectionless communication allows data transfers between active endpoints
without going through a connection setup process. The diagram below shows
the basic components needed to setup connection-less communication.
Connectionless communication setup differs from UDP sockets in that it
requires that the remote addresses be stored with libfabric.

```

1 insert_addr() 2 send() | | /Address <–3 lookup–> / Active

/ /

```
libfabric requires the addresses of peer endpoints be inserted into a local
addressing table, or address vector, before data transfers can be initiated
against the remote endpoint. Address vectors abstract fabric specific
addressing requirements and avoid long queuing delays on data transfers
when address resolution is needed. For example, IP addresses may need to be
resolved into Ethernet MAC addresses. Address vectors allow this resolution
to occur during application initialization time. libfabric does not define
how an address vector be implemented, only its conceptual model.
All connection-less endpoints that transfer data must be associated with an
address vector.
# Endpoints
At a low-level, endpoints are usually associated with a transmit context, or
queue, and a receive context, or queue.  Although the terms transmit and
receive queues are easier to understand, libfabric uses the terminology
context, since queue like behavior of acting as a FIFO (first-in, first-out)
is not guaranteed.  Transmit and receive contexts may be implemented using
hardware queues mapped directly into the process’s address space.  An endpoint
may be configured only to transmit or receive data.  Data transfer requests
are converted by the underlying provider into commands that are inserted into
hardware transmit and/or receive contexts.
Endpoints are also associated with completion queues. Completion queues are
used to report the completion of asynchronous data transfer operations.
## Shared Contexts
An advanced usage model allows for sharing resources among multiple endpoints.
The most common form of sharing is having multiple connected endpoints
make use of a single receive context.  This can reduce receive side buffering
requirements, allowing the number of connected endpoints that an application
can manage to scale to larger numbers.
# Data Transfers
Obviously, a primary goal of network communication is to transfer data between
processes running on different systems. In a similar way that the socket API
defines different data transfer semantics for TCP versus UDP sockets, that is,
streaming versus datagram messages, libfabric defines different types of data
transfers. However, unlike sockets, libfabric allows different semantics over
a single endpoint, even when communicating with the same peer.
libfabric uses separate API sets for the different data transfer semantics;
although, there are strong similarities between the API sets.  The differences
are the result of the parameters needed to invoke each type of data transfer.
## Message transfers
Message transfers are most similar to UDP datagram transfers, except that
transfers may be sent and received reliably.  Message transfers may also be
gigabytes in size, depending on the provider implementation.  The sender
requests that data be transferred as a single transport operation to a peer.
Even if the data is referenced using an I/O vector, it is treated as a single
logical unit or message.  The data is placed into a waiting receive buffer
at the peer, with the receive buffer usually chosen using FIFO ordering.
Note that even though receive buffers are selected using FIFO ordering, the
received messages may complete out of order.  This can occur as a result of
data between and within messages taking different paths through the network,
handling lost or retransmitted packets, etc.
Message transfers are usually invoked using API calls that contain the string
"send" or "recv".  As a result they may be referred to simply as sent or
received messages.
Message transfers involve the target process posting memory buffers to the
receive (Rx) context of its endpoint.  When a message arrives from the network,
a receive buffer is removed from the Rx context, and the data is copied from
the network into the receive buffer.  Messages are matched with posted receives
in the order that they are received.  Note that this may differ from the order
that messages are sent, depending on the transmit side's ordering semantics.
Conceptually, on the transmit side, messages are posted to a transmit (Tx)
context.  The network processes messages from the Tx context, packetizing
the data into outbound messages.  Although many implementations process the
Tx context in order (i.e. the Tx context is a true queue), ordering guarantees
specified through the libfabric API determine the actual processing order.  As
a general rule, the more relaxed an application is on its message and data
ordering, the more optimizations the networking software and hardware can
leverage, providing better performance.
## Tagged messages
Tagged messages are similar to message transfers except that the messages
carry one additional piece of information, a message tag.  Tags are application
defined values that are part of the message transfer protocol and are used to
route packets at the receiver.  At a high level, they are roughly similar to
message ids.  The difference is that tag values are set by the application,
may be any value, and duplicate tag values are allowed.
Each sent message carries a single tag value, which is used to select a receive
buffer into which the data is copied.  On the receiving side, message buffers
are also marked with a tag.  Messages that arrive from the network search
through the posted receive messages until a matching tag is found.
Tags are often used to identify virtual communication groups or roles.
In practice, message tags are typically divided into fields.  For example, the
upper 16 bits of the tag may indicate a virtual group, with the lower 16 bits
identifying the message purpose.  The tag message interface in libfabric is
designed around this usage model.  Each sent message carries exactly one tag
value, specified through the API.  At the receiver, buffers are associated
with both a tag value and a mask.  The mask is used as part of the buffer
matching process.  The mask is applied against the received tag value carried
in the sent message prior to checking the tag against the receive buffer.  For
example, the mask may indicate to ignore the lower 16-bits of a tag.  If
the resulting values match, then the tags are said to match.  The received
data is then placed into the matched buffer.
For performance reasons, the mask is specified as 'ignore' bits. Although
this is backwards from how many developers think of a mask (where the bits
that are valid would be set to 1), the definition ends up mapping well with
applications.  The actual operation performed when matching tags is:

```

send_tag | ignore == recv_tag | ignore

/* this is equivalent to: * send_tag & ~ignore == recv_tag & ~ignore */ ```

Tagged messages are equivalent of message transfers if a single tag value is used. But tagged messages require that the receiver perform a matching operation at the target, which can impact performance versus untagged messages.

### RMA
RMA operations are architected such that they can require no processing by the CPU at the RMA target. NICs which offload transport functionality can perform RMA operations without impacting host processing. RMA write operations transmit data from the initiator to the target. The memory location where the data should be written is carried within the transport message itself, with verification checks at the target to prevent invalid access.

RMA read operations fetch data from the target system and transfer it back to the initiator of the request, where it is placed into memory. This too can be done without involving the host processor at the target system when the NIC supports transport offloading.

The advantage of RMA operations is that they decouple the processing of the peers. Data can be placed or fetched whenever the initiator is ready without necessarily impacting the peer process.

Because RMA operations allow a peer to directly access the memory of a process, additional protection mechanisms are used to prevent unintentional or unwanted access. RMA memory that is updated by a write operation or is fetched by a read operation must be registered for access with the correct permissions specified.

### Atomic operations
Atomic transfers are used to read and update data located in remote memory regions in an atomic fashion. Conceptually, they are similar to local atomic operations of a similar nature (e.g. atomic increment, compare and swap, etc.). The benefit of atomic operations is they enable offloading basic arithmetic capabilities onto a NIC. Unlike other data transfer operations, which merely need to transfer bytes of data, atomics require knowledge of the format of the data being accessed.

A single atomic function operates across an array of data, applying an atomic operation to each entry. The atomicity of an operation is limited to a single data type or entry, however, not across the entire array. libfabric defines a wide variety of atomic operations across all common data types. However support for a given operation is dependent on the provider implementation.

### Collective operations
In general, collective operations can be thought of as coordinated atomic operations between a set of peer endpoints, almost like a multicast atomic request. A single collective operation can result in data being collected from multiple peers, combined using a set of atomic primitives, and the results distributed to all peers. A collective operation is a group communication exchange. It involves multiple peers exchanging data with other peers participating in the collective call. Collective operations require close coordination by all participating members, and collective calls can strain the fabric, as well as local and remote data buffers.

Collective operations are an area of heavy research, with dedicated libraries focused almost exclusively on implementing collective operations efficiently. Such libraries are a specific target of libfabric. The main object of the libfabric collection APIs is to expose network acceleration features for implementing collectives to higher-level libraries and applications. It is recommended that applications needing collective communication target higher-level libraries, such as MPI, instead of using libfabric collective APIs for that purpose.

## AUTHORS
OpenFabrics.

branch: sisyphus


title: fi_direct(7)

text:
# fi_direct(7)

## NAME
fi_direct - Direct fabric provider access

## SYNOPSIS

```
-DFABRIC_DIRECT
#define FABRIC_DIRECT

```

Fabric direct provides a mechanism for applications to compile against a specific fabric providers without going through the libfabric framework or function vector tables. This allows for extreme optimization via function inlining at the cost of supporting multiple providers or different versions of the same provider.

## DESCRIPTION
The use of fabric direct is intended only for applications that require the absolute minimum software latency, and are willing to re-compile for specific fabric hardware. Providers that support fabric direct implement their own versions of the static inline calls which are define in the libfabric header files, define selected enum values, and provide defines for compile-time optimizations. Applications can then code against the standard libfabric calls, but link directly against the provider calls by defining FABRIC_DIRECT as part of their build.

In general, the use of fabric direct does not require application source code changes, and, instead, is limited to the build process.

Providers supporting fabric direct must install `direct' versions of all libfabric header files. For convenience, the libfabric sources contain sample header files that may be modified by a provider. The `direct' header file names have `fi_direct' as their prefix: fi_direct.h, fi_direct_endpoint.h, etc.

Direct providers are prohibited from overriding or modifying existing data structures. However, provider specific extensions are still available. In addition to provider direct function calls to provider code, a fabric direct provider may define zero of more of the following capability definitions. Applications can check for these capabilities in order to optimize code paths at compile time, versus relying on run-time checks.

## CAPABILITY DEFINITIONS
In order that application code may be optimized during compile time, direct providers must provide definitions for various capabilities and modes, if those capabilities are supported. The following #define values may be used by an application to test for provider support of supported features.

FI_DIRECT_CONTEXT

The provider sets FI_CONTEXT or FI_CONTEXT2 for fi_info:mode. See fi_getinfo for additional details. When FI_DIRECT_CONTEXT is defined, applications should use struct fi_context in their definitions, even if FI_CONTEXT2 is set.

FI_DIRECT_LOCAL_MR

The provider sets FI_LOCAL_MR for fi_info:mode. See fi_getinfo for additional details.

## SEE ALSO
fi_getinfo(3), fi_endpoint(3), fi_domain(3)

## AUTHORS
OpenFabrics.


title: fi_efa(7)

text:
# fi_efa(7)

## NAME
fi_efa - The Amazon Elastic Fabric Adapter (EFA) Provider

## OVERVIEW
The EFA provider supports the Elastic Fabric Adapter (EFA) device on Amazon EC2. EFA provides reliable and unreliable datagram send/receive with direct hardware access from userspace (OS bypass). For reliable datagram (RDM) EP type, it supports two fabric names: efa and efa-direct. The efa fabric implements a set of wire protocols to support more capabilities and features beyond the EFA device capabilities. The efa-direct fabric, on the contrary, offloads all libfabric data plane calls to the device directly without wire protocols. Compared to the efa fabric, the efa-direct fabric supports fewer capabilities and has more mode requirements for applications. But it provides a fast path to hand off application requests to the device. To use efa-direct, set the name field in fi_fabric_attr to efa-direct. More details and difference between the two fabrics will be presented below.

## SUPPORTED FEATURES
The following features are supported:

Endpoint types

The provider supports endpoint type FI_EP_DGRAM, and FI_EP_RDM on a new Scalable (unordered) Reliable Datagram protocol (SRD). SRD provides support for reliable datagrams and more complete error handling than typically seen with other Reliable Datagram (RD) implementations.

RDM Endpoint capabilities

For the efa fabric, the following data transfer interfaces are supported: FI_MSG, FI_TAGGED, FI_SEND, FI_RECV, FI_RMA, FI_WRITE, FI_READ, FI_ATOMIC, FI_DIRECTED_RECV, FI_MULTI_RECV, and FI_SOURCE. It provides SAS guarantees for data operations, and does not have a maximum message size (for all operations). For the efa-direct fabric, it supports FI_MSG, FI_SEND, FI_RECV, FI_RMA, FI_WRITE, FI_READ, and FI_SOURCE. As mentioned earlier, it doesn’t provide SAS guarantees, and has different maximum message sizes for different operations. For MSG operations, the maximum message size is the MTU size of the efa device (approximately 8KiB). For RMA operations, the maximum message size is the maximum RDMA size of the EFA device. The exact values of these sizes can be queried by the fi_getopt API with option names FI_OPT_MAX_MSG_SIZE and FI_OPT_MAX_RMA_SIZE

DGRAM Endpoint capabilities

The DGRAM endpoint only supports FI_MSG capability with a maximum message size of the MTU of the underlying hardware (approximately 8 KiB).

Address vectors

The provider supports FI_AV_TABLE. FI_AV_MAP was deprecated in Libfabric 2.x. Applications can still use FI_AV_MAP to create an address vector. But the EFA provider implementation will print a warning and switch to FI_AV_TABLE. FI_EVENT is unsupported.

Completion events

The provider supports FI_CQ_FORMAT_CONTEXT, FI_CQ_FORMAT_MSG, and FI_CQ_FORMAT_DATA. FI_CQ_FORMAT_TAGGED is supported on the efa fabric of RDM endpoint.

The efa and efa-direct fabrics for RDM endpoints support FI_WAIT_UNSPEC and FI_WAIT_FD wait objects for blocking CQ operations (fi_cq_sread).

DGRAM endpoints do not support wait objects.

Modes

The provider requires the use of FI_MSG_PREFIX when running over the DGRAM endpoint. And it requires the use of FI_CONTEXT2 mode for DGRAM endpoint and the efa-direct fabric of RDM endpoint. The efa fabric of RDM endpoint doesn’t have these requirements.

Memory registration modes

The efa fabric of RDM endpoint does not require memory registration for send and receive operations, i.e. it does not require FI_MR_LOCAL. Applications may specify FI_MR_LOCAL in the MR mode flags in order to use descriptors provided by the application. The efa-direct fabric of FI_EP_RDM endpint and the FI_EP_DGRAM endpoint only supports FI_MR_LOCAL.

Progress

RDM and DGRAM endpoints support FI_PROGRESS_MANUAL. EFA erroneously claims the support for FI_PROGRESS_AUTO, despite not properly supporting automatic progress. Unfortunately, some Libfabric consumers also ask for FI_PROGRESS_AUTO when they only require FI_PROGRESS_MANUAL, and fixing this bug would break those applications. This will be fixed in a future version of the EFA provider by adding proper support for FI_PROGRESS_AUTO.

Threading

Both RDM and DGRAM endpoints supports FI_THREAD_SAFE.

## LIMITATIONS

### Completion events
•

DGRAM endpoints do not support synchronous CQ reads (fi_cq_sread) or wait objects.

•

FI_WAIT_FD is not supported for RDM endpoints with SHM transfers enabled. Valid wait objects are FI_WAIT_NONE or FI_WAIT_UNSPEC. Blocking read via fi_cq_sread() is supported and will wait on SHM completions. When SHM transfers are disabled, FI_WAIT_FD wait objects are supported.

### RMA operations
•

Completion events for RMA targets (FI_RMA_EVENT) is not supported.

•

For the efa-direct fabric, the target side of RMA operation must insert the initiator side’s address into AV before the RMA operation is kicked off, due to a current device limitation. The same limitation applies to the efa fabric when the FI_OPT_EFA_HOMOGENEOUS_PEERS option is set as true.

### Zero-copy receive mode
•

Zero-copy receive mode can be enabled only if SHM transfer is disabled.

•

Unless the application explicitly disables P2P, e.g. via FI_HMEM_P2P_DISABLED, zero-copy receive can be enabled only if available FI_HMEM devices all have P2P support.

### fi_cancel support
•

fi_cancel is only supported in the non-zero-copy-receive mode of the efa fabric. It’s not supported in efa-direct, DGRAM endpoint, and the zero-copy receive mode of the efa fabric in RDM endpoint.

When using FI_HMEM for AWS Neuron or Habana SynapseAI buffers, the provider requires peer to peer transaction support between the EFA and the FI_HMEM device. Therefore, the FI_HMEM_P2P_DISABLED option is not supported by the EFA provider for AWS Neuron or Habana SynapseAI.

## PROVIDER SPECIFIC ENDPOINT LEVEL OPTION
FI_OPT_EFA_RNR_RETRY

Defines the number of RNR retry. The application can use it to reset RNR retry counter via the call to fi_setopt. Note that this option must be set before the endpoint is enabled. Otherwise, the call will fail. Also note that this option only applies to RDM endpoint.

FI_OPT_EFA_EMULATED_READ, FI_OPT_EFA_EMULATED_WRITE, FI_OPT_EFA_EMULATED_ATOMICS - bool

These options only apply to the fi_getopt() call. They are used to query the EFA provider to determine if the endpoint is emulating Read, Write, and Atomic operations (return value is true), or if these operations are assisted by hardware support (return value is false).

FI_OPT_EFA_USE_DEVICE_RDMA - bool

This option only applies to the fi_setopt() call. Only available if the application selects a libfabric API version >= 1.18. This option allows an application to change libfabric’s behavior with respect to RDMA transfers. Note that there is also an environment variable FI_EFA_USE_DEVICE_RDMA which the user may set as well. If the environment variable and the argument provided with this variable are in conflict, then fi_setopt will return -FI_EINVAL, and the environment variable will be respected. If the hardware does not support RDMA and the argument is true, then fi_setopt will return -FI_EOPNOTSUPP. If the application uses API version < 1.18, the argument is ignored and fi_setopt returns -FI_ENOPROTOOPT. The default behavior for RDMA transfers depends on API version. For API >= 1.18 RDMA is enabled by default on any hardware which supports it. For API<1.18, RDMA is enabled by default only on certain newer hardware revisions.

FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES - bool

This option only applies to the fi_setopt() call. It is used to force the endpoint to use in-order send/recv operation for each 128 bytes aligned block. Enabling the option will guarantee data inside each 128 bytes aligned block being sent and received in order, it will also guarantee data to be delivered to the receive buffer only once. If endpoint is not able to support this feature, it will return -FI_EOPNOTSUPP for the call to fi_setopt().

FI_OPT_EFA_WRITE_IN_ORDER_ALIGNED_128_BYTES - bool

This option only applies to the fi_setopt() call. It is used to set the endpoint to use in-order RDMA write operation for each 128 bytes aligned block. Enabling the option will guarantee data inside each 128 bytes aligned block being written in order, it will also guarantee data to be delivered to the target buffer only once. If endpoint is not able to support this feature, it will return -FI_EOPNOTSUPP for the call to fi_setopt().

FI_OPT_EFA_HOMOGENEOUS_PEERS - bool

This option only applies to the fi_setopt() call for RDM endpoints on efa fabric. RDM endpoints on efa-direct fabric are unaffected by this option. When set to true, it indicates all peers are homogeneous, meaning they run on the same platform, use the same software versions, and share identical capabilities. It accelerates the initial communication setup as interoperability between peers is guaranteed. When set to true, the target side of a RMA operation must insert the initiator side’s address into AV before the RMA operation is kicked off, due to a current device limitation. The default value is false.

FI_OPT_EFA_USE_UNSOLICITED_WRITE_RECV - bool

This option only applies to the fi_setopt() call. It is used to disable unsolicited write recv for this endpoint, which can reduce the likelihood of CQ overflow. The default value is true. For efa-direct, FI_RX_CQ_DATA is required when FI_OPT_EFA_USE_UNSOLICITED_WRITE_RECV is false, or it will return -FI_EOPNOTSUPP for the call to fi_setopt().

## PROVIDER SPECIFIC DOMAIN OPS
The efa provider exports extensions for operations that are not provided by the standard libfabric interface. These extensions are available via the “fi_ext_efa.h” header file.

### Domain Operation Extension
Domain operation extension is obtained by calling fi_open_ops (see fi_domain(3))

```
int fi_open_ops(struct fid *domain, const char *name, uint64_t flags,

    void **ops, void *context);

```

Requesting FI_EFA_DOMAIN_OPS in name returns ops as the pointer to the function table fi_efa_ops_domain defined as follows:

```
struct fi_efa_ops_domain {

    int (*query_mr)(struct fid_mr *mr, struct fi_efa_mr_attr *mr_attr);
};

```

### query_mr
This op queries an existing memory registration as input, and outputs the efa specific mr attribute which is defined as follows

```
struct fi_efa_mr_attr {

    uint16_t ic_id_validity;

    uint16_t recv_ic_id;

    uint16_t rdma_read_ic_id;

    uint16_t rdma_recv_ic_id;
};

```

ic_id_validity

Validity mask of interconnect id fields. Currently the following bits are supported in the mask:

FI_EFA_MR_ATTR_RECV_IC_ID: recv_ic_id has a valid value.

FI_EFA_MR_ATTR_RDMA_READ_IC_ID: rdma_read_ic_id has a valid value.

FI_EFA_MR_ATTR_RDMA_RECV_IC_ID: rdma_recv_ic_id has a valid value.

recv_ic_id

Physical interconnect used by the device to reach the MR for receive operation. It is only valid when ic_id_validity has the FI_EFA_MR_ATTR_RECV_IC_ID bit.

rdma_read_ic_id

Physical interconnect used by the device to reach the MR for RDMA read operation. It is only valid when ic_id_validity has the FI_EFA_MR_ATTR_RDMA_READ_IC_ID bit.

rdma_recv_ic_id

Physical interconnect used by the device to reach the MR for RDMA write receive. It is only valid when ic_id_validity has the FI_EFA_MR_ATTR_RDMA_RECV_IC_ID bit.

### Return value
query_mr() returns 0 on success, or the value of errno on failure (which indicates the failure reason).

To enable GPU Direct Async (GDA), which allows the GPU to interact directly with the NIC, request FI_EFA_GDA_OPS in the name parameter with efa-direct fabirc. This returns ops as a pointer to the function table fi_efa_ops_gda defined as follows:

```
struct fi_efa_ops_gda {

    int (*query_addr)(struct fid_ep *ep_fid, fi_addr_t addr, uint16_t *ahn,

              uint16_t *remote_qpn, uint32_t *remote_qkey);

    int (*query_qp_wqs)(struct fid_ep *ep_fid, struct fi_efa_wq_attr *sq_attr, struct fi_efa_wq_attr *rq_attr);

    int (*query_cq)(struct fid_cq *cq_fid, struct fi_efa_cq_attr *cq_attr);

    int (*cq_open_ext)(struct fid_domain *domain_fid,

               struct fi_cq_attr *attr,

               struct fi_efa_cq_init_attr *efa_cq_init_attr,

               struct fid_cq **cq_fid, void *context);

    uint64_t (*get_mr_lkey)(struct fid_mr *mr);
};

```

### query_addr
This op queries the following address information for a given endpoint and destination address.

ahn

Address handle number.

remote_qpn

Remote queue pair Number.

remote_qkey

qkey for the remote queue pair.

### Return value
query_addr() returns FI_SUCCESS on success, or -FI_EINVAL on failure.

### query_qp_wqs
This op queries EFA specific Queue Pair work queue attributes for a given endpoint. It retrieves the send queue attributes in sq_attr and receive queue attributes in rq_attr, which is defined as follows.

```
struct fi_efa_wq_attr {

    uint8_t *buffer;

    uint32_t entry_size;

    uint32_t num_entries;

    uint32_t *doorbell;

    uint32_t max_batch;
};

```

buffer

Queue buffer.

entry_size

Size of each entry in the queue.

num_entries

Maximal number of entries in the queue.

doorbell

Queue doorbell.

max_batch

Maximum batch size for queue submissions.

### Return value
query_qp_wqs() returns 0 on success, or the value of errno on failure (which indicates the failure reason).

### query_cq
This op queries EFA specific Completion Queue attributes for a given cq.

```
struct fi_efa_cq_attr {

    uint8_t *buffer;

    uint32_t entry_size;

    uint32_t num_entries;
};

```

buffer

Completion queue buffer.

entry_size

Size of each completion queue entry.

num_entries

Maximal number of entries in the completion queue.

### Return value
query_cq() returns 0 on success, or the value of errno on failure (which indicates the failure reason).

### cq_open_ext
This op creates a completion queue with external memory provided via dmabuf. The memory can be passed by supplying the following struct.

```
struct fi_efa_cq_init_attr {

    uint64_t flags;

    struct {

        uint8_t *buffer;

        uint64_t length;

        uint64_t offset;

        uint32_t fd;

    } ext_mem_dmabuf;
};

```

flags

A bitwise OR of the various values described below.

FI_EFA_CQ_INIT_FLAGS_EXT_MEM_DMABUF: create CQ with external memory provided via dmabuf.

ext_mem_dmabuf

Structure containing information about external memory when using FI_EFA_CQ_INIT_FLAGS_EXT_MEM_DMABUF flag.

buffer

Pointer to the memory mapped in the process’s virtual address space. The field is optional, but if not provided, the use of CQ poll interfaces should be avoided.

length

Length of the memory region to use.

offset

Offset within the dmabuf.

fd

File descriptor of the dmabuf.

### Return value
cq_open_ext() returns 0 on success, or the value of errno on failure (which indicates the failure reason).

### get_mr_lkey
Returns the local memory translation key associated with a MR. The memory registration must have completed successfully before invoking this.

lkey

local memory translation key used by TX/RX buffer descriptor.

### Return value
get_mr_lkey() returns lkey on success, or FI_KEY_NOTAVAIL if the registration has not completed.

## Traffic Class (tclass) in EFA
To prioritize the messages from a given endpoint, user can specify fi_info->tx_attr->tclass = FI_TC_LOW_LATENCY in the fi_endpoint() call to set the service level in rdma-core. All other tclass values will be ignored.

## RUNTIME PARAMETERS
FI_EFA_IFACE

A comma-delimited list of EFA device, i.e. NIC, names that should be visible to the application. This paramater can be used to include/exclude NICs to enforce process affinity based on the hardware topology. The default value is “all” which allows all available NICs to be discovered.

FI_EFA_TX_SIZE

Maximum number of transmit operations before the provider returns -FI_EAGAIN. For only the RDM endpoint, this parameter will cause transmit operations to be queued when this value is set higher than the default and the transmit queue is full.

FI_EFA_RX_SIZE

Maximum number of receive operations before the provider returns -FI_EAGAIN.

## RUNTIME PARAMETERS SPECIFIC TO RDM ENDPOINT
These OFI runtime parameters apply only to the RDM endpoint.

FI_EFA_RX_WINDOW_SIZE

Maximum number of MTU-sized messages that can be in flight from any single endpoint as part of long message data transfer.

FI_EFA_TX_QUEUE_SIZE

Depth of transmit queue opened with the NIC. This may not be set to a value greater than what the NIC supports.

FI_EFA_RECVWIN_SIZE

Size of out of order reorder buffer (in messages). Messages received out of this window will result in an error.

FI_EFA_CQ_SIZE

Size of any cq created, in number of entries.

FI_EFA_MR_CACHE_ENABLE

Enables using the mr cache and in-line registration instead of a bounce buffer for iov’s larger than max_memcpy_size. Defaults to true. When disabled, only uses a bounce buffer

FI_EFA_MR_MAX_CACHED_COUNT

Sets the maximum number of memory registrations that can be cached at any time.

FI_EFA_MR_MAX_CACHED_SIZE

Sets the maximum amount of memory that cached memory registrations can hold onto at any time.

FI_EFA_MAX_MEMCPY_SIZE

Threshold size switch between using memory copy into a pre-registered bounce buffer and memory registration on the user buffer.

FI_EFA_MTU_SIZE

Overrides the default MTU size of the device.

FI_EFA_RX_COPY_UNEXP

Enables the use of a separate pool of bounce-buffers to copy unexpected messages out of the pre-posted receive buffers.

FI_EFA_RX_COPY_OOO

Enables the use of a separate pool of bounce-buffers to copy out-of-order RTS packets out of the pre-posted receive buffers.

FI_EFA_MAX_TIMEOUT

Maximum timeout (us) for backoff to a peer after a receiver not ready error.

FI_EFA_TIMEOUT_INTERVAL

Time interval (us) for the base timeout to use for exponential backoff to a peer after a receiver not ready error.

FI_EFA_ENABLE_SHM_TRANSFER

Enable SHM provider to provide the communication across all intra-node processes. SHM transfer will be disabled in the case where ptrace protection is turned on. You can turn it off to enable shm transfer.

FI_EFA_ENABLE_SHM_TRANSFER is parsed during the fi_domain call and is related to the FI_OPT_SHARED_MEMORY_PERMITTED endpoint option. If FI_EFA_ENABLE_SHM_TRANSFER is set to true, the FI_OPT_SHARED_MEMORY_PERMITTED endpoint option overrides FI_EFA_ENABLE_SHM_TRANSFER. If FI_EFA_ENABLE_SHM_TRANSFER is set to false, but the FI_OPT_SHARED_MEMORY_PERMITTED is set to true, the FI_OPT_SHARED_MEMORY_PERMITTED setopt call will fail with -FI_EINVAL.

FI_EFA_SHM_AV_SIZE

Defines the maximum number of entries in SHM provider’s address vector.

FI_EFA_SHM_MAX_MEDIUM_SIZE

Defines the switch point between small/medium message and large message. The message larger than this switch point will be transferred with large message protocol. NOTE: This parameter is now deprecated.

FI_EFA_INTER_MAX_MEDIUM_MESSAGE_SIZE

The maximum size for inter EFA messages to be sent by using medium message protocol. Messages which can fit in one packet will be sent as eager message. Messages whose sizes are smaller than this value will be sent using medium message protocol. Other messages will be sent using CTS based long message protocol.

FI_EFA_FORK_SAFE

Enable fork() support. This may have a small performance impact and should only be set when required. Applications that require to register regions backed by huge pages and also require fork support are not supported.

FI_EFA_RUNT_SIZE

The maximum number of bytes that will be eagerly sent by inflight messages uses runting read message protocol (Default 307200).

FI_EFA_INTER_MIN_READ_MESSAGE_SIZE

The minimum message size in bytes for inter EFA read message protocol. If instance support RDMA read, messages whose size is larger than this value will be sent by read message protocol. (Default 1048576).

FI_EFA_INTER_MIN_READ_WRITE_SIZE

The mimimum message size for emulated inter EFA write to use read write protocol. If firmware support RDMA read, and FI_EFA_USE_DEVICE_RDMA is 1, write requests whose size is larger than this value will use the read write protocol (Default 65536). If the firmware supports RDMA write, device RDMA write will always be used.

FI_EFA_USE_DEVICE_RDMA

Specify whether to require or ignore RDMA features of the EFA device. - When set to 1/true/yes/on, all RDMA features of the EFA device are used. But if EFA device does not support RDMA and FI_EFA_USE_DEVICE_RDMA is set to 1/true/yes/on, user’s application is aborted and a warning message is printed. - When set to 0/false/no/off, libfabric will emulate all fi_rma operations instead of offloading them to the EFA network device. Libfabric will not use device RDMA to implement send/receive operations. - If not set, RDMA operations will occur when available based on RDMA device ID/version.

FI_EFA_USE_HUGE_PAGE

Specify Whether EFA provider can use huge page memory for internal buffer. Using huge page memory has a small performance advantage, but can cause system to run out of huge page memory. By default, EFA provider will use huge page unless FI_EFA_FORK_SAFE is set to 1/on/true.

FI_EFA_USE_ZCPY_RX

Enables the use of application’s receive buffers in place of bounce-buffers when feasible. (Default: 1). Setting this environment variable to 0 can disable this feature. Explicitly setting this variable to 1 does not guarantee this feature due to other requirements. See https://github.com/ofiwg/libfabric/blob/main/prov/efa/docs/efa_rdm_protocol_v4.md for details.

FI_EFA_USE_UNSOLICITED_WRITE_RECV

Use device’s unsolicited write recv functionality when it’s available. (Default: 1). Setting this environment variable to 0 can disable this feature.

FI_EFA_INTERNAL_RX_REFILL_THRESHOLD

The threshold that EFA provider will refill the internal rx pkt pool. (Default: 8). When the number of internal rx pkts to post is lower than this threshold, the refill will be skipped.

FI_EFA_USE_DATA_PATH_DIRECT

Use the direct data path implementation that bypasses rdma-core on data path, including the CQ polling and TX/RX submissions, when it’s available. Setting this variable as 0 will disable this feature (Default: true).

FI_EFA_TRACK_MR

Enable tracking of memory registrations to detect if any outstanding operations still reference an MR when it is closed. When enabled, the provider will print a warning message if an MR is closed while TX or RX operations still reference it. This is useful for debugging memory registration issues. (Default: false).

## SEE ALSO
fabric(7), fi_provider(7), fi_getinfo(3)

## AUTHORS
OpenFabrics.


title: fi_guide(7)

text:
# fi_guide(7)

## NAME
fi_guide - libfabric programmer’s guide

## OVERVIEW
libfabric is a communication library framework designed to meet the performance and scalability requirements of high-performance computing (HPC) applications. libfabric defines communication interfaces that enable a tight semantic map between applications and underlying network services. Specifically, libfabric software interfaces have been co-designed with network hardware providers and application developers, with a focus on the needs of HPC users.

This guide describes the libfabric architecture and interfaces. Due to the length of the guide, it has been broken into multiple pages. These sections are:

Introduction fi_intro(7)

This section provides insight into the motivation for the libfabric design and underlying networking features that are being exposed through the API.

Architecture fi_arch(7)

This describes the exposed architecture of libfabric, including the object-model and their related operations

Setup fi_setup(7)

This provides basic bootstrapping and setup for using the libfabric API.

## AUTHORS
OpenFabrics.


title: fi_hook(7)

text:
# fi_hook(7)

## NAME
fi_hook - The Hook Fabric Provider Utility

## OVERVIEW
The hooking provider is a utility function that can intercept calls to any provider. The hook provider is always available, but has zero impact on calls unless enabled. It is useful for providing performance data on selected calls or debugging information.

## SUPPORTED FEATURES
Hooking support is enabled through the FI_HOOK environment variable. To enable hooking, FI_HOOK must be set to the name of one or more of the available hooking providers. When multiple hooks are specified, the names must be separated by a semi-colon. To obtain a list of hooking providers available on the current system, one can use the fi_info utility with the `–env' command line option. Hooking providers are usually identified by `hook' appearing in the provider name.

Known hooking providers include the following:

ofi_hook_perf

This hooks `fast path' data operation calls. Performance data is captured on call entrance and exit, in order to provide an average of how long each call takes to complete. See the PERFORMANCE HOOKS section for available performance data.

ofi_hook_trace

This hooks most of API calls for fabric communications. The APIs and their runtime parameters are logged to provide detail trace information for debugging. See the TRACE HOOKS section for the APIs enabled for trace.

ofi_hook_profile

This hooks data operation calls, cq operation calls and mr registration calls. The API calls and the amount of data being operated are accumulated, to provide a view of APIs’ usages and a histogram of data size operated in a workload execution. See the PROFILE HOOKS section for the report in the detail.

ofi_hook_monitor

Similar to ofi_hook_profile, this hooks data operation calls, cq operation calls and mr registration calls. The API calls and the amount of data being operated are accumulated and made available for export via an external sampler through a shared communication file. See the MONITOR HOOKS section for more details.

## PERFORMANCE HOOKS
The hook provider allows capturing inline performance data by accessing the CPU Performance Management Unit (PMU). PMU data is only available on Linux systems. Additionally, access to PMU data may be restricted to privileged (super-user) applications.

Performance data is captured for critical data transfer calls: fi_msg, fi_rma, fi_tagged, fi_cq, and fi_cntr. Captured data is displayed as logged data using the FI_LOG_LEVEL trace level. Performance data is logged when the associated fabric is destroyed.

The environment variable FI_PERF_CNTR is used to identify which performance counter is tracked. The following counters are available:

cpu_cycles

Counts the number of CPU cycles each function takes to complete.

cpu_instr

Counts the number of CPU instructions each function takes to complete. This is the default performance counter if none is specified.

## TRACE HOOKS
This hook provider allows tracing each API call and its runtime parameters. It is enabled by setting FI_HOOK to “trace”.

The trace data include the provider’s name, API function, and input/output parameter values. The APIs enabled for tracing include the following:

data operation calls

This include fi_msg, fi_rma, fi_tagged all data operations. The trace data contains the data buffer, data size being operated, data, tags, and flags when applicable.

cq operation calls

This includes fi_cq_read, fi_cq_sread, fi_cq_strerror and all cq operations. The trace data contains the cq entries based on cq format.

cm operation calls

This includes fi_getname, fi_setname, fi_getpeer, fi_connect and all cm operations. The trace data contains the target address.

resource creation calls

This includes fi_av_open, fi_cq_open, fi_endpoing, fi_cntr_open and fi_mr operations. The trace data contains the corresponding attributes used for resource creation.

The trace data is logged after API is invoked using the FI_LOG_LEVEL trace level

## PROFILE HOOKS
This hook provider allows capturing data operation calls and the amount of data being operated. It is enabled by setting FI_HOOK to “profile”.

The provider counts the API invoked and accumulates the data size each API call operates. For data and cq operations, instead of accumulating all data together, it breaks down the data size into size buckets and accumulates the amount of data in the corresponding bucket based on the size of the data operated. For mr registration operations, it breaks down memory registered per HMEM iface. At the end when the associated fabric is destroyed, the provider generates a profile report.

The report contains the usage of data operation APIs, the amount of data received in each CQ format and the amount of memory registered for rma operations if any exist. In addition, the data operation APIs are grouped into 4 groups based on the nature of the operations, message send (fi_sendXXX, fi_tsendXXX), message receive (fi_recvXXX, fi_trecvXXX), rma read (fi_readXXX) and rma write (fi_writeXXX) to present the percentage usage of each API.

The report is in a table format which has APIs invoked in rows, and the columns contain the following fields:

API

The API calls are invoked.

Size

Data size bucket that at least one API call operates data in that size bucket. The pre-defined size buckets (in Byte) are [0-64] [64-512] [512-1K] [1K-4K] [4K-64K] [64K-256K] [256K-1M] [1M-4M] [4M-UP].

Count

Count of the API calls.

Amount

Amount of data the API operated.

% Count

Percentage of the API calls over the total API calls in the same data operation group.

% Amount

Percentage of the amount of data from the API over the total amount of data operated in the same data operation group.

The report is logged using the FI_LOG_LEVEL trace level.

## MONITOR HOOKS
This hook provider builds on the “profile” hook provider and provides continuous readout capabilities useful for monitoring libfabric applications. It is enabled by setting FI_HOOK to “monitor”.

Similar to the “profile” hook provider, this provider counts the number of invoked API calls and accumulates the amount of data handled by each API call. Refer to the documentation on the “profile” hook for more information.

Data export is facilitated using a communication file on the filesystem, which is created for each hooked libfabric provider. The monitor hook expects to be run on a tmpfs. If available and unless otherwise specified, files will be created under the tmpfs /dev/shm.

A sampler can periodically read this file to extract the gathered data. Every N intercepted API calls or “ticks”, the provider will check whether data export has been requested by the sampler. If so, the currently gathered counter data is copied to the file and the sampler informed about the new data. The provider-local counter data is then cleared. Each sample contains the counter delta to the previous sample.

Communication files will be created at path $FI_OFI_HOOK_MONITOR_BASEPATH/<uid>/<hostname> and will have the name: <ppid>_<pid>_<sequential id>_<job id>_<provider name>. ppid and pid are taken from the perspective of the monitored application. In a batched environment running SLURM, job id is set to the SLURM job ID, otherwise it is set to 0.

See fi_mon_sampler(1) for documentation on how to use the monitor provider sampler.

### CONFIGURATION
The “monitor” hook provider exposes several runtime options via environment variables:

FI_OFI_HOOK_MONITOR_BASEPATH

String to basepath for communication files. (default: /dev/shm/ofi)

FI_OFI_HOOK_MONITOR_DIR_MODE

POSIX mode/permission for directories in basepath. (default: 01700)

FI_OFI_HOOK_MONITOR_FILE_MODE

POSIX mode/permission for communication files. (default: 0600)

FI_OFI_HOOK_MONITOR_TICK_MAX

Number of API calls before communication files are checked for data request. (default: 1024)

FI_OFI_HOOK_MONITOR_LINGER

Whether communication files should linger after termination. (default: 0) This is useful to allow the sampler to read the last counter data even if the libfabric application has already terminated. Note: Using this option without a sampler results in files cluttering FI_OFI_HOOK_MONITOR_BASEPATH. Make sure to either run a sampler or clean these files manually.

## LIMITATIONS
Hooking functionality is not available for providers built using the FI_FABRIC_DIRECT feature. That is, directly linking to a provider prevents hooking.

The hooking provider does not work with triggered operations. Application that use FI_TRIGGER operations that attempt to hook calls will likely crash.

## SEE ALSO
fabric(7), fi_provider(7)

## AUTHORS
OpenFabrics.