Fusing Matmul Epilogs with nvmath-python NEWS DETAIL

nvmath-python (Beta) provides Python access to NVIDIA CUDA-X mathematical libraries, including low-level bindings and higher-level Python abstractions. For neural-network linear layers, its Matmul API can fuse selected operations with matrix multiplication through MatmulEpilog, reducing the need to express bias, ReLU, or related backward-pass work as separate Python-level array operations.

The problem addressed by fused epilogs

A typical linear-layer forward pass multiplies an input batch by a weight matrix, adds a bias to each output column, and applies ReLU. Written as separate operations, this sequence requires the matrix multiplication result to be handled again for the bias addition and activation. The source demonstrates an alternative in which MatmulEpilog.RELU_BIAS combines bias addition and ReLU with the matrix multiplication operation.

For a workload expressed with CuPy arrays, the source uses nvmath.linalg.advanced.Matmul, a stateful API that separates setup and planning from execution. This pattern is particularly relevant when an application executes multiple similar matrix multiplications, because planning can be completed before repeated calls to execute().

Forward-pass capabilities in the source

The demonstrated forward path starts with weights, an input batch, and a bias vector. Instead of manually adding the bias and applying ReLU after mm.execute(), the plan can be configured with MatmulEpilog.RELU_BIAS and an epilog_inputs dictionary containing bias. Execution then returns the output matrix.

When the subsequent backward pass needs to know which pre-ReLU values were negative or non-negative, the source uses MatmulEpilog.RELU_AUX_BIAS. This epilog returns both the computed output and auxiliary outputs. The auxiliary dictionary includes relu_aux, a bit-encoded ReLU mask intended for use by compatible backward-pass epilogs rather than manual inspection.

• RELU_BIAS: combines matrix multiplication, bias addition, and ReLU for the illustrated forward computation.
• RELU_AUX_BIAS: performs the same illustrated forward work while also returning relu_aux.
• Stateful Matmul: allows planning to be separated from repeated execution.

Backward-pass path shown in the example

The source also covers a backward calculation involving a transposed weight matrix and an output gradient. A straightforward implementation would execute matrix multiplication, manually apply the ReLU mask, and sum across the batch dimension to obtain the bias gradient.

Its fused alternative uses MatmulEpilog.DRELU_BGRAD. The epilog receives the previously produced relu_aux as input, applies the ReLU-related mask to the multiplication result, and returns an auxiliary output named drelu_bgrad. In the example, that auxiliary output supplies the column-wise sum used as the bias gradient. This creates a paired forward/backward design: retain the auxiliary ReLU information from RELU_AUX_BIAS, then provide it to DRELU_BGRAD during backpropagation.

Suitable evaluation scenarios

This approach is suited to Python applications that already work with CuPy or PyTorch-compatible workflows and repeatedly run matrix multiplications with the specific fused patterns shown. The source illustrates float16 matrix multiplications of shapes (65536, 16384) and (16384, 8192), followed by forward or backward operations, measured on an NVIDIA H200 GPU. Those measurements are an example configuration, not a general performance guarantee for other GPUs, matrix shapes, data types, software versions, or training graphs.

1. Identify repeated linear-layer operations that match the required bias-and-ReLU or ReLU-gradient pattern.
2. Use the stateful Matmul API and plan the epilog before the repeated execution path.
3. For backpropagation, preserve relu_aux from the forward result and pass it to DRELU_BGRAD.
4. Validate output values, auxiliary-output handling, memory behavior, and end-to-end performance in the target environment.

FAQ

Does RELU_BIAS provide the ReLU mask needed for the illustrated backward pass?

No. The source identifies RELU_AUX_BIAS as the forward epilog that returns the relu_aux auxiliary output. That auxiliary information is used by DRELU_BGRAD in the shown backward path.

Can the source’s performance result be applied to every workload?

No. The reported comparison is tied to the stated float16 matrix sizes and NVIDIA H200 measurement environment. A project should verify supported epilogs, API behavior, compatibility, and performance in dated official nvmath-python documentation and its own representative tests.

Conclusion

nvmath-python (Beta) exposes a focused way to combine matrix multiplication with selected neural-network operations through MatmulEpilog. The source’s forward and backward examples show how RELU_AUX_BIAS and DRELU_BGRAD can share ReLU auxiliary data, but deployment decisions should be based on the complete software documentation and measured behavior of the intended workload.

After reviewing Fusing Matmul Epilogs with nvmath-python, continue with buyer selection questions for related evaluation paths.