FP8 Training: Capabilities, Debugging, and Evaluation NEWS DETAIL

FP8 can reduce compute and memory pressure in large-model training, but it requires disciplined numerical validation. The source describes FP8 as a training precision option for newer GPUs: for compute-intensive operations such as matrix multiplication, NVIDIA Tensor Cores can provide twice the peak performance of BF16 and four times the acceleration of TF32. Because FP8 carries less data, it can also reduce memory-traffic pressure for memory-intensive operations. These potential benefits must be balanced against FP8's smaller dynamic range and precision relative to FP16, BF16, and FP32.

What FP8 training is intended to address

FP8 is suited to teams seeking to shorten the time spent on compute-intensive training operations while reducing data movement. The source also notes an operational consideration: training in FP8 can make it more straightforward to align training with low-precision FP8 inference. That alignment is not automatic, however. Model behavior, dataset characteristics, training configuration, scaling method, and inference handling can all affect the result.

The source does not establish that FP8 will improve a particular model, workload, or business outcome. Teams should compare an FP8 run with a BF16 baseline using the same model, data, training plan, and downstream evaluation process.

Training issues to investigate

The source groups practical FP8 concerns into loss spikes, loss growth or divergence, and downstream-score gaps.

• Loss spikes: A spike is not unique to FP8; it may also occur with BF16. If FP8 spikes resemble BF16 behavior, the issue may be general to the workload. More frequent FP8 spikes or spikes requiring repeated recovery attempts warrant further FP8-specific investigation.
• Loss divergence: Divergence at the start of training may indicate a software issue. The source recommends checking the versions of NVIDIA NeMo, Megatron Core, and Transformer Engine in use. It also recommends isolating recently enabled configuration features, such as CPU offloading or FP8 parameters, by disabling them for comparison.
• Mid-training instability: If loss rises or diverges after training has progressed, the source suggests testing another scaling recipe, including current scaling or Fangrand scaling, or falling selected layers back to BF16. It identifies the first and last layers as potentially more sensitive based on cited recent research.
• Downstream-score gaps: Check whether inference uses the intended scaling factors and weights. When FP8 training is paired with BF16 inference, test FP8 inference as well, because the source notes that BF16 inference may introduce additional error for an FP8-trained model.

FP8 Debug tool capabilities described in the source

The source describes an FP8 Debug tool that was in internal testing as of the source publication date. It is designed to observe FP8 training status and compare FP8 with BF16. Its reported metrics include mean squared error (MSE), cosine similarity, tensor underflow and overflow, and the difference between delayed-scaling and current-scaling factors.

The tool can dump selected tensors and associate them with tensor and layer names. Examples in the source include forward GEMM inputs and weights, plus the backward-propagation Dy tensor. Periodic output across training steps supports comparison of AMin, AMax, current scaling, delayed scaling, quantization error, and underflow or overflow ratios. Saved FP8 tensors can support later analysis.

The source states that the tool can work with any version of NVIDIA NeMo Megatron without changing framework internals. Before relying on that statement for a deployment, verify compatibility, supported versions, access conditions, and operational requirements with dated official documentation or the relevant NVIDIA technical contact.

A practical evaluation path

1. Establish a BF16 baseline with recorded loss behavior and downstream-task results.
2. Run FP8 with comparable model, data, and evaluation conditions.
3. Separate early divergence, mid-run instability, and downstream-score discrepancies rather than treating them as one problem.
4. Inspect affected layers and tensors using MSE, cosine similarity, underflow, overflow, and scaling-factor observations.
5. Test configuration changes one at a time, including scaling choices, selected BF16 fallback layers, and the training-to-inference precision combination.
6. Confirm the final candidate through project-specific training and inference tests; the source's internally constructed examples do not establish general performance thresholds.

FAQ

Does a stable FP8 loss curve prove that the model is ready?

No. The source treats loss curves and downstream-task metrics as complementary checks. A run may have acceptable loss behavior while downstream scores remain below the BF16 baseline, so inference inputs, scaling factors, weights, recipes, and precision choices should also be evaluated.

What does a high underflow ratio indicate?

It can indicate that FP8's smaller dynamic range is causing values to underflow, which may contribute to accuracy issues. The source's example highlights a forward FC2 input tensor as an area to examine and suggests considering BF16 fallback or another scaling strategy. The appropriate threshold and remedy must be validated for the specific model and workload.

Conclusion

FP8 training offers a path to faster compute-intensive processing and lower data movement, with the strongest case built through controlled comparison against BF16. Use tensor-level diagnostics to locate numerical risk, test scaling and fallback choices methodically, and validate both training behavior and downstream inference results before adopting a configuration.

After reviewing FP8 Training: Capabilities, Debugging, and Evaluation, continue with buyer selection questions for related evaluation paths.