Nemo Mbridge Perf Moe Hardware Configs

MoE Hardware Configuration Reference

Stable docs: @docs/training/moe-optimization.md Card: @skills/nemo-mbridge-perf-moe-hardware-configs/card.yaml

Quick Platform Playbook

These rows are search seeds, not hardware defaults or throughput promises.

First Answer Checklist

For hardware playbook questions, answer from these canonical rows before adding throughput caveats:

For Qwen3 235B on GB200, explicitly say VPP=unspecified; do not invent or extrapolate VPP=12 unless a measured row provides it. Treat TE-scoped CUDA graph scopes (attn, moe_router, moe_preprocess) as profile-driven candidates, CUDA_DEVICE_MAX_CONNECTIONS selection, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, NCCL_GRAPH_REGISTER=0, GB200/GB300 CPU-side tuning, and the warning not to cargo-cult tracker rows.

Rounded Performance Bands

These are intentionally rounded so the document stays durable as the tracker moves. Treat them as planning ranges, not exact promises.

Representative Config Families

DSV3 on H100

Dispatcher: DeepEPTP=2  EP=64  PP=8  VPP=4Routing: force balanceRecompute: light-to-moderate selective recomputePriority: overlap communication and keep PP efficient

DSV3 on B200

Dispatcher: DeepEPTP=1  EP=32  PP=8  VPP=2 or similarPrecision: MXFP8-classRecompute: selective recompute around MLA up-projection and MLP-side modulesPriority: container quality, PP layout, and DeepEP SMS tuning

DSV3 on GB200 or GB300

Dispatcher: HybridEPTP=1  EP=64  PP=4  VPP=4Precision: MXFP8-classCUDA Graph: attn + moe_router + moe_preprocessPriority: HybridEP, CPU optimization, and graph-friendly static shapes

Qwen3 235B on H100

Dispatcher: alltoall in the current canonical recipe; re-screen flex backends on the target stackTP=2  EP=32  PP=8  VPP=4Recompute: none in the current canonical recipePriority: communication overlap and router-path cleanup

Qwen3 235B on GB200

Dispatcher: HybridEPTP=1 or 2  EP=32 to 64  PP=4  VPP=unspecified unless measuredCUDA Graph: attn + moe_router + moe_preprocessRecompute: moe_act, mlp, or norm depending on memory pressurePriority: balance throughput against memory headroom

Qwen3 30B-A3B on 16 H100

Dispatcher: HybridEPTP=1  EP=16  PP=1  CP=1Precision: BF16Sequence: 4096Batch: MBS1 GBS1024Routing: force balanceEP overlap: enabledDelayed wgrad: disabledCUDA Graph: moe_router + moe_preprocessHybridEP: permute fusion, 32 SMs, 64-token combine chunksMeasured: 20.14729s/step, 299.352 model TFLOPS/GPU over iterations 41-50Rank-0 peak allocated memory: 62.166 GiB

The current number is the final multi-knob canonical recipe result. An earlier matched A/B isolated plain EP overlap: 244.039 to 287.305 TFLOPS/GPU, with communication hidden by GEMM/attention increasing from 0.11% to 36.55%. Do not attribute the later 299.352 result entirely to overlap.

Qwen3-Next 80B on GB200

Dispatcher: HybridEPTP=1  EP=32  PP=2  VPP around 4CUDA Graph: attn + moe_router + moe_preprocessPriority: pipeline layout and grouped GEMM quality

Cross-Cutting Patterns

PP layout

• E = embedding
• t = transformer
• m = MTP
• L = loss
• | = stage boundary

The biggest platform difference is usually not just the dispatcher. It is the combination of dispatcher, PP shape, and whether VPP keeps each stage balanced.

Recompute strategy

Environment variables

CUDA_DEVICE_MAX_CONNECTIONS=1CUDA_DEVICE_MAX_CONNECTIONS=32   # common when EP overlap and CUDA graphs are combinedPYTORCH_CUDA_ALLOC_CONF=expandable_segments:TrueNCCL_GRAPH_REGISTER=0

CPU-side tuning

On GB200 and GB300, CPU affinity and general host-overhead cleanup can move the needle almost as much as a dispatcher swap. Treat them as first-class tuning work, not as afterthoughts.

Pitfalls

1. Do not cargo-cult a tracker row: the winning config usually depends on routing mode, container, and PP layout as much as on hardware name.

2. Container quality matters: large regressions can come from the software stack rather than the model recipe.

3. VPP must be intentional: a bad VPP split can erase the gain from a better dispatcher.

4. Compare absolute throughput, not only MFU: MFU can mislead when switching between BF16, FP8, and other precision modes.

5. Force-balance routing is benchmark-only: it can control routing variance, but it changes semantics. Keep routing fixed within an A/B and validate natural routing separately for training acceptance.

6. Do not treat the dispatcher table as a hard platform rule: HybridEP is the validated winner for the canonical 16×H100 Qwen3 30B shape, while the current 256×H100 Qwen3 235B recipe uses alltoall. Benchmark backend compatibility and throughput in the production container.

7. Separate screening, causality, and acceptance: short runs reject weak candidates, matched one-variable A/Bs explain a mechanism, and a 50-step final run validates the complete winner.

Last signature refresh: 2026-08-03.