01 Hardware briefing
NVIDIA GB300 NVL72
One rack. 72 GPUs working together.
The GB300 NVL72 is a liquid-cooled AI system. A high-speed network called NVLink connects all 72 Blackwell Ultra GPUs so they can tackle workloads that need more than one server.
- Blackwell Ultra GPUs
- 72
- Grace CPUs
- 36
- Compute trays
- 18
Compute trays
Each of the 18 trays holds 4 GPUs and 2 Grace CPUs. In this lab, each tray is a separate Kubernetes node.
02 NVSentinel
Fault lab
Simulate a GPU failure and see what happens to the AI job that was using it.
Recovery walkthrough
Ready to start
A GPU fails during an AI job
Server 07 is running a job. A spare is ready to take over if it fails.
- Detect
- Isolate
- Move job
- Repair
- Return
Rack monitor
All servers healthyServer 07 is running an AI job.
Technical details Custom fault, node map, and event log
Node map
16 running · 2 sparesServer 07 · Ready · Running trainer-06
Event log
Newest firstEvents appear here when you simulate a fault.
03 Field notes
How recovery works
How does Kubernetes use this rack?
The lab models 18 compute trays as 18 Kubernetes nodes. Sixteen nodes run one pod each, using four GPUs per pod. Two empty nodes act as spares. Real jobs spanning trays over NVLink can use the NVIDIA DRA driver's ComputeDomains.
What do NVSentinel and the GPU Operator do?
NVSentinel monitors for faults and coordinates the response. It first cordons the affected node to block new scheduling. After workload eviction and a successful repair, it removes the cordon. The GPU Operator supplies the driver, runtime tools, GPU discovery, and monitoring services the node needs.
What does the simulation simplify?
Every fault in the lab follows a reboot and recovery sequence. Real policies can choose other actions, such as resetting a GPU, and draining depends on workload constraints and available capacity. The drawn cartridges and console represent software. The lab handles one fault at a time. In a real cluster, a quarantine circuit breaker can limit how many nodes are cordoned.