Fig 1GB300 on KubernetesClick a tray to fault it18 nodes · 72 GPUs · all healthy
Cordon bar: the node is unschedulablePod cartridge: one pod claiming a node's 4 GPUsFaulted GPUOperator carriage: the node's DaemonSets coming backOne GB300 NVL72 rack is 18 Kubernetes nodes (compute trays of 4 Blackwell Ultra GPUs and 2 Grace CPUs) joined by 9 NVLink switch trays into one 72-GPU NVLink domain. The GPU Operator puts the driver, container toolkit, device plugin, DCGM exporter and feature discovery on every node. NVSentinel watches each node through DCGM and the system log: on a fault it cordons the node, drains its pod to a spare, has the janitor reboot it, and uncordons it once its health checks pass. Its quarantine circuit breaker stops cordoning if half the nodes are cordoned within 5 minutes.