When GPUs Fail Quietly: Separating GPU Hardware Degradation from Monitoring Failure under Degraded Observability

Abstract

A missing GPU measurement in a production supercomputer is ambiguous: either the device has failed, or the pipeline that reports on it has. The two call for different responses – an engineer sent to the hardware, or a ticket raised against the collector – so the ambiguity has to be resolved before either one is right. We ask which half of that decision needs a model. Whether the monitoring path has degraded is close to directly readable: one lightweight statistic that is never fitted, the gap between the data-delivery rates on the out-of-band hardware-sensor path and the in-band host path, ranks monitoring incidents at AUPRC 0.237 where they are only 5.2 % of the units ranked, and no learned component we trained improves on it. Deciding whether already-degraded evidence has a physical cause is harder, and needs both workload-aware residuals on the hardware sensors and an explicit model of the observation process. PARALLAX supplies these, tracking device health and monitoring health as separate latent states over out-of-band IPMI measurements, in-band telemetry, workload context, and structured missingness; the configuration we recommend is the hybrid that takes its physical risk from PARALLAX and its monitoring risk from the lightweight statistic. On 29 months of a 988-node production cluster, restricted to a pre-specified subset whose evidence has already degraded so that nothing can score by detecting absence, it reaches macro-F1 0.455 against 0.349 for the strongest classical detector and AUPRC 0.603 against 0.184: the four-way verdict is right more often across all four states, and a worklist ranked by physical risk holds far more genuine faults near its top, which is what an operator with a few inspections a day actually spends. These are triage figures as absolute precision falls towards the class base rate beyond moderate recall, so AUPRC supports ranking near the top of a worklist rather than an unattended alarm, and both numbers are measured against constructed weak labels on one system. In practice that detector files 81 % of real hardware failures as monitoring problems, four in five sent to the wrong queue, and the hybrid cuts this to 20 % while escalating no healthy node. The coupled state, in which both causes act at once, is too rare in this corpus to be resolved and is left open. The contribution is a boundary rather than a system: use simple, interpretable indicators for monitoring health, and reserve explicit latent modelling for the regime in which observability itself has already degraded.