Caladrius Workload Console
Get more useful work from every dollar of GPU spend.
You pay for GPU-hours; you get paid for finished training runs and served tokens. When a job stalls or slows on GPUs you rent, the gap between the two is the hours spent finding out why. Caladrius runs on your own nodes, names the cause across device, fabric, storage, and workload, fixes what is on your side at the automation level you set, hands what is below the node to your provider with the evidence attached, and verifies the fix held. It often identifies a problem while it's still forming.
Is it my model or the infra?
Every stalled or slowed run starts with the same question, and answering it by hand means an engineer correlating GPU telemetry, NCCL timings, host metrics, and the job's own logs for hours, while the rented GPUs bill at full price. Caladrius correlates those signals on the nodes you run and names which layer owns the problem: the model and its configuration, the GPUs, the fabric as your nodes see it, or storage.
For anything on your side, that is a named root cause with the fix attached: a dead rank, an NCCL binding fault, a data-loader stall, an OOM restart loop, a config or version regression after a framework bump. For anything below the node, it is the evidence your provider needs and the list of what you have already ruled out.
Your environment, discovered directly
The Workload Console deploys as agents on your own nodes and discovers your GPU assets directly, within the scope your provider exposes to you. There is no provider API to integrate, so nothing waits on the provider. That puts every environment in scope, including hyperscaler tenancies.
Your nodes are the line. Caladrius works on both sides of it.
On your side
Named, fixed, verified.
Below the node
What you see, what you ruled out, what remains.
If your provider runs Caladrius too
What Caladrius names on rented GPUs
On your side
Named and fixed- Comms-bound stallthe collective waits; one rank is the reason
- Dead ranka process gone, the job hung on it
- NCCL binding faultwrong interface, wrong topology, slow from the first step
- Fail-slow stragglerone GPU at half speed drags every step
- Data-loader stallGPUs idle behind input, not compute
- OOM restart looplong context or batch size, then a cold start, repeatedly
- KV-cache thrashprefix misses under serving load
- Config or version regressionthroughput drops after a framework or driver bump
Below the node
What node-side symptoms point to- Fabric degradationHCA link errors and NCCL timeouts on one path
- Throttling GPUclocks halve; the cause is thermal or power on the provider's side
- ECC-driven draincorrectable errors climbing toward a retirement
- Flapping linkintermittent resets on one port
- Degraded storage nodecheckpoint writes stretch; the mount is fine, the backend is not
- Unannounced maintenance or reclaima node drained on the provider's schedule
What the agents see, and what they cannot
Renting GPUs | |
|---|---|
Where the agents run | Inside your own rented nodes: Kubernetes, Slurm, or bare metal |
What they collect | GPU telemetry (XID, ECC, thermal, clocks), the HCA's own InfiniBand or RoCE counters, RDMA and NCCL timings, PCIe, host metrics, mounted file systems, workload and serving metrics |
What they cannot see | The provider's switches and spine, the NVSwitch fabric, the storage backend, the far side of any link |
What proof means here | Node-side evidence plus everything on your side ruled out. Never a measurement of the provider's gear. |
Where the fix runs | On your nodes, at the automation level you set. Below the node, an escalation with the evidence attached. |
Where the agents run
Inside your own rented nodes: Kubernetes, Slurm, or bare metal
What they collect
GPU telemetry (XID, ECC, thermal, clocks), the HCA's own InfiniBand or RoCE counters, RDMA and NCCL timings, PCIe, host metrics, mounted file systems, workload and serving metrics
What they cannot see
The provider's switches and spine, the NVSwitch fabric, the storage backend, the far side of any link
What proof means here
Node-side evidence plus everything on your side ruled out. Never a measurement of the provider's gear.
Where the fix runs
On your nodes, at the automation level you set. Below the node, an escalation with the evidence attached.
At the automation level you set
Every remediation on your side runs the way you decide: recommend only, execute on your approval, or execute automatically for the playbooks you have promoted, within the permissions you grant. Start with recommendations, watch what Caladrius would have done, and promote one playbook at a time.
Whichever level runs it, the fix is not done until it is verified. Caladrius checks that throughput recovered and errors cleared against the job's own indicators, and rolls back if they did not, so a wrong fix costs minutes rather than a run.
Every environment, one answer
Rented capacity, cloud tenancies, GPUs you own, or a mix: Caladrius runs the same analysis everywhere and gives you one answer instead of a different tool's opinion in each. Move a workload and the diagnosis moves with it.
hours that advance the model, out of hours paid for
jobs finish sooner and cost less to run
Higher goodput
Close the gap between the hours you pay for and the hours that actually advance your model.
Less wasted spend
Waste broken down by cause (idle, comms-bound, or failed runs), so you cut the biggest line item first.
Fewer lost runs
Faults in a long job caught and fixed while they are still forming.
Engineering time back
Stop your team debugging its own stack for problems that live below it; escalate those with evidence instead.
The same expensive GPUs deliver more finished training and inference, at a lower cost per result.