Caladrius for inference providers
Accountable for the customer experience, on infrastructure you don't own.
You run the models your customers depend on, on GPU clusters rented from neoclouds and hyperscalers, and you answer for the whole path. That path splits at a demarcation line between your serving layer and the rented infrastructure. Caladrius runs on your own nodes, on every cluster you rent. When latency starts to bite or requests fail, it names the cause in minutes, not hours, and shows which side of the line it sits on. Above the line, it drives the fix at the automation level you set and verifies it held. Below the line, it localizes the fault to a specific neocloud, with everything in your platform already ruled out: the evidence to escalate. It often identifies a degradation while it is still forming, before your customers feel it.
Rented infrastructure
Your customers
Your serving layer
Platform or neocloud?
Every inference incident lands on you first, and it opens with a question whose answer can sit where you cannot look: the serving platform, or the GPU infrastructure underneath it? By hand, the answer costs hours: an engineer correlating serving signals against GPU, fabric, storage, and host telemetry—TTFT against XID codes, queue depth against NCCL timings—on every neocloud, while requests time out and the customer waits.
Caladrius sees what your clusters see, on every neocloud at once, and correlates the full signal set to name which side of the line owns the problem. In your platform, that is a named root cause with the fix ready to run at the automation level you set—recommend only, on your approval, or automatic for the playbooks you have promoted. A serving regression after an engine bump, an autoscaler lagging a burst, a router landing off-prefix, a replica in a restart loop: named, fixed, verified. Below the line, it is the evidence that localizes the fault to a specific neocloud, with everything in your platform already ruled out. The neocloud starts from your evidence instead of from scratch: a faster fix on their side, and no finger-pointing on the way.
You straddle the line. Caladrius works in both directions.
Facing your neoclouds
Out of your reach, not out of view.
Facing your customers
Answer with authority.
If a neocloud you rent from runs Caladrius too
One pane across every neocloud
Each neocloud can show you its own estate at best. Caladrius gives you the view none of them can offer: every customer deployment, mapped to the GPUs, fabric, storage, and neocloud running it, in one pane. Visibility is node-side and tenant-isolated, with every deployment scoped to its customer and access on grant: no rack access, no neocloud cooperation, and agents deployed in your own clusters in an afternoon.
What goes wrong—and who owns the fix
The platform layer
You fix itThe inference-serving layer you operate. Caladrius names the cause, drives the fix, and verifies it held.
- TTFT p99 breachqueue time climbs while every node metric stays green
- Streaming stall mid-responseTPOT degrades under KV pressure
- Cold start on scale-uplong seconds to first token from zero
- KV prefix miss or cache thrashthe router lands off-prefix
- Engine version regressionthroughput drops after a vLLM or TensorRT bump
- Queue depth explosionadmission sheds load and customers see 429s
- Noisy tenant starves the poolone customer's burst, everyone's latency
- Replica restart loopOOM at long context, then a cold start, repeatedly
Demarcation line
The neocloud
You escalate itNode-side symptoms localize the fault to a specific neocloud; your platform is already ruled out.
- NVLink or NVSwitch degradedintra-node collectives slow
- Silent degradation, no XIDmeasurably slower, nothing logged
- Thermal or power throttleclocks halve; the cause is inlet temperature or power on the neocloud's side
- NCCL collective hangthe job wedges with no useful stack
- PFC pause storm or congestiona saturated uplink you cannot see, showing up as link errors you can
- Storage-fabric MTU or PFCcheckpoint and model loads slow to a crawl, silently
- Unannounced maintenance or reclaimnodes drained on the neocloud's schedule
- Spot preemptionseconds of warning, no reason given
What the agents see, and what they cannot
Serving on rented neoclouds | |
|---|---|
Where the agents run | Inside your clusters, on every neocloud in use: Kubernetes, and the same node-side footprint on each |
What they collect | Inference-serving signals (TTFT, TPOT, tokens per second, KV-cache, queue depth, cold-start latency) alongside GPU telemetry (XID, ECC, thermal, clocks), the HCA's own InfiniBand or RoCE counters, RDMA and NCCL timings, PCIe, host metrics, and mounted file systems |
What they cannot see | Each neocloud's switches and spine, its NVSwitch fabric, its storage backend, the far side of any link |
What proof means here | Node-side evidence plus everything in your platform ruled out, and which neocloud the fault localizes to. Never a measurement of a neocloud's gear. |
Where the fix runs | In your serving layer, at the automation level you set. Below the node, an escalation to the right neocloud with the evidence attached. |
Where the agents run
Inside your clusters, on every neocloud in use: Kubernetes, and the same node-side footprint on each
What they collect
Inference-serving signals (TTFT, TPOT, tokens per second, KV-cache, queue depth, cold-start latency) alongside GPU telemetry (XID, ECC, thermal, clocks), the HCA's own InfiniBand or RoCE counters, RDMA and NCCL timings, PCIe, host metrics, and mounted file systems
What they cannot see
Each neocloud's switches and spine, its NVSwitch fabric, its storage backend, the far side of any link
What proof means here
Node-side evidence plus everything in your platform ruled out, and which neocloud the fault localizes to. Never a measurement of a neocloud's gear.
Where the fix runs
In your serving layer, at the automation level you set. Below the node, an escalation to the right neocloud with the evidence attached.
Act on what you own, at the automation level you set
Every remediation in your own layer runs the way you decide: recommend only, execute on your approval, or execute automatically for the playbooks you have promoted, within the permissions you grant. Serving config, autoscaler settings, routing, a replica restart: start with recommendations, watch what Caladrius would have done, and promote one playbook at a time.
Whichever level runs it, the fix is not done until it is verified. Caladrius checks that latency came back and errors cleared against the deployment's own indicators, and rolls back if they did not, so a wrong fix costs seconds of one deployment rather than a customer.
Pass the visibility down
Every deployment is scoped to its customer, and its reliability picture can be passed to the customer as part of what they buy from you: what happened, where it sat on the line, what was done, and whether it held. The same Workload Console that answers your platform team's questions gives each customer a per-deployment view of theirs. Reliability becomes something you show, not something you promise.
often caught before customers feel them
GPUs serving, not stalled behind faults
Fewer SLA breaches customers see
Degradations often caught while still forming, before they reach your customers' users.
Less time lost to 'platform or neocloud?'
A named cause in your platform or a fault localized to a specific neocloud, in minutes, not hours, with the evidence to act on it, so incidents resolve or escalate fast instead of consuming engineering days.
Hold neoclouds accountable
Escalate the right neocloud with node-side evidence and enforce the SLA you pay for; faults you would otherwise absorb become their problem to fix.
A trust advantage
Pass per-deployment reliability visibility to your customers as part of what makes your platform worth choosing.
Lower cost per million tokens
GPUs serving requests instead of stalling behind unresolved faults, across every neocloud.
Fewer breaches, cheaper inference, and reliability you can both defend and sell.
Often, the customer never notices
Caladrius often identifies the cold-start storm building, the KV-cache filling, or the link degrading while the problem is still forming, with the action to take. The deployment keeps serving and the question never gets asked.