Why We Built Paralleliq Like a Kubernetes Operator, Not a Scheduler
Most infrastructure tools work the same way: scan on a timer, compare against a threshold, raise an alert. That model breaks down the moment you actually want a system to act on what it finds. Here's the pattern we borrowed instead — and the one place we deliberately broke from it.
Why "Scan and Alert" Doesn't Scale
Most infrastructure tools work the same way: run a periodic check, compare a metric against a threshold, throw an alert. That's fine for a dashboard someone glances at. It falls apart the moment you want a system to actually act on what it finds — because "check on a timer" has no memory of what it already knows, no way to distinguish something that just happened from something that's been quietly true for hours, and no natural place to plug in an approval step before anything changes.
We ran into this directly while building out how Paralleliq watches a GPU fleet. The answer wasn't a smarter alerting rule. It was borrowing a pattern that already exists, one level down in the stack we run on.
The Pattern: Reconciliation, Not Polling
Kubernetes itself isn't built around alerting — it's built around reconciliation. You declare a desired state ("I want 3 replicas of this pod running"), and a controller continuously watches the actual state, compares it to what was declared, and takes action to close any gap. That loop — watch, compare, act — is how the whole system stays correct without anyone polling a dashboard.
A Kubernetes Operator extends that same loop to domains Kubernetes has no built-in understanding of. A Postgres Operator, for example, encodes everything needed to keep a Postgres cluster healthy — failover, backups, safe migrations — domain knowledge that's specific to Postgres, not something generic pod-and-container mechanics can express. The Operator watches, reconciles, and acts, the same way Kubernetes' own controllers do, just for a much more specialized problem.
Applying the Same Loop to GPU Fleets
That's the shape we built Paralleliq around, with the specialized domain being GPU-fleet configuration and the models running on it, rather than one stateful application. The loop watches the fleet continuously, reasons about whether its current configuration is actually correct given what's running on it, and proposes a fix when it isn't — the same watch-compare-act structure, aimed at a different problem than any existing Operator solves.
One thing worth being precise about: in a typical Operator, the desired state is something a person wrote down — "3 replicas" in a spec. In this case, "the right configuration for this workload" isn't declared anywhere in advance; it has to be derived from understanding the workload itself. That's a harder version of the same problem, not a different one — the loop still watches, compares, and acts, it just has more work to do to know what "correct" even means before it can compare against it.
The One Deliberate Departure: A Human in the Loop
A standard Operator reconciles on its own — it notices drift and closes the gap without asking anyone. We built Paralleliq's loop to do everything the same way, right up until the last step. It watches continuously, reasons continuously, and proposes a fix the moment it's confident one is needed — but the actual act — changing something on the fleet — waits for a named person to approve it.
That's a narrow exception inserted at one specific point in an otherwise standard pattern, not a different architecture. And it's there for a specific reason: a fully autonomous loop has no way to produce the thing a regulated buyer's auditor actually asks for — a record of who reviewed a change and signed off on it before it happened. Adding that record isn't a feature bolted onto the side of the system. It's a gate built into the loop itself, at the one point where it matters most.
Why This Matters
None of this is a novel, unproven approach. It's a well-understood pattern — the same one running underneath most production Kubernetes clusters already — pointed at a problem (GPU-fleet correctness) that doesn't have a standard Operator for it yet, with one deliberate, load-bearing exception where a human has to say yes before anything changes. Familiar mechanics, applied somewhere new, with the one departure that makes it usable by the buyers who need proof of every change, not just a fast reaction to it.
Curious how this compares to whatever's watching your own fleet today? We'd genuinely like to compare notes.