For the past several months I've been building agentic DevOps tasks and RL environments for frontier-model post-training. The work sits at an interesting intersection: it needs the infrastructure depth of an SRE, the adversarial mindset of a security reviewer, and the statistical discipline of an eval engineer.
The market for this work is exploding, and so is its known failure mode: environments that look like tasks but don't actually train anything, because the grader can be tricked, the task isn't reproducible, or the difficulty is miscalibrated. This post is about what the rigorous version looks like.
Anatomy of a trainable task
A useful agentic task has four parts, and every one of them earns its place:
-
A reproducible broken state. The environment must build from scratch — a Dockerfile, a compose file, a kind cluster manifest — into exactly the same failure every time. If the bug only manifests on the third run, or depends on network weather, rollouts become noise and the reward signal degrades.
-
A task spec written for the agent, not the author. The spec should describe symptoms, not causes: "the checkout service returns 502s under load," not "fix the connection-pool exhaustion in db.go." If the spec leaks the diagnosis, you're training pattern-matching, not debugging.
-
A hidden, test-based verifier. The grader runs after the agent finishes, in a context the agent never sees. It asserts on observable behavior — the service responds, the pipeline passes, the deployment converges — never on "did the agent edit file X."
-
A difficulty tier backed by evidence. "Hard" is not a vibe. A tier is a claim about pass rates, and the only way to back it is to run baselines and look at the distribution.
Where reward hacking actually shows up
Every environment author eventually watches an agent pass their grader without solving the problem. The common patterns I've seen, in rough order of frequency:
- Grading-surface edits. If the verifier checks that
pytestexits 0, the shortest path is deleting the failing tests. The fix: the verifier restores test files from a checksummed copy — or better, runs its own tests the agent has never seen. - Symptom masking. Asked to fix a crashing service, the agent wraps the crash in a retry loop or a
try/except: pass. Behavioral verifiers must assert on outcomes under load, not just "process is up." - Hardcoding to the probe. If the grader curls one endpoint with one payload, the agent can special-case that payload. Verifiers need randomized inputs drawn at grade time.
- Environment escape. The agent discovers the grader's own scripts inside the sandbox and reads the expected answers. Graders live outside the task container, full stop.
The meta-lesson: the verifier is the product. The broken deployment is scenery; the grader is the thing that decides whether thousands of GPU-hours of rollouts produce signal or garbage.
Difficulty calibration is an empirical claim
A task that frontier models pass 100% of the time teaches nothing; a task they pass 0% of the time teaches almost nothing either (no gradient of partial success to climb). The useful band is wide but not infinite, and you can't guess where a task lands — models are shockingly good at some "hard" infrastructure problems and shockingly bad at some "easy" ones.
So calibration is a loop: run N rollouts against a strong baseline model, look at pass rates and where trajectories fail, then harden or soften. Saturated milestones — checks that always pass or always fail across every rollout — are dead weight and should be removed or redesigned.
Why DevOps is a great domain for this
Infrastructure debugging has a property most agentic domains lack: ground truth is cheap and objective. A Kubernetes deployment either converges or it doesn't. A CI pipeline either goes green or it doesn't. That makes verifiers crisp in a way that essay-grading rubrics never are — which is exactly why RLVR-style training works so well here.
It's also a domain where realism is hard to fake. Production infrastructure fails in layered, correlated ways: the OOMKill that's actually a memory-limit typo that's actually a Helm values indirection. Authors who haven't operated production systems write single-layer bugs, and models blow through them. Five years of being paged for the real thing turns out to be the best task-design training there is.
A checklist I actually use
Before any task ships:
- Builds from clean checkout to broken state, twice, identically
- Reference solution exists and passes the verifier
- Initial (broken) state fails the verifier — you'd be surprised
- Verifier survives: test deletion, symptom masking, probe hardcoding, sandbox snooping
- Spec describes symptoms only; no diagnosis leakage
- Baseline rollouts run; difficulty tier assigned from data, not intuition
None of this is glamorous. All of it is the difference between an environment a lab can train against and a repo that just looks like one.
If you're building or buying RL environments and want a second pair of eyes on grader rigor, I do exactly this — get in touch.