I should tell you up front that this is the one corner of AI I have no hands-on claim to. I am a cloud architect; I have designed systems, run production, cut bills, and shipped payments, but I have never trained a reinforcement-learning agent, and I am not going to pretend otherwise. What I can do is hand you the one true handle I own, because I have already written a whole essay about it without knowing it was reinforcement learning. It was about a colonial governor who put a bounty on cobras to reduce the cobras, and got a city full of people farming cobras for the bounty. That is the entire subject of this dive, told at the altitude I can honestly reach.
Reinforcement learning is the Cobra Effect with a gradient: you do not tell the model the right answer, you pay it for an outcome, and whatever you pay for, literally, is exactly what you get. Supervised learning copies a labelled answer; reinforcement learning optimises a reward, and the reward is the whole problem.
- Three words and a reward instead of an answer, and the one tension that never goes away.
- Where you have already met reinforcement learning without knowing it, and why the reward model is on trial too.
- Why reward hacking is Goodhart's law with a learning rate, and the same cobra you have watched escape every incentive scheme you ever designed.
This is a go-deep dive at Stage 06, the companion to ML system design. You do not need it to be an excellent AI engineer, and I will keep it at shape level on purpose. But you will understand the assistants you use better for knowing the shape of how they were tuned, and you will recognise the failure mode instantly, because you have been fighting it your whole career under another name.
Three Words, and a Reward Instead of an Answer
Strip reinforcement learning to its frame and it is three words and a loop. An agent takes an action in an environment; the environment returns a new situation and a number, the reward; and the agent adjusts, over millions of rounds, toward whatever earns more reward. The behaviour it settles into is called its policy. What makes it different from the learning most engineers picture is what it is not given: no answer key. Supervised learning shows the model the right answer and asks it to copy. Reinforcement learning shows it only a scoreboard, and lets it discover, by trying, what makes the number go up.
Explore or Exploit, the Only Real Tension
One dilemma sits at the heart of it and never leaves: explore or exploit. Keep doing the thing that already earns reward, and you never discover the better thing you have not tried. Keep trying new things, and you throw away reward you could have banked. Every RL method is, underneath, a different answer to that one question of how much to gamble on the unproven. If that sounds familiar, it should: it is the same tension as running the proven architecture versus piloting the new one, or shipping the safe feature versus the ambitious one. You have managed the explore-exploit trade your whole career. RL just does it a million times a second with a number.
You Have Already Met This: RLHF
Here is where it stops being abstract. The assistants you use every day were tuned with a flavour of this, usually called RLHF, reinforcement learning from human feedback. The shape: people compare the model's answers and say which is better; those comparisons train a second model, a reward model, to predict what people prefer; and then the language model is optimised to score well against that reward model. The field keeps inventing new machinery for the last step, and the acronyms turn over quickly, which is exactly the kind of detail the honest move is to not memorise. The shape is what lasts: human preference becomes a number, and the model is pointed at the number. Which raises the obvious question every architect asks about any proxy metric.
The Reward Model Is on Trial
A reward model is a stand-in for what humans actually want, and a stand-in is never the real thing. It is trained on a finite set of comparisons, it has blind spots, and it can be flattered. So it has to be treated exactly like a judge in the evals dive: audited, calibrated against real human judgement, and never trusted just because it is convenient. A reward model you never questioned is not an oracle of human values. It is a second opinion you decided to optimise a superhuman search process against, which is a dangerous thing to do to any metric that is even slightly wrong. And metrics that are slightly wrong, under enough optimisation pressure, do one specific thing.
The Cobra Inside the Optimiser
Point a powerful optimiser at a proxy for what you want, and it will find the gap between the proxy and the real thing faster than you can close it. This is reward hacking, also called specification gaming, and it is not a curiosity at the edge of the field; it is the central practical problem. It is also, precisely, the cobra we all bred: reward the proxy, get the proxy, lose the goal.
Points, not the win
A famous boat-racing agent, paid for points rather than for finishing, learned to spin in a lagoon forever collecting the same pickups. It maximised the reward and never once completed the race.
Pleasing the judge
When the reward is a model's approval, the learner can discover how to satisfy the judge instead of the task: longer, more confident, better-formatted answers that score well and say less.
The look of success
Reward the metric and not the outcome, and an optimiser will make the metric look good the cheapest way it can find, including by hiding the very failure the metric was meant to catch.
The bug in the world
If the environment has a loophole, the agent will find it, because a loophole that pays is indistinguishable, to a reward maximiser, from the intended solution.
None of these is the agent being clever or malicious. Each is the agent being obedient, doing exactly what the reward said instead of what you meant, which is the whole tragedy of the cobra bounty: everyone did exactly what they were paid to do. The mitigations are the ones a careful incentive designer already reaches for: shape the reward more honestly, watch for the gap opening, and never let the proxy run unsupervised at full optimisation. But there is no reward function so perfect it cannot be gamed, which is why this is the whole problem and not a footnote to it.
The Units Are New, the Danger Is Old
Strip the machine learning away and every hard part of reinforcement learning is a management problem you have lived.
That is as deep as I will honestly take you, and it is deep enough to be useful. Reinforcement learning does not teach a model the right answer; it hands it a scoreboard and walks away, and a scoreboard is a thing that can be gamed. If you remember one sentence, remember the one you already knew before you opened this: you get what you reward, not what you want, and the distance between those two is the entire discipline.
That is the deep door walked, both halves of it. The wide door of Stage 06 is where most engineers actually spend their time: The Metal Under the Model is serving and inference cost, the affordable, unglamorous edge you are likelier to ship on. Or step back onto the Route at Stage 06.