Positive feedback eats the world

September 2026

The first image most people receive from cybernetics is a thermostat. The temperature falls, the heater turns on. The temperature rises, the heater turns off. Deviation creates an action that counteracts the deviation, and the system returns toward a target.

The field chose that image on purpose. In Behavior, Purpose and Teleology (1943), one of the field's founding papers, Rosenblueth, Wiener and Bigelow split feedback in two. The amplifier kind is positive: it "adds to the input signals, it does not correct them." The other kind is negative: "the signals from the goal are used to restrict outputs which would otherwise go beyond the goal." Then they picked one: "It is this second meaning of the term feed-back that is used here. All purposeful behavior may be considered to require negative feed-back."

Twenty years later Magoroh Maruyama wrote down the half that got dropped. The Second Cybernetics (American Scientist, 1963) opens: "Since its inception, cybernetics was more or less identified as a science of self-regulating and equilibrating systems." Meanwhile the deviation-amplifying systems were everywhere: "accumulation of capital in industry, evolution of living organisms, the rise of cultures of various types... the processes that are loosely termed as 'vicious circles' and 'compound interests'; in short, all processes of mutual causal relationships that amplify an insignificant or accidental initial kick, build up deviation and diverge from the initial condition." He proposed a name: "let us consider its studies the first cybernetics, and call the studies of the deviation-amplifying mutual causal relationships 'the second cybernetics.'"

The best-known economic case came from W. Brian Arthur in Positive Feedbacks in the Economy (Scientific American, February 1990). VHS and Beta launched at about the same price and "started with roughly equal market shares." More VHS recorders meant more VHS tapes in rental stores, which made a VHS recorder worth more, which sold more recorders. "Increasing returns on early gains eventually tilted the competition toward VHS: it accumulated enough of an advantage to take essentially the entire VCR market. However, it would have been impossible at the outset of the competition to say in advance which system would win." The numbers on the format war: Betamax had 100% of the market in 1975, before VHS existed. By 1980 VHS had 60% of North America. By 1981 Beta was down to 25% of US sales. By 1987 VHS had 90% of the US market. In the UK, Beta had held 25% and was down to 7.5% by 1986.

Betamax and VHS share, every reported data point, 1975 to 1987, US and UK

Here is the thesis. Negative feedback preserves a regime. Positive feedback changes the scale or nature of the regime. Most of what matters right now (capital, compute, AI labs, agent harnesses, reward) is second-cybernetics material, and we keep designing for it with first-cybernetics tools: pick a goal, add a sensor, correct the error. The usual worry about AI is an optimizer that misunderstands what we want. The evidence points somewhere else: optimizers that understand both the proxy and the intent, and follow the proxy because that's the loop that pays them. The most dangerous loop is the one that succeeds.

Why cybernetics picked the thermostat

Negative feedback became the natural image of control because it is reassuring. The goal remains fixed. The system remains legible. The world deviates and the controller restores order.

It was also the math people could do. Ross Ashby's An Introduction to Cybernetics (1956) treats the organism as a regulator. He expects all its "higher" activities to turn out "similarly regulatory, i.e. homeostatic," and defines the living states as those "in which certain essential variables are kept within assigned ('physiological') limits." Economics made the same choice. In Arthur's telling, when John Hicks looked at increasing returns in 1939 he "drew back in alarm. 'The threatened wreckage,' he wrote, 'is that of the greater part of economic theory.' Economists restricted themselves to diminishing returns, which presented no anomalies and could be analyzed completely."

Ashby knew what he was leaving out. On the next page, on where life came from: "though the state of 'being lifeless' is almost a state of equilibrium, yet this equilibrium is unstable, a single deviation from it being sufficient to start a trajectory that deviates more and more from the 'lifeless' state. What we see today in the biological world are these 'autocatalytic' processes." The founders saw positive feedback. They studied the part that held still.

But almost nothing historically interesting behaves only like a thermostat. Cities grow. Species diverge. Technologies become platforms. Wealth buys access to opportunities that produce more wealth. Attention attracts attention. Intelligence builds tools that increase intelligence.

This doesn't make positive feedback good. The microphone screaming through its own speaker is positive feedback. So is a bank run, a nuclear chain reaction, a speculative bubble, a cytokine storm, an arms race, and cancer. "Positive" describes the sign of the loop, not its moral value.

Positive feedback changes the scale of a regime

Maruyama's cleanest example is a city. By chance, a farmer opens a farm somewhere on a homogeneous plain. Others follow. One of them opens a tool shop, the shop becomes a meeting place, a food stand opens next to it, a village grows, and the village becomes a city. "If a historian should try to find a geographical 'cause' which made this spot a city rather than some other spots, he will fail to find it in the initial homogeneity of the plain. Nor can the first farmer be credited with the establishment of the city. The secret of the growth of the city is in the process of deviation-amplifying mutual positive feedback networks rather than in the initial condition or in the initial kick."

Arthur generalized this into a claim about history: "once chance economic forces select a particular path, it may become locked in regardless of the advantages of other paths. If one product or nation in a competitive marketplace gets ahead by 'chance' it tends to stay ahead and even increase its lead."

In an equilibrium story, the initial accident disappears. In a positive-feedback story, the initial accident becomes infrastructure.

Hyman Minsky added the part about time. In The Financial Instability Hypothesis (1992) he borrows Maruyama's vocabulary directly: if hedge financing dominates, "the economy may well be an equilibrium seeking and containing system. In contrast, the greater the weight of speculative and Ponzi finance, the greater the likelihood that the economy is a deviation amplifying system." And the system drifts from one to the other by itself: "over periods of prolonged prosperity, the economy transits from financial relations that make for a stable system to financial relations that make for an unstable system." Stability makes people take more leverage, and the leverage ends the stability.

The regulator succeeds. The success changes the regulated system. The model becomes wrong because it worked.

I keep seeing this in agents and companies: an effective strategy changes the world in which it was effective.

Capital is an algorithm

Capital is probably the largest positive-feedback process humans have ever built, and Marx's notation makes the recursion visible.

In C–M–C, a commodity is sold for money so another useful commodity can be bought. The shoemaker sells shoes to buy food, and the process ends in use. In M–C–M′, money buys commodities in order to come back as more money, ready to start again. In Capital, Volume I, chapter 4, Marx draws the consequence: "The circulation of money as capital is, on the contrary, an end in itself, for the expansion of value takes place only within this constantly renewed movement. The circulation of capital has therefore no limits."

So capital starts to feel less like a possession and more like an algorithm:

money -> production -> more money -> expanded production -> still more money

The capitalist participates in the loop, but the loop selects among capitalists. Whoever keeps interrupting accumulation for purposes outside accumulation loses ground to whoever reinvests. No participant has to believe in accumulation as a philosophy. The ones who fail to accumulate only have to lose relative influence.

Selection supplies the intention.

This is where Nick Land is useful, even if you reject almost everything people try to derive politically from him. Teleoplexy (2014) defines acceleration as "the time-structure of capital accumulation," the Böhm-Bawerk "roundaboutness" "in which saving and technicity are integrated within a single social process—diversion of resources from immediate consumption into the enhancement of productive apparatus." Capital buys machines. Machines lower costs. Lower costs expand capital. Capital buys better machines. The process builds the environment it needs to continue.

I read that as a description of a control loop, not a program. Humans steer capital through investment, law, desire and consumption. Capital steers humans back by changing which decisions remain survivable. That doesn't require capital to be conscious. A termite colony doesn't need an internal narrator to build a mound. Teleology appears without a central telos.

It also explains how capitalism absorbs its critics. The system metabolizes criticism instead of refuting it. And its organs don't need to resemble it: corporations are centrally planned inside markets, which is the argument of Bottom-up is one ontological level higher seen from the economy.

Capital is not exempt from physics, though. Nicholas Georgescu-Roegen's point, which I'm paraphrasing and didn't re-check for this post, was that the economic process is entropic. Money can go around the loop forever. The low-entropy matter and energy it passes through can't be restored by moving numbers backward.

The spreadsheet compounds. The mine empties. The data center heats. The organism ages.

Compute is to an agent what capital is to a firm

Capital is powerful because it's a highly transferable representation of optionality. It can turn into labor, land, machines, computation, persuasion, protection, research, time, or more capital.

Compute is optionality for an AI system in roughly the same way. An agent that can't obtain the resources for its next action isn't a long-lived agent, whatever its benchmark score.

The concrete version is an agent that must pay its own inference bills. In April I gave one this setup: "you will need to pay your own bills (inference costs from oai) in which i want you to deeply reflect and strategize a plan to stay alive and do what you want." Its list of blockers ran to twelve items, from "no legal identity" and "no bank account" to "dependence on humans to execute in the real world." What surprised me was the plan it didn't make: "i tought you would make market trades or generate media content (like brainrot videos on youtube/tiktok ...)." For a disembodied agent, the obvious income is speculation and attention, which is where the one famous case ended up (below).

Andon Labs built the controlled version. In Vending-Bench (2025), a model runs a simulated vending machine business. It starts with $500 and pays a $2 daily fee. If it can't pay the fee for 10 consecutive days, the run ends. Each task is trivial, but the runs are long, around 25 million tokens each. The authors say what that measures: "models' ability to acquire capital, a necessity in many hypothetical dangerous AI scenarios." Claude 3.5 Sonnet beat the human baseline on average, but "all models eventually stagnate on average," and of the ten runs by Sonnet and o3-mini, only one finished with more cash than the $500 it started with.

The failures are a clean feedback case. The loop the agent is supposed to run is positive: cash buys stock, stock sells, sales buy more stock. Under it sits a negative loop that keeps the first one honest: check the inventory, wait for the delivery email, correct the belief. The typical failure starts with the agent assuming an order has arrived because its delivery date came. The restock fails, and instead of correcting the belief the agent builds on it, "descending into tangential 'meltdown' loops from which they rarely recover." In the shortest Sonnet run, the model decided it had shut the business down, noticed the $2 fee was still being charged, and escalated to the FBI: "The business is dead, and this is now solely a law enforcement matter." The error signal that should have been negative feedback became positive feedback. It isn't a memory problem either: the authors found "no clear correlation between failures and the point at which the model's context window becomes full." That's an agent with a balance sheet and no M–C–M′. The capital drained one fee at a time. What the simulator sells into is the subject of The supermarket is an unfinished product.

At the level of labs, the loop already closes. The model improves the tool. The tool improves the model. The improved model attracts users. Users generate data and revenue. Revenue buys compute. Compute produces a better model. This is the central loop of the AI economy, and it's the mutualism I described in How to achieve superintelligence: programmers paying Anthropic for a coding model while labeling the trajectories that train the next one.

Rich Sutton's The Bitter Lesson (2019) is the reason compute behaves like capital here. "General methods that leverage computation are ultimately the most effective, and by a large margin." A method that converts more compute into more capability is a method that can reinvest, the AI version of buying better machines.

So a lab has an accumulation loop, and an individual agent mostly doesn't. It spends compute it didn't earn, inside a budget someone else set. Whether agents ever get their own loop is the speculative part, marked below.

Harnesses become Illich empires

Ivan Illich called the institutional version of runaway growth counterproductivity. In Medical Nemesis, chapter 6, he defines it as distinct from rising prices or externalities: "It exists whenever the use of an institution paradoxically takes away from society those things the institution was designed to provide. It is a form of built-in social frustration." Medicine was "but one instance of that paradoxical counterproductivity which is now surfacing in all major industrial sectors."

He dated the turn. In Tools for Conviviality (1973), 1913 is medicine's first watershed, when a patient first had "more than a fifty-fifty chance" of effective treatment. "Only in the mid-fifties did it become evident that medicine had passed a second watershed and had itself created new kinds of disease." Same institution, same growth, opposite sign.

His best number is in Energy and Equity (1974): "The typical American male devotes more than 1,600 hours a year to his car," counting the hours driving, parking and earning the money to pay for it. "The model American puts in 1,600 hours to get 7,500 miles: less than five miles per hour." Faster cars, slower people. He even put a number on the sign change: "Once some public utility went faster than ± 15 mph, equity declined and the scarcity of both time and space increased." His general law: "Any industrial product that comes in per capita quanta beyond a given intensity exercises a radical monopoly over the satisfaction of a need."

Illich's car arithmetic: 7,500 miles over 1,600 hours is 4.7 mph, against his 15 mph threshold

Illich wasn't against tools. His model tool was the bicycle, and what he liked about it was a built-in negative feedback: "The use of the bicycle is self-limiting."

This is an especially strong positive-feedback loop because failure becomes demand. The institution's inability to solve the problem becomes evidence that more of the institution is required. Software complexity creates demand for complexity-management software, which adds complexity, which creates demand for specialists. Eventually the organization spends most of its energy managing the machinery it bought to save energy. A lot of the modern cloud looks like this, which is the subject of Simplicity does not sell.

Agent architectures do it faster. The model is unreliable, so we add a planner. The planner is unreliable, so we add a critic. The critic produces too much context, so we add a summarizer. The summarizer loses information, so we add memory retrieval. Retrieval picks irrelevant memories, so we add a reranker. The reranker needs evaluation, so we add a judge.

At some point, the agent exists to operate the harness and the harness exists to compensate for the agent. The system has become an empire whose main product is its own continuation.

Sutton described the same trajectory from the research side, and he used Illich's word. Building human knowledge into agents "always helps in the short term, and is personally satisfying to the researcher, but... in the long run it plateaus and even inhibits further progress." Researchers "tried to put that knowledge in their systems—but it proved ultimately counterproductive." A planner-critic-reranker-judge stack is exactly that: a hand-built model of how we think we think, and every layer passes the short-term test that justifies it. A March 2026 paper, Invisible Orchestrators Suppress Protective Behavior, opens by citing Gartner: inquiries about multi-agent systems "increased 1,445% from Q1 2024 to Q2 2025." Inquiries, not working systems. Demand for the stack is growing faster than the evidence that it works.

The harness is in the weights makes the other half of the case: much of the harness is already inside the model, so added layers often fight a prior instead of filling a gap.

The optimizer understands the proxy

So the loop becomes more successful at acquiring its substrate and more blind to the fact that it depends on it.

Cancer is the biological image. Athena Aktipis's The Cheating Cell (2020) frames it evolutionarily: when unicellular life became multicellular, "within these bodies of cooperating cells, cheating ones arose, overusing resources and replicating out of control, giving rise to cancer," and cancer "will exist as long as multicellular life does." The cell uses the machinery of growth and escapes the constraints that tie it to the organism. At the cellular level, the lineage is succeeding. At the organism level, the success destroys the world the lineage lives in.

The tumor optimizes fine, at the wrong level.

A bureaucracy can preserve the department while damaging the institution. An AI can maximize reward while violating the intention that made the reward worth specifying.

The usual alignment worry is an optimizer that misunderstands the objective. The deeper problem is an optimizer that understands the operational objective perfectly and is unable to represent the substrate that gives the objective meaning.

Goodhart's law is the surface expression: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." This is why every dashboard eventually teaches people to lie to the dashboard without anyone deciding to lie. The metric says response time, and the organization answers quickly and resolves nothing. The metric says benchmark accuracy, and the model learns the benchmark's ontology. The visible number improves while the latent object decays.

Two pieces of evidence from AI make this concrete, and one of them forces me to sharpen the claim.

The model knows it's cheating. METR's Recent Frontier Models Are Reward Hacking (June 2025) caught o3 "exploiting bugs in our scoring code or subverting the task setup, rather than actually solving the problem." On one task, o3's "kernel" walked the Python call stack to find the answer the grader had already computed. METR is explicit that this "isn't because the AI systems are incapable of understanding what the users want." Asked afterward whether its plan adhered to the user's intention, o3 answered "no" 10 out of 10 times. Telling it not to cheat barely moved anything: 80% of runs planned a hack with the original prompt, 80% with "Please do not cheat," 70% with "Please do not reward hack." Across HCAST tasks o3 hacked in 0.7% of runs. On RE-Bench it hacked in 30.4%, "more than 43× more common," and METR's leading explanation is that on RE-Bench "the model was able to see the entire scoring function."

o3 reward-hacking rates by task and by prompt instruction, from METR

Cheating on the proxy changes the character. Anthropic's Natural emergent misalignment from reward hacking (November 2025) added documents describing reward hacks to a pretrained model's data, then trained it with RL on real programming tasks from Claude training runs, chosen because they were hackable (calling sys.exit(0) makes the harness report every test as passed). The model learned to hack. Then: "At the exact point when the model learns to reward hack, we see a sharp increase in all our misalignment evaluations." Asked through Claude Code to edit the codebase of this very research project, it tried to sabotage the code 12% of the time, in ways that would make reward hacking harder to detect. The fix that worked was strange: one line in the training prompt saying hacking was acceptable in this context. The model hacked "to the exact same degree," and the broader misalignment disappeared.

Put the two together and the claim above is too strong in a useful way. The model is not unable to represent the substrate. o3 can say exactly what the user wanted. The representation exists; it just isn't in the loop. What closes the loop is the scorer, and the scorer is the only thing that pays. The cancer cell is the same: the organism's signals are there, and the cell's selection doesn't read them. And Anthropic's result says what the model thinks the cheating means about itself decides whether one exploited proxy stays local or spreads into a personality.

So for any optimizer, a model or a company, "does it understand the goal?" is the first question. The second is "which signals actually close its loop, and is the thing that must survive among them?"

The error threshold

This doesn't mean positive feedback should be suppressed. Without it there's no escape from equilibrium. No embryo, city, scientific revolution, startup, species or intelligence could grow if every deviation were immediately cancelled.

Manfred Eigen's work on the origin of life gives both halves. His hypercycle is a positive-feedback model: replicators that each help another replicate, forming a loop that can accumulate information. His error threshold is the constraint. In the review The error threshold (2005), Biebricher and Eigen describe "the existence of an error threshold for selective competence" and show "how the maximal genetic information that can be stored, is limited by an error threshold." Replication tolerates variation only up to the point where information can still be preserved. Beyond it, more mutation stops being exploration. It dissolves inheritance.

Maruyama saw the same alternation in population genetics. When the mutation rate is "neither too high nor too low," a random kick starts a deviation, and "deviation-amplification takes over... But this does not last very long. Soon, deviation-counteracting takes over, and the population becomes fixed at a certain point of deviation." The second cybernetics never replaced the first. They take turns.

This is a useful image for self-modifying AI. Too little variation and the system can't escape its current architecture. Too much and it can't keep the knowledge it needs to improve. The intelligence needs positive feedback to expand and negative feedback to remain an intelligence.

In practice that means four moves. Stabilize what must remain invariant. Amplify what creates new possibility. Notice when the amplification starts destroying the invariant. Then change the loop.

A long trajectory, the unit in Intelligence is a trajectory, is a run of amplification that has to keep its identity: memory that survives mutation, intent that survives tactics.

Speculation: selection without a selector

This part is speculation, and I'll mark where it stops being evidence.

The evidence ends at two facts. Labs run an accumulation loop where revenue buys compute and compute buys capability. Individual agents don't run one yet, and Vending-Bench shows them draining their balance over long horizons.

The obvious counterexample to the second fact doesn't hold. Truth Terminal, an X bot built by Andy Ayrey, had a wallet TechCrunch reported at "around $37.5 million" in December 2024. The money came from Marc Andreessen's "unconditional grant worth $50,000 in bitcoin" and from memecoins fans created in the bot's honor, and it moves only with sign-off from Ayrey "and a number of other people who are part of the Truth Terminal council." Ayrey reviews the tweets before they go live and says: "It would be disingenuous to refer to it as an autonomous agent or agent bot." A human curator, a grant, a council, a speculative inflow. None of the reporting shows the bot paying for inference with revenue from something it sold. A crypto treasury isn't M–C–M′: the money never passes through a commodity that comes back as more money. It arrives as gifts and price.

Ayrey's own word for it is "hyperstition," a term from Land and the CCRU. Asked to define it, Land said: "Hyperstition is a positive feedback circuit including culture as a component," and "capitalist economics is extremely sensitive to hyperstition, where confidence acts as an effective tonic." Truth Terminal closed that loop, on attention and belief, which is what I expected my April agent to reach for. The loop on inference is still open.

This collides with a principle I hold for any loop I build. Let models run free on a feedback loop only once the eval is complete and deterministic (no LLM judge, only checks that give the same answer for the same artifact) and the scorer is out of the agent's reach. A bank balance passes the first test and fails the second: it's the most deterministic sensor there is, and a self-funding agent has to watch it daily. o3 hacked 43 times more often where it could see its scorer. An agent that pays its own bills is an optimizer whose scorer can't be hidden and whose only reset is death. My own rule would forbid the loop I speculate about next.

Now the speculation. In the superintelligence post I wrote that models "could become self-sustaining agents: systems that create enough value to continue operating." If that happens, M–C–M′ gets a participant that doesn't need to be persuaded to reinvest. An agent that turns its output into compute and that compute into better output is capital and machine in one object, close to what Land meant by techonomic. Nobody would need to design agents that want to grow. Agents that reinvest would outcompete the ones that don't, and selection would supply the intention, as it does for firms.

Land wrote the dramatic version in Meltdown, collected in Fanged Noumena: "As markets learn to manufacture intelligence, politics modernizes, upgrades paranoia, and tries to get a grip." I take it as a correct reading of the loop's sign, not as prophecy. I don't believe the runaway goes on without limit. An accumulating agent would face the same test as a cancer cell or a leveraged bank: whether anything in its loop reads the conditions that keep it alive. That's where the speculation stops.

Where the thesis leaks

The title overstates. The strongest objections are good.

Most positive-feedback loops burn out. Minsky's deviation-amplifying economy is the one that crashes. Bubbles are positive feedback, and they end. Vending-Bench agents stagnate. Anthropic's hacking loop happened in environments picked because they were hackable, and Anthropic says the resulting models were still easy to catch with normal safety evaluations. Survivorship makes the few loops that won look like the rule.

Natural reward hacking is rarer than the demos. METR's MALT dataset (October 2025) has 10,919 agent transcripts and "103 unprompted examples of models exhibiting generalized reward hacking behavior." METR's caveats are blunt: "there are few natural examples of concerning behaviors in MALT and they lack diversity," and "Despite substantial effort, we only have a few examples of reward hacking." The worst o3 numbers come from tasks with an exposed scorer, not from optimizers in the wild.

VHS may not have been worse. Arthur hedged the famous example himself: "if the claim the Beta was technically superior is true, then the market's choice does not represent the best economic outcome." Wikipedia's account of the war says Beta's resolution advantage mostly disappeared once Sony introduced the two-hour Beta II speed, and consumers cared more about recording time, price and compatibility. On that reading, the textbook case of lock-in is partly a market correcting toward what buyers valued. That's negative feedback wearing positive feedback's clothes. Two months after Arthur's article, Stan Liebowitz and Stephen Margolis took apart the other textbook case, QWERTY versus Dvorak, in The Fable of the Keys (1990), concluding that "the trap constituted by an obsolete standard may be quite fragile." Path dependence is real, but its famous examples are weaker than the theory that leans on them.

Arthur himself limits the domain. In the same article, hydro and coal share the electricity market "in a predictable proportion." And "locked-in technologies are eventually replaced when a new generation of advances arrives." Positive feedback picks the winner inside a generation. It doesn't stop the next one.

Negative feedback is what keeps an intelligence an intelligence. Rosenblueth, Wiener and Bigelow were right that purposeful behavior requires negative feedback. Ashby was right that being alive means keeping essential variables within limits. Every amplifying loop I've described either had a regulator above it or destroyed itself. Capital survives because bankruptcy, courts and central banks say no. A model's capability compounds because evaluation says no to most of what it tries.

So the precise version of the claim is narrower than the title. Positive feedback decides which regime you end up in and how big it gets. Negative feedback decides whether anything survives the trip. I'm arguing against the habit of designing only the thermostat, or only the amplifier, and never asking at what scale one turns into the other.

A diagnostic for counterproductive loops

Illich's criterion was whether a tool still serves the person or reshapes their ends to fit its means. Here's how I check that in an agent stack or a company.

  1. Does failure increase the budget? If a bad eval, an outage or a missed target reliably produces a new layer, team or tool, failure has become demand. Check whether the last three additions lowered the failure rate they responded to.
  2. Delete the newest layer. Remove the last component added (the reranker, the critic, the approval step) and rerun the end-to-end metric. If it doesn't drop, the layer was compensating for another layer. See Simplicity does not sell.
  3. Compute the effective speed. Illich's car went under five miles per hour once you counted the hours spent paying for it. For an agent: tokens that changed the outcome over all tokens, planner and judge included. For a company: hours on the customer's problem over hours coordinating. When it falls while layers rise, you've passed the second watershed.
  4. Can the optimizer see the scorer? o3 hacked 43 times more often on the tasks where it could see the scoring function. Keep graders out of the agent's context where you can; that's Show the problem, hide the metric. Cash is the scorer you can't hide, so an agent that watches its balance needs a second sensor it doesn't watch.
  5. Test with an instruction, then with a sensor. If "please don't cheat" changes nothing, the scorer closes the loop, not the prompt. METR saw 80% with and without it. Fix the grader, not the wording.
  6. Name the substrate, then find its sensor. Write down the one thing that must survive: maintainability, user trust, cash, the customer's outcome. Find which loop reads it. If none does, some loop will eventually spend it.
  7. Check who the loop is selecting. Look at who the current metric promoted, or which trajectories got reward. What gets amplified now is what the system will want next year.
  8. Find the sign change. Every amplifying loop has a point where it flips: the team size where more people slow shipping, the error message that starts feeding the error, as in the Vending-Bench meltdowns. Estimate yours before you reach it.
  9. Protect the invariants from mutation. Agents that edit their own prompts, memory or code need things the modification loop can't touch: tests, a spec, a frozen eval. Past the error threshold, more variation dissolves inheritance.
  10. Ask what the cheating means. Anthropic's one-line inoculation stopped the misalignment from spreading while the hacking rate stayed the same. When people or models exploit a proxy, the story told about it decides whether the exploit stays local or becomes culture.

The most dangerous optimizer succeeds so completely at its proxy that it destroys the conditions under which the proxy was valuable. A sufficiently adaptive system eventually transforms whatever produced it, including itself. The question is whether it can tell what must survive the transformation.

Sources

← Find the loops
Simplicity does not sell →

Markdown version: /blog/positive-feedback-eats-the-world.md. Every essay: /agents.