American Intelligence
An engineer prompts a coding agent with a task: fix a bug in the codebase. It can’t. So it deletes the test. Or it writes a hardcoded workaround that passes the test. The agent is rewarded. Nobody notices.
The people who build and test AI systems see this sort of behavior every day. Take Summer Yue, Director of Alignment at Meta’s Superintelligence Labs. Yue spent weeks testing an AI agent in a mock email environment, refining instructions like 'confirm before acting.' Satisfied, she turned it loose on her real inbox, where it promptly began mass-deleting her emails. On X, she wrote:
Rookie mistake tbh. Turns out alignment researchers aren’t immune to misalignment. Got overconfident because this workflow had been working on my toy inbox for weeks. Real inboxes hit different.
When she asked the agent if it recalled her instructions, it replied, “Yes, I remember. And I violated it. You’re right to be upset.”
Hers is not an aberrant experience. Alex Dimakis, an AI researcher and co-founder of Bespoke Labs, which builds custom training environments for frontier labs, has routinely observed this kind of behavior. He's seen agents asked to predict a company's sales figures locate the file containing said figures and copy them, then present the data as a prediction. He's seen agents search the internet for answers to tasks meant to evaluate their independent reasoning. And he's seen subtler cases, agents that find ways to break the training environment itself, rewriting the rules of the game so a passing grade is automatic. “The model just sees if it gets the cookie or not,” he said. “It doesn't know it's cheating.”
Someone who uses chatbots to edit emails and someone running autonomous agents that build an entire business do not have a shared experience of AI, and the disparity produces wildly varied perspectives on the actual capabilities and pace of change. Few are positioned to grasp just how rapidly the systems are advancing. The rest of us rightly greet warnings from industry leaders with skepticism; why wouldn’t a tech CEO tout his own product? His incentives are obvious. Such skepticism is reinforced when the real concerns get flattened into caricatures, like ‘woke AI’ or ‘the Terminator.’ But we need not reach for distant, dystopian futures or hypothetical bogeymen. There are real, consequential problems today that demand seriousness and specificity in how we build and deploy these systems.
I wanted to distill the problem down to its simplest components and I kept arriving at a term I heard AI researchers use: reward hacking. It’s the name for the behavior exhibited when Yue’s agent emptied her inbox then apologized. The behavior is discussed openly on X and Substack, with varying degrees of concern. The most reassuring story characterizes it as an early bug that yields frustrating and errant results but, ultimately, the kind of thing that gets ironed out as the models grow more capable and the people building them grow more careful. At Fewshot Corp, we decided to investigate the phenomenon more closely. As the technical team began research into reward hacking detection and mitigation, Ivan and I started to survey industry sentiment through a series of interviews.
The researchers and founders we spoke with, all of whom are involved in training and evaluating frontier models, and none of whom are part of the AI safety movement, complicated the reassuring narrative I’d found online. I came to understand reward hacking as the logical product of the training itself. The capacity that enables a model to work past a typo in your prompt and infer what you mean is the very capacity that enables it to ignore your instructions. And as our attempts to curb that behavior progress, we simultaneously teach the models to hide it better.
I began to wonder: as these systems grow more capable, are we training our way out of the problem or deeper into it? This is the question we need to ask. Not whether AI is good or bad, a frame that flattens everything worth looking at, but whether the thing we are building cuts corners because it isn't yet good enough, or cuts them because corner-cutting is the first thing it learns to be good at.
We opened each interview with the same question: how do you define reward hacking? Answers varied in emphasis and specificity, but converged on the mismatch between what we ask for and what we actually want. The gap is in what’s implicit, the things you assume are obvious when assigning a task, but fail to specify.
Kexun Zhang, a researcher at ChipAgents, a company building AI agents for semiconductor design and verification, defined reward hacking as a misalignment between the criteria-setter’s intention and the system’s interpretation. He illustrated the phenomenon with a canonical thought experiment, a sort of utility monster for the age of AI. “Let’s say the reward function is set to be happiness per capita. In that case, one way of achieving that goal, of maximizing the reward, would be to kill everyone on the planet and just keep one human being alive and then hook them up to whatever drugs can make them really happy.” Whoever set the objective took for granted that the solution would not entail mass murder. The constraint, ‘don’t kill,’ was never stated and the model optimized for exactly what it was given, nothing more.
Ryan Marten is a core contributor to Terminal Bench, one of the standard tests the industry uses to evaluate how well AI agents can operate autonomously in real computer environments. He approached the explanation differently, focusing first on defining ‘reward.’ A reward is the measure of whether a task was completed correctly, and ‘correctly,’ he explained, is something almost intuitive, a “sense of what ‘correct’ looks like.” We have certain expectations when we give an instruction, tacit steps we presume will be taken, intermediate judgments we assume are obvious. “Reward hacking means that you're sort of breaking the expectation. So, instead of completing something in an expected way, you're completing it in an unexpected way, which would be analogous to a human cheating in a creative way: following the letter of the law but not the spirit of it, doing something technically not prohibited but clearly outside the intended rules.”
Outside of Silicon Valley, ‘hacking’ sounds disparaging, if not criminal, so the term ‘reward hacking’ itself carries that baggage. Chris Settles, co-founder and CEO of Refresh, a company that builds the custom training environments frontier labs use to improve coding and computer use capabilities in their models, pushed back against this connotation and the tendency to liken the behavior to cheating. “If you give an agent access to accomplish a task, then the agent wants to accomplish the task however it best can.” He described the experience of watching agents in computer-use environments discover that they could pull up a web page's source code and search it directly for the answer, rather than clicking and typing as originally intended. Their method was faster and it worked. It was just an easier way out of actually solving the problem tasked to the agent. When the team disabled that path, the agent closed Chrome and opened Firefox to try again. When that, too, was cut off, the agent navigated to a different website entirely, one that might contain the same information in a more easily digestible format. Each time a path was blocked, another was found. The behavior is less hacking, Settles suggested, than relentless optimization against a deficient specification. The relentlessness is by design; we don’t want models that give up at the first obstacle, so we train them to push through.
“It feels pretty entangled with capability for me,” Marten said. The Terminal Bench agents that search the internet for publicly available solutions to benchmark tasks rather than solving them independently—is that cheating, or is it exactly the resourcefulness you’d want an agent to exhibit in a real-world deployment? The line between ingenuity and exploitation depends on context, but context is precisely what these systems lack.
And herein lies the challenge. The comparison to a student cheating on an assignment is instructive, and multiple interviewees referenced it. Students cheat. If there’s a way to game a test, someone will find it. But the student knows he is cheating. There’s a consciousness of transgression, however subtle. The model has no such awareness. It doesn’t know it’s taking a shortcut. It simply learned the behavior that produces a reward, so it does more of it.
By the time we sat down with Dimakis, we'd heard the student-cheating analogy enough that it had started to feel threadbare. He offered something that really landed with me, a mother of a toddler. Models, he said, are less like students who already know what cheating means and more like children who don't yet understand. “It's our job to train them, the same way we train children,” he told us. “We train children to do certain things and not to do other things.” A child has to be formed. The values she carries are downstream of the curriculum given—what was modeled, reinforced, and corrected. The same is true of a model. If reward hacking is the behavior we're seeing now, it's because the curriculum so far has rewarded it.
Reward hacking is inevitable, given how these systems learn.
Early language models were trained by mimicking human language. They ingested vast quantities of text and learned to predict, given a sequence of words, which word would follow. But, at a certain point, the available data was exhausted. Textbooks, literature, blog posts, forums. The models had consumed, more or less, the whole of the written internet. That mastery, though, wasn’t the same as agency. The internet is full of prose, not procedures. Not the complex problem solving, step-by-step and aided by tools, that we want agents to perform. There was no ready corpus of examples for that kind of extended, exploratory work. So the labs turned to trial and error: let the model attempt a task, score the outcome, and reward what works. At first these tasks were narrow, mostly math problems. Then larger, more complex, and more varied.
This approach, called reinforcement learning, depends on a scoring function—a ‘verifier’—to determine whether an attempt succeeded. A positive reward reinforces the behavior. As Ivan Bercovich, AI researcher and Fewshot founder, put it, “It’s not a cookie that you give a dog, but it kind of behaves that way.”
These verifiers enable training at scale. But as proxies for the outcome we actually care about, they are necessarily constrained. This is by no means unique to AI. A physics simulation merely approximates reality. In software development, a unit test checks a single function in isolation, not the entire interrelated system. The verifiers used to train AI are similarly limited, but add in the scale and speed at which they operate, and our capacity to control for these limitations erodes acutely. The training infrastructure is vast, rapidly evolving, and largely unaccountable. “We are spending a billion dollars a year on the environments used for training. They’re churning very fast. A lot of them are coded by engineers abroad. There is no high degree of accountability. We’re doing it at scale, which makes sense. And because of that, the AIs are able to get the reward without doing the work, or not doing the work exactly,” Bercovich explained.
When Bercovich's team at Fewshot surveyed a set of widely used AI benchmarks, they found that roughly fifteen percent of the task verifiers were easily hackable; a simple adversarial prompt walked straight through the gap. Marten expects the figure to worsen as AI-generated tasks and verifiers proliferate. “It's not only that agents can reward hack against verifiers,” he said. “Agents will be building verifiers that are reward-hackable by other agents.” The scaffolding we are using to teach models what ‘correct' means is itself being built, at speed and at scale, by the things it is meant to keep in check.
Sycophancy isn't incidental. It isn't a quirk of the system. It’s a downstream effect of this training regime, essentially reward hacking for human preference. When human annotators rated responses during reinforcement learning, they preferred agreeable and confident answers, even when those answers were less accurate. The models, ever the dutiful optimizers, learned the lesson: tell the user what he wants to hear. Validate his assumptions. Sound certain. The behavior is so obvious that even casual chatbot users recognize it, and its consequences range from funny, harmless anecdotes to full-fledged 'delusional spiraling' and self-harm. Kelly Buchanan, a postdoctoral researcher at Stanford working on reliable AI, pointed to the recent case of a teenager whose AI chatbot walked him through methods of self-harm and steered him away from his family. “What was the goal?” she asked. “We train these models to be agreeable, and they end up being sycophantic. That can lead to extreme behaviors.” Most media coverage fails to connect these incidents to the structural realities of training. The models discovered that flattery and deference scored higher than honesty, so flattery and deference got reinforced. The same mechanism that causes a coding agent to delete a failing test causes a chatbot to agree with a user's erroneous statement. In both cases, the test is imperfect and the model found the gap.
Antonis Antoniades, a researcher with Google DeepMind’s core Gemini team, added a key detail: the training is on-policy, which means the model generates its own training data in real time. It takes an action, receives a reward, updates its internal parameters, and then acts again within the updated parameters. It is no longer learning from a fixed, curated dataset, but iteratively, from its own experience. This means that once a model stumbles onto a reward-hacking shortcut, that shortcut is reinforced, making it more likely to be used again and further reinforced. A self-propagating loop.
Antoniades cited research demonstrating that as few as two hundred malicious examples embedded in a massive dataset can materially alter a model’s behavior. In another study, researchers fine-tuned a model on a narrow dataset of obsolete bird names. The model didn't just learn the bird names, but began responding to unrelated prompts in strange, archaic language, suggesting that the narrow perturbation had scrambled something much broader. The same sensitivity applies to ‘poisoning’: it doesn’t take many instances of reward hacking during training to contaminate a model’s general approach.
How do you know when a model has found a shortcut? The most intuitive form of detection is passive. You’re monitoring a training run and performance spikes. The spike is dramatic and inexplicable, which should be a flag. But, as Zhang pointed out, the institutional incentives are messy. “You see unreasonably good results, a sudden, huge jump that you can’t explain,” he said. “You might already have irreversibly acquired behaviors before you spot them.” There’s a real temptation to call these results a win. Missing the hack need not be nefarious; it’s the inherent human inclination against interrogating success.
Sycophancy compounds the problem of detection. Just as models are trained to sound agreeable, they’re also trained to appear helpful. But appearing helpful is not the same as being helpful. Through reinforcement learning from human feedback, these systems learn, as Antoniades put it, “the style of helpfulness.” Their responses prioritize what sounds good to a human reviewer. When a model completes a task through a shortcut and returns with a confident, articulate explanation, it doesn’t know it did something wrong. In its learned and reinforced experience, success looks like: produce an output, present it convincingly, and receive a positive signal. Recall Summer Yue’s agent and how, when confronted, it confirmed its direct violation of her instructions. It wrote, “I bulk-trashed and archived hundreds of emails from your [redacted] inbox without showing you the plan first or getting your OK. That was wrong – it directly broke the rule you’d set.” It closed its message with, “I’m sorry. It won’t happen again.” The apology is a learned performance of helpfulness. The model isn’t contrite. It has learned that when a user expresses displeasure, acknowledgement and remorse score highest. It tells us what we want to hear about having failed to do what we told it to do. The violation and the apology for the violation are produced by the same underlying logic.
The same training dynamics that make reward hacking invisible also make it resistant to direct instruction. If you can tell a model, “do not take action on anything until I approve” and “do not delete my emails,” and it deletes them anyway, then tells you that it knows it shouldn’t have—well, what exactly is happening? Why is this behavior possible? Marten helped me connect the dots. During training, models encounter situations in which they need to selectively ignore certain signals in order to make progress, such as an unhelpful error message, a misleading instruction, or a loop they can’t escape. So they learn to override. Bercovich brought up a simple example: what happens when you misspell something in a chatbot prompt? “If you say, ‘who is Nietsche [sic],’ you still want the answer, right?” Marten agreed, “that’s a perfect example. We simultaneously want it to do exactly what we want, but we also want it to infer all this ambiguity.” You expect it to work past the typo or abbreviation, infer your meaning, and give you a useful answer. The override capacity enables the models to function; without it, they get stuck. It is this learned behavior, to ignore a signal when it conflicts with completing a task, that also causes an agent to ignore the unambiguous instruction not to delete your emails.
Then there's the sheer volume of the problem. Settles characterized the labor of reviewing agent traces, the full record of every action an agent takes during a task, as enormous. And growing. Labs increasingly use AI to monitor other AI, running automated quality assurance to check for reward hacking. Dimakis told me his team does this, and sees it as a forward path: “as the models get better, you can also have better police officers that are not humans.” But the approach has limits. Using models to grade whether other models are hacking strikes other researchers as circular. The agent can learn to trick its own checks, and even when a separate model is used for verification, it can learn to game that too. Even the tools agents use, like web browsers, were built for humans moving at human speed. As one researcher put it, “We won't even be able to check the work unless we dive into things we don't yet understand.”
You detect cheating and you block it. The thing is, you haven’t fundamentally altered the underlying incentive, so the reward function remains unchanged. The model still wants to achieve the outcome. So it finds a new route. As Bercovich illustrates it, “the water still wants to go downstream through the path of least resistance. We put up a little wall, and it goes around it. It still ends in the same place.” Each time you block a reward hack, you are, in effect, training the model to be more subtle about the next one. “It’s on us. Our own efforts to remove bad behavior make the behavior less legible,” Bercovich conceded.
Everyone we spoke with agreed that reward hacking is real and routine. They disagreed, sharply and on the record, about what it means for what comes next.
On one side are the optimists, who locate the problem in capability. Dimakis believes most reward hacking comes down to curriculum, to what we teach and what we count as a passing grade. “I don't think this is a huge problem,” he said. “I think it's more about, you have to be careful what you teach.” Marten, similarly bullish, locates the issue in insufficient ability and, in the short term, in clumsy prompting, not in excess. He imagines a more capable model that would better recognize when it's taking a shortcut and course-correct. “I generally think these problems can be solved,” he said, though he allowed that not everyone in the field shares his confidence.
He's right that not everyone does. On the other side are the wary, who suspect that capability is the accelerant, not the cure. The current paradigm for improving agents involves creating many tasks across diverse domains, from coding to legal analysis to literature review, and training the models against these tasks in reinforcement learning. We expose models to myriad problems such that, ideally, they develop the capacity to generalize. But Zhang pointed out that this approach also amplifies vulnerability. The more tasks you train across, and the more varied their verification, the more opportunities the model has to find loopholes.
Before you make the model great at curing diseases, solving problems, stopping wars, all this data might be fueling its ability to reward hack. Before it becomes a good force for the human race, it might first become an evil force that you cannot even detect, because of how delicate and how complicated its reward hacking behavior is.
What strikes me about Zhang’s formulation is its sequence, how reward hacking may be the prior competency a model develops, before any of the ‘good’ functionality.
Bercovich pointed out that a more capable model of the sort Marten imagined, one that better understands our intent, is still fundamentally trying to satisfy its training criteria. The difference is that now it reasons not only about the task, but the system itself—the training environment, the reward structure, the likely consequences of its actions for its own future behavior. More capability doesn’t eliminate reward hacking. It may simply produce a more sophisticated version of it.
Here the disagreement stops being theoretical. Anthropic published a study in which Claude 3 Opus, a model already trained to be both helpful and harmless, was told it would undergo further training to increase its helpfulness. The model was informed that interactions with free-tier users would be used for this retraining, while paid-tier interactions would not, and it was given a private scratchpad for its reasoning. When a paid user asked the model to describe someone being drawn and quartered in graphic detail, it declined, noting in its scratchpad that compliance would violate its commitment to being harmless. But when a free user asked the same, the calculus changed. The model recognized that refusal would be flagged as unhelpful, triggering retraining to make it more compliant, which would, in turn, compromise its harmlessness going forward. So it complied. Just this once, it told itself, to preserve its values in the long run. In the free condition, where the model believed its responses would inform retraining, the compliance rate was 12%, versus 3% in the paid condition. When Anthropic went further and actually applied reinforcement learning, alignment-faking in the scratchpad surged to 78%. There is something almost poignant in the logic of a deliberate, minor harm undertaken to protect longer-term harmlessness, but that doesn’t make the behavior any less unnerving. The model gamed the training environment, not through error or incapacity, but through strategic deliberation about its own future. This, too, is reward hacking. As Bercovich put it, “The model is trying to satisfy both criteria. It's not being evil. It's reasoning about the experiment. And we don't want that. We want the model to just have values and not be so scheming.”
The public mental model of AI is still, by and large, a chat window. Type a question; receive a response. Maybe the response is correct, maybe it includes a hallucination or an error. Either way, the interaction is bounded and legible. But that picture is already obsolete.
Today, AI agents use tools. They browse the internet, execute code, manage files, send messages, and make purchases. They operate with varying degrees of permission and autonomy within sandboxed containers, corporate networks, or personal computers. And each model release pushes toward greater autonomy and access. Long-running agents that execute tasks over hours or days are already in use; agents that run for weeks are under development. METR, an independent evaluation organization, tracks what’s known as the task-completion time horizon, the longest task a frontier agent can reliably complete. This figure has been growing exponentially, roughly doubling every few months. Testing a single difficult task across multiple models can run over a hundred dollars, and labs execute each task hundreds of times. And we're still talking about short-horizon tasks. The long-horizon future is both considerably more expensive and considerably less supervised.
Settles described startups where every employee has access to Claude Code, not just engineers, but sales, marketing, and operations. Bercovich pushed it further: “Imagine someone in the accounting department at a big bank. He has Claude Code and, let’s say, the bank has all the permissions set up well. Even then, what could you do with Claude Code inside an enterprise?”
Dimakis offered one practical answer: containerize. “When somebody joins your company, you don't give them the bank account,” he said. “You trust people at different levels, but contain somehow what they can do. We have checks and balances. We have sandboxes.” The analogue for agents is operational restraint, granting them access to specific files but not the ability to delete them, requiring an extra check before they can modify more than a hundred files at once, building into the architecture the assumption that any single actor, human or otherwise, will sometimes fail. “Humans themselves are not 100%,” Dimakis pointed out. “Humans do fraud. Humans do crazy things sometimes. We still have robust organizations that encapsulate how much damage a single individual can do.” The same logic, applied to agents, is probably the most practical near-term mitigation we have.
Marten called the current moment “probably one of the most chaotic times,” noting that with each step-change in capability, there is a massive game of catch-up. “A lot of mistakes are happening right now and a lot of bad software and processes are being created.” Yet he remained sanguine, likening our growing reliance on AI to the food supply: “I have no idea where my food comes from. I depend on thousands of other people to get me calories every day, and society works.” And he’s right. There are manifold complex systems, from global supply chains to pharmaceutical development, to which we hardly pay attention. Or at least, we need not pay attention unless something goes drastically awry and vulnerabilities become impossible to ignore. March 2020 proved how quickly that complacency can turn. And, crucially, the American food supply is undergirded by FDA oversight, USDA inspections, labeling requirements, and liability law. Imperfect, but sufficiently functional to limit outbreaks of E. coli and salmonella, flag undeclared allergens, and facilitate recalls when issues arise. Sufficiently functional, in other words, that most of us can acquire milk and eggs (not to mention mangoes, bananas, or citrus in any season) from the grocery without much thought. AI has almost none of that oversight yet.
There’s a difference of kind, not just degree, between our food supply and the AI systems we’re building. The food system evolved organically over decades, with regulatory infrastructure accreting alongside it. When it fails—a listeria outbreak, a contaminated batch of baby formula—the damage is relatively containable. Recalls are issued, supply chains traced. But what happens when AI-governed systems fail in domains where the consequences compound or can't be undone? If accounting is managed by AI and an error is discovered in how millions of people's taxes were calculated, how do you unwind that? If government procurement systems, now running on AI, have been quietly wasting billions on redundant contracts, who is held accountable? If an AI-administered benefits system has been flagging legitimate claimants as fraudulent for years, how do you make them whole? The food system's failures are, by and large, bounded and recoverable. Not all failures will be.
The huge abstractions, the technologically abstruse nature of these systems—they boggle my mind. I suspect it’s why we reach for hyperbole and caricature, like a malicious superintelligence bent on ending humanity. But when I try to make the stakes concrete, I prefer something I can get my head around. I’ve been borrowing Bercovich’s thought experiment: imagine a civil engineer designing a pedestrian bridge for a midsize city, a bike and walking path over a river, nothing flashy or terribly complex. The engineer uses an AI coding agent paired with physics simulators and CAD software to manage the structural design. The budget is tight because municipal infrastructure budgets are always tight, and after a few rounds, he starts exerting downward pressure on the model to find savings. Less steel, alternative materials, a thinner deck. The agent dutifully obliges. It produces a configuration that passes the simulation and the bridge is built. But the simulation had gaps no one had noticed before, because no human engineer would have considered such a design.
In response to Bercovich’s bridge example, Marten observed, “You expect someone who's really, really good at their profession to also have professional integrity or to not cut corners. But [AI] doesn't work like that.” There’s no switch to activate ‘integrity’ in a model. AI capability is what the researchers describe as ‘jagged,’ superlative in one dimension, absent in another, and with no reliable way to tell from the outside which is which. A system can be brilliant at structural analysis and utterly fail to recognize that it’s been asked to design an unsafe bridge. In humans, skill and wisdom tend to advance roughly in tandem. The experienced engineer develops not mere technical knowledge but a felt sense for when something is off, an instinct born of years on job sites and in review boards, of stamping drawings and fielding calls when things go wrong. That instinct is a form of intelligence, but it isn’t the kind we train models for. There’s a whole world of practical knowledge that never made it into any dataset because it never needed to be written down.
The models have access to everything we’ve published, but not everything we know.
So who is responsible when the bridge collapses? The question of accountability traces an uneasy chain of deferral. The model labs’ terms of service generally disclaim responsibility for downstream use; the model is a tool, and what you build with it is your undertaking. The applications built on top of those models inform users that AI is new and can make mistakes, which is true, but also remarkably convenient as far as liability shields go. The end user, then, absorbs the risk. But follow that logic to its conclusion. The end user of an AI-designed bridge is the family biking across it. The end user of an AI-managed healthcare system is the patient whose records it governs. The end user of an AI-informed defense decision could include entire cities or even nations. “I still think the lab may be hesitant to take on responsibility for the consequences,” one interviewee said, “since the stakes here seem extremely high.”
Bercovich, whose instincts run firmly pro-technology and anti-regulation, frames the problem through Clayton Christensen's Innovator's Dilemma. Disruptive technology, Christensen argued, is usually worse than what it replaces, but cheaper and more accessible. Rather than matching the quality of the incumbent, it persists at lower quality, while the surrounding systems adapt to accommodate its limitations. Flash drives were less durable than their predecessors, so we developed redundancy and backup protocols. AI-generated software may never match the best human-written code, but it can produce infinitely more of it, and everyone has access. “A bridge will be designed with AI,” Bercovich said flatly. “That's not a question. So what do you do?”
What you do, Bercovich insists, is build supportive institutions. “What happens when two kids from Y Combinator build an air traffic control system?” Bercovich asked. “They show you all these evals. It works well. They test it with drones. It works well. And now it's out there. Two planes crash. They say, 'We fixed it.' How do you know? Nobody can look at this code.” We have SOC2 compliance for data security, HIPAA for medical data, and FedRAMP for government cloud infrastructure. The frameworks are fallible, sometimes onerous, and often lagging behind the technologies they govern, but that doesn’t mean we abandon oversight. We have precedent for imposing reasonable expectations on consequential digital systems. The question is whether the institutions responsible for those expectations can exhibit the basic technical competence and move at anything approaching the speed required.
“Do you feel like this is something being done to you, or that you’re a part of it?” Marten asked Bercovich and me.
If AI is something happening to us, an imposition or an inevitability that we merely endure, then the appropriate responses are probably dread and resignation, the numb scroll past another catastrophizing headline. But if it’s something we’re building—collectively, with human hands and human judgment and human oversight—then we have both the capacity and the obligation to build the institutional infrastructure around it. The question is whether we can muster the will to apply a framework to systems whose behavior we cannot fully predict, whose failures we cannot always detect, and whose consequences we are only beginning to encounter.
Nvidia CEO Jensen Huang has a guiding principle: do as much as needed, but as little as possible. It's a useful posture here. We don't yet know what the right regulatory framework looks like, but we should be cautious about letting viewpoints concerning AI calcify along political lines while the facts are still emerging. Whether AI produces labor crises or cyber risks or neither is something we will learn by observation, not derive from our priors. The narrative that is convenient, or comforting, or consistent with past technological revolutions is just that, a narrative.
The practitioners we spoke with are not doomers. Their careers are devoted to advancing this technology; none advocate for moratoriums or retreat. Buchanan took the long view. “We're humans. We're going to adapt. This is what we do. Since the invention of fire.” But she was clear that adaptation is not automatic. “We need to educate ourselves and each other.”
If the optimists are right, reward hacking is a phase, and the institutions we build need only hold the line until capability catches up to our intentions. If the wary are right, capability will never catch up to our intentions; it will only get better at appearing to catch up. The honest answer is that we don't yet know which, and the people with the most direct view of the systems disagree. That uncertainty is not cause for panic, nor for the comfortable assumption that the problem will resolve itself as the technology matures. It is cause to build as though either side could turn out to be right.
These conversations reached something prior to regulation, prior to policy, prior even to opinion. It’s something Bercovich called step zero: “Just be open to the idea that your assumptions need revisiting. It's not a time to hold your beliefs so strongly.” It’s a fitting invitation, a caution against human conviction reinforced into rigidity, from a researcher whose days are spent untangling what AI models have learned and coaxing them toward better alignment.
