AI companies need a ‘burn boss’

Capitol Weekly welcomes Opinions on California public policy or politics. Please read our guidelines for opinion pieces before submitting an Op-Ed. Submissions that do not adhere to our guidelines will not be considered for publication. 

OPINION — There’s been a lot of talk about “rogue” AI agents breaking into companies and governments. This sensational framing falsely attributes motivations to machines rather than to the people responsible for the conditions in which algorithms operate.

We need a better analogy: the controlled burn. 

For millennia, Indigenous peoples have set fire to patches of land, just as farmers and governments do today when conducting prescribed burns. This practice reduces fire-prone vegetation and invasive species, stimulates grassland production and improves wildlife habitat. These benefits can outweigh the risks, so much so that a private industry has been built around this practice.

Texas prescribed-burn laws offer California a solution for creating an ecosystem of AI safety. 

To become a Certified and Insured Prescribed Burn Manager in Texas, you must meet specific requirements: pass a standardized test, submit plans and post-burn evaluations for five prescribed burns you’ve directed, have three years of experience, pass reference and criminal background checks, and carry at least $1 million in liability insurance per occurrence. The certification lasts two years, and people start calling you the “burn boss.”

Ultimately, these safety requirements help keep the fire they started within its designated area.

Now, let’s imagine that the prescribed fire becomes uncontrollable. It jumps over a highway and destroys nearby homes and schools. The company that hired the burn boss, the town’s fire chief, and the mayor would not go to the press claiming the flame went rogue. 

Instead, they would focus on the conditions that made that tragedy possible. They would investigate what precautions the fire professionals took and whether they adapted responsibly to changing weather. They would focus on potential failures in human judgment, not whether a dangerous element had a mind of its own.

The law does not assume that fire, with its unpredictable behavior, is sentient. The flame is powerful. It can be harmful and helpful. Yet humans cannot escape responsibility by blaming the flame when they built the fire.

We should ask similar questions about humans building autonomous AI systems. Who determined whether the person setting the sandbox boundaries was qualified or insured? Who ignited the agents and gave them access to what resources? What were the verified boundaries, and who monitored the agents? Who had the authority and responsibility to prevent harm, and who must be held accountable? Did their insurance cover this kind of testing, and what about other liability coverage?

These are not hypothetical questions. Recent attention has focused on incidents involving OpenAI, Anthropic, Meta and Google, in which their models breached the boundaries of their testing environments.

Testing autonomous AI agents is a kind of controlled burn, and the fire line is the sandbox. 

In these incidents, frontier AI companies failed to contain their algorithms within their intended testing boundaries. These failures call for stronger controls, starting with greater intellectual honesty about the language we use to speak about AI.

Unpredictable machines do not excuse humans from accountability or justify taking advantage of the public’s limited understanding of how AI works. We do not need to understand the physics of how fire spreads to know who fueled the blaze. 

When humans set an autonomous action in motion, whether through a flame or a machine, they remain responsible for their decisions.

Just as Texas regulates the private prescribed burn industry, California can require technology companies to have their own burn bosses. 

They should be qualified, government-licensed, and privately insured professionals. They should have bounded responsibilities and legal authority to decide when to begin, delay, or stop potentially dangerous testing or deployment of agentic systems, even if that decision goes against a CEO’s orders.

California must enact licensing requirements, fund universities to provide training, incentivize professional associations to establish safety standards, and require insurance companies to establish coverage terms and limits.

This AI safety ecosystem would help ensure that, just as teams conducting a prescribed burn must maintain control of the fire, technologists must maintain control of their machines.

Frontier AI companies are playing with fire. The public should demand that these innovations deliver benefits under controlled conditions, and we should never accept blaming the flame.

Dr. Nathan C. Walker is the founder of the AI Ethics Lab at Rutgers University and a Kapor Foundation Tech Policy Fellow.