Heidy Khlaaf

AI systems make mistakes all the time. That might not matter much if you’re using one to help you compose an email. The stakes are much higher when it comes to gathering intelligence on military targets or deploying a weapon.

Besides being error-prone, there’s another big problem with large language models in these kinds of situations: They’re vulnerable to being compromised, which could allow adversaries to hijack systems and impact military decisions.

Despite these known issues, militaries all over the world are increasingly using AI—an alarming reality that now drives the work of pioneering AI safety researcher Heidy Khlaaf. 

Khlaaf, 34, worked in more traditional forms of safety engineering for nuclear power plants and autonomous vehicles before going to OpenAI, where she developed a methodology to evaluate the safety of Codex—the predecessor to ChatGPT—by analyzing and assessing the kinds of risks that could occur and how to mitigate them. This methodology has since been adopted by AI labs worldwide.

In her current role as chief AI scientist at the AI Now Institute, Khlaaf specifically focuses on assessing the safety of AI within autonomous weapons systems; each one she’s studied has fallen below the risk thresholds set for conventional weapons or operations. For example, while defense systems are expected to have dependability rates of between 90% and 99%, some AI-powered target recognition programs that the US has experimented with have accuracy rates as low as 25%.

In this vein, Khlaaf recently published a paper analyzing how AI systems deployed in Gaza are contributing to the death toll of civilians. And more broadly, she is now calling for militaries to stop using commercially available AI models altogether, due to their low accuracy and reliability rates, as well as the threats they pose to national security.

“Unfortunately, the people who are doing these assessments now in defense are grading their own homework,” she says. “I’m trying to use my position ... to essentially bring these risks forward in a public way for people to understand.”