Nur Muhammad "Mahi" Shafiullah

Today’s robots are pretty good at doing one thing over and over again on an assembly line or in a warehouse. But if they’re ever going to become reliable enough to help in our homes, they need to be able to tackle lots of tasks in unfamiliar surroundings.

That’s an incredible technical challenge, though—largely because there isn’t much data to teach robots to navigate messy, ever-changing environments like our homes. People don’t tend to post videos of themselves washing dishes or tidying their messy sock drawers.

This gulf in data has been a focus for Nur Muhammad "Mahi" Shafiullah, 27, since he was a PhD student at New York University.

There, he was part of a team that came up with smart, scalable ways of collecting data that show how people complete domestic chores, and collated it into datasets that could be used to train an AI model and, later, robots. These included attaching an iPhone to a grabber stick to record video of people opening cabinet doors or drawers, picking up napkins or paper bags, and reorienting fallen objects. These datasets have been used by Nvidia, Microsoft, and Google, among others. Shafiullah was also part of a major cross-institution collaboration to build a dataset of 527 different robot skills—consisting of images, videos, actions, and text instructions—that can help machines perform actions they’ve never seen before, such as moving toy fruit and vegetables around a kitchen, unfolding wrinkled towels, and organizing shoes and mugs. “This dataset was a stepping stone to robots that exhibit more general behavior, like picking arbitrary handheld objects, or rearranging things in an unseen home,” Shafiullah says.

Now, starting as a postdoc researcher on Meta’s Fundamental AI Research (FAIR) team, he’s excited by the prospect of using other types of hardware, like AR smart glasses and headsets, to collect videos to train robots. Shafiullah also has set his sights on developing ways to teach robots to carry out tasks autonomously for longer periods of time, to help pave the way toward making them reliable enough to act unsupervised in our homes. It’s a complex and difficult problem, he acknowledges—but it’s a fun one.