Connecting Language and Robotic Capabilities: Spotlight on Visiting Fellow Stefanie Tellex
Stefanie Tellex is Associate Professor of Computer Science at Brown University. Her work at the AI Institute focuses on developing foundation models for navigation and skill learning.
What are you working on right now?
My group focuses on figuring out how we can build models to achieve new levels of robotic capability and autonomy. There’s been a lot of work in the field to make robots understand language, where you go from language to a simplified, high-level abstraction to robot actions. But the problem is that symbolic representation, or the list of things a robot can do, is very limited. Basically, robots can’t do very much. So even though we have large language models and large vision models that are very powerful, using them to make robots do a lot of different things is still an unsolved problem.
The plan we are pursuing at the AI Institute is to scale up the capabilities of robots. Instead of a robot being able to do five things or ten things, we hope to have them do a thousand things. Once we get there, then we have the ability for the robot to do anything a person can tell the robot to do. But in order to get to that level of functionality, we need to gather a lot of data and develop an approach that makes it easier to teach robots new capabilities.
How did you become interested in the research you're doing?
I’ve always been interested in language, and language understanding by computers. I came to robotics as a PhD student because I got frustrated with statistical natural language processing or NLP. I felt like it wasn’t connecting language with the world in a meaningful way. For my post-doc research, I joined a lab at MIT CSAIL with Nick Roy and Seth Teller where they had really cool, really robust robots like robotic forklifts and robotic wheelchairs. I got to work with these robots and help them understand language, and it was amazing to see how fast robotics was advancing. There were very few people in robotics thinking about language at that time, so I wrote some of the first papers on robotic language understanding with statistical learning-based methods. Then I started at Brown with a focus on language and robots, but robotics is extremely broad. If we want language to map to everything a robot can see and do, we need to think about everything the robot can see and do. So I started thinking about robot perception, planning, skills, and hierarchical abstraction, and how to connect those fields to language.
Where do you hope to see the field in 5-10 years?
One of the parts of robotics that is lagging behind right now is mobile manipulation. There are a lot of skills a person can perform with manipulation, but right now we don’t have the software or the data we need to translate those skills to robots. In 5-10 years, I hope we have a mobile manipulator that you can take to any environment and, in a couple of days in that environment, teach the robot to do anything it is physically capable of doing. Right now, robots are blocked more by software than by hardware. Even though we can teleoperate robots to perform tasks, we don't have the algorithms to perform the tasks autonomously. That’s not to say that the physical, mechanical parts are fully solved — they're not — but we need to make our software better so that we’re blocked by the mechanical components again. If we can get to that point, that would be really cool. I’m full-time at the AI Institute now because here we have the vision and resources to address these problems. Robots have enormous potential to make our lives better and it's really exciting to be in a place where we can move the field forward in a profound way.
