Researchers at Carnegie Mellon University have developed a system that enables everyday objects to move autonomously and anticipate user actions.
The technology integrates cameras, artificial intelligence models, and small wheeled platforms to allow items such as coffee mugs, staplers, and trivets to relocate themselves proactively. This approach aims to enhance convenience without introducing additional dedicated robots into existing environments, according to Violet Han, a Ph.D. student at CMU and lead author of a paper on the research.
Scientists mounted objects including a pencil tray, stapler, and coffee mug onto wheeled platforms controlled by an AI system in Carnegie Mellon University’s Interactive Structures Lab. Each platform uses a Bluetooth-enabled microcontroller with motors and batteries for mobility. The objects themselves do not inherently contain AI; rather, a central AI system oversees the entire environment.
The system utilizes a camera to stream image frames to AI models that process and identify ongoing activities and relevant objects. Large language models with reasoning capabilities predict subsequent actions. A knowledge base embedded in the system informs it about typical human-object interactions, such as positioning a mug handle towards a user for convenience.
This research addresses challenges associated with larger humanoid robots, which include potential damage from malfunctions, human-like appearance concerns, and difficulties in achieving reliable dexterity. Alexandra Ion, an assistant professor at CMU’s Human-Computer Interaction Institute and who leads the Interactive Structures Lab, noted that existing environments are often built with the assumption of human dexterity.
While the technology is “not that far off” from deployment, according to Ion, implementation depends on public acceptance of overhead cameras. Potential applications include a key tray shaking keys when a user is about to leave without them, or an AI system moving a hidden stapler into view. Privacy and security concerns, such as the use of overhead cameras, represent new challenges that require policy and regulatory solutions, alongside models operating on local hardware not connected to the internet.
Discussions regarding the actuation of objects, such as knives, highlight safety considerations. Researchers configured a moving knife to ensure its blade always faced away from people. The goal involves ensuring that robotic actions within home environments promote safety and align with user intent, enabling robots to understand user preferences, Han said.
