Researchers from MIT and other institutions have discovered an unexpected phenomenon known as the ‘indoor training effect’ where AI agents trained in a completely different, noise-free environment perform better than those trained in the same noisy environment.
Here is the improved marked-down text:
New Training Approach Could Help AI Agents Perform Better in Uncertain Conditions
Imagine a home robot trained to perform tasks in a factory. It might struggle when deployed in a user’s kitchen because the environment differs from its training space. Engineers usually try to match the training environment as closely as possible with the real world.
However, researchers from MIT and other institutions have discovered that training AI in a completely different, noise-free environment can sometimes yield better results than training in the same noisy environment where the agent will be tested. This unexpected phenomenon is known as the ‘indoor training effect’.
The researchers trained AI agents to play Atari games with added unpredictability. They found that the indoor training effect appeared consistently across different games. They hope this will inspire further research to improve AI training methods.
To understand why AI agents trained in one environment perform poorly in different environments, the researchers used ‘reinforcement learning‘ which involves trial and error to learn actions that maximize rewards. They added noise to a key part of the training, the transition function, which defines the probability of moving from one state to another.
For example, Pac-Man might determine the ghosts’ movements. When they trained the agent with added noise, performance dropped. However, when they trained the agent in a noise-free environment and tested it with noise, the agent performed better than the one trained with noise from the start.
AI agents are trained using various machine learning algorithms and techniques.
Supervised learning involves providing the agent with labeled data to learn from, while unsupervised learning requires the agent to identify patterns on its own.
Reinforcement learning trains the agent through trial and error by rewarding desired actions.
Training datasets can be sourced from various domains, including text, images, and audio.
The choice of training method depends on the complexity and requirements of the AI task.
The researchers tested various environments by adding different noise levels to the transition function. This made the games less realistic, with more noise-causing ghosts in Pac-Man to teleport randomly. They adjusted probabilities to check if the indoor training effect worked in normal Pac-Man games, so ghosts moved up and down more often.
AI agents trained in noise-free environments still performed better in these adjusted, realistic games. The researchers were surprised that this effect seemed to be a general property of reinforcement learning, not just due to their noise adjustments.
When the researchers investigated further, they noticed patterns in how AI agents explore their training spaces. If both agents explore similar areas, the one trained in a noise-free environment does better because it can learn the game’s rules more easily.
If their exploration patterns differ, the agent trained in the noisy environment performs better, likely because it learns patterns it wouldn’t encounter in the noise-free environment. ‘if I learn to play tennis only with my forehand in a non-noisy environment, but then in a noisy one, I have to also play with my backhand, I won’t play as well in the non-noisy environment.’ said Serena Bono, a research assistant in the MIT Media Lab and lead author of a paper.
Looking ahead, the researchers plan to study the indoor training effect in more complex reinforcement learning environments and apply it to other areas like computer vision and natural language processing. They aim to create training environments that use the indoor training effect to help AI agents perform better in unpredictable settings. This could significantly enhance the versatility and robustness of AI systems in the real world.
The Indoor-Training Effect: Unexpected Gains from Distribution Shifts in the Transition Function
Journal Reference:
Serena Bono, Spandan Madan, Ishaan Grover, Mao Yasueda, Cynthia Breazeal, Hanspeter Pfister, Gabriel Kreiman. The Indoor-Training Effect: unexpected gains from distribution shifts in the transition function.
arXiv:
2401.15856v2
