Researchers at MIT have developed a new method for solving contextual reinforcement learning problems that achieves 5-50x better sample efficiency on standard and traffic benchmarks. The approach, called Model-Based Transfer Learning (MBTL), selects the most promising tasks to train on based on their potential for improving overall performance.
A New Approach to Reinforcement Learning Could Improve Complex Tasks Involving Variability
The researchers developed an algorithm called Model-Based Transfer Learning (MBTL) to identify which tasks to select and train on. MBTL models how well each algorithm would perform if trained independently on one task, as well as how much its performance would degrade when transferred to other tasks.
The MBTL Algorithm Works by:
-
Modeling the performance of each algorithm on a given task
-
Estimating the generalization performance of each algorithm across multiple tasks
-
Selecting the most promising tasks to train on based on their potential for improving overall performance
This Approach Was Tested and Found to be 5-50x More Efficient Than Other Methods
The researchers tested MBTL on simulated tasks, including controlling traffic signals, managing real-time speed advisories, and executing classic control tasks. The results showed that MBTL achieved significantly better sample efficiency than other methods.
Implications for Complex Tasks Involving Variability
MBTL’s ability to improve performance with a smaller amount of training data has significant implications for complex tasks involving variability. This approach could lead to more efficient and reliable AI systems in fields such as robotics, medicine, and political science.
Future Plans
The researchers plan to extend MBTL to more complex problems, such as high-dimensional task spaces. They also aim to apply their approach to real-world problems, particularly in next-generation mobility systems.
Funding and Related Research
The research is funded by a National Science Foundation CAREER Award, the Kwanjeong Educational Foundation PhD Scholarship Program, and an Amazon Robotics PhD Fellowship. The researchers are affiliated with the Laboratory for Information and Decision Systems, Institute for Data, Systems, and Society, Department of Civil and Environmental Engineering, School of Engineering, and MIT Schwarzman College of Computing.
Related Articles
-
New AI model could streamline operations in a robotic warehouse
-
AI accelerates problem-solving in complex scenarios
-
The curse of variety in transportation systems
-
On the road to cleaner, greener, and faster driving
Introduction
Reinforcement learning (RL) has been shown to be surprisingly brittle to contextual variations in tasks. However, researchers at MIT have developed a new method for solving contextual RL problems that achieves 5-50x better sample efficiency on standard and traffic benchmarks.
The Challenge of Training AI Agents
Training an algorithm to control traffic lights at many intersections in a city is a complex task. Engineers typically choose between two main approaches: training one algorithm for each intersection independently, or training a larger algorithm using data from all intersections and then applying it to each one. However, each approach comes with its share of downsides.
The New Method
Wu and her collaborators sought a sweet spot between these two approaches. They chose a subset of tasks and trained one algorithm for each task independently. Importantly, they strategically selected individual tasks that were most likely to improve the algorithm’s overall performance on all tasks. This method leverages zero-shot transfer learning, in which an already trained model is applied to a new task without being further trained.
Results
The researchers found that their technique was between five and 50 times more efficient than standard approaches on an array of simulated tasks. This gain in efficiency helps the algorithm learn a better solution in a faster manner, ultimately improving the performance of the AI agent.
