MIT Researchers Develop an Efficient Way to Train more Reliable AI Agents

Fields varying from robotics to medication to government are attempting to train AI systems to make meaningful choices of all kinds. For example, using an AI system to wisely control traffic in a busy city could assist motorists reach their destinations faster, while improving safety or sustainability.

Unfortunately, teaching an AI system to make excellent choices is no simple job.

Reinforcement knowing designs, which underlie these AI decision-making systems, still often stop working when faced with even little variations in the jobs they are trained to perform. In the case of traffic, a model may have a hard time to control a set of crossways with various speed limitations, varieties of lanes, or traffic patterns.

To boost the reliability of support learning designs for intricate tasks with irregularity, MIT scientists have introduced a more efficient algorithm for training them.

The algorithm strategically selects the very best jobs for training an AI agent so it can effectively carry out all jobs in a collection of associated jobs. When it comes to traffic signal control, each task could be one crossway in a job space that consists of all intersections in the city.

By focusing on a smaller sized variety of intersections that contribute the most to the algorithm’s general effectiveness, this technique optimizes performance while keeping the training cost low.

The researchers found that their technique was between five and 50 times more efficient than basic methods on an array of simulated tasks. This gain in efficiency helps the algorithm learn a much better option in a much faster manner, eventually enhancing the performance of the AI agent.

“We were able to see amazing efficiency improvements, with a very simple algorithm, by thinking outside the box. An algorithm that is not extremely complex stands a much better chance of being embraced by the community because it is easier to carry out and much easier for others to understand,” says senior author Cathy Wu, the Thomas D. and Virginia W. Cabot Career Development Associate Professor in Civil and Environmental Engineering (CEE) and the Institute for Data, Systems, and Society (IDSS), and a member of the Laboratory for Information and Decision Systems (LIDS).

She is joined on the paper by lead author Jung-Hoon Cho, a CEE college student; Vindula Jayawardana, a graduate trainee in the Department of Electrical Engineering and Computer Science (EECS); and Sirui Li, an IDSS college student. The research study will exist at the Conference on Neural Information Processing Systems.

Finding a middle ground

To train an algorithm to manage traffic lights at numerous crossways in a city, an engineer would normally select between 2 primary approaches. She can train one algorithm for each crossway separately, utilizing just that intersection’s data, or train a larger algorithm using information from all crossways and after that use it to each one.

But each technique features its share of drawbacks. Training a different algorithm for each task (such as an offered intersection) is a lengthy process that requires a massive quantity of information and computation, while training one algorithm for all tasks typically leads to substandard efficiency.

Wu and her collaborators sought a sweet area in between these two techniques.

For their method, they select a subset of jobs and train one algorithm for each job individually. Importantly, they strategically select specific jobs which are more than likely to enhance the algorithm’s overall performance on all tasks.

They leverage a typical trick from the support learning field called zero-shot transfer knowing, in which an already trained design is used to a brand-new task without being further trained. With transfer knowing, the model frequently performs incredibly well on the brand-new next-door neighbor task.

“We know it would be perfect to train on all the tasks, but we wondered if we could get away with training on a subset of those tasks, apply the result to all the jobs, and still see an efficiency increase,” Wu states.

To recognize which jobs they need to pick to take full advantage of predicted efficiency, the scientists developed an algorithm called Model-Based Transfer Learning (MBTL).

The MBTL algorithm has two pieces. For one, it models how well each algorithm would perform if it were trained individually on one job. Then it designs just how much each algorithm’s efficiency would degrade if it were transferred to each other job, a concept called generalization efficiency.

Explicitly modeling generalization efficiency allows MBTL to estimate the value of training on a new job.

MBTL does this sequentially, selecting the job which leads to the greatest efficiency gain first, then choosing additional tasks that offer the biggest subsequent marginal improvements to general efficiency.

Since MBTL just concentrates on the most promising jobs, it can considerably enhance the performance of the training process.

Reducing training costs

When the scientists evaluated this strategy on simulated tasks, including controlling traffic signals, handling real-time speed advisories, and carrying out numerous classic control jobs, it was 5 to 50 times more efficient than other techniques.

This suggests they might reach the very same option by training on far less information. For example, with a 50x efficiency boost, the MBTL algorithm could train on just 2 tasks and attain the exact same efficiency as a basic approach which uses information from 100 tasks.

“From the point of view of the 2 main methods, that indicates information from the other 98 tasks was not required or that training on all 100 tasks is puzzling to the algorithm, so the efficiency winds up even worse than ours,” Wu states.

With MBTL, adding even a small quantity of additional training time could lead to far better performance.

In the future, the researchers plan to design MBTL algorithms that can encompass more complex issues, such as spaces. They are likewise interested in applying their approach to real-world issues, especially in next-generation movement systems.