In an effort to make the Internet more accessible for people with disabilities, researchers at The Ohio State University began developing an artificial intelligence agent that could complete complex tasks on any website using simple language commands.
In the three decades since it was first released into the public domain, the world wide web has become an incredibly complex, dynamic system. However, because the Internet is now so integral to the well-being of society, its complexity also makes it significantly more difficult to navigate.
Today there are billions of websites available to help access information or communicate with others, and many tasks on the Internet take more than a dozen steps to complete. That’s why Yu Su, co-author of the study and an assistant professor of computer science and engineering at Ohio State, said their work, which uses information obtained from live websites to create web agents — online artificial intelligence assistants — is a step in the direction of making the digital world less confusing.
“For some people, especially those with disabilities, it’s not easy to browse the internet,” Su said. “We rely more and more on the computing world in our daily lives and work, but there are more and more barriers to that access, which, to some extent, widens the gap.”
The study was presented in December at the Thirty-Seventh Conference on Neural Information Processing Systems (NeurIPS), a leading conference for artificial intelligence and machine learning research.
By harnessing the power of large language models, the agent works similarly to how people behave when browsing the web, Su said. The Ohio State team showed that their model was able to understand the layout and functionality of different websites using only its ability to process and predict language.
The researchers started the process by creating Mind2Web, the first data set for general web agents. Although previous efforts to create web agents focused on game simulation websites, Mind2Web fully embraces the complex and dynamic nature of real-world websites and emphasizes an agent’s ability to generalize to completely new websites that it has never seen before. Su said much of their success is due to their agent’s ability to handle the ever-evolving learning curve of the Internet. The team developed over 2,000 open-source tasks from 137 different real-world sites, which they then used to train the agent.
Some of the tasks included booking one-way and round-trip flights, following celebrity Twitter accounts, browsing comedy movies from 1992 to 2017 via streaming on Netflix, and even scheduling driving tests at the DMV. Many of the tasks were very complex — for example, booking one of the international flights used in the model would require 14 actions. This effortless flexibility allows for varied coverage across a number of websites and opens up a new landscape for future models to explore and learn autonomously, Su said.
“It is only possible to do this because of the recent development of large language models like ChatGPT,” said Su. Since the chatbot went public in November 2022, millions of users have used it to automatically generate content ranging from poetry and jokes to cooking tips and medical diagnoses.
However, because a website could contain thousands of raw HTML elements, it would be very expensive to feed so much information into a single large model language. To address this gap, the study also introduces a framework called MindAct, a two-pronged agent that uses both small and large language models to perform these tasks. The team found that by using this strategy, MindAct significantly outperforms other common modeling strategies and is able to understand various concepts at a decent level.
In more detail, the study notes, the model could likely be used alongside large open- and closed-source language models such as Flan-T5 or GPT-4. However, their work highlights an increasingly important ethical problem in creating flexible artificial intelligence, Su said. While it could certainly serve as an enabler for people surfing the web, the model could also be used to improve systems like ChatGPT and turn the entire Internet into an unprecedentedly powerful tool, Su said.
“On the one hand, we have great potential to improve our efficiency and allow us to focus on the more creative part of our work,” he said. “But on the other hand, there’s a huge potential for harm.” For example, autonomous agents able to translate online steps into the real world could influence society by taking potentially dangerous actions, such as misusing financial information or spreading misinformation.
“We should be extremely cautious about these factors and make a concerted effort to try to mitigate them,” Su said. But as AI research continues to evolve, he notes that society is likely to see significant growth in the commercial use and performance of generalist Web agents in the coming years, especially since the technology has already gained so much popularity with the public.
“Throughout my career, my goal has always been to try to bridge the gap between human users and the computing world,” Su said. “That said, the real value of this tool is that it will really save people time and make the impossible possible.”
The research was supported by the National Science Foundation, the US Army Research Laboratory and the Ohio Supercomputing Center. Other co-authors were Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, and Huan Sun, all at Ohio State.