Reference: Full blog post

There has been a lot of hype about the promise of autonomous agent-based LLM workflows. By now, all major LLMs are capable of interacting with external tools and functions, letting the LLM perform sequences of tasks automatically.

But reality is proving more challenging than anticipated.

The WebArena leaderboard, which benchmarks LLMs agents against real-world tasks, shows that even the best-performing models have a success rate of only 35.8%.

Challenges in Practice

After seeing many attempts to AI agents, I believe it's too early, too expensive, too slow, too unreliable.
It feels like many AI agent startups are waiting for a model breakthrough that will start the race to productize agents.

• Reliability: As we all know, LLMs are prone to hallucinations and inconsistencies. Chaining multiple AI steps compounds these issues, especially for tasks requiring exact outputs.
• Performance and costs: GPT-4o, Gemini-1.5, and Claude Opus are working quite well with tool usage/function calling, but they are still slow and expensive, particularly if you need to do loops and automatic retries.
• Legal concerns: Companies may be held liable for the mistakes of their agents. A recent example is Air Canada being ordered to pay a customer who was misled by the airline's chatbot.
• User trust: The "black box" nature of AI agents and stories like the above makes it hard for users to understand and trust their outputs. Gaining user trust for sensitive tasks involving payments or personal information will be hard (paying bills, shopping, etc.).

Real-World Attempts

Several startups are tackling the AI agent space, but most are still experimental or invite-only:

• adept.ai - $350M funding, but access is still very limited
• MultiOn - funding unknown, their API-first approach seems promising
• HypeWrite - $2.8M funding, started with an AI writing assistant and expanded into the agent space
• minion.ai - created some initial buzz but has gone quiet now, waitlist only

Only MultiOn seems to be pursuing the "give it instructions and watch it go" approach, which is more in line with the promise of AI agents.
All others are going down the record-and-replay RPA route, which may be necessary for reliability at this stage.

Large players are also bringing AI capabilities to desktops and browsers, and it looks like we'll get native AI integrations on a system level:

• OpenAI announced their Mac desktop app that can interact with the OS screen.
• At Google I/O, Google demonstrated Gemini automatically processing a shopping return.
• Microsoft announced Copilot Studio, which will let developers build AI agent bots.

Screenshot Screenshot

These tech demos are impressive, but we'll see how well these agent capabilities will work when released publicly and tested against real-world scenarios instead of hand-picked demo cases.

The Path Forward

AI agents overhyped and it's too early.
However, the underlying models continue to advance quickly, and we can expect to see more successful real-world applications.
Instead of trying to have one large general purpose agent that is hard to control and test, we can use many smaller agents that basically just pick the right strategy for a specific sub-task in our workflows. These "agents" can be thought of as medium-sized LLM prompts with a) context and b) a set of functions available to call.

The most promising path forward likely looks like this:

1. Narrowly scoped, well testable automations that use AI as an augmentation tool rather than pursuing full autonomy
2. Human-in-the-loop approaches that keep humans involved for oversight and handling edge cases
3. Setting realistic expectations about current capabilities and limitations

By combining tightly constrained agents, good evaluation data, human-in-the-loop oversight, and traditional engineering methods, we can achieve reliably good results for automating medium-complex tasks.

Will AI agents automate tedious repetitive work, such as web scraping, form filling, and data entry? Yes, absolutely.

Will AI agents autonomously book your vacation without your intervention? Unlikely, at least in the near future.