Automatic Design of Agentic Systems

‍

Every time AI has hit a ceiling, the response has followed the same pattern.

‍

1. Hand-craft a solution
2. Learn what can be automated
3. Hand-craft the automation
4. Learn what can be automated from that
5. Repeat...

‍

We did it with features, then with architectures, then with training pipelines. In a recent Passion Academy session, our Chief Scientific Officer Nadine Kroher asked whether we're about to do it again, this time with agentic systems themselves.

‍

‍

Slides

‍

‍

‍

‍

‍

We're Still in the Hand-Craft Stage

‍

Today's multi-agent systems are built almost entirely by hand.

‍

Developers decide:

• How many agents to use
• What roles they play
• How they communicate
• What prompts they receive
• How outputs get combined

‍

Every design choice is tweaked manually until the system performs well enough, then, it gets shipped.

‍

This is exactly how early machine learning worked before we learned to automate feature engineering. And it's exactly how neural network design worked before neural architecture search. The question ADAS (Automatic Design of Agentic Systems) asks is a simple one: can we automate the design of agentic systems too? Not just the prompts, the whole architecture.

‍

‍

How Meta Agent Search Works

‍

The methodology at the heart of ADAS is called Meta Agent Search. The idea is to use an LLM (in this case GPT-4) as a meta agent whose job is not to solve tasks directly, but to design other agents that can.

‍

The process works as a loop. The meta agent looks at an archive of previously generated agent designs along with their performance results. It uses this as context to propose a new, interesting agent design. This is complete with a high-level description, a name and working code built on a lightweight framework. That new agent is then tested on a benchmark. If it produces errors, a self-reflection loop kicks in to fix them. If it performs well, it gets added to the archive. The cycle repeats.

‍

The archive starts simple and gets progressively richer as better designs accumulate. Each new agent is informed by everything that came before it.

‍

‍

What It Actually Discovers

‍

The system was evaluated on the ARC Challenge: a benchmark designed to test abstract reasoning by showing input-output grid examples and asking the model to infer the transformation rule and apply it to a new grid. It's a task that requires genuine pattern recognition rather than memorisation.

‍

What the meta agent discovered across iterations is revealing. Early on it tested high-level strategies before filling in low-level details. It then introduced dynamic memory to enable more refinements. It scaled successful ideas and eventually it converged on the best-performing design: a structured feedback and ensemble agent that generates five candidate answers using chain-of-thought reasoning. It passes them through three specialist experts (an efficiency expert, a readability expert and a simplicity expert alongside a human-like critic) refines answers three times based on combined feedback and selects the top answer from the final ensemble.

‍

Nobody designed that architecture, the system found it by iterating over what worked and building on it. Performance increased steadily across iterations, eventually surpassing all hand-crafted baselines including chain-of-thought, self-refine and LLM debate.

‍

‍

Where It Falls Short

‍

The tasks used for evaluation are relatively clean and well-defined. They don't reflect the messiness of real-world deployments (no web searches, no database calls, no retrieval systems and no ambiguous inputs). For ADAS to work, you need a large, unambiguous benchmark dataset to evaluate against. That exists for ARC but it doesn't exist for most business problems.

‍

There's also the deeper question of what "interesting" means when the meta agent is prompted to propose new designs. The system is exploring a search space, but that space is defined by what the LLM can imagine and express as code. This is in itself shaped by what humans have already built and written about. It's not clear how far outside existing patterns it can genuinely reach.

‍

‍

The Broader Picture

‍

The current results are proof of concept rather than production-ready technology. Despite this, But the direction is clear. As the benchmark coverage improves and the search process matures, the gap between what humans design by hand and what a meta agent can discover is likely to narrow. Whether that's exciting or unsettling probably depends on how much of your current work involves designing agentic systems by hand.

‍

References

‍

Hu, Shengran, Cong Lu, and Jeff Clune. "Automated Design of Agentic Systems." International Conference on Learning Representations, 2025 (Source).