Exploring the Intersection of Robots and LLMs for Numerical Optimisation

Large Language Models (LLMs) like Gemini and GPT aren’t just text generators anymore. Researchers are exploring how they can be used for numerical optimisation and even for controlling robots with plain language instructions. It’s a surprising crossover of math, language, and robotics.

‍

#1 LLMs as Optimisers

Optimisation is at the heart of machine learning e.g. adjusting parameters to minimise a loss function. Traditional methods like gradient descent have been used for decades.

‍

But what if an LLM could take on this role?

‍

In a recent DeepMind paper, researchers asked Gemini 1.5 to perform optimisation tasks. Given problem descriptions, sample values, and previous attempts, the LLM had to propose the next candidate solution step by step.

‍

The surprising result: Gemini outperformed standard algorithms in low dimensions (2–3) and remained competitive even in 8 dimensions, all while providing reasoning for each choice.

‍

#2 Robots That Follow Language Prompts

‍

The paper also explored a more playful domain: robot arms trained to play table tennis.

‍

Normally, these robots learn one basic behaviour, hit the ball back. But with LLM-driven optimisation, researchers asked:

‍

Could a robot adapt to natural language commands like “hit the ball as high as possible” or “send it far to the right”?

‍

Here’s how it worked:

‍

• The robot’s behaviour is nudged by a small set of control parameters (not its entire neural policy).
• The LLM receives the problem description and previous execution traces.
• It summarises the data, identifies which parameters correlated with success, and proposes new values.
• Over iterations, the robot improves at following the human’s instruction.

‍

The result? The ball trajectory shifted closer and closer to the desired behaviour, without rewriting the policy or retraining from scratch.

‍

Why This Is Exciting

‍

• Language as control: Instead of designing complex reward functions, humans can give plain instructions.
• Transparent reasoning: At each step, the LLM explains its logic, offering insights into decision-making.
• Hybrid approaches: While toy examples for now, this line of work hints at future systems where optimisation, robotics, and language are tightly integrated.

‍

Of course, open questions remain. Gradient descent is cheap and reliable, so why use a billion-parameter LLM? For now, the answer lies in flexibility and reasoning. But the potential for human–robot collaboration through natural language is hard to ignore.

‍

Looking Ahead

‍

This research is early-stage, but it shows the expanding scope of LLMs. Beyond chatbots, they may soon become active participants in optimisation, robotics, and control systems, bridging the gap between language and action. This means continuing to create AI that listens, reasons, and adapts in the physical world.

‍