Navigating by Sound: Teaching AI Agents to Find a Voice Using Reinforcement Learning

A person can walk into a crowded room and usually work out where a particular voice is coming from. We use differences in volume, the direction of sound and the way voices change as we move. But what happens when an AI agent has to do the same thing without being able to see anything?

‍

In a recent Passion Academy session, we explored a research paper that uses deep reinforcement learning to teach an agent to navigate towards a particular speaker using only stereo audio. We then took the research further, rebuilding the environment and testing what happens when the agent encounters obstacles, more speakers and increasingly complex rooms.

‍

The results offer an interesting look at how reinforcement learning can teach an agent to use sound as a way of understanding and interacting with its surroundings.

‍

‍

Slides

‍

‍

‍

The Research: Finding One Voice in a Room

‍

The original research, A Deep Reinforcement Learning Approach to Audio Based Navigation in a Multi Speaker Environment, by P. Giannakopoulos, A. Pikrakis and Y. Cotronis, was presented at IEEE ICASSP in 2021.

‍

The task sounds relatively simple. An agent is placed in a two dimensional room with three stationary speakers, each playing different audio recordings. One is the target speaker, while the other two act as distractions.

‍

The agent has three objectives:

‍

• Find and reach the target speaker.
• Avoid the other two speakers.
• Stay within the boundaries of the room.

‍

There is one important restriction: the agent can only use raw stereo audio to navigate. It has no camera, no visual information and no access to the positions of the speakers.

‍

This is different from many existing approaches to audio based navigation, which combine sound with visual information or use separate systems to identify and locate sound sources.

‍

Here, speaker detection and navigation are learned together. The agent has to work out how to use the audio to decide where to move, without being given an explicit location for any of the speakers.

‍

‍

How Does the Agent Learn to Navigate?

‍

The researchers use deep reinforcement learning, a type of machine learning in which an agent learns by interacting with an environment and receiving feedback on its actions. Rather than being given a set of instructions telling it how to reach the target, the agent tries different movements and learns which decisions lead to success.

‍

The task is modelled as a Markov Decision Process (MDP), which provides a framework for describing how an agent observes its environment, takes an action and receives a reward.

‍

There are three main components.

‍

1. State: What the Agent Hears

‍

• The agent's input is raw audio from two channels: left and right.
• At each decision step, it receives approximately 21 milliseconds of audio, sampled at 48 kHz.
• The two channels are combined into a single input containing 2,048 values.

‍

Importantly, the agent does not receive a processed description of the sound. It is not told that a speaker is to its left or that another is getting closer. Instead, it has to learn useful patterns directly from the audio waveform.

‍

One of the main spatial clues is volume. As the agent moves towards a speaker, the sound becomes louder. As it moves away, the sound becomes quieter. The researchers use a simplified linear relationship between volume and distance to provide this information.

‍

2. Actions: Where the Agent Moves

‍

The agent controls its movement along two axes, horizontal and vertical. At each step, it outputs two continuous values representing its velocity in the X and Y directions. These values determine how it moves through the room. Its orientation is fixed, meaning it cannot turn its head or rotate to listen in a different direction. This makes the task particularly interesting. The agent has to learn to navigate using changes in the sound it receives as it moves, rather than actively turning towards a sound source.

‍

3. Rewards: Learning What Works

‍

The agent receives feedback based on what happens during each attempt.

‍

The reward system is relatively simple:

• Reaching the target speaker: +1.0
• Touching a distractor speaker: −1.0
• Leaving the room: −1.0
• Every movement step: −0.001

‍

An attempt ends when the agent reaches a speaker or leaves the room. The small penalty for every step encourages the agent to reach its target efficiently rather than wandering around indefinitely. Over repeated attempts, the agent learns which movements are more likely to lead to a positive outcome.

‍

‍

The Model Behind the Agent

‍

The researchers use Proximal Policy Optimisation (PPO), a popular reinforcement learning algorithm that helps agents learn effective strategies while keeping updates to their behaviour relatively stable.

‍

The neural network itself is fairly small. It takes the 2,048 audio values as input, processes them through two fully connected layers of 256 neurons each, and produces two outputs representing the agent's horizontal and vertical movement. The important point is that the network learns a direct relationship between what the agent hears and how it should move.

‍

There is no separate speaker identification model or sound localisation system telling it where to go. The agent learns the navigation behaviour through interaction with the environment.

‍

‍

How Well Does It Work?

‍

The original paper reports some impressive results. The researchers trained the agent using recordings from three speakers, two male and one female. They used 500 training utterances and 100 test utterances per speaker, with the recordings divided using voice activity detection (VAD) to separate speech from silence. The agent was trained for six million steps and evaluated across 100 test episodes for each target speaker, using previously unseen utterances.

‍

The trained agent successfully reached its target in 98% of test episodes, compared with just 15% for a random walking agent.

‍

The researchers also tested how well the agent generalised to changes in the audio. When the pitch of the speakers was shifted by 4 to 8%, the success rate remained at 93%.

‍

This is important because it suggests that the agent was not simply memorising individual recordings. It had learned patterns that allowed it to navigate towards speakers even when the test audio differed from what it had encountered during training.

‍

However, the original experiment takes place in a relatively controlled environment: an empty rectangular room, stationary speakers and simplified sound behaviour. That raises a more interesting question. What happens when the environment becomes less predictable?

‍

‍

Beyond the Paper: Our Own Experiments

‍

For the second part of the session, we rebuilt the paper's environment and agent in Python, keeping the PPO training approach and original reward structure as our starting point.

‍

We then explored two of the limitations identified by the original researchers: navigating around obstacles and dealing with more speakers in larger environments. This is where the experiments became particularly interesting. We found that an agent that performs extremely well in a simple room does not necessarily know how to handle a more complicated one.

‍

‍

Experiment 1: What Happens When an Obstacle Blocks the Sound?

‍

We first introduced a silent box into the room. The box does not make any sound and does not block the speakers. However, it can sit directly between the agent and its target, making the target's voice weaker as the agent approaches. The agent now has a new problem. It needs to recognise that the target is nearby but that moving directly towards it is not necessarily the right decision. It needs to find a way around the obstacle, using LiDAR to detect its surroundings and audio to identify the target.

‍

Giving the Agent LiDAR

‍

LiDAR is a sensing technology that measures distances by sending out laser pulses and measuring how long they take to return. It is commonly used in robotics to help machines understand the position of nearby objects. We added a two dimensional LiDAR sensor that sends out 64 rays across a full 360 degrees, measuring the distance to the nearest wall or obstacle up to a range of 10 metres.

‍

Even without walls in the room, adding LiDAR improved the agent's success rate from 82 to 86% with audio alone to 92% with audio and LiDAR. It also reduced the frequency of the agent leaving the room from 8% to 1.3%. This demonstrates the value of combining different types of information. Audio helps the agent identify where its target is, while LiDAR gives it information about the physical environment around it. Detecting an obstacle is not the same as knowing how to navigate around it.

‍

The video below shows how it receives information from its environment, combining audio with LiDAR measurements.

‍

‍

‍

‍

‍

The Importance of Memory

‍

One of the biggest challenges was that the box could hide the target voice.

‍

The agent could detect the box using LiDAR, but the sound behind it became too weak to provide a reliable direction. Without any memory of what it had heard previously, it struggled to understand how the situation was changing.

‍

We tested two different ways of handling this. First, we introduced a more realistic volume model, where sound fades with distance in a way that more closely resembles how it behaves in air.Second, we gave the agent a two second audio memory. Rather than relying only on what it hears at the current moment, it can use a history of sound levels across different frequency bands.

‍

This allows it to recognise how the sound changes as it moves, rather than treating every moment as an entirely new situation. The difference was substantial. With no memory, success in the standard one box test was just 3.3% using the realistic volume model. Adding two seconds of sound memory increased this to 79%. The experiment suggests that remembering how the sound changes over time can be much more useful than relying on a single audio snapshot.

‍

Teaching the Agent That Getting Closer Matters

‍

We also found that the original reward system had a limitation. As the agent only received a positive reward when it reached the target, it could spend a long time wandering around without making meaningful progress.

‍

To address this, we introduced a 60 second time limit and a reward for getting closer to the target along the shortest available path around the obstacles. This path was calculated using Dijkstra's algorithm, a method for finding the shortest route through a network of connected points.

‍

The idea was to give the agent more useful feedback during navigation, rather than only rewarding it at the end. This made a significant difference. In the test without memory, adding the closeness reward increased success from 23% to 71.3%, with half the training. With memory, the same reward improved success in the target hiding scenario from 45.3% to 59.4%.

‍

We also experimented with harsher movement penalties and giving additional value to the final metres of a journey. Neither improved performance, showing that simply making the reward system more complicated does not necessarily make an agent better. Our best result in the one box experiments reached 94% success, with the box appearing in random positions.

‍

‍

Experiment 2: More Voices, Bigger Rooms and More Obstacles

‍

Once the agent could handle a single obstacle, we increased the difficulty further by introducing two sets of environments.

‍

• ‍Normal scenes contained a 10 metre room with three speakers and one or two boxes. The target was hidden behind an obstacle in 45% of episodes.‍
• Hard scenes contained larger rooms, between 15 and 20 metres, with four or five speakers and two to four boxes. The target was hidden in 64% of episodes.

‍

We also expanded the range of voices used during training. Instead of relying on the original three speakers, we introduced 45 speakers from the VCTK speech dataset, with five additional unseen speakers used as distractors during testing. The aim was to see whether the agent could learn a more general navigation strategy rather than becoming too specialised in a particular set of voices or room layouts.

‍

Training Different Agents

‍

We compared several versions of the agent. Some were trained from scratch using the original environment, while others were trained with more varied voices and then fine tuned using additional environments (Fine tuning means taking a model that has already learned something and continuing its training on a different or more challenging set of examples).

‍

This allowed us to compare the benefits of training for longer in a familiar environment against introducing more variety into the training process. All agents used the same underlying PPO approach. What changed was the environment they experienced during training and how long they were trained for.

‍

The following video shows how two models perform when faced with more complex environments, including additional obstacles and voices.

‍

‍

‍

What Did We Find?

‍

The results showed that there is a trade off between specialisation and generalisation.

‍

On normal scenes, training for longer on the original environment produced particularly strong results. This suggests that when the task remains relatively familiar, additional training on that specific setup can be very effective.

‍

However, the picture changed when the environments became harder.

‍

Our final agent, trained using varied voices and more challenging scenes, achieved a 61% success rate in the hard environment. This compared with 52% for the next best agent and just 13% for the original paper's agent. The final agent was also tested in environments it had never encountered during training. These super hard scenes included five speakers and three to five large obstacles, with the target hidden in 68% of episodes. The final agent achieved 41.5% success, compared with 38.7% for the next best agent and 8.6% for the original paper's agent.

‍

These results are particularly interesting because they suggest that exposing an agent to more varied training environments can help it handle situations beyond the ones it was originally trained for. However, the results are still preliminary. The experiments used one training seed for the final agents, so further testing across multiple independent training runs would be needed to establish how consistently these improvements hold.

‍

‍

What We Learned

‍

1. More information does not automatically mean better decisions.

‍

Adding LiDAR gave the agent information about obstacles but it still needed to learn how to use that information alongside sound. Combining different inputs is only useful if the agent can learn how they relate to one another.

‍

2. Memory can be just as important as perception.

‍

The two second sound memory made a substantial difference when the target was hidden behind an obstacle. Instead of reacting only to what it could hear at that moment, the agent could use information about how the sound had changed over time.

‍

3. How an agent is rewarded shapes how it behaves.

‍

The original reward system encouraged the agent to reach its target, but did not necessarily teach it how to make progress along the way. Adding a reward for getting closer helped address this. Designing a good reward function is therefore a significant part of building an effective reinforcement learning system.

‍

4. Training variety matters for generalisation.

‍

An agent that performs well in one environment may struggle when the room, obstacles or voices change. Training across more varied conditions helped our final agent perform better in harder environments it had not encountered before.

‍

The Takeaway

‍

Teaching an AI agent to navigate using sound alone is already an interesting challenge. However, the more important question is whether it can continue to make good decisions when the environment changes.

‍

The original research demonstrated that deep reinforcement learning can teach an agent to find a particular speaker with very high accuracy using only stereo audio. Our experiments built on that foundation, showing how memory, additional sensing, reward design and more varied training environments can help extend its capabilities. There is still a considerable gap between a controlled simulation and real world audio navigation. Real rooms introduce echoes, background noise, moving people and unpredictable sound sources, all of which make the problem considerably harder.

‍

Regardless, the broader idea is exciting. Agents do not necessarily need to be given an explicit understanding of their surroundings to learn how to navigate them. Through repeated interaction, they can learn to interpret signals, make decisions and adapt their behaviour.

‍

It is a useful reminder of the potential of reinforcement learning beyond traditional games and simulations, particularly in robotics and other applications where AI systems need to interact with the physical world.

‍

‍

References

‍

• Giannakopoulos, P., Pikrakis, A., and Cotronis, Y. (2021). A Deep Reinforcement Learning Approach to Audio Based Navigation in a Multi Speaker Environment. IEEE ICASSP 2021, pp. 3475–3479. DOI: 10.1109/ICASSP39728.2021.9415013.
• VCTK Corpus: English multi speaker speech dataset used in the extended experiments.
• Experimental implementation: AudioRL on GitHub.

‍