Prefix Tuning for Audio Captioning: Teaching GPT-2 to Describe Sound Without Retraining It

If you show an AI model a photo, it can now write a decent sentence describing it. There’s a lot of training data to make that possible. Do the same thing with a short audio clip, “a dog barks as a car drives past,” and the AI has far less to learn from. In a recent Passion Academy session, we looked at a paper from POSTECH and Yonsei University that gets around this shortage with a clever trick: instead of teaching a language model to describe sound, it teaches a much smaller system to translate sound into something a language model already understands.

‍

‍

What Is Automated Audio Captioning?

‍

Automated audio captioning (AAC) describes an audio clip in natural language: which sounds occur, what makes them and in what order.

‍

Audio roughly splits into three categories:

‍

Environmental sounds (“a dog barks as a car passes by”)
Speech (“a man speaks loudly”)
Music (“a piano plays a slow melody”).

‍

This paper focuses specifically on environmental sounds. This is the category where the task is simply naming the acoustic source and sequence of events (rather than transcribing words),.

‍

‍

Why This Is Harder Than It Sounds

‍

The reason this is a genuinely hard problem isn’t that audio is more complicated than images. It’s that there’s so much less training material to learn from.

‍

Here’s a comparison. COCO, the dataset most image-captioning models train on, has 413,915 captions. Clotho, one of the main datasets for audio captioning, has 14,465. That’s a 29-fold difference. AudioCaps, the other major audio dataset, has 38,118.

‍

If you tried to train a caption-writing model from scratch on a dataset that small, the usual thing happens: it memorises the quirks of its own training examples instead of learning something that generalises. Show it a new recording it hasn’t seen anything like before and it falls apart. So the real challenge isn’t “how do we understand audio”, it’s “how do we build a good writer without enough examples to teach one from scratch.”

‍

‍

The Idea: Don’t Train a Language Model, Borrow One

‍

GPT-2 already knows how to write fluent, grammatical English, it learned that from a huge amount of text on the internet, long before anyone cared about audio captioning. What it doesn’t know is how to listen...

‍

So instead of training a new writer from the small pile of audio-caption examples, the paper keeps GPT-2 exactly as it is and never touches its underlying writing ability at all. What it trains instead is a small, separate system whose only job is to turn a sound into something that looks (from GPT-2’s point of view) like a normal sentence it’s already halfway through reading.

‍

Concretely, that means three things happen:

‍

GPT-2 is frozen: Its internal settings never change during training. AKA It still writes exactly the way it always has.
‍A small audio system is trained from scratch: This is the only part of the whole setup that actually learns anything new. Its job is to listen to the clip and pull out what’s relevant.‍
A translator sits in between: This is the clever part, it takes what the audio system picked up and turns it into a handful of “words” (not real words, but things shaped exactly like the word-tokens GPT-2 reads) and hands those to GPT-2 as if they were the start of a sentence. GPT-2 then just... keeps writing, the same way it always would.

‍

‍

‍

What Is a Log-Mel Spectrogram?

‍

Before any of this can happen, the raw sound file needs to become something a computer can actually look at. The way this is done is to turn it into what’s called a log-mel spectrogram, which is really just a picture of sound. Time runs left to right. Pitch runs bottom to top. Brightness shows how loud that pitch is at that moment. It looks a bit like a heat map, but for sound instead of temperature. This image, not the raw waveform, is what the system actually examines.

‍

‍

The Audio Encoder: Two Views of the Same Sound

‍

Once that picture exists, a specialised audio model (itself already pre-trained, on a large dataset of labelled sounds called AudioSet) looks at it in two different ways at once:

‍

what is this clip about, overall?: This gives one summary of the whole thing, something like “a busy street with traffic.” This is the global view.
what’s happening, moment by moment?: This keeps track of the order things occur in, so the system can tell the difference between “a dog barks, then a car passes” and “a car passes, then a dog barks.” Call this the temporal view.

‍

‍

Mapping Networks: Translating Audio Into Prefixes

‍

This is the part doing the real work of connecting two very different worlds. Think of it like this: a handful of empty “slots” are handed the audio information and each slot’s job is to pay attention to the sound and soak up whatever seems relevant to it. Once that’s done, the original audio data is thrown away, only the slots, now filled with something useful, are kept. Those filled slots are what get handed to GPT-2.

A few practical details about this translation step matter:

It always produces the same amount of output: A five-second clip and a thirty-second clip both end up as exactly 26 of these “fake word” slots, 11 from the global view and 15 from the temporal view. GPT-2 never has to deal with a variable amount of input.
It genuinely looks like real input to GPT-2: Each of these slots is shaped exactly like a normal word-piece inside GPT-2, so the model doesn’t need any special handling, it just reads them the way it reads anything else.
The global and temporal views get their own, separate translators: Only the one handling the temporal view keeps track of time order; the other is just summarising.

‍

‍

Training: One Frozen Model, One Simple Objective

‍

The way this gets trained is refreshingly simple: the system tries to guess the next word of a real, human-written caption, gets corrected when it’s wrong and adjusts. The twist is in which part of the system is allowed to adjust. The audio listener and the translator both learn from this process. GPT-2 itself never updates, its writing ability was already there before training started, and stays exactly as it was afterward.

‍

‍

Datasets and Three Test Settings

‍

Tested against six previous approaches on the standard benchmarks, this method came out ahead on 8 out of 9 scoring measures on one dataset (Clotho) and 5 out of 9 on the other (AudioCaps). That’s a genuine improvement given how little new training data the system actually needed.

‍

The researchers also tested something more interesting than just “does it do well on the dataset it trained on.” They tried training on one dataset and testing on a completely different one, and training on both combined and testing on each separately, essentially checking whether the model actually understood something general about sound, or whether it had just memorised patterns specific to one dataset. This is where keeping GPT-2 frozen paid off the most: a version of this system that had instead trained everything, including the language model, together from scratch did noticeably worse on recordings unlike anything in its training set. Freezing the parts that didn’t need to change turned out to be exactly what let the system handle genuinely new audio well.

‍

Evaluation used two standard AAC datasets. AudioCaps contains roughly 50,000 short clips (10 seconds each) sourced from YouTube. Clotho contains roughly 5,000 clips (15–30 seconds each) from Freesound, each with 5 captions. Generated captions are scored against these human-written references, with SPIDEr as the main metric (higher is better).

‍

The paper tested three different settings to understand not just performance, but generalisation:

‍

Same dataset: train and test on the same dataset. This is the usual benchmark test.
Swap datasets: train on one dataset, test on the other, checking whether the model still works on recordings it’s never seen anything like during training.
All data: train on both datasets combined, test on each separately, checking whether a single model can handle both distributions at once.

‍

‍

Results: Beating the Previous State of the Art

‍

In the standard same-dataset setting, compared against six previous methods, this approach beat all prior methods on 8 out of 9 metrics on Clotho, and 5 out of 9 metrics on AudioCaps. That’s a meaningful improvement given how little trainable audio-text data was available to work with, and it’s a direct result of the frozen language model doing the heavy lifting on actually producing coherent, well-formed English.

‍

‍

What to Remember

‍

Three takeaways from the session capture the core argument of the paper.

‍

A frozen LM means better generalisation. Keeping GPT-2 fixed, and learning only how to map audio into its prefix vocabulary, outperformed jointly-trained baselines specifically when tested out-of-domain, that is, in the swap and combined-dataset settings where the model has to handle recordings unlike its original training distribution.

‍

Prefixes are what bridges the two modalities. The mapping networks are the crucial component in the whole system, and the global and temporal features are genuinely complementary: using both consistently captured more of what’s actually happening in a clip than either feature alone could.

‍

Little data is enough. Because only the audio side of the system is ever trained, even a comparatively small audio-text dataset is sufficient to produce rich, accurate captions. The language modelling itself, the hardest part of producing fluent, grammatical English, is entirely inherited for free from GPT-2’s original pretraining.

‍

The Broader Picture

‍

This paper is a clean example of a pattern that shows up again and again when a target domain has too little labelled data to train a full generative model from scratch: rather than fighting the data shortage directly, borrow a capability that already exists elsewhere, and train only the small connecting piece that’s actually domain-specific. GPT-2 already knew how to produce fluent English; what it didn’t know was how to listen. Prefix tuning teaches it to listen without ever touching what it already knew how to say.

‍

References

‍

Kim et al., “Prefix Tuning for Automated Audio Captioning,” ICASSP 2023 (POSTECH & Yonsei University)

‍

< back to academy

< previous

Next >