Semantic Similarity: Word Embeddings

‍

In 2021, one of our engineers built a semantic matching tool for a car insurer. The problem was simple to describe: a user types their job title in free text ("Senior Product Manager at a Fintech Startup" ) and the system needs to automatically map it to the closest match in a predefined list of around 2,000 occupations. ChatGPT didn't exist yet. The state of the art looked nothing like it does today.

‍

So why are we revisiting five-year-old work? Because the fundamentals haven't moved, specialised systems are starting to consistently outperform general-purpose LLMs on narrow tasks. Furthermore, evaluation remains the bottleneck most teams underestimate. In a recent Passion Academy session, we traced the full arc of how machines learned to understand meaning and what that tells us about building reliable systems today.

‍

‍

Slides

‍

‍

‍

‍

Teaching Machines What Words Mean

‍

The idea that words can be represented mathematically goes back to 1954, when linguist Zellig Harris proposed what became known as the distributional hypothesis: words that appear in similar contexts tend to have similar meanings. A word isn't defined by what it is, it's defined by the company it keeps.

‍

That idea sat mostly in academia until the tooling caught up. TF-IDF in 1972 gave us a statistical way to weight terms by how distinctive they are across documents. Word2Vec in 2013 turned it into a neural problem: training shallow networks to predict words from their neighbours and producing dense vector representations where similar words cluster together in space. GloVe in 2014 refined this by learning from global co-occurrence statistics rather than local windows alone.

‍

Then in 2017, the Transformer architecture arrived with the paper "Attention is All You Need" and changed everything. BERT in 2018 showed that context matters (i.e. the word "bank" means something different beside "river" than beside "interest rate") and a model should represent it differently in each case. Sentence-BERT in 2019 extended this to full sentences, making it practical to compare the meaning of entire phrases rather than individual words.

‍

That's the lineage that made the 2021 job-matching tool possible and it's the same lineage underpinning most semantic search, recommendation and classification systems today.

‍

‍

Why Specialised Systems Still Win

‍

There's a tempting assumption that as general-purpose LLMs get more powerful, the need for purpose-built systems disappears. The session pushed back on this directly.

‍

For narrow, well-defined tasks, a fine-tuned small model will consistently outperform a GPT-5-class system on cost, latency and accuracy. The reasons are intuitive once you think about it. A large general model has to balance an enormous range of capabilities. A small model trained specifically on your data, your vocabulary and your task has nothing else to worry about.

‍

‍

Evaluation Is the Real Bottleneck

‍

Building a semantic similarity system is not the hard part. Knowing whether it actually works on your data, for your users, in your context is. Most teams invest heavily in model selection and architecture decisions and underinvest badly in evaluation. They ship something that performs well on a benchmark and then discover it behaves unexpectedly in production because the benchmark didn't reflect the real distribution of inputs.

‍

‍

The Broader Picture

‍

Semantic similarity is one of the most quietly foundational capabilities in modern AI. It underlies search, recommendation, classification, deduplication, question answering and more. The surface-level technology has changed dramatically since 1954. The core idea (that meaning lives in context and context can be measured) has not.

‍

As more decisions get delegated to these systems, the cost of getting similarity wrong goes up. A job title misclassified on an insurance form might seem trivial. The same failure pattern in a hiring tool, a content moderation system or a medical record classifier is not. The fundamentals matter more now.

‍

‍

References

‍

• Harris, Z. S. (1954). Distributional Structure. Word, 10(2–3), 146–162.
• Jones, K. S. (1972). A Statistical Interpretation of Term Specificity and Its Application in Retrieval. Journal of Documentation, 28(1).
• Mikolov, T., et al. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781.
• Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global Vectors for Word Representation. EMNLP.
• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS.
• Devlin, J., et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805.
• Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. EMNLP.
• Dusserre, M., & Padró, L. (2017). Bigger Does Not Mean Better! We Prefer Specificity. IWCS.
• Chandrasekaran, M., & Mago, V. (2021). Evolution of Semantic Similarity: A Survey. ACM Computing Surveys, 54(2).
• Office for National Statistics. (2020). Standard Occupational Classification (SOC) 2020, Volume 1.

‍

‍