Mapping meaning: what embeddings reveal and what they hide
A customer searches for my parcel has not arrived, while the help page calls the problem delivery delay. A search based only on matching words may miss the connection. An embedding model can help by representing the query and the passages as lists of numbers, then comparing those lists to find related text.
That ordered list of numbers is called a vector. Each position in the list is a dimension. In a small illustration, we might choose one dimension for delivery topics and another for refund topics. A trained model’s numbers are more complicated: they are learned for its task, and a single dimension usually does not have an obvious label that a person can read. The vector is a useful representation for calculation, not a dictionary definition of the passage.
A sentence embedding gives a sentence or passage a vector that can be compared with other vectors. Some embedding models are trained to retrieve relevant documents for a query; others focus on comparing sentence meanings. Related objectives do not make the models interchangeable. The question is whether the representation helps find the right evidence for the actual searches people make.
There is another distinction inside language models. A token is a piece of text in the model’s vocabulary, such as a word or part of a word. A static token embedding is the initial list of numbers looked up for that token. After the model processes the sequence, it produces a contextual representation that also reflects the surrounding text. The word bank can begin with the same lookup vector and then acquire different contextual representations beside river or loan. A chart of isolated word vectors does not, by itself, show that later processing.
One common way to compare vectors is cosine similarity. Imagine the vectors as arrows starting at the same point. Cosine similarity compares the directions of those arrows, rather than just their lengths. For the chosen lists [1, 0] and [1, 1], the calculation gives about 0.707. That is a result for those two lists, not a measured similarity between real sentences. A similarity of 0.707 is also not a 70.7% probability that two passages mean the same thing.
Topic similarity and agreement are different. A policy allowing refunds within thirty days and one refusing refunds after thirty days both discuss refunds and time limits. They may be useful neighbors in a search result, but the words, dates and exceptions still decide what each passage supports. Include such distinctions when reviewing search results. Finding related evidence is a step toward answering a question, not proof that the answer is correct.
Embedding vectors can have many dimensions, so a chart usually compresses them into two positions on a screen. That compression is called a projection. It necessarily leaves information out, and different drawing methods preserve different relationships. Two points that look close on the chart may be less close in the original vectors. The example below deliberately drops an important distinction so you can compare the calculation with the picture.
Loading the interactive example.
Compare a full vector with its picture
For this example, we choose the numbers ourselves rather than asking an embedding model to produce them. Each fictional support request has a list of six values, in this order: cancel, delivery, return or refund, address, login and negation. Negation records wording such as not that reverses an instruction. Cancel my order has [1, 0, 0, 0, 0, 0]. Do not cancel my order adds a 1 in the last position. These named positions make the example easy to inspect; the dimensions of a trained embedding usually do not have such convenient labels.
Select Do not cancel my order, then find Cancel my order in the results. The direction similarity, calculated using all six values, is about 0.707. Now compare their numbered points on the map. The points sit in exactly the same place because the drawing ignores the last value, negation. A distinction that matters to the instruction has disappeared from the picture.
Nearest requests in six dimensions
A map that leaves information out
On narrow screens, scroll the map horizontally to inspect all numbered points.
The numbers on the map identify the requests in the menu. The selected request has a green point with a dark ring. Several requests can share a position. To keep the drawing simple, we use cancellation and delivery values to set the horizontal position, and return, address and login values to set the vertical position. Negation is deliberately left out. The full similarity calculation still uses all six values.
Inspect the example vectors
A trained model's static token embedding and its contextual representation are different objects. Neither appears here. A two-dimensional projection cannot show all original relationships or reveal a model's thoughts.
The shared point does not mean the two instructions are interchangeable. It means the drawing left out the number that distinguishes them. In a real embedding map, the lost information is usually harder to name so neatly. Treat the picture as a way to ask questions about the data, then check those questions using the original vectors and text. It is not a view of the model’s thoughts or proof that the map’s clusters are distinct concepts.
For a search system, collect examples of what users ask and have someone who knows the documents identify passages that would help answer them. Compare ordinary keyword search with embedding search and a combination of both. Look at whether the useful passages appear near the top, whether important exceptions remain visible and whether irrelevant passages push better evidence down the list. A small collection of difficult, reviewed searches can be more useful than a visually tidy map.
Keep the information around each vector too: who may read the document, when it was written and where the original passage can be found. Apply access restrictions before releasing search results. If you change the embedding model, rebuild the stored vectors deliberately rather than mixing representations from different models without checking compatibility. Repeat the reviewed searches after the change. A model that improves broad topic matching may still lose an exact policy exception that the original system found.
Further reading
Sentence Transformers: Semantic textual similarity