Engineering
Six years of three different people naming things three different ways. What actually works, and what only works in benchmarks.
Every operating catalogue we have worked on has the same archaeology. An original naming scheme, a second one introduced when a new hire took over, a third that arrived with a marketplace integration, and a layer of emergency edits made during a stock-out in 2023. Nobody is going to clean it. The system has to work anyway.
Pure vector search does well on natural language and badly on the things operations queries are actually made of: SKUs, model numbers, dimensions, pack sizes. "SKU-4471-B" and "SKU-4471-D" are nearly identical in embedding space and are different products, sometimes at different prices.
Hybrid retrieval — lexical BM25 alongside dense vectors, fused with reciprocal rank — is unglamorous and reliably better on this kind of corpus. In our benchmarks on real client catalogues it beats dense-only by 12–20 points on top-5 recall, and the gap is widest exactly where the money is: variant-level disambiguation.
Fixed-size chunking splits a sizing table across two chunks and destroys it. Catalogue and policy data has structure — product, variant, attribute group, clause — and chunking on those boundaries preserves the thing that makes a chunk answerable.
For supplier terms and returns policy we chunk per clause and carry the parent heading into each chunk's text. It costs a little duplication and it means a retrieved clause arrives knowing which policy and which market it belongs to, which is the difference between a citable answer and a confident guess.
The instinct is to clean the catalogue on the way in. It fails, because the mess keeps arriving and because you have now got two sources of truth that drift apart.
We normalise on both sides at query time instead: units, pack notation, colour synonyms and common misspellings expanded into a query variant set, matched against a lightly normalised index. The original record is never modified. When the client eventually does a data-quality programme, nothing we built has to be redone.
Most retrieval failures in operations are not ranking failures. They are scope failures — the right answer was for a different market, a discontinued variant, or last season's policy.
Filtering hard on market, status, effective date and channel before ranking removes a category of error that no amount of reranking fixes. It is boring, it is a WHERE clause, and it is responsible for more accuracy in our deployments than any model upgrade we have made.
A cross-encoder reranker over the top 50 candidates helps on ambiguous natural-language queries and does very little on SKU lookups. Since it adds latency and cost per query, we route: precise identifier queries skip it, descriptive queries get it.
That routing decision comes out of the evaluation set, not from a preference. On one deployment reranking everything would have added 180ms and 22% cost for a 1.4-point gain — real, but not worth it on that queue.
Every response carries the record IDs and clause references it used. Two consequences follow, and both matter more than the accuracy number.
First, staff can verify in seconds instead of re-deriving the answer, which is what makes them actually adopt it. Second, when the answer is wrong you can see immediately whether it was a retrieval failure or a reasoning failure — and those have completely different fixes. Systems without citations force you to debug by guessing.
None of this is novel and that is rather the recommendation. Retrieval over real operational data is won with careful, dull engineering, and the teams that go looking for a clever technique usually end up back at this list a quarter later.
Keep reading
Recognise the process?
Bring one process that is eating your team's week. We will tell you on the call whether it is a good candidate — and we say no to roughly a third of what we are asked about.
