Instant search: semantic re-rank on a settled query

How it works

The original Jev experiment fired one request per keystroke carrying 35 questions: a five-level relevance score for each of the 30 lexical candidates, plus five query-level questions — intent_category, wants_cheap, wants_premium, is_gift and sort_preference. Here the same judgments are three calls on a settled query, because this page shares one demo key with every other visitor.

Relevance — one /yes-no call, 30 texts

Each candidate becomes one text (title. description Category: … Price $… Rated … stars.) and both statements are built from what you typed:

• This product is the kind of thing a shopper asking for <query> is shopping for.
• This product does everything the request asks for: <query>.

Hints are shared by the batch: when_true “The product's own stated features cover the whole request.”, when_false “The product misses any part of the request, or only shares a word with it.” That second clause matters — every candidate already shares a word with the query by construction, so the question has to be about intent or it adds nothing. The two probabilities become the two bars beside each row. In practice the second one saturates — measured on “rugged phone case not bulky” it read 0.95–1.00 for all 30 candidates, including a laptop bag, while the first ranged 0.07–0.99 — so the relevance score is 0.75 × right kind + 0.25 × does everything asked, and the relevance floor sits at 0.40. A plain mean would have lifted that laptop bag to 0.51 and kept it above the floor. Both bars still show the raw probabilities.

Shopper intent — one /yes-no call, 4 statements

One text (the query) against four short declaratives: wants cheap, wants premium, buying a gift, wants the best-rated. when_true “The words of the search say so.”, when_false “The words of the search do not say so.” The sort preference is not asked at all — it is derived in code from those four probabilities, the same rule the original used.

Department — one /classify call, 6 described labels

A six-way judgment, so it goes to /classify with a description per label: electronics, kitchen, outdoors, toys, office, and the original's any — “The search does not point to one department”. Without any the model had to pick an aisle for every query: “rugged phone case not bulky” came back outdoors at 89% and earned camping gear a bonus on a phone-accessory search. With it, that query reads any at 91% and no department bonus applies. A named department under 60% confidence also changes nothing.

The bonus is deliberately tiny — 0.02, against a relevance term worth up to 1.0. The department a query points at is not always the catalog category the right answer is filed under: “something to keep coffee hot on a hike” classifies as outdoors at 100% because of the hike, while vacuum flasks are filed under kitchen. At the original weight every outdoors water bottle jumped the flasks the model rated higher. It breaks ties inside the aisle; it does not move rows past one another.

Batching and budget

One settled query costs three calls and about 2,500 input tokens, for 65 decisions: 30 candidates × 2 statements, 4 flags and 1 department. Typing alone sends nothing. “Re-rank” sends one group, an example chip types the query out at 70 ms per character and then sends one group, and “Live as you type” debounces 500 ms and stops itself after 60 seconds or 12 dispatches. Answers are cached by query, so re-clicking a chip costs nothing. The simulated-slow baseline still serves its 2.5 s from cache, otherwise it would stop showing the thing it exists to show. The sliders re-rank what is already painted and never send anything.

Reading the metrics strip

The three calls in a group are not the same size, so their latencies are not pooled. The two call tiles describe the 30-candidate re-rank call only; model comes from x-inference-ms, which sums model time over all 60 judgments inside that one call and is therefore larger than its own round trip. to paint is the whole group, measured in the browser from dispatch to the moment the right column renders, and decisions/s is 65 ÷ that. Those are the numbers a visitor actually experiences. All three reset when you switch mode, so one run of the 2.5 s baseline cannot sit in decision-machine-1's median.

Where the code does the work

Final score = w_lexical·lex + w_relevance·spread(relevance) + w_department·p(department) + w_price·price fit + w_gift·p(gift).

spread is the part worth explaining. A yes-no probability saturates: on the coffee query the model puts a coffee maker at 0.89 and a vacuum flask at 0.99, a gap of 0.10 that a lexical term of 0.25 walks straight over — and the coffee maker matches “coffee” and “hot” word for word, so it takes lex 1.00 while the flask takes 0.53. The original had room to spare here because its relevance was a 0–4 ordinal, three whole levels between those two items. So the ranker works in log-odds instead, min-max normalised across the 30 candidates, which makes the last two points of probability worth as much as the first eighty. Rows still print the raw probability, and the relevance floor still reads it, so nothing on screen is rescaled.

Candidates under the relevance floor drop to the bottom, and an inferred price or rating sort is applied only within relevance bands 0.1 wide on the printed score, so a cheap unrelated item can never outrank a good match and the column never reorders two rows whose scores read differently. If a call fails the right column falls back to the regex baseline and says so.