Srijith Ravikumar is a Principal Engineer at Amazon building AI-powered recommendation systems at scale. Published researcher at AAAI.
In conversations with retail technology leaders, I keep hearing a version of the same plan. They’ve watched a competitor announce an AI-powered shopping experience, and their CEO is asking when their site is getting one. They’re evaluating language models, lining up vendors and sizing GPU budgets.
Most of them are solving the wrong problem. The question every retail CTO is actually facing isn’t which model to choose. It’s whether the generative search experience they’re about to ship will quietly destroy their conversion rate.
A 2017 Akamai Technologies research report found that every 100 milliseconds of added latency can drop conversion rates by 7%. A frontier large language model running synchronously behind a search bar adds two to three full seconds to a request once retrieval and reranking are included (median TTFT for Llama 3.3 70B sits around 1.7 seconds across providers, before any RAG overhead). The math is brutal, and most teams don’t run it before they commit to the architecture.
The most useful framing I’ve found is this. Generative search is a latency decision, not an AI decision. The model you pick matters less than where you put the inference call in the request path.
Three strategic choices determine whether you ship a conversion-positive product or a conversion-negative one:
1. What is generative search actually for?
Retail query distributions are punishing. A small handful of head queries make up the majority of traffic, and most of those don’t need an LLM at runtime. The interesting work happens in the long tail, where users are exploring, comparing and asking questions traditional search can’t handle.
If your team is planning to put a model in front of every query, you’re spending frontier compute on the easy questions and leaving the hard ones underserved. Define the slice of traffic where generative search creates real lift, and architect for that slice. Treat the rest as a precomputation problem.
2. What runs at build time, and what runs at runtime?
This is what separates teams that ship sub-second experiences from teams that don’t. Most of the value an LLM produces in retail (review summaries, structured attribute extraction, category-level answers, semantic embeddings over the catalog) can be generated offline and indexed.
By the time a user types a query, the heavy work is already done. The runtime path becomes a vector lookup, which fits comfortably inside the latency budget. Teams that try to generate everything live are paying frontier prices for predictable work and absorbing the latency hit on top.
3. Which model serves which query?
The instinct to standardize on a single capable model is understandable from a procurement standpoint and wrong from an economic one. Most queries in a retail pipeline (intent classification, entity extraction, query reformulation) are small-model problems. A 3-billion-parameter model (such as Microsoft’s Phi-4-mini, with 3.8B parameters and a 128K context window) running on modest hardware handles them in under 50 milliseconds at roughly 1% of the cost of a frontier call.
RouteLLM demonstrated 85% cost reduction on MT Bench and 45% on MMLU when routing between GPT-4 and Mixtral 8x7B while preserving 95% of GPT-4’s response quality. Reserving expensive models for queries that genuinely need them is where most of the cost optimization in production retail AI actually lives.
The teams I see succeed with generative search treat it as a retrieval system that happens to use AI, not as an AI feature with retrieval attached. They measure tail latency, not average latency, because the 99th percentile is what users remember (a principle that Jeffrey Dean and Luiz André Barroso laid out in “The Tail at Scale,” which is still the canonical reference on why p50 metrics lie at scale). They run their cost projections on real query distributions, not vendor demo traffic, and accept that the right answer often involves not calling a language model at all.
The next 18 months will produce a clear divide in retail. Some teams will ship generative experiences that lift conversion. Others will ship experiences that look impressive in pitch decks and quietly drag down their core funnel. What separates the two camps comes down to a single decision: whether they treated the rollout as an AI project or a latency project.
If you’re about to green-light a generative search initiative, ask your engineering organization for a latency budget before you ask it for a model evaluation. The order matters.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
