Speak to almost any CTO or senior engineering leader (as The Stack does regularly) and the focus is rarely on the what’s-hot-and-new performance of a given frontier model; those building agent-heavy systems are mixing and matching models for a combination of low latency, low cost, high performance – and picking an LLM or SLM of choice for a given workload.
Take Bank of America’s CTO Hari Gopalikrishan. He told The Stack earlier this month that the bank is rebooting its customer services chatbot Erica (built on a “set of open source small language models”) on an eclectic “combination of small language models and large language models.”
The raison d'être is simple: “90% of customer interactions are actually small bursts of conversations [like] ‘hey, what's my balance?’ [that do] not require a massive [LLM] using complex GPU resources on the cloud to do.”
