Big Cloud™ touts swappable AI models, wants the harness

Speak to almost any CTO or senior engineering leader (as The Stack does regularly) and the focus is rarely on the what’s-hot-and-new performance of a given frontier model; those building agent-heavy systems are mixing and matching models for a combination of low latency, low cost, high performance – and picking an LLM or SLM of choice for a given workload. 

Take Bank of America’s CTO Hari Gopalikrishan. He told The Stack earlier this month that the bank is rebooting its customer services chatbot Erica (built on a “set of open source small language models”) on an eclectic “combination of small language models and large language models.”

The raison d'être is simple: “90% of customer interactions are actually small bursts of conversations [like] ‘hey, what's my balance?’ [that do] not require a massive [LLM] using complex GPU resources on the cloud to do.”