The unpredictable nature of the commodities market, heavily influenced by external events like natural disasters and geopolitical turbulence, means commodities companies increasingly rely on access to fast, accurate intelligence to navigate such events and aid their decision-making.

The irony is not lost on the Independent Commodity Intelligence Services (ICIS), whose commodity prices have been the backbone of thousands of transactions for decades, that it deals in what is arguably now the most valuable commodity of all: data. Part of LexisNexis Risk Solutions, ICIS’s role as a source of commodities intelligence makes it a vital cog as the global economy becomes more hungry for information.

“Coming from an IoT and smart systems background, I sometimes joke that the commodities industry is not really about big data in a storage sense,” says Christian Mastrodonato, Senior Director of Software Engineering at LexisNexis Risk Solutions Group. “It's more about data quality, versus the sheer amount of data you can get from billions of sensors across the field. 

“Our data landscape is varied. We have a lot of numerical data, mainly on prices and what we call fundamental data, which is based on supply and demand across different industries. But we also have a lot of content, as we have editors and journalists doing research, talking to people all the time and writing new content. So what makes ICIS special is having this very interesting combination of quantitative and qualitative data that you have to match all of the time.”

No end to transformation

Transformation is a constant for ICIS, which for a digital company is now a necessity to ensure it doesn’t fall behind the curve, Mastrodonato proclaims. With numerous channels through which customers can consume its data, one of the key pillars of ICIS’s current transformation is evolving to a data-as-a-service model and investing in what it calls ‘cloud DB’.

Specifically, ICIS is building a ‘central truth’ repository to transform how it exchanges data across its channels and ultimately manage and structure data across the business more consistently. ICIS is also utilising a micro frontend architecture in the way it builds websites, which is helping to completely redesign the workflows and experience of its editors. MongoDB has been fundamental as a persistent layer for this transformation of ICIS’s service model and web applications.

But successful transformation means keeping one eye on the future, and ICIS’s positive experience with MongoDB in its core business spurred it to experiment with generative AI and retrieval-augmented generation (RAG). These powerful technologies, which could transform commodities intelligence, are included in the suite of features of MongoDB Atlas. 

First of a kind

Recently launched in production, using MongoDB Atlas as the underpinning vector database, “Ask ICIS” is a first-of-its-kind generative AI assistant for commodity market intelligence. Subscribers can tailor responses directly to their roles, market segments and priorities, and “Ask ICIS”’s citation-backed content can be provided in various forms, from short summaries to long reports.

While the tool is still in its early days, the sheer pace at which it enables energy and chemical professionals to access insights is already unparalleled. It is fitting, therefore, that speed was also the core KPI when building the application and interface. 

“The key point for us when building the RAG application was getting it in front of the customer as soon as we could,” says Mastrodonato. “That was a key driver for us choosing MongoDB Atlas as our vector database, along with quality. We also had the internal knowledge, as we're already using MongoDB for our core web platform, so it was our first choice. We tried it, it worked out of the box, that was it. We also leveraged Azure Cognitive Services and a blend of cloud LLMs.

“When it comes to RAG and generative AI, the real transformation is in how our customers interact with our data, which is completely different to before. The landscape is moving so quickly and the reality is most of the value comes from customers putting real questions into the system every day. It helps us to understand how the user experience is going to change.”

Future of commodities intelligence

ICIS’s heterogeneous IT provides the flexibility to opt for either a SQL or NoSQL back-end when developing a new application. Such choices typically fall back to a discussion of speed versus quality, with SQL chosen when the focus is more on slow-moving data. “That might represent 20 to 30% of our use cases,” Mastrodonato says. “But when we go closer to the customers, and therefore transaction speed starts to become more important, MongoDB is becoming the default.”

While the deployment itself was fast, ICIS has spent several years getting its data in shape, which has been integral to “Ask ICIS”’s early success. Unlike other AI assistants which rely on OpenAI's ChatGPT for source data and can therefore lack accuracy and transparency, “Ask ICIS” is developed and fed exclusively with ICIS’s own database of trusted insights and news content.

“We've also pushed towards a data mesh architecture that allows us to have a much more structured way to manage our data models,” Mastrodonato adds. “We are getting to a place where it's relatively easier to implement a RAG application compared to others. Also by working and operating in such a clear bound of context, commodities, it's relatively easy to control the system, mainly via prompts, to get meaningful answers or no answers at all.”

There are no doubt plenty of opportunities for “Ask ICIS” to improve, Mastrodonato concedes, but the positive rates of adoption, which surpassed expectations given the conservative nature of the commodities industry, shows the direction of travel is very clearly towards generative AI.

“We actually have customers talking to other customers saying, ’It's really cool, you have to try it.’ We're definitely managing to find a baseline of people who understand the value and the potential, and they can use it in their daily work to reduce their workload. Of course, there are still customers, either because they are more conservative or they are extremely sophisticated users, who struggle to get the benefit. But overall we are positively impressed with the results.”

Delivered in partnership with MongoDB

The link has been copied!