How the cloud unlocked a new generation of enterprise integration
When the rise of the cloud pushed the traditional way of enterprise integration past its sell-by date, SnapLogic tapped MongoDB to execute its vision for its integration platform as a service.
Cloud computing triggered a lot of changes in how technology had traditionally been supporting enterprises in the couple of decades prior to its ascent. One of the less talked about areas is integration tools, which had grown crucial for ensuring information and processes could flow between all the back-end software services that companies had adopted to run their business.
From Salesforce to Workday, SAP to Oracle, as the leading enterprise applications migrated to the cloud, it became apparent that IT teams no longer need to maintain the instances, as this is done by cloud providers, and the people managing them are now in the relevant business units.
These changes, while not removing the need of integration, did warrant a new, end-to-end approach to integration to support enterprises as they went through their own cloud migration journeys. Gaurav Dhillon, who previously co-founded Informatica, saw the golden opportunity.
Rather than take on one area of integration – traditionally companies like Informatica only did data integration and other companies only did app integration, such as TIBCO – Dhillon’s vision was, because the user is changing, one tool should be able to do all of it. SnapLogic was born.
“Gaurav wanted to build a new generation of integration software for the cloud with an easy to use interface. People would later call this integration platform as a service (iPaaS) – the glue that integrates all back-end services together,” says Vaikom Krishnan, who previously led engineering for Informatica Cloud and now leads SnapLogic engineering as Chief Product Officer.
Building the engine
Building an iPaaS system was a challenge, Krishnan admits, not least because it had never been done before, effectively designing from scratch an engine that could handle all modes of enterprise integration and manage the document-centric flow of cloud-based applications.
SnapLogic’s choices for its own technology stack were heavily influenced by being born in the cloud and wanting to avoid a relational database due to the technical restrictions it poses on the most prominent form of development today, through documents like JSON and XML. The company ultimately opted to self-manage the MongoDB Community Edition on AWS servers.
“Relational is a bad choice for the current mode of development,” says Krishnan. “We had a couple of choices at that time for a document-based database. We did the evaluation and we decided that MongoDB was easier to manage and had better capabilities.”
The first version of SnapLogic came out in 2014, and soon after SnapLogic was being deployed by real customers.
Avoiding bottlenecks
Dhillon’s prediction about user needs’ changing proved to be spot on and it didn’t take long for a large customer to take a bet on SnapLogic’s integration offering. Adobe had decided to completely shift to the cloud, moving to a subscription model, and to make that happen the company had to go through a lot of business process changes which it felt a new tool was required to execute.
Just like Dhillon had envisioned when dreaming up his vision for a new generation of enterprise integration, rather than a small set of IT developers dealing with integrations Adobe wanted to farm them out to business units to do their own integrations and business process engineering.
“They realised the value of what we built. They've grown to be a huge company and all of their business processes run on SnapLogic,” says Krishnan. “Typically our customers are moving to the cloud, replacing their current stack. They want to be more agile, not bottlenecked because 100 business units want to do something but only ten people in IT can deal with the integration.
“That’s going to slow any business down. Within Adobe, there are about 4,000 users that use SnapLogic [because it's embedded in the business units]. If you think about the previous generation of tooling, the largest Informatica installation probably had 100 or 200 people using it. They were complex products, you needed training and all that. But SnapLogic is easy to use and that is what our customers usually tell us – it's easier, faster and cheaper with SnapLogic.”
Changing course
Having previously opted to self-manage, SnapLogic’s growth has caused a change in direction. Starting with just a single, shared database cluster, which held all of the metadata, as the needs of the application grew SnapLogic split that into multiple clusters.
About a year and a half ago, however, there was a creeping realisation of the operational challenges of self-maintaining SnapLogic’s infrastructure. In fact, it’s the costliest part of their entire hosting bill.
There is also the consideration that currently SnapLogic is running MongoDB clusters in one single region, but it wants to make it a multi-region shared instance to provide more disaster recovery capabilities for its customers. All of this pointed the company towards MongoDB Atlas on AWS.
“If we continued to do it on our own, we’d have to build some MongoDB-synced technology alongside,” Krishnan says. “As we are a start-up, cost is an important issue. Ease of management is also important. It turned out MongoDB could promise us wins on both. It was actually cheaper for us to move to Atlas and decommission the servers we were running MongoDB Community on.
“On top of that, we get a fully-managed MongoDB database with Atlas, which is a great advantage as we can focus on building the data models and the access technology, rather than worrying about how to provision and manage servers and make sure multi-region cluster works.
“We have a path of migration and we are well on it. MongoDB's team has been helping very well and they have a good set of developers in the migration team who can help. They've seen this multiple times, people moving from Community to Atlas, so they know what are the pitfalls, what are the things that they need to look at, and they have been immensely helpful in moving us.”
