I recently got my hands into a product called CluedIn. Not only does it have a cool tech stack (built cloud native for deployment in a Kubernetes cluster with fine grained scaling and throttling for cost control) but it also offers a novel approach to how to solve the challenge of delivering high quality data for business intelligence across system silos.
Once you have built an organisation with effective operational workflows supported by the best enterprise solutions that money can buy, you want to extract data to track and mine for insights about your business. Data driven decisions based on accurate and fresh data.
What typically happens
The traditional approach to business intelligence is that you build an expensive data warehouse where the different systems feed into one big data model. Creating the common model and implementing the system integrations turn into a monster project that take months if not longer. It can be done and I have seen many such solutions. They can work. For a while. What typically happens next is that your business or organisation change and the data warehouse starts breaking down.
A key employee moves on to new challenges. A new system appears through merger and acquisitions. One business unit decides to replace a system to better support operations. A workflow changes.
Suddenly that expensive, custom built data warehouse needs to be updated. It’s expensive and time consuming and your priorities for the organisation are elsewhere. So you accept that the data warehouse is not up to date for a while. You no longer have a single source of truth. You lose confidence in what the data warehouse tells you. That value you have fought for and paid for erodes over time.
What CluedIn offers instead
This is where CluedIn changes the rules of the game.
Instead of making data integrations long running technical projects, CluedIn puts the business in the driving seat.
You connect to a new data source in whatever state the source data is in. It can be as simple as uploading an Excel sheet. As soon as data is in the system, CluedIn starts to make sense of it: It identifies common entities like persons and organisations and finds connections to data you already have. Next you clean and enrich data easily in CluedIn by training the system with cleaning rules. Spelling mistakes and typos? Mapping from one terminology to another? That one bad record from that hard to get to system that no one can update right now? No problem, create and maintain your cleaning rules in CluedIn and apply them when needed. Or write back the cleaned data directly to the source systems. Once data quality is good, you can publish multiple models for your different consumers with full control over sensitive data, who gets to see exactly which data.
CluedIn is where you see and maintain your golden records, your single source of truth across your business. You know exactly what data you have, where it came from, what transformations were made, if it is sensitive and who has access to it.
An organisation is rarely at rest. There is always something fuzzy and changing. This is the normal state. Take the data as is and give the people with insight into what the data means a powerful tool to analyse, clean and connect data to find out what is going on and to discover business insights and opportunities.