Data Lake vs. Customer Data Platform: What Is Right for My Organization?

Key Takeaways:

1. Increasing amounts of data, from many different sources and run by siloed teams, create barriers to achieving a customer 360 view.
2. Understanding where your data is sourced, creating a standardized data layer and determining where it needs to flow helps decide the right solution for your organization.
3. Implemented correctly either alternative is a worthy investment that will reduce costs and create demand-generation opportunities, which means few potential projects have the same ROI potential. 

As someone who’s worked in marketing and operations management for nearly 20 years I can’t believe how many times I’ve come across a challenge and thought, “there’s got to be a better way.” 

My experience isn’t different from most, as people, processes or technology often become barriers to success rather than catalysts for it. This is too often true when it comes to customer data and how companies collect, store and utilize customer information. 

This problem often leads to digital transformation or customer 360 initiatives; a PWC survey of CEOs found that 95% of executive leaders believe data about customer and client preferences is critically important. The same survey found that 90% of CEOs believe the same regarding data about your brand and reputation. These findings contradict what is actually happening at most companies, where an August 2019 CMO Survey found that marketing analytics is only used in decision-making 39% of the time.

Let’s first talk about the reason for this disconnect. Then we will dive into two competing schools of thought on how to solve this problem: using a data lake (or data warehouse) or a customer data platform (CDP).

It’s not a stretch to say all companies struggle to effectively use customer data. This stems from three primary causes and one effect.

The first cause is the explosive growth of data. There were over 26 billion internet of things (IOT) devices in use in 2019, and that number is forecast to grow to over 31 billion by the end of 2021. And that is for a world population of approximately 8 billion. Those devices create an incredible amount of data to be processed and attributed back to a specific user who, in most cases, has multiple devices (phone, work computer, home computer, tablet, etc.).

The second cause is the increasing growth of channels, especially digital, in recent years. If you go back 30 years you could point to three channels — print, TV and radio. Mobile communications devices, coupled with increased bandwidth and the internet, seem to create new channels on a daily basis. The first Google search didn’t happen until 1998. Facebook wasn’t a thing until 2004. The iPhone didn’t release until 2007, and the much-talked-about Tiktok was a figment of someone’s imagination until 2016.

Marketers must decide which channels to advertise on and how to distribute spend across them to reach customers where they are. From an application standpoint many of these channels are operationalized by new SaaS products — CRM, marketing automation, ad platforms — designed to enable marketers to execute on all of these new channels.

The third cause is the increasing legal ramifications surrounding privacy and security. CAN-SPAM passed in 2003, GDPR (European Union) became enforceable in 2018 and CCPA (California) went into force in 2020. Coupled with the forthcoming elimination of third-party cookies, a company’s master data management strategy has never been more important. 

The effect of these three causes has been the proliferation of data silos within organizations, which has caused the need for these data initiatives. The sales team owns CRM and its data. Marketing automation is owned by the email channel. Social media is owned by that team. Paid is owned by the digital team or by an outsourced ad agency. IT owns data security and compliance. And finance owns the ERP system. 

This means you took an inherent business problem (cross-functional departments) and stacked the aforementioned challenges on top of it. The result is where we find ourselves today, with nearly every company embarking on a digital transformation initiative.

Customer data platforms and data lakes/data warehouses address these problems, albeit in different ways. Before we discuss their differences let’s discuss their similarities. In both cases data is abstracted from source systems into a centralized repository where it is standardized and cleaned in some manner to provide for an output for business purposes. A master data management strategy that maps data attributes into a common format and determines when to create, update or overwrite data from source systems is critical for success. The value of this integrated data set could be in the form of analytics and reporting, including machine learning/artificial intelligence, or it could be utilized for personnel who interact with customers, namely sales, marketing or customer service. 

While these projects are complex, take time and come at a considerable cost, their potential value cannot be underestimated. In the age of big data intelligence, getting more value out of data can be the competitive advantage of the 21st century.

Here’s why. With more complex customer journeys and numerous interactions, the ability to personalize experiences for customers as well as being able to quantify when touch points create value impact both sides of the profitability equation — reducing costs while increasing revenue — all while compressing the time from interest to purchase.

Data Lake/Data Warehouse

Now that we understand the potential value of data let’s talk about how a data lake or data warehouse attempts to employ technology to create this value. Amazon Web Services (AWS) defines a data warehouse as “a central repository of information that can be analyzed to make better informed decisions. Data flows into a data warehouse from transactional systems, relational databases, and other sources, typically on a regular cadence. Business analysts, data scientists, and decision makers access the data through business intelligence (BI) tools, SQL clients, and other analytics applications.”

These data marts can be on-premise or in the cloud, though many businesses are moving to the cloud due to cost advantages. These are IT-driven projects that involve moving data at regular intervals from source systems using connection end points or APIs with ETL (extract, transform and load) tools to move data between applications. This data can be difficult to access by non-technical users due to security, because not every user needs or even should have access to all of the data. The technical skill sets needed to operate this technology are high and the demand for these resources generally outstrips the supply. 

There also is a constant learning curve to understand not only data repositories but also the data input and output sources. The time to value can be slow as each new data source needs to be evaluated to research how to bring in, map and clean data. Furthermore, these integrations potentially break every time an app is updated on one side or the other. So if it takes three months to integrate a new data set into your data lake and you have 16 data sets it will be a four year project to make these connections — after the initial lake is stood up — with the increasing likelihood that new data sources will be added and others deleted as you work through the process so prioritization and agile development are key to success. 

The other key is how you stitch together a customer record and how various activities should hang off of that user. It is important to devise how you create a unified profile and how the different IDs map into the same record from multiple sources.

Customer Data Platform

At its foundation, a CDP is a data-as-a-service infrastructure that unifies and persists customer profile and other data, from any source, legally and securely into a single database with a comprehensive view of all customer activities or behaviors. Those activities are given rules-based attributes to create personas in real time that can be activated for any customer-facing communication in every customer channel. This data can also be utilized for analytics, reporting and data science, including machine learning and AI.

Once correctly implemented, non-technical teams should be able to utilize data within applications using configuration or low or no-code technology. This means marketing, sales and customer service teams can own the outputs from the CDP within applications they use every day — CRM, marketing automation, customer service systems and ad platforms — with minimal IT involvement.

CDPs generally are SaaS platforms that run on the cloud. They are the hub system that takes data from many spokes. In many cases the CDP platform will have integrations to bring in client-side data using tags and server-side data from other platforms. They have a built-in data layer that helps map similar fields form source applications that are uniquely named into a standardized format. This data layer is the engine that makes the CDP work. Using logic users and their activities are stitched together with one or more primary keys like email address, birthdate, Social Security number or some other distinguishing ID. It then has integrations with activation channels to take action based on data to do everything from sending an email to personalizing a page in real time or sending a lead to sales.

Because the CDP vendor manages integrations between systems, non-technical users can manipulate data with low- or no-code resources and create the right audiences to make an impact. While some integrations likely will rely on technical users the prebuilt connectors drastically reduce the time to value. This means you will need fewer technical resources and fewer hours to become operational than with a data lake strategy. This labor offset will help pay for the CDP license costs. 

Here are some signs your company might be ready to consider a CDP system: 

• You need a complete and consistent customer view.
• You want to personalize the experiences of your customers in real time (sub-200 milliseconds).
• You need to tie offline and online customer data together.
• You have technology evaluations underway for CRM, email, tag management, etc.
• You’ve had issues integrating customer data to CRM, email, ad tech, cloud providers, etc.
• You have an upcoming website redesign.
• You lack the resources and skills to manage your data.

Not all systems advertised as a CDP are equal; however, by doing your due diligence or working with a trusted partner you can find the right solution for your company.

Recommendation

As we’ve stated before both systems offer significant value over the status quo. Most leaders quickly understand the value of this initiative; however, it is first important to make sure that the project sponsor understands the effort and timeline of the initiative.

Second, it is important to pursue these projects using an agile methodology and not as a waterfall project. Due to the expensive nature of the effort, if you can’t show value quickly and routinely bring features online to show impact (even with a scaled down version), there is a chance the project could be scrapped before it shows value.

Third, lean into data governance and the risk of inaction. Deloitte and Forrester conducted a survey in late 2019 that found only 38% of companies strongly agree that they know where all of their customer data is stored. This presents huge headaches and missed revenue potential but also risks the possibility of financial penalties being assessed due to noncompliance. The goal of data governance should be to get marketers the data needed to drive revenue while reducing the risk for technology teams and the wider enterprise. This is an often-overlooked benefit of master data management — that is, until your legal team becomes involved due to a compliance mistake.

Fourth, develop a coalition of advocates. Marketing, sales, customer service, IT and analytics departments all benefit from these initiatives. Sell the value proposition to each stakeholder in terms that benefit their line of business to gain as much buy-in as possible. It will be needed as you execute on the project.

Fifth, start at the end and work backward. Data and the customer experience don’t work in a linear fashion. Understand the outcomes you hope to drive — increased conversion rates, personalization, reduced ad costs, machine learning/AI, better intel for customer-facing roles and/or regulatory compliance. This helps you determine not only the data sets but also the methods and transformation needed to reach your desired outcomes.

Sixth, understand the complexity of your data sources. Your level of SaaS maturity will dictate the best course of action. Do you have all home-built applications? Many systems or few? Which department owns each data platform? What is the technical expertise of your staff? What is your budget?

Seventh, develop a roadmap for the project led by someone who is able to bring the right people to the table and has the clout to keep different departments on track. It’s also helpful to communicate successes to the wider organizations while being willing to modify plans as things invariably change.

Eighth, be sure to focus on the integration of additional systems with your current infrastructure. While good planning can overcome potential conflicts of selecting “best of breed” software, be sure to look at every new technology stack acquisition from a 30,000-foot perspective to make sure it doesn’t cause unintended problems. For example, selecting a CRM and a marketing automation system that don’t have a native integration is problematic. If two marketing automation systems both have a similar integration and one is better than the other, your risk is lower. 

Ninth and finally, don’t be afraid to get another perspective. There is huge value to maintaining and growing talent from within an organization; however, there is a downside when one’s business becomes too insular. It’s been said that a smart person learns from his or her mistakes but a wise person learns from the mistakes of others. That’s where a first-rate consultant is worth their weight in gold as they’ve seen what works — and even more importantly what doesn’t. Be willing to go outside of your comfort zone to explore different viewpoints and technologies. You don’t want to be in a situation where you invest in a strategic initiative and find out six months down the road that you made the wrong decision.

While we might be biased, with all of the moving pieces required for a successful data lake or CDP project, trust a partner to help chart your course forward. Zirous can help guide you along your digital transformation path. Contact us today to learn more.