What is Change Data Capture (CDC)?

If you’re a data practitioner, you’ve probably heard of Change Data Capture (CDC) and may even incorporate it in your data architecture, but do you understand how it works? Often, data teams employ a particular type of CDC without realizing that there may be more efficient implementations; I know this because it happened to me. This post will define Change Data Capture, its benefits, and the various implementation approaches.

What is Change Data Capture (CDC)?

Change data capture is a proven data integration pattern to track when and what changes occur in data and then alert other systems and services that must respond to those changes. Change data capture helps maintain consistency and functionality across all systems that rely on data.

Data is fundamental to every business. However, the challenge is that data is constantly being updated and changed. And enterprises must keep up with those changes. Whether it’s transactions, orders, inventory, or customers – having current, real-time data is vital to keeping your business running. When a purchase order is updated, a new customer is onboarded, or a payment is received, applications across the enterprise need to be informed to complete critical business processes.

How does CDC work?

As the modern data center has evolved, so have changed data capture methods. In fact, many of the early rudimentary approaches to change data capture were born out of the necessity to poll databases effectively. Over the years, some techniques, such as table differencing and change-value selection, gained more traction than others. However, many of these more rudimentary methods incurred substantial resource overhead in the system.

Today’s CDC strategies work by supplying the sourcing mechanism within a data pipeline with only data that has changed. Businesses can accomplish this in several ways, such as polling or triggering.

The Benefits of Change Data Capture

CDC captures changes from database transaction logs. Then it publishes changes to a destination such as a cloud data lake, cloud data warehouse, or message hub. This has several benefits for the organization:

• Greater efficiency: With CDC, only data that has changed is synchronized. This is exponentially more efficient than replicating an entire database. Continuous data updates save time and enhance the accuracy of data and analytics. This is important as data moves from master data management (MDM) systems to production workload processes.
• Faster decision-making: Change Data Capture helps organizations make faster decisions. It’s important to be able to find, analyze and act on data changes in real-time. Then you can create hyper-personal, real-time digital experiences for your customers. For example, real-time analytics enables restaurants to create personalized menus based on historical customer data. Data from mobile or wearable devices deliver more attractive deals to customers. Online retailers can detect buyer patterns to optimize offer timing and pricing.
• Lower impact on production: Moving data from a source to a production server is time-consuming. CDC captures incremental updates with a minimal source-to-target impact. It can read and consume incremental changes in real-time. The analytics target is then continuously fed data without disrupting production databases. This opens the door to high-volume data transfers to the analytics target.
• Improved time to value and lower TCO: CDC lets you build your offline data pipeline faster. This saves you from the worries that come with scripting. It means that data engineers and data architects can focus on important tasks that move the needle for your business. It also reduces dependencies on highly skilled application users. This lowers the total cost of ownership (TCO).

Change Data Capture in ETL

Change data capture is a method of ETL (Extract, Transform, Load) where data is extracted from a source, transformed, and then loaded to a target repository such as a data lake or data warehouse. Let’s walk through each step of the ETL pipeline.

Extract. Historically, data would be extracted in bulk using batch-based database queries. The challenge comes as data in the source tables is continuously updated. Completely refreshing a replica of the source data is not suitable and therefore these updates are not reliably reflected in the target repository.

Change data capture solves this challenge, extracting data in a real-time or near-real-time manner and providing you a reliable stream of change data.

Transformation. Typically, ETL tools transform data in a staging area before loading. This involves converting a data set’s structure and format to match the target repository, typically a traditional data warehouse. Given the constraints of these warehouses, the entire data set must be transformed before loading, so transforming large data sets can be time intensive.

Today’s datasets are too large and timeliness is too important for this approach. In the more modern ELT pipeline (Extract, Load, Transform), data is loaded immediately and then transformed in the target system, typically a cloud-based data warehouse, data lake, or data lakehouse. ELT operates either on a micro-batch timescale, only loading the data modified since the last successful load, or CDC timescale which continually loads data as it changes at the source.

Load. This phase refers to the process of placing the data into the target system, where it can be analyzed by BI or analytics tools.

Change Data Capture Use Cases and Examples

From reducing some of the operational overhead in the traditional data pipeline architecture to supporting real-time data replication, there are many reasons an organization might consider implementing CDC into tier overall data integration strategy. Ultimately, CDC will provide your organization with greater value from your data by providing quicker access, less resource overhead, and a data integration strategy that is less prone to errors or data loss.

Let’s look at a couple of common use cases for Change Data Capture to give more context around how CDC can help align your organization for success.

• Streaming Data Into a Data Warehouse – One of the core functions of today’s data pipelines is to move data from a source database to a data warehouse. This is because most operational databases aren’t designed to support intensive analytical processing, whereas a data warehouse is perfect for these types of operations. Here, CDC is a critical step in the data pipeline architecture that facilitates data migration from the source to the target data warehouse.
• Migrating On-premises Data to the Cloud – When organizations want to perform resource-intensive operations such as artificial intelligence, machine learning, or deep learning, they’ll often look to cloud-based data warehousing to facilitate the highly resource-intensive data processing process. The reason for this is that the operational costs will be much lower running these operations in the cloud over an on-premises deployment, making the pay-as-you-go model of the cloud a great option. Here, CDC can play an important role in facilitating the data migration from on-premises to the cloud.

Conclusion

Change Data Capture is a data replication method with several advantages, especially if using log-based CDC. It’s one of the best approaches that data teams can use to enable near real-time analytics without compromising performance. But even if implementing log-based CDC in your data stack comes with many benefits, it doesn’t mean that it’s easy. It requires data engineering knowledge and expertise. Fortunately, modern data integration platforms can help with the heavy lifting.