How to copy production Kafka data to a test cluster with personal data masked

How to copy production Kafka data to a test cluster with personal data masked

Copy production Kafka topics to a test cluster with personal fields masked

To copy production Kafka data to a test cluster without exposing personal data, mask the personal fields while the records are in transit, so the test topic never receives the original values. In Kadeck you define a data protection policy for the sensitive fields and then create a pipe from the production topic to the test cluster; the pipe applies the policy to everything it writes.

The useful property is the default: production data leaves masked without anyone having to remember a step. Topic names such as customers and orders below are examples.

Why the obvious approaches hurt

A hand-written consumer and producer

A script that consumes from production, rewrites a few fields and produces to test is quick to start and slow to finish. You own serialization, schema handling, offsets, retries and the question of what happens to records that fail. The masking rules live in the script, so every new sensitive field is a code change that nobody outside the team sees.

MirrorMaker 2

MirrorMaker 2 is built to replicate topics between clusters, and it does that well. Its goal is a faithful copy, which is the opposite of what you want here. Changing fields on the way means adding custom transformation code to a replication tool.

Kafka Connect with single message transforms

Kafka Connect can change records with single message transforms (SMTs), and Apache Kafka ships a MaskField transform. In practice you need a Connect cluster, a connector that moves data from one Kafka cluster to another, converters that match your formats, and a transform chain per topic. The built-in transforms are simple; nested fields or partial masking usually need custom or third-party transforms. That is reasonable for a permanent pipeline and a lot for a one-off copy of last week's orders.

How masking and copying fit together

A data protection policy states which fields are masked, on which connections and topics, and with which method. A Kadeck Pipe writes the records of a topic view to another topic, on the same or another cluster, with no code to deploy.

The link between them is the pipe's author, the user who creates it. A pipe applies its author's policies when a record is read, before the view's JavaScript Processor and filters run, and it covers keys, values and headers. The processor therefore only ever sees masked values. A policy is skipped only if the author belongs to one of that policy's excluded user groups.

The steps a pipe applies to each record: read, decode, mask, process, filter, encode, write. Masking by data protection policies happens before the JavaScript Processor.
A pipe masks each record before the JavaScript Processor and filters see it.

In the pipe dialog this is the Data masking setting under Governance, and it is on by default. Administrators also have an Enforce data protection setting for pipes, on by default as well. While it is on, the dialog shows masking as "Always on" and authors cannot switch it off.

For how policies work in general, see the post on data protection policies. Pipe behavior is documented under Configuration, guarantees and permissions.

Step by step

  1. Define the policy. In the administration area, add a data protection policy for the production connection and topics. Select fields by name (a regular expression works) or by path, for example email and phone, and give each a masking method: full masking, a random value, only the initials, the first or last few characters, parts of an email address, zero or null.
  2. Create the target topic on the test cluster. It must exist before the pipe starts, and it cannot be the source topic.
  3. Open the production topic in the Explorer. Optionally use the JavaScript Processor tab to remove fields no policy covers, for example delete rec.value.internalNote; return rec;, or add a filter to copy only part of the topic.
  4. Click Pipe in the view's toolbar. In the Convert to Kadeck Pipe dialog, choose the target cluster and topic and name the pipe.
  5. Set Apply. A view shows at most 50 results per partition by default, and with Entire view the pipe writes no more than that. To copy the whole topic, choose Selected parts and turn End condition off.
  6. Set Run to Run once or Keep running, and On error to Drop or Dead letter topic.
  7. Check Write as. The key and value codecs are prefilled from the view. Schemas are looked up in the Schema Registry of the target cluster, not the source.
  8. Click Test run. It takes one record through decoding, processing and encoding and shows the record the pipe would write, without writing anything. Check that the personal fields are masked and that the record still fits the target schema, then click Start pipe.
Example pipe settings: target customers on the test cluster, Apply set to Selected parts with End condition off, Run set to Keep running, On error set to Dead letter topic, Data masking always on.
An example configuration for a test topic that stays in sync with production.

The Pipes page then shows the pipe's state, lag, and how many records were written, dropped or dead-lettered.

What to check before you start the pipe

  • The author. The author cannot be changed later. Create the pipe as a user who is not in an excluded group of the policy, and confirm the result in the test run. A pipe keeps running if its author later loses access, so review pipes when people change teams.
  • Permissions. Creating a pipe needs TopicAccessRead on the source topic and TopicAccessWrite on the target topic, plus TopicAccessWrite on the dead letter topic if you use one. Kadeck's cluster connection must be allowed to commit offsets for consumer groups named kadeck-pipe-<id> on the source cluster and to write to the target topic.
  • Run once or keep running. Run once ends at the view's end condition or at the end of the topic as it was when the pipe started. Keep running processes every arriving record until you stop it. A pipe cannot be edited; use Duplicate to change it.
  • Error handling. Records that fail to decode, process, encode or be accepted by the target are dropped or dead-lettered, and the pipe keeps running. Records sent to the dead letter topic are also written masked when a policy applies.
  • Delivery. Pipes deliver at least once: a record may be written twice but is never left out. Timestamps are kept. Kafka assigns the partition on write, so the source partition is not kept.
  • Codecs. Masking needs structured data, such as JSON, Avro or Protobuf. A policy does not accept the plain String or CSV codec on the part it protects.
  • Join fields. If your tests join orders to customers on an ID, choose the masking method for that field with care. A random value in each topic will no longer match.
  • Kadeck Desktop. A pipe that runs locally in Kadeck Desktop does not apply data masking. Run this pipe on Kadeck Platform.

Questions

Can MirrorMaker 2 mask personal data while replicating?

Not out of the box. It is designed to replicate topics faithfully; masking fields on the way requires transformation code that you write, deploy and maintain.

Does masking change the data in the production topic?

No. The pipe only reads from the source topic and keeps its position in its own consumer group.

Can the test topic stay in sync with production?

Yes. Choose Keep running and the pipe masks and writes each new record as it arrives. If the policy changes, a running pipe picks up the change within about a minute.

Is masked production data still personal data under GDPR?

It can be. Masking lowers exposure, but whether the result counts as anonymous depends on which fields remain and whether people can be re-identified by combining them. Review the policy's field list with whoever is responsible for data protection.

Plans and next steps

Kadeck works with Apache Kafka, Redpanda, Amazon MSK and Aiven. Pipes are included in the Free plan with one active pipe; data masking and data protection policies are part of Kadeck Platform, which is priced per user, from 10 users, billed annually. You can get started here and compare plans on the pricing page.