How to copy production Kafka data to a test cluster with personal data masked
Copy production Kafka topics to a test cluster with personal fields masked

Copy production Kafka topics to a test cluster with personal fields masked
To copy production Kafka data to a test cluster without exposing personal data, mask the personal fields while the records are in transit, so the test topic never receives the original values. In Kadeck you define a data protection policy for the sensitive fields and then create a pipe from the production topic to the test cluster; the pipe applies the policy to everything it writes.
The useful property is the default: production data leaves masked without anyone having to remember a step. Topic names such as customers and orders below are examples.
A script that consumes from production, rewrites a few fields and produces to test is quick to start and slow to finish. You own serialization, schema handling, offsets, retries and the question of what happens to records that fail. The masking rules live in the script, so every new sensitive field is a code change that nobody outside the team sees.
MirrorMaker 2 is built to replicate topics between clusters, and it does that well. Its goal is a faithful copy, which is the opposite of what you want here. Changing fields on the way means adding custom transformation code to a replication tool.
Kafka Connect can change records with single message transforms (SMTs), and Apache Kafka ships a MaskField transform. In practice you need a Connect cluster, a connector that moves data from one Kafka cluster to another, converters that match your formats, and a transform chain per topic. The built-in transforms are simple; nested fields or partial masking usually need custom or third-party transforms. That is reasonable for a permanent pipeline and a lot for a one-off copy of last week's orders.
A data protection policy states which fields are masked, on which connections and topics, and with which method. A Kadeck Pipe writes the records of a topic view to another topic, on the same or another cluster, with no code to deploy.
The link between them is the pipe's author, the user who creates it. A pipe applies its author's policies when a record is read, before the view's JavaScript Processor and filters run, and it covers keys, values and headers. The processor therefore only ever sees masked values. A policy is skipped only if the author belongs to one of that policy's excluded user groups.

In the pipe dialog this is the Data masking setting under Governance, and it is on by default. Administrators also have an Enforce data protection setting for pipes, on by default as well. While it is on, the dialog shows masking as "Always on" and authors cannot switch it off.
For how policies work in general, see the post on data protection policies. Pipe behavior is documented under Configuration, guarantees and permissions.
email and phone, and give each a masking method: full masking, a random value, only the initials, the first or last few characters, parts of an email address, zero or null.delete rec.value.internalNote; return rec;, or add a filter to copy only part of the topic.
The Pipes page then shows the pipe's state, lag, and how many records were written, dropped or dead-lettered.
TopicAccessRead on the source topic and TopicAccessWrite on the target topic, plus TopicAccessWrite on the dead letter topic if you use one. Kadeck's cluster connection must be allowed to commit offsets for consumer groups named kadeck-pipe-<id> on the source cluster and to write to the target topic.orders to customers on an ID, choose the masking method for that field with care. A random value in each topic will no longer match.Not out of the box. It is designed to replicate topics faithfully; masking fields on the way requires transformation code that you write, deploy and maintain.
No. The pipe only reads from the source topic and keeps its position in its own consumer group.
Yes. Choose Keep running and the pipe masks and writes each new record as it arrives. If the policy changes, a running pipe picks up the change within about a minute.
It can be. Masking lowers exposure, but whether the result counts as anonymous depends on which fields remain and whether people can be re-identified by combining them. Review the policy's field list with whoever is responsible for data protection.
Kadeck works with Apache Kafka, Redpanda, Amazon MSK and Aiven. Pipes are included in the Free plan with one active pipe; data masking and data protection policies are part of Kadeck Platform, which is priced per user, from 10 users, billed annually. You can get started here and compare plans on the pricing page.