Documenting your business’s real-time API with Kadeck

Data streaming is at the heart of many businesses' real-time infrastructure. Companies use Apache Kafka, Red Panda, and Amazon Kinesis to reliably deliver streaming data from their point of origin to people and systems in real time. The more accessible this data becomes, the more rapidly companies can innovate new user experiences and improve business efficiency via real-time recommendations, vehicle tracking, monitoring & alerting, fraud prevention, and more.

The first step in making real-time streaming data more accessible is to communicate its availability to people, and how to make use of it. This is achieved by documenting message “topics.” Topics and topic message structure compose a business’s real-time API. 

Documenting topics and topic message structures is very challenging because there can be very many topics and frequent changes to them. Keeping topic documentation current is time consuming, and unfortunately many organizations fail to update it or even create it in the first place.

In this article we explain who needs to understand the real-time API and how to document it using Kadeck's visual collaboration platform (use it for free) for Apache Kafka, Amazon Kinesis, and Red Panda. 

The importance of documenting Kafka Topics

The primary audiences for topic documentation are:

• Streaming data platform developers and operators - create the data streams.
• Streaming data application developers and analysts - use the streaming data.

There are multiple reasons why creating API documentation for data streaming topics matters:

• Higher ROI on streaming infrastructure: Make data stakeholders aware of the real-time data available to them and how to use it. Accelerates real-time analytics and automation innovation across your company.
• Onboarding: New engineers can quickly understand each topic's purpose, its messages, and the data schema.
• Troubleshooting: Having comprehensive documentation helps in identifying and resolving problems faster.
• Collaboration: When everyone understands the topic's schema and functionality, they can work together more efficiently.
• Domain clarity: Especially when applying Domain-Driven Design principles, documenting your Kafka topics aids in defining and communicating the bounded context and domain events related to each topic.

Streaming data topic documentation template

Creating documentation for Kafka topics is very straightforward. At a high level, you will want to make it clear:

• What topics exist (making it easy to find and browse them)
• What the message structure is for each topic
• Which systems produce and consume each topic
• Which team member(s) maintain each topic

Below, we present a step-by-step guide based on Apache Kafka Topic API Documentation Template in Kadeck:

1. Topic Name: The name of the topic.
2. Partitions: The number of partitions.
3. Schema: Provide the schema for both the key and the value. This should define the type of the key and value, along with a brief description. Also note, if a Schema Registry is used.
4. Description: What the topic is used for. This could include its role in the system, the kind of data it deals with, and any other important details.
5. Domain Define the bounded context in which the topic operates. This helps clarify the topic's role in your system and the business events it relates to. Also, define key terms used within the topic's domain to ensure consistent understanding.
6. Data: Type of data, expected volume and frequency of messages.
7. Messages: List of domain events and messages associated with this topic.
8. Producers and Consumers: Where is the data coming from in the software landscape? Who is consuming it?
9. Data Owner: Indicate the individual or team responsible for this topic. This promotes accountability and clarity about who to contact for queries or issues related to the topic.
10. Notes: Additional notes, such as project specific details.
11. Change Log: Keep a history of significant changes to the topic, its schema, or its use. This provides a clear record of how the topic has evolved over time.

Kadeck saves time and assures accuracy by automatically updating many of these template sections, including Schema, Producers and Consumers, and others.

Creating API documentation for Kafka topics in Kadeck

Kadeck is a free, visual collaboration layer for Kafka, Kinesis, and Red Panda, and it makes managing and troubleshooting streaming data easier for teams. Kadeck is comprehensive, and many thousands of people use it to get insights into and manage clusters, topics, consumers, producers, and users. 

API documentation is created via the Topic Details page in Kadeck. 

A unique advantage to using Kadeck is that a topic’s consumers, producers and schemas are always kept up to date and can be viewed as part of the documentation. This information tends to be outdated quickly in written documents.

If you still want to keep the documentation in tools such as Confluence, it is possible to link to those documents in Kadeck to get the best of both worlds.

Documenting topic data owners for clear accountability

One of the standout features of Kadeck is its ability to assign data owners for each Kafka topic. This feature ensures clear lines of responsibility within your team, fostering effective collaboration and communication

Organizing API documentation with labels

As your Kafka topics grow in number, so does the complexity of their management. Kadeck's labeling feature acts as a flexible filing system for your topics, allowing you to categorize, filter, and search them based on functionality, data type, team, or any other relevant metric. This feature refines your workflow, keeping you focused on data management rather than getting lost in the labyrinth of topics.

Document your real-time API for free with Kadeck

Kadeck empowers you to craft API documentation for every topic, thereby significantly: 

• Improving topic comprehension, 
• Facilitating  troubleshooting, 
• Making the process of integrating new team members more efficient,
• Making it easier for other teams to tap into streaming data to build new real-time applications

Consider incorporating Kadeck into your streaming data development arsenal today and transform the way you put real-time streaming data to work.

Download and run Kadeck UI for free

‍