Apache Kafka is a distributed streaming platform.

Apache Kafka is a tool that allows for the real-time processing of data streams, enabling the transfer of large amounts of data between systems and applications in a scalable and fault-tolerant manner. It provides a publish-subscribe messaging system that allows producers to send messages to a topic, which can then be consumed by one or more consumers. Kafka also offers features such as data replication, fault tolerance, and horizontal scalability, making it a popular choice for building real-time data pipelines and streaming applications.

Settings

Topic

The name of the Kafka topic to connect to.

Group ID

The ID of the consumer group to which the client belongs.

Bootstrap Servers

The list of Kafka brokers to connect to.

Primary Keys

The field(s) used as the primary key for the messages.

Use Message Key

Whether to use the message key as the primary key.

Initial Start Time

The timestamp from which to start consuming messages.

Max Runtime (ms)

The maximum amount of time the client will run before shutting down.

Commit Internal (ms)

The interval at which the client will commit offsets.

Consumer Timeout (ms)

The maximum amount of time the client will wait for new messages.

Session Timeout (ms)

The maximum amount of time a consumer session can be inactive before being considered dead.

Heartbeat Interval (ms)

The interval at which the client will send heartbeat messages to the broker.

Max Poll Records

The maximum number of records to fetch in a single poll.

Max Poll Interval (ms)

The maximum amount of time to wait for new records in a single poll.

Message Format

The format of the messages being consumed.

Protobuf Schema

The schema for the Protobuf messages being consumed.

Protobuf Classes Directory

The directory containing the generated Protobuf classes.