Kafka

Kafka Connection

Description

The "Kafka" connection type is used to connect a flow/agent to an external Kafka messaging broker to either send or receive messages.

Configuration Properties

• Brokers: A list of one or more broker URLs, for example: broker1.example.com:9092.

• Topic Template: A template that contains the full topic-name structure, into which the topicName provided by the flow will be added:

  • For example: myenv.myapplication.${topicName}

  • The flow provides a topic name in the kafka node configuration (e.g. mytopic), and the resulting fully qualified topic name is: myenv.myapplication.mytopic

• Mechanism: The SASL mechanism for the connection.

  1. PLAIN:

     • Description: Uses plain text username and password for authentication.

     • Usage: Simple to set up but less secure, typically used with SSL/TLS to protect credentials.

  2. SCRAM-SHA-256 and SCRAM-SHA-512:

     • Description: Uses Salted Challenge Response Authentication Mechanism with 256-bit or 512-bit hashing for more secure authentication than PLAIN.

     • Usage: Preferred for environments requiring enhanced security.

  3. OAUTHBEARER:

     • Description: Uses OAuth 2.0 Bearer tokens for authentication, suitable for integrating with OAuth 2.0 identity providers.

     • Usage: Ideal for scenarios needing single sign-on (SSO) or token-based authentication.

• Partitioner: The partitioner is responsible for determining which partition a record should be sent to within a Kafka topic. Different partitioners use various strategies to distribute records across partitions. The two values you mentioned, "legacy" and "default," refer to different partitioning strategies used in Kafka Connect.

  • Description: The legacy partitioner provides backward compatibility with older versions of Kafka Connect. It uses the same logic as the default partitioner used in Kafka 0.8.2 and earlier. This logic typically involves simpler and potentially less efficient distribution algorithms.

  • Usage: Primarily used for maintaining compatibility with systems that rely on the older partitioning logic. It ensures that upgrades to Kafka Connect do not disrupt the existing partitioning scheme.

• Environment Variables: Additional environment variables can be added and used in the topic template.