Configuring Apache Kafka Wire Protocol Service
Describes the Apache Kafka Wire Protocol Service configuration files and instructions.
The following sections provide configuration information for DAG (Data Access Gateway) and Apache Kafka clients and applications:
Configuring Apache Kafka Wire Protocol Service
DAG packages include the following configuration files for Apache Kafka Wire Protocol Service:
kafka-server.conf
-
- This file is stored in
/opt/mapr/data-access-gateway/conf
and loaded from the local file system. - This file contains node-specific options, such as the port number and security protocol. Set the options in this file on a per-node basis.
- By default, the Apache Kafka Wire Protocol Service listens on port 9092.
- If the DAG service instances are behind a firewall, the firewall rules must be configured to allow this traffic.
For a list of configurable parameters, see Apache Kafka Wire Protocol Service Settings.
- This file is stored in
kafka-cluster.conf
-
- This file is stored in
/opt/kafka-wire-protocol/default-cluster/conf
and loaded from the distributed file system. - This file contains cluster-wide settings, including the topic mapping rules you configure to map topics to streams. See Mapping Topics to Streams.
- A template of this file is located in
/opt/mapr/data-access-gateway/conf
askafka-cluster.conf.src
. - You must copy the template file (
kafka-cluster.conf.src
) to the distributed file system under/opt/kafka-wire-protocol/default-cluster/conf
and save it askafka-cluster.conf
.
- This file is stored in
Create Topic Mapping Rules and Streams
Create topic mapping rules and corresponding streams, as described in Mapping Topics to Streams.
IMPORTANT If Apache Kafka Wire Protocol Service does not
find the streams configured in
kafka-cluster.conf
in the Data Fabric
cluster upon DAG start up, the following message is logged in the DAG log file and
Apache Kafka Wire Protocol Service
is
disabled:ERROR [main] KafkaServer - The stream '</path/to/configured/stream>' is specified in the Kafka Wire Protocol configuration file (or loaded from default configuration), but it doesn't exist in the MapR Filesystem. Kafka Wire Protocol service will be disabled.
To
resolve the issue, create the missing stream(s) in the Data Fabric cluster and restart the
DAG service.Configuring Apache Kafka Clients and Applications
No special configuration is required on the client side; however, an application must modify the connection string (bootstrap server parameter) to point to the DAG hostname/IP (Apache Kafka Wire Protocol Service). This is similar to pointing to another Kafka cluster when using Apache Kafka.