Configuring Apache Kafka Wire Protocol Service

Describes the Apache Kafka Wire Protocol Service configuration files and instructions.

The following sections provide configuration information for DAG (Data Access Gateway) and Apache Kafka clients and applications:

Configuring Apache Kafka Wire Protocol Service

DAG packages include the following configuration files for Apache Kafka Wire Protocol Service:
kafka-server.conf
  • This file is stored in /opt/mapr/data-access-gateway/conf and loaded from the local file system.
  • This file contains node-specific options, such as the port number and security protocol. Set the options in this file on a per-node basis.
  • By default, the Apache Kafka Wire Protocol Service listens on port 9092.
  • If the DAG service instances are behind a firewall, the firewall rules must be configured to allow this traffic.

For a list of configurable parameters, see Apache Kafka Wire Protocol Service Settings.

kafka-cluster.conf
  • This file is stored in /opt/kafka-wire-protocol/default-cluster/conf and loaded from the distributed file system.
  • This file contains cluster-wide settings, including the topic mapping rules you configure to map topics to streams. See Mapping Topics to Streams.
  • A template of this file is located in /opt/mapr/data-access-gateway/conf as kafka-cluster.conf.src.
  • You must copy the template file (kafka-cluster.conf.src) to the distributed file system under /opt/kafka-wire-protocol/default-cluster/conf and save it as kafka-cluster.conf.

Create Topic Mapping Rules and Streams

Create topic mapping rules and corresponding streams, as described in Mapping Topics to Streams.
IMPORTANT If Apache Kafka Wire Protocol Service does not find the streams configured in kafka-cluster.conf in the Data Fabric cluster upon DAG start up, the following message is logged in the DAG log file and Apache Kafka Wire Protocol Service is disabled:
ERROR [main] KafkaServer - The stream '</path/to/configured/stream>' is specified in the Kafka Wire Protocol configuration file (or loaded from default configuration), but it doesn't exist in the MapR Filesystem. Kafka Wire Protocol service will be disabled.
To resolve the issue, create the missing stream(s) in the Data Fabric cluster and restart the DAG service.

Configuring Apache Kafka Clients and Applications

No special configuration is required on the client side; however, an application must modify the connection string (bootstrap server parameter) to point to the DAG hostname/IP (Apache Kafka Wire Protocol Service). This is similar to pointing to another Kafka cluster when using Apache Kafka.