Migrating Kafka Python Applications to MapR Event Store For Apache Kafka
With some modification, you can use existing confluent-kafka python applications to consume and produce topics in MapR Event Store For Apache Kafka. The MapR Event Store For Apache Kafka Python Client is a binding for Apache librdkafka that works with MapR Event Store For Apache Kafka.
- Install the MapR Event Store For Apache Kafka Python Client. NOTE This required that you also install and configure the MapR Event Store For Apache Kafka C Client. See Configuring the MapR Event Store For Apache Kafka C Client.
- Do one of
the following depending on whether you are using the EEP 5.0 (or higher) MapR Event Store For Apache Kafka Python
Client or the EEP 3.0 (or higher) MapR Event Store For Apache Kafka Python Client.
- If you are using MapR Event Store For Apache Kafka Python EEP 5.0 (or higher), skip this step. The references to
confluent_kafka
should be retained. - If you are using MapR Event Store For Apache Kafka Python EEP 3.0 (or higher), update import statements to refer
to the MapR Stream Python API. References to
confluent_kafka
should be updated tomapr_streams_python
.NOTE For example, updatefrom confluent_kafka import Consumer
tofrom mapr_streams_python import Consumer
.
- If you are using MapR Event Store For Apache Kafka Python EEP 5.0 (or higher), skip this step. The references to
- When you refer to a topic in the application code, include the path and name of the stream
in which the topic is located:
/<path and name of stream>:<name of topic>
For example, you might have a stream in a MapR Data Platform cluster that is named stream_A, and the stream might be in a volume named IoT and in a directory named automobile_sensors. You want to redirect a producer application to a topic in that stream. The syntax of the path to the topic might look like this:
/mapr/IoT/automobile_sensors/stream_A:<name of topic>
.NOTE Optionally, use thestreams.consumer.default.stream
andstreams.producer.default.stream
configuration parameters. When you configure these parameters, applications can specify just the topic name to write or read from the default stream. - Review the APIs that are supported and make changes to your application, as needed. See API for MapR Event Store For Apache Kafka Python Client.
- See Configuration Properties for MapR Event Store For Apache Kafka Python Client for the list of supported configuration
parameters and make changes to your application, as needed. NOTE SSL-related configuration parameters are ignored. When you set these parameters, the MapR Event Store For Apache Kafka Client issues a warning indicating that the parameters are not supported.