Read HPE Ezmeral Data Fabric Streams Using a Kafka Source
IMPORTANT This component is deprecated. Hewlett Packard
Enterprise recommends using an alternate product. For more information, see Discontinued Ecosystem Components.
The following parameters are required when configuring Kafka source to read data from MapR
Streams topics:
Property Name | Description |
---|---|
channels | If the source writes events to a channel, this is the name of the channel that the source writes events to. |
type | This property must be set
to
|
kafka.consumer.group.id | A unique identifier of the consumer group. This defaults to
flume . Setting the same id in multiple sources or agents indicates that they
are part of the same consumer group; when these sources or agents are running at the
same time, they will each read a unique set of partitions for the topics in the group.
|
kafka.topics | A comma separated list of HPE Ezmeral Data Fabric Streams topics where each topic is specified with the
volume path and stream name. For example:
NOTE It
is critical that the path to the topic starts with a slash(/), as the slash is what
distinguishes the topic as a HPE Ezmeral Data Fabric Streams topic. |
batchDurationMillis | Maximum number of milliseconds to wait before closing a batch. The default value is 1000. |
For additional properties that you may want to configure, see the Flume
documentation. However, note that kafka.bootstrap.servers
is not
required for reading HPE Ezmeral Data Fabric Streams.
Tip: To increase throughput, set the batchSize
to a higher value. The
batchSize
is the maximum number of messages written to Channel in one
batch. By default, it is set to 1000.