Distributed Worker Configuration Options

group.id

A unique string that identifies the Connect cluster group that the worker belongs to.

config.storage.topic

The name of the MapR Event Store For Apache Kafka topic to store connector and task configuration data in. This must be the same for all workers with the same group.id.

For example: /path/to/stream:topic-prefix-

status.storage.topic

The name of the MapR Event Store For Apache Kafka topic where connector and task configuration updates are stored. This must be the same for all workers with the same group.id.

offset.storage.topic

The MapR Event Store For Apache Kafka topic to store offset data for connectors in. This must be the same for all workers with the same group.id.

For example: /path/to/stream:topic-prefix-

heartbeat.interval.ms

The expected time between heartbeats to the group coordinator when using Kafka’s group management facilities.

Heartbeats are used to ensure that the worker’s session stays active and to facilitate rebalancing when new members join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

session.timeout.ms

The timeout used to detect failures when using Kafka’s group management facilities.

connections.max.idle.ms

Close idle connections after the number of milliseconds specified by this config.

receive.buffer.bytes

The size of the TCP receive buffer (SO_RCVBUF) to use when reading data.

request.timeout.ms

The configuration controls the maximum amount of time the client will wait for the response of a request.

If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

send.buffer.bytes

The size of the TCP send buffer (SO_SNDBUF) to use when sending data.

worker.sync.timeout.ms

When the worker is out of sync with other workers and needs to resynchronize configurations, wait up to this amount of time before giving up, leaving the group, and waiting a backoff period before rejoining.

worker.unsync.backoff.ms

When the worker is out of sync with other workers and fails to catch up within worker.sync.timeout.ms, leave the Connect cluster for this long before rejoining.

client.id

An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just IP/port by allowing a logical application name to be included in server-side request logging.

metadata.max.age.ms

The period of time in milliseconds after which we force a refresh of metadata even if we haven’t seen any partition leadership changes to proactively discover any new brokers or partitions.

metric.reporters

A list of classes to use as metrics reporters. Implementing the MetricReporterinterface allows plugging in classes that will be notified of new metric creation. The JmxReporter is always included to register JMX statistics.

metrics.num.samples

The number of samples maintained to compute metrics.

metrics.sample.window.ms

The number of samples maintained to compute metrics.

reconnect.backoff.ms

The amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all requests sent by the consumer to the broker.

retry.backoff.ms

The amount of time to wait before attempting to retry a failed fetch request to a given topic partition. This avoids repeated fetching-and-failing in a tight loop.