HPE Ezmeral Data Fabric Event Store Java Applications
This section contains information on developing client applications with Java including information about the HPE Ezmeral Data Fabric Event Store and Apache Kafka Java APIs, configuration parameters, and compiling and running producers and consumers.
Apache Kafka Support
MapR version | Apache Kafka API |
---|---|
As of 6.2 | 2.1 |
As of 6.1 | 1.1 |
As of 6.0.1 | 1.0 |
6.0.0 and earlier | 0.9.0 |
Log Compaction
As of MapR 6.1, log compaction is supported. Log compaction can be enabled for streams created with MapR core 6.1 and later. In addition, clients older than MapR 6.1 are prevented from consuming from streams that have had log compaction enabled on them at least once in their lifetime.
- If a replica cluster has been upgraded and the stream data for a source cluster is compacted (that is, one or more messages have been deleted), then the source cluster replicates the compacted data to the replica cluster.
- If a replica cluster has not been upgraded, then the source cluster fails the replication and an error is generated that requests an replica cluster upgrade.
In the context of a scan by a client that is not upgraded, the (upgraded) server inspects the row header to check if it is serving a compacted row. If it is serving a compacted row, then the server fails the consumer request. This behavior applies both to a stream that is explicitly configured for compaction and a replica that has received a compacted row.
-force
option can be used. The -force
option should only be used when ALL clients
have been upgraded to MapR 6.1.Idempotent Producer
As of MapR 6.1, the idempotent producer (exactly once) feature is supported. You can implement the idempotent producer with MapR core 6.1 and later.
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true)
The idempotent producer feature is supported by MEP MapR 6.0 clients and MapR 6.1.0 servers.
- You must upgrade all servers to v6.1.0 and enable all the v6.1.0 features, before you enable the idempotent producer.
- If you use a pre-MapR 6.1 client and a MapR 6.1 server, and if a group of messages are atomically persisted without a valid producer ID, the server treats the request as a non-idempotent producer.
- If you use a MapR 6.1 client and a pre-MapR 6.1 server, the idempotent producer is not
supported. In this case, the idempotent producer fails to produce to the stream and the
following excepton is
thrown:
Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: Operation not permitted (1) null at com.mapr.streams.impl.producer.MarlinFuture.valueOrError(MarlinFuture.java:46) at com.mapr.streams.impl.producer.MarlinFuture.get(MarlinFuture.java:41) at com.mapr.streams.impl.producer.MarlinFuture.get(MarlinFuture.java:17) at com.mapr.qa.marlin.common.StandaloneProducer.main(StandaloneProducer.java:75) Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: Operation not permitted (1) null
TimestampType Permissions
The following discussion describes the Access Control Expression (ACE) permissions that you need when using the timestamp type parameter. See Stream Security for general information about HPE Ezmeral Data Fabric Event Store streams security.
A HPE Ezmeral Data Fabric Event Store stream topic inherits the default timestamp type value from its stream. To override the stream's default value, set the timestamp type for the topic to a different value.
- Setting the value at the stream-level requires
adminperm
permissions. The stream-level timestamp type parameter isdefaulttimestamptype
. See stream create and stream edit for more information on setting this parameter using themaprcli
command. - Setting the
timestamptype
at the topic-level requirestopicperm
permissions. The topic-level timestamp type parameter istimestamptype
. See stream topic create and stream topic edit for more information on setting this parameter using themaprcli
command.
User Impersonation
As of MapR 6.0, user impersonation is supported for HPE Ezmeral Data Fabric Event Store.
You can set up user
impersonation programmatically. To do so, use the UserGroupInformation.doAs()
method in the Hadoop documentation. See Class UserGroupInformation for more
information.
If you are setting up user impersonation in a secure cluster, you need to generate an impersonation ticket. See the Generating and Printing Service with Impersonation Ticket section in the maprlogin Command Examples topic.
- Ensure that user
mapruser1
has read permissions on the ticket. - If you moved the ticket file to a different location, set the
$MAPR_TICKETFILE_LOCATION
environment variable with the appropriate path.
Backward Compatibility
As of MapR 6.0.1, along with the support of Apache Kafka, the
java.util.Collection
interface is being used. This impacts applications
using certain APIs. See HPE Ezmeral Data Fabric Event Store Java API Library for detailed information.
References
- HPE Ezmeral Data Fabric Event Store Sample Programs on GitHub.
- Getting Started with MapR Streams on the MapR blog site.