Home
6.0 Development
This section contains information related to application development for ecosystem components and MapR products including MapR-DB (binary and JSON), MapR-FS, and MapR Streams.
MapR-ES and Apps
MapR-ES brings integrated publish and subscribe messaging to the MapR Converged Data Platform.
MapR-ES Java Applications
This section contains information on developing client applications with Java including information about the MapR-ES and Apache Kafka Java APIs, configuration parameters, and compiling and running producers and consumers.

MapR 6.0 Documentation

6.0 Development
This section contains information related to application development for ecosystem components and MapR products including MapR-DB (binary and JSON), MapR-FS, and MapR Streams.
- Application Development Process
  Before you start developing applications on MapR’s Converged Data Platform, consider how you will get the data onto the platform, the format it will be stored in, the type of processing or modeling that is required, and how the data will be accessed.
- MapR-FS and Apps
  The following sections provide information about accessing MapR-FS with C and Java applications.
- MapR-DB and Apps
  This section contains information about developing client applications for JSON and binary tables.
- MapR-ES and Apps
  MapR-ES brings integrated publish and subscribe messaging to the MapR Converged Data Platform.
  - Getting Started with MapR-ES
    If you have a basic understanding of MapR-ES's components and the typical flow of messages from producers to consumers, you can give MapR-ES a try.
  - Consuming CDC Records
    The OJAI changelog interfaces are used to consume changed data records (propagated by the Change Data Capture feature).
  - Consuming Audit Logs
    Audit Streaming (available from v6.0.1) provides a way to process the audit data in real-time.
  - MapR-ES Java Applications
    This section contains information on developing client applications with Java including information about the MapR-ES and Apache Kafka Java APIs, configuration parameters, and compiling and running producers and consumers.
    - MapR-ES Java API Library
      Use the MapR-ES Admin Java API library as an alternative to maprcli commands and the REST APIs for performing administrative tasks on streams and topics. This library can also be used for analysis of the contents of streams.
    - Apache Kafka Java APIs
      MapR-ES supports these Apache Kafka Java APIs.
    - Configuration Parameters
      This topic describes configuration parameters that are either specific to MapR-ES or supported from Apache Kafka.
    - Compiling and Running MapR-ES Java Apps
      For producer and consumer applications that use the MapR-ES Java API, use Maven to compile and determine the application's dependencies. Then, when you run the application, specify those dependencies in the application's classpath.
    - Migrating Apache Kafka Java Applications to MapR-ES
      There are only two steps that you need to follow to migrate applications written with the Apache Kafka Java API to MapR-ES.
    - Differences between MapR-ES and Apache Kafka Configuration
      Describes the MapR-ES supportability of Apache Kafka configuration parameters for producers and consumers.
  - MapR-ES C Applications
    C applications can be developed for MapR-ES (as of MapR 5.2.1). The MapR-ES C Client is a distribution of librdkafka that works with MapR-ES. The MapR-ES C Client is available in MapR Ecosystem Pack (MEP) 3.0 or higher.
  - MapR-ES Python Applications
    As of MapR 5.2.1, you can create python applications for MapR-ES using the MapR-ES Python client. The MapR-ES Python client is a binding for librdkafka and the MapR-ES C Client is a distribution of librdkafka that works with MapR-ES.
  - MapR-ES C#/.NET Applications
    As of MapR 6.0.1/MEP5.0, you can create C#/.NET applications for MapR-ES using the MapR-ES C#/.NET client. The MapR-ES C#/.NET client is a binding for librdkafka and the MapR-ES C Client is a distribution of librdkafka that works with MapR-ES.
  - Utilities for MapR-ES
    MapR-ES provides the utilities for operating on streams and topics.
  - Configuring Properties for Message Size
    Describes the message.max.bytes and receive.message.max.bytes properties for configuring message size.
- MapReduce and Apps
  This section contains information associated with developing YARN applications.
- MapR Data Science Refinery
  The MapR Data Science Refinery is an easy-to-deploy and scalable data science toolkit with native access to all platform assets and superior out-of-the-box security.
- MapR Data Fabric for Kubernetes FlexVolume Driver
  This section describes how to use and troubleshoot the MapR Data Fabric for Kubernetes FlexVolume Driver.
- Ecosystem Components
  The following sections provide information about each open source project that MapR supports.
- Maven and MapR
  This section discusses topics associated with Maven and MapR.
- Developer's Reference
  This section contains in-depth information for the developer.
- API Documentation
  MapR supports public APIs for MapR-FS, MapR-DB, and MapR-ES. These APIs are available for application development purposes.

MapR-ES Java Applications

This section contains information on developing client applications with Java including information about the MapR-ES and Apache Kafka Java APIs, configuration parameters, and compiling and running producers and consumers.

Apache Kafka Support

The following Apache Kafka Java API versions are supported:

Table 1. Supported Apache Kafka APIs
MapR version	Apache Kafka API
As of 6.0.1	1.0
6.0.0 and earlier	0.9.0

TimestampType Permissions

The following describes the ACE permissions are needed when using the timestamp type parameter. See Stream Security for general information about MapR-ES streams security.

NOTE: The TimestampType parameter is new as of 6.0.1.

A MapR-ES stream topic inherits the default timestamp type value from its stream. To override the stream's default value, set the timestamp type for the topic to a different value.

Setting the value at the stream-level requires adminperm permissions. The stream-level timestamp type parameter is defaulttimestamptype. See stream create and stream edit for more information on setting this parameter through the maprcli.
Setting the timestamptype at the topic-level requires topicperm permissions. The topic-level timestamp type parameter is timestamptype. See stream topic create and stream topic edit for more information on setting this parameter through the maprcli.

User Impersonation

NOTE: As of MapR 6.0, user impersonation is supported for MapR-ES.

You can set up user impersonation programmatically. To do so, use the UserGroupInformation.doAs() method in the Hadoop documentation. See Class UserGroupInformation for more information.

If you are setting up user impersonation in a secure cluster, you need to generation an impersonation ticket. See the Generating and Printing Service with Impersonation Ticket section in the maprlogin Command Examples topic.

After generating the ticket:

Ensure that mapruser1 has read permissions on the ticket.
If you moved the ticket file to a different location, set the $MAPR_TICKETFILE_LOCATION environment variable.

For more information about impersonation, see:

Backward Compatibility

As of MapR 6.0.1, along with support of Apache Kafka the java.util.Collection interface is being used. This impacts applications using certain APIs. See MapR-ES Java API Library for detailed information.

References

MapR-ES Sample Programs on GitHub.
Getting Started with MapR Streams on the MapR blog site.

(Topic last modified: 2018-09-18)