Using the MapR Event Store For Apache Kafka for S3 Bucket Event Notifications

You can use the MapR Event Store For Apache Kafka to receive event notifications for buckets. For example, when an object is created in a bucket. Install the event store, create a stream and topic, and then enable notifications in /opt/mapr/objectstore-client/objectstore-client-<version>/conf/config.json. You can then configure S3 event notifications through the S3 REST API, and use the AWS CLI to manage the notifications.

About this task

NOTICE The S3 gateway is included in EEP 6.0.0 - EEP 8.0.0 repositories.

About this task

To enable bucket-notification functionality, complete the following steps:

Procedure

Install the mapr-kafka* package:
```
zypper|yum|apt-get install mapr-kafka
```

Create a stream and topic:

maprcli stream create -path /path/to/stream
maprcli stream topic create -path /path/to/stream -topic nameOfTopic

Set up /opt/mapr/objectstore-client/objectstore-client-<version>/conf/config.json in block notifications:

"notify": {
  "kafka": {
    "notifyBlockName": {
      "enable": true,
      "brokers": [""],
      "topic": "/path/to/stream:nameOfTopic"
      }
   }
}

Restart the S3 gateway.

What to do next

Install and configure the AWS CLI to manage notifications from the command line. See Install and Configure the AWS CLI for installation and configuration details.

The following examples demonstrate some useful commands:

To read a topic:

./kafka/kafka-<version>/bin/kafka-console-consumer.sh --topic /path/to/stream:nameOfTopic --bootstrap-server some:9092 --from-beginning

To add notifications to a bucket:

aws s3api --endpoint-url http://node1.cluster.com:9000 put-bucket-notification-configuration  --notification-configuration file://fileWithNotificationRules.json --bucket bucketName

To get notifications for a bucket:

aws s3api --endpoint-url https://node1.cluster.com:9000 get-bucket-notification-configuration  --bucket bucketName

Configure S3 Event Notifications

You can configure event notifications in the S3 REST API if you want to receive notifications about specific events that happen in a bucket. To enable notifications, add a notification configuration that identifies the events you want published to a topic. For additional details and instruction, see Configuring Amazon S3 Event Notifications.

About this task

The following examples demonstrate how to configure event notifications:

Create an object event for all names

{
  "QueueConfigurations": [
  {
   "Id": "1",
   "QueueArn":  "arn:minio:sqs::notifyBlockName:kafka",
   "Events": [
       "s3:ObjectCreated:*"
      ],
   "Filter": {
       "Key": {
       "FilterRules": [
      {
        "Name": "prefix",
        "Value": "*"
       }
     ]
    }
   }
  }
 ]
}

Remove object events that have names with the mybuck prefix

{
  "QueueConfigurations": [
    {
     "Id": "1", 
     "QueueArn": "arn:minio:sqs::notifyBlockName:kafka", 
     "Events": [
          "s3:ObjectRemoved:*"
         ], 
       "Filter": {
            "Key": {
            "FilterRules": [
              {
                "Name": "prefix", 
                "Value": "mybuck"
              }
            ]
          }
        }
      }
    ]
 }