KSQL

KSQL is an open-source streaming SQL engine that implements continuous, interactive queries.

Use KSQL to query, read, write, and process data in real-time, at scale, through SQL commands. KSQL interacts directly with the Kafka Streams API, eliminating the need for a Java application.

The KSQL data flow architecture is designed such that the user interacts with the KSQL server and the KSQL server interacts with the MapR Event Store For Apache Kafka server.

Use Cases

Common use cases include fraud detection, personalization, notifications, real-time analytics, and sensor data and IoT.

Architecture

A set of KSQL processes run as a cluster, and the KSQL server process completes queries. You can dynamically add more processing capacity by starting more instances of the KSQL server. These instances are fault-tolerant: if one fails, the others continue the work. Queries are launched using the interactive KSQL command line client, which sends commands to the cluster over a REST API. The command line allows you to inspect the available streams and tables, issue new queries, check the status of and terminate running queries.

KSQL Server
The KSQL server runs the engine that completes KSQL queries. This includes processing, reading, and writing data to and from the target Kafka cluster. KSQL servers form KSQL clusters and can run in containers, virtual machines, and bare-metal machines. You can add and remove servers in a KSQL cluster during live operations to elastically scale processing capacity. You can deploy different KSQL clusters to achieve workload isolation.
KSQL CLI
You can interactively write KSQL queries through the KSQL command line interface (CLI). The KSQL CLI acts as a client to the KSQL server.

KSQL Deployment Modes

You can deploy KSQL queries through Interactive or Non-Interactive mode.

Interactive Mode

In Interactive mode, users interact with the KSQL server through a REST API, such as the KSQL CLI. Interactive mode is useful when users need to write and verify their queries interactively on a shared KSQL cluster. In interactive KSQL clusters, the authenticated KSQL user must have open access to create, read, write, delete topics, and use of any consumer group.

Interactive KSQL clusters are not supported in a production environment. When you deploy queries to the production environment, lock-down access to KSQL servers by deploying a non-interactive (headless) environment.

Non-Interactive Mode (Headless Mode)

Non-interactive mode supports locked-down or “headless” deployment scenarios where interactive use of the KSQL cluster is disabled, thereby preventing KSQL CLI access. In this mode, you can write queries to an SQL file, which allows for version control, and lock down access to KSQL servers to prevent users from interacting directly with the KSQL cluster.

Non-interactive KSQL clusters (headless mode) is the main deployment model for KSQL queries on MapR (similar to the Apache Kafka model). Use non-interactive mode for production scenarios.

For More Information