Spark 1.6.1-1707 Release Notes
This section provides reference information, including new features, fixes, known issues, and limitations for Spark 1.6.1-1707.
The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 1.6.1 Release Notes.
Spark Version | 1.6.1 |
Release Date | August 2017 |
MapR Version Interoperability | See EEP Components and OS Support. |
Source on GitHub | https://github.com/mapr/spark |
GitHub Release Tag | 1.6.1-mapr-1707 |
Maven Artifacts | https://repository.mapr.com/maven/ |
Package Names | See Package Names for Ezmeral Ecosystem Packs (EEPs) |
IMPORTANT
- To integrate Spark 1.6.1 with MapR Streams, you must install the latest Kafka 0.9.0.0 package.
- Full support of MapR Streams is available only on MapR 5.2 and later clusters.
- When integrating Hive with Spark 2.0.1-1707, use Hive 1.2.-1707, which contains the fix for MAPR-26310.
Hive Support
This version of Spark supports integration with Hive. However, note the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but it is not fully compatible with Hive. For details, see the Apache Spark documentation and the MapR Spark documentation.
Fixes
This MapR release includes the following new fixes since the latest MapR Spark 1.6.1 release. For details, refer to the commit log for this project in GitHub.
GitHub Commit | Date (YYYY-MM-DD) | MapR Fix Number and Description |
---|---|---|
c352e23 | 2017/05/16 | [MAPR-27339] Fix failures when Spark jobs write to Hive tables. |
0fdd46e | 2017/05/15 | [SPARK-16664][SQL] Fix persist calls on DataFrames with more than 200 columns. |
Known Issues and Limitations
- MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
- MAPR-19761: On a secure cluster, MapR software does not support the Spark SQL Thrift JDBC server. When the cluster is secure, the Spark Thrift server will not start.
- Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability
Resolved Issues
None.