Spark 2.0.1-1707 Release Notes
This section provides reference information, including new features, fixes, known issues, and limitations for Spark 2.0.1-1707.
The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.0.1 Release Notes.
| Spark Version | 2.0.1 |
| Release Date | August 2017 |
| MapR Version Interoperability | See EEP Components and OS Support. |
| Source on GitHub | https://github.com/mapr/spark |
| GitHub Release Tag | 2.0.1-mapr-1707 |
| Maven Artifacts | https://repository.mapr.com/maven/ |
| Package Names | See Package Names for Ezmeral Ecosystem Packs (EEPs) |
IMPORTANT
- Full support of MapR Streams is available only on MapR 5.2 and later clusters.
- You cannot submit Spark jobs in YARN mode when the cluster is running in MRv1(classic) mode. This applies even if you have installed and configured YARN in your cluster.
- When integrating Hive with Spark 2.0.1-1707, use Hive 1.2.-1707, which contains the fix for MAPR-26310.
Hive Support
This version of Spark supports integration with Hive. However, note the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but it is not fully compatible with Hive. For details, see the Apache Spark documentation and the MapR Spark documentation.
Fixes
This MapR release includes the following new fixes since the latest MapR Spark 2.0.1 release. For details, refer to the commit log for this project in GitHub.
| GitHub Commit Number | Date (YYYY-MM-DD) | MapR Fix Number and Description |
|---|---|---|
| bcb1640 | 2017/05/22 | [MAPR-27519] Improve performance of calculating web UI counters for Kafka-streaming. |
| ab0a3f2 | 2017/05/16 | [SPARK-19019][PYTHON][BRANCH-2.0] Fix hijacked `collections.namedtuple`. Port cloudpickle changes needed for PySpark to work with Python 3.6.0. |
| 9faac4a | 2017/05/05 | [MAPR-26414] Fix Spark History Server memory leak. |
| 46b1913 | 2017/05/04 | [SPARK-33] Fix streaming example. |
| ab1f040 | 2017/05/04 | [SPARK-19146][CORE] Drop more elements when stageData.taskData.size > retainedTasks. |
| 5e61eb7 | 2017/05/04 | [MAPR-26287] Remove unnecessary code from hadoop-version-picker.sh. |
| 2b16ad3 | 2017/05/04 | [MAPR-26414] Fix Spark History Server memory leak. |
Known Issues and Limitations
- MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
- MAPR-25052: Spark Thrift Server does not start on clusters secured by MapR-SASL.
- Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability
Resolved Issues
None.