Spark 2.1.0-1710 Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 2.1.0-1710.
The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.1.0 Release Notes.
Spark Version | 2.1.0 |
Release Date | November 2017 |
MapR Version Interoperability | See EEP Components and OS Support. |
Source on GitHub | https://github.com/mapr/spark |
GitHub Release Tag | 2.1.0-mapr-1710 |
Maven Artifacts | https://repository.mapr.com/maven/ |
Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
IMPORTANT
- Full support of MapR Streams is available only on MapR 5.2 and later clusters.
- Spark 2.1 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
- Spark Standalone and Spark on YARN can only run on clusters in MRv2 (YARN) mode. They are not supported on clusters in MRv1 (classic) mode.
- MapR 6.0 and EEP 4.0 introduce "Simplified Security". If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.
Hive Support
This version of Spark supports integration with Hive. However, note the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but it is not fully compatible with Hive. For details, see the Apache Spark documentation and the MapR Spark documentation.
New in This Release
- Simplified Security - Starting in the MapR 6.0 and EEP 4.0 releases, you can use the
"Enable Security" check box in the installer to enable security for the core platform and
the installed ecosystem components. Alternatively, running
configure.sh -R
enables Spark security features if you have enabled security on your MapR cluster. See Security with Spark Standalone and Security with Spark on YARN for more information.
Fixes
This MapR release includes the following new fixes since the latest MapR Spark 2.1.0 release. For details, refer to the commit log for this project in GitHub.
GitHub Commit | Date (YYYY-MM-DD) | Comment |
---|---|---|
f1a4e96 | 2017/08/03 | [SPARK-16845][SQL] Fix code generation to prevent `GeneratedClass$SpecificOrdering` from growing beyond 64 KB |
13def70 | 2017/08/07 | [MAPR-28339] Fix race condition resulting in delete failures, when Spark SQL queries use "SaveAsTable" API |
c64da80 | 2017/08/09 | [MAPR-20331][SQL] Enhance Hive partition pruning predicate pushdown |
b27ea82 | 2017/08/17 | [MAPR-28705] Fix late arriving message |
c170554 | 2017/08/17 | [MAPR-18971][CORE] Upgrade Netty to 4.0.43.Final version |
0de12ba | 2017/08/18 | [MAPR-28659] Fix issue where executor threads in Spark executor are stuck in a lock state |
6a33e6f | 2017/08/28 | [MAPR-19307][PYSPARK] Ensure user conf is propagated to SparkContext |
85004b1 | 2017/08/29 | [MAPR-28460] Fix impersonation when data read from HPE Ezmeral Data Fabric Database via Spark-Hive |
a75bbe8 | 2017/08/29 | [SPARK-39] Remove ambiguous dependencies from Spark classpath |
c5a87b0 | 2017/09/06 | [MAPR-29052] Use waitForConsumerAssignment() instead of consumer.poll(0) to avoid initialization error in HPE Ezmeral Data Fabric Streams client |
cf96fdd | 2017/09/11 | [MAPR-29106] - Fix unsafe deserialization in Apache Spark launcher API |
7942de3 | 2017/09/18 | [SPARK-18991][CORE] Change ContextCleaner.referenceBuffer to use ConcurrentHashMap to make it faster |
3d44f64 | 2017/09/18 | [SPARK-45] Move Spark-OJAI connector code to Spark github repo |
e2a4e2a | 2017/09/25 | [SPARK-20358][CORE] Fix executors failing stage on interrupted exception by cancelled tasks |
e39c4b7 | 2017/10/03 | [SPARK-69] Fix license problem when reading from JSON and writing to HPE Ezmeral Data Fabric Database |
d444627 | 2017/10/06 | [MAPR-29014]Fix message offset for MapR Core 6.0 |
Known Issues
- MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
- Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability
Resolved Issues
None.