Spark 2.1.0-1803 (EEP 3.0.3) Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 2.1.0-1803.
The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.1.0 Release Notes.
| Spark Version | 2.1.0 |
| Release Date | March 2018 |
| MapR Version Interoperability | See EEP Components and OS Support. |
| Source on GitHub | https://github.com/mapr/spark |
| GitHub Release Tag | 2.1.0-mapr-1803 |
| Maven Artifacts | https://repository.mapr.com/maven/ |
| Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
IMPORTANT
- Spark 2.2 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
- Spark Yarn and Standalone modes are supported only on clusters in MRv2 (YARN) mode. It is not supported on clusters in MRv1 (classic) mode.
Hive Support
This version of Spark supports integration with Hive. However, note the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but it is not fully compatible with Hive. For details, see the Apache Spark documentation and the MapR Spark documentation.
New in This Release
- None.
Fixes
This MapR release includes the following new fixes since the latest MapR Spark 2.2.1 release. For details, refer to the commit log for this project in GitHub.
| GitHub Commit | Date (YYYY-MM-DD) | Comment |
|---|---|---|
| 0a08b10 | 2018/01/18 | [MAPR-30536] Spark SQL queries on Map column failed after upgrade |
| 46da505 | 2018/02/07 | [MAPR-30583] PartitioningAwareFileIndex changed to getFileBlockLocations in a parallel way |
| bf7296a | 2018/02/08 | MapR [SPARK-152] Incorrect date string parsing fixed |
| 4c01f15 | 2018/03/01 | [SPARK-21181] Release byteBuffers to suppress netty error messages |
| 9473a5f | 2018/03/07 | MapR [SPARK-171] StackOverflowException in equals method in DBMapValue |
| 3b5d608 | 2018/03/19 | MapR [SPARK-190] Spark R NoSuchElementException during start |
Known Issues
- The following error is displayed at times during the running of Spark application:
ERROR MapRFileSystem: Failed to delete path, error: No such file or directory (Tickets: MapR-28620, MapR-25770, MFS-1897) - Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability
Resolved Issues
None.