Spark 2.1.0-1801 Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 2.1.0-1801.
The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.1.0 Release Notes.
Spark Version | 2.1.0 |
Release Date | February 2018 |
MapR Version Interoperability | See EEP Components and OS Support. |
Source on GitHub | https://github.com/mapr/spark |
GitHub Release Tag | 2.1.0-mapr-1801 |
Maven Artifacts | https://repository.mapr.com/maven/ |
Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
IMPORTANT
- Full support of MapR Streams is available only on MapR 5.2 and later clusters.
- Spark 2.1 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
- Spark Standalone and Spark on YARN can only run on clusters in MRv2 (YARN) mode. They are not supported on clusters in MRv1 (classic) mode.
- MapR 6.0 and EEP 4.0 and later introduce built-in security. If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.
Hive Support
This version of Spark supports integration with Hive. However, note the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but it is not fully compatible with Hive. For details, see the Apache Spark documentation and the MapR Spark documentation.
New in This Release
- Support for Java and Python APIs for HPE Ezmeral Data Fabric Database OJAI connector. See HPE Ezmeral Data Fabric Database OJAI Connector for Apache Spark.
Fixes
This MapR release includes the following new fixes since the latest MapR Spark 2.1.0 release. For details, refer to the commit log for this project in GitHub.
GitHub Commit | Date (YYYY-MM-DD) | Comment |
---|---|---|
5430a1d | 2017/10/31 | MapR [SPARK-107] idField information is lost in MapRDBDataFrameWriterFunctions.saveToMapRDB |
68c211b | 2017/11/16 | MapR [SPARK-113] Hit java.lang.UnsupportedOperationException: empty.reduceLeft during loadFromMapRDB |
a325770 | 2017/11/27 | MapR [SPARK-125] Enable handling default value of idFieldPath parameter |
737e2ac | 2017/11/28 |
[SPARK-18827][CORE] Fix cannot read broadcast on disk |
f02a1dc | 2017/11/28 | [SPARK-19104][BACKPORT-2.1][SQL] Lambda variables in ExternalMapToCatalyst is made global |
594d5d4 | 2017/11/29 | MapR [SPARK-121] Spark OJAI JAVA: Read to Dataset functionality implementation |
2a8a6c1 | 2017/11/29 | MapR [SPARK-128] HPE Ezmeral Data Fabric Database connector - Fix wrong handle of null fields when nullable is false |
06c6597 | 2017/12/05 | MapR [SPARK-131] Exception when trying to save JSON table with Binary _id field |
b273661 | 2017/12/05 | MapR [SPARK-118] Spark OJAI Python: Read implementation |
b8adcd0 | 2017/12/05 | MapR [SPARK-117] Spark OJAI Python: Save functionality implementation |
ef88f8a | 2017/12/13 | MapR [SPARK-118] Spark OJAI Python: Move HPE Ezmeral Data Fabric Database Connector class importing in order to fix MapR [ZEP-101] interpreter issue |
3d7e193 | 2017/12/13 | MapR [SPARK-118] Spark OJAI Python: Missed DataFrame import while moving imports in order to fix MapR [ZEP-101] interpreter issue |
7e3e1e7 | 2017/12/14 | MapR [SPARK-121] Spark OJAI JAVA: Update functionality removed |
5f2dd1d | 2017/12/26 | MapR [SPARK-140] Change the option name "tableName" to "tablePath" in the Spark/HPE Ezmeral Data Fabric Database connectors |
c7f2f8a | 2017/12/28 | MapR [SPARK-139] Remove "update" related APIs from connector |
496f040 | 2017/12/28 | [SPARK-21321][SPARK CORE] Spark very verbose on shutdown |
0a08b10 | 2018/01/18 | [MAPR-30536] Spark SQL queries on Map column fails after upgrade |
Known Issues
- Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability
Resolved Issues
Users logged in with a normal user account (not mapr or root) can run spark jobs on the cluster without disabling Spark SSL.