Spark 2.1.0-1801 Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.1.0-1801.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.1.0 Release Notes.

Spark Version 2.1.0
Release Date February 2018
MapR Version Interoperability See EEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.1.0-mapr-1801
Maven Artifacts https://repository.mapr.com/maven/
Package Names Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names.
IMPORTANT
  • Full support of MapR Streams is available only on MapR 5.2 and later clusters.
  • Spark 2.1 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
  • Spark Standalone and Spark on YARN can only run on clusters in MRv2 (YARN) mode. They are not supported on clusters in MRv1 (classic) mode.
  • MapR 6.0 and EEP 4.0 and later introduce built-in security. If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

New in This Release

Fixes

This MapR release includes the following new fixes since the latest MapR Spark 2.1.0 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
5430a1d 2017/10/31 MapR [SPARK-107] idField information is lost in MapRDBDataFrameWriterFunctions.saveToMapRDB
68c211b 2017/11/16 MapR [SPARK-113] Hit java.lang.UnsupportedOperationException: empty.reduceLeft during loadFromMapRDB
a325770 2017/11/27 MapR [SPARK-125] Enable handling default value of idFieldPath parameter
737e2ac 2017/11/28

[SPARK-18827][CORE] Fix cannot read broadcast on disk
f02a1dc 2017/11/28 [SPARK-19104][BACKPORT-2.1][SQL] Lambda variables in ExternalMapToCatalyst is made global
594d5d4 2017/11/29 MapR [SPARK-121] Spark OJAI JAVA: Read to Dataset functionality implementation
2a8a6c1 2017/11/29 MapR [SPARK-128] HPE Ezmeral Data Fabric Database connector - Fix wrong handle of null fields when nullable is false
06c6597 2017/12/05 MapR [SPARK-131] Exception when trying to save JSON table with Binary _id field
b273661 2017/12/05 MapR [SPARK-118] Spark OJAI Python: Read implementation
b8adcd0 2017/12/05 MapR [SPARK-117] Spark OJAI Python: Save functionality implementation
ef88f8a 2017/12/13 MapR [SPARK-118] Spark OJAI Python: Move HPE Ezmeral Data Fabric Database Connector class importing in order to fix MapR [ZEP-101] interpreter issue
3d7e193 2017/12/13 MapR [SPARK-118] Spark OJAI Python: Missed DataFrame import while moving imports in order to fix MapR [ZEP-101] interpreter issue
7e3e1e7 2017/12/14 MapR [SPARK-121] Spark OJAI JAVA: Update functionality removed
5f2dd1d 2017/12/26 MapR [SPARK-140] Change the option name "tableName" to "tablePath" in the Spark/HPE Ezmeral Data Fabric Database connectors
c7f2f8a 2017/12/28 MapR [SPARK-139] Remove "update" related APIs from connector
496f040 2017/12/28 [SPARK-21321][SPARK CORE] Spark very verbose on shutdown
0a08b10 2018/01/18 [MAPR-30536] Spark SQL queries on Map column fails after upgrade

Known Issues

Resolved Issues

Users logged in with a normal user account (not mapr or root) can run spark jobs on the cluster without disabling Spark SSL.