Spark 2.2.1-1803 (EEP 5.0.0) Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 2.2.1-1803.
The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.2.1 Release Notes.
| Spark Version | 2.2.1 |
| Release Date | March 2018 |
| MapR Version Interoperability | See EEP Components and OS Support. |
| Source on GitHub | https://github.com/mapr/spark |
| GitHub Release Tag | 2.2.1-mapr-1803 |
| Maven Artifacts | https://repository.mapr.com/maven/ |
| Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
IMPORTANT
- Spark 2.2 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
- Starting from Spark 2.2.1 and EEP 5.0.0 Kafka is updated to 1.0.1.
- MapR 6.0 and EEP 5.0 and later introduce built-in security. If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.
- Spark Master port was changed to 8580.
Hive Support
This version of Spark supports integration with Hive. However, note the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but it is not fully compatible with Hive. For details, see the Apache Spark documentation and the MapR Spark documentation.
New in This Release
- Support for Structured Streaming. See Structured Spark Streaming.
- Structured Streaming HPE Ezmeral Data Fabric Database OJAI Sink. See Structured Spark Streaming.
- PAM for Spark Web UIs on secure clusters. See PAM Authentication for Spark.
Fixes
This MapR release includes the following new fixes since the latest MapR Spark 2.2.1 release. For details, refer to the commit log for this project in GitHub.
| GitHub Commit | Date (YYYY-MM-DD) | Comment |
|---|---|---|
| 5430a1d | 2018/01/22 | MapR [SPARK-16] Spark 2.2.1 porting |
| b3a4ffa | 2018/01/22 | [MAPR-30228] Spark example job failed with "/opt/mapr/conf/ssl_keystore (Permission denied)" error on Spark 2.1.X EEP 4.0 - MapR SASL security enabled |
| aeb9e9d | 2018/01/22 | MapR [SPARK-144] Add insertToMapRDB method for rdd for Java API |
| 489e21d | 2018/01/23 | MapR [SPARK-16] Change Spark version in Warden files and configure.sh |
| 548e492 | 2018/02/01 | MapR [SPARK-143] Added PAM for Spark UIs for secure clusters |
| 2499aba | 2018/02/06 | MapR [SPARK-135] Spark 2.2 with MapR Streams (Kafka 1.0) |
| 1b37b08 | 2018/02/07 | MapR [SPARK-21] Structured Streaming HPE Ezmeral Data Fabric Database Sink created |
| 2d9e466 | 2018/02/07 | [MAPR-30583] InMemoryFileIndex changed to getFileBlockLocations in a parallel way |
| 1a2e864 | 2018/02/08 | MapR [SPARK-152] Incorrect date string parsing fixed |
| cca410c | 2018/02/15 | MapR [SPARK-153] Exception in spark job with configured labels on yarn-client mode |
| 6f83937 | 2018/02/20 | MapR [SPARK-159] Added possibility to configure secure ports as part of security by default |
| f341e85 | 2018/02/20 | MapR [SPARK-155] Changed Spark Master port from 8080 |
| f417a68 | 2018/02/26 | MapR [SPARK-161] Include Kafka Structured streaming jar to Spark package |
| 77ed36f | 2018/02/28 | MapR [SPARK-164] Update Kafka version to 1.0.1-mapr in Spark Kafka Producer module |
| 5a390c1 | 2018/03/07 | MapR [SPARK-170] StackOverflowException in equals method in DBMapValue |
| 45cea4a | 2018/03/19 | MapR [SPARK-154] Spark R NoSuchElementException during start |
| 3725eed | 2018/03/21 | MapR [SPARK-191] Incorrect work of HPE Ezmeral Data Fabric Database Sink 'complete' and 'update' modes fixed |
| 844642e | 2018/03/23 | MapR [SPARK-143] Spark History Server does not require login for secured-by-default clusters |
| 71cd6d2 | 2018/03/25 | MapR [SPARK-194] Redirect to Spark History server |
| 1c093b1 | 2018/03/26 | MapR [SPARK-188] Could not connect to thrift server via spark beeline on kerberos cluster |
| 4eff5df | 2018/04/02 | [SPARK-198] Update hadoop dependency version to 2.7.0-mapr-1803 for Spark 2.2.1 |
| d324bfa | 2018/04/04 | [SPARK-205] Change Kafka dependencies versions |
Known Issues
Spark versions up to and including 2.3.0 have the following security vulnerability:
Resolved Issues
None.