Spark 2.2.1-1901 (EEP 5.0.2) Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 2.2.1.
The notes below relate specifically to the MapR Distribution for Apache Hadoop. This release of Spark has backward-compatibility changes, see the open-source Spark 2.1.0 Release Notes for more information.
These release notes contain only MapR-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.
Spark Version | 2.2.1 |
Release Date | February 2019 |
MapR Version Interoperability | See EEP Components and OS Support. |
Source on GitHub | https://github.com/mapr/spark |
GitHub Release Tag | 2.2.1-mapr-1901 |
Maven Artifacts | https://repository.mapr.com/maven/ |
Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
- Spark 2.2 can connect to Hive Metastore 2.1. But, features of Hive added after Hive 1.2 are not supported by Spark.
- Starting from Spark 2.2.1 and EEP 5.0.0 Kafka version is updated to 1.0.1.
- Spark Yarn and Standalone modes are supported only on clusters in MRv2 (YARN) mode. It is not supported on clusters in MRv1 (classic) mode.
- MapR 6.0 and EEP 5.0 and later introduce security by default. If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.
Hive Support
This version of Spark supports integration with Hive. However, note the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but it is not fully compatible with Hive. For details, see the Apache Spark documentation and the MapR Spark documentation.
New in This Release
None.
Fixes
This MapR release includes the following new fixes since the latest MapR Spark 2.2.1 release. For details, refer to the commit log for this project in GitHub.
GitHub Commit | Date (YYYY-MM-DD) | Comment |
---|---|---|
e444b4a | 2018/10/01 | MapR [SPARK-316] Backporting request for SPARK-22062 and SPARK-21475 |
fc8076b | 2018/10/01 | [SPARK-22033][CORE] BufferHolder, other size checks should account for the specific VM array size limitations |
f1dd6d9 | 2018/10/17 | [MAPR-32263] Seek called on unsubscribed partitions |
874aefc | 2018/10/24 | [MAPR-32290] Spark processing offsets when messages are already ttl in first batch |
7ecab19 | 2018/11/19 | [SPARK-357] consumer groups are prepeneded with a "service_" prefix |
9cf9c83 | 2018/12/21 | [MAPR-26258] hbasecontext.HBaseDistributedScanExample fails |
f0bb7de | 2019/01/04 | MapR [SPARK-390] Fix for hive version |
b346a9f | 2019/01/04 | MapR [SPARK-311] Spark beeline uses default ssl truststore instead of mapr ssl truststore |
22d8192 | 2019/01/08 | MapR [SPARK-382] Codegen issue fixed |
66ada4a | 2019/01/08 | MapR [SPARK-319] Fix for sparkR version check |
70fc922 | 2019/01/22 | MapR [SPARK-419] Update hive-maprdb-json-handler jar for spark 2.2.1 |
Known Issues
- None.
Resolved Issues
- None.