Spark 2.3.2.0-1901 (EEP 6.1.0 and EEP 6.0.1) Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 2.3.2.0.
The notes below relate specifically to the MapR Distribution for Apache Hadoop. This release of Spark has backward-compatibility changes, see the open-source Spark 2.3.2.0 Release Notes for more information.
These release notes contain only MapR-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.
Spark Version | 2.3.2.0 |
Release Date | February 2019 |
MapR Version Interoperability | See EEP Components and OS Support. |
Source on GitHub | https://github.com/mapr/spark |
GitHub Release Tag | 2.3.2-mapr-1901 |
Maven Artifacts | https://repository.mapr.com/maven/ |
Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
- Starting with EEP 6.0.0,
keyStore
andtrustStore
passwords can be removed from thespark-defaults.conf
file and can be set in the/opt/mapr/conf/ssl-client.xml
file. - Starting with EEP 6.0.0, after an upgrade,
configuration files of previous versions are saved in the
/opt/mapr/spark
directory. - The MapR 6.1 and EEP 6.0.0 release introduces "Simplified Security". If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.
Hive Support
This version of Spark supports integration with Hive. However, note the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but it is not fully compatible with Hive. For details, see the Apache Spark documentation and the MapR Spark documentation.
New in This Release
For a complete list of all new features, refer to the open source documentation.
Fixes
This MapR release includes the following new fixes since the latest MapR Spark 2.3.1 release. For details, refer to the commit log for this project in GitHub.
GitHub Commit | Date (YYYY-MM-DD) | Comment |
---|---|---|
fddc84f | 2018/10/01 | MapR [SPARK-325] Add examples for work with the MapR-DB JSON connector into the Spark project |
a2b88ef | 2018/10/16 | MapR [SPARK-331] Remove snapshot versions of MapR dependencies from Spark |
4f380b4 | 2018/10/23 | [MAPR-32263] Seek called on unsubscribed partitions |
e8d59b9 | 2018/10/24 | [MAPR-32290] Spark processing offsets when messages are already TTL in the first batch |
58b3115 | 2018/11/14 | MapR [SPARK-352] Spark shell fails with "NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream" if java is not available in PATH |
fcee5ab |
2018/11/15 |
MapR [SPARK-350] Deprecate Spark Kafka-09 package |
a2a3a0f | 2018/11/19 | MapR [SPARK-326] Investigate possibility of writing Java example for the MapR-DB OJAI connector |
da4e0b3 | 2018/11/27 | MapR [SPARK-356] Merge MapR changes from kafka-09 package into the kafka-10 |
3608c7b | 2018/11/29 | MapR [SPARK-319] Fix for SparkR version check |
2aaa6a5 | 2018/12/03 | MapR [SPARK-349] Update OJAI client to v3 for Spark MapR-DB JSON connector |
71f6153 | 2018/12/10 | MapR [SPARK-137] Analyze the warning during compilation of OJAI connector |
f62df7a | 2018/12/13 | MapR [SPARK-369] Spark 2.3.2 fails with error related to zookeeper |
a2bf94f | 2018/12/18 | [MAPR-26258] hbasecontext.HBaseDistributedScanExample fails |
cd75768 | 2018/12/27 | MapR [SPARK-372] Support 4 digits version scheme for Spark |
182a7b1 | 2019/01/08 | [MSPARK-24355] Spark external shuffle server improvement to better handle block fetch requests. |
6b28052 | 2019/01/09 | MapR [SPARK-381] Kafka version changed to '1.1.1-mapr-SNAPSHOT' |
fec2162 | 2019/01/11 | MapR [SPARK-374] Spark Hive example fails when we submit job from another(simple) cluster user |
609c0a8 | 2019/01/11 | MapR [SPARK-363] Hive requests fail with ClassNotFoundException |
e6a8a6b | 2019/01/17 | MapR [SPARK-410] All submitted jobs fail for Spark 2.3.2.0 |
f032694 | 2019/01/18 | MapR [SPARK-363] Hive version changed to '1.2.0-mapr-spark-MEP-6.0.0' |
2686b34 | 2019/01/21 | MapR [SPARK-373] Unexpected behavior during job running in standalone cluster mode |
b991202 | 2019/01/22 | MapR [SPARK-419] Update hive-maprdb-json-handler jar for spark 2.3.2.0 |
63acd83 | 2019/01/23 | MapR [SPARK-374] Spark Hive example fails when we submit job from another (simple) cluster user |
Known Issues
- None.
Resolved Issues
- None.