Spark 2.4.7.0 - 2101 (EEP 7.0.1) Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 2.4.7.0.
The notes below relate specifically to the HPE Distribution for Apache Hadoop. For more information, you may also wish to consult the open-source Spark 2.4.7 Release Notes.
These release notes contain only HPE-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.
| Spark Version | 2.4.7 |
| Release Date | January 2021 |
| MapR Version Interoperability | See Component Versions for Released EEPs and EEP Components and OS Support. |
| Source on GitHub | https://github.com/mapr/spark |
| GitHub Release Tag | 2.4.7-mapr-2101 |
| Maven Artifacts | https://repository.mapr.com/maven/ |
| Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
IMPORTANT
- Beginning with EEP 6.0.0, the keyStore and
trustStore password can be removed from
spark-defaults.confand set in/opt/mapr/conf/ssl-client.xml. - Beginning with EEP 6.0.0, after an upgrade,
the previous version's configuration files are saved in the
/opt/mapr/sparkdirectory. - MapR 6.1.0 with EEP 6.0.0 and later support simplified security. If you enable security on your data-fabric cluster, HPE scripts automatically configure Spark security features.
- Beginning with MapR 6.2 and EEP 7.0, Spark supports SSL for WebUI.
Hive Support
This version of Spark supports integration with Hive, but has the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but is not fully compatible with Hive. See the Apache Spark documentation and the MapR Spark documentation for details.
New in This Release
- For a complete list of new features, see the open-source Spark 2.4.7 Release Notes.
Fixes
This HPE release includes the following new fixes since the latest data-fabric Spark release. For details, refer to the commit log for this project in GitHub.
| GitHub Commit | Date (YYYY-MM-DD) | Comment |
|---|---|---|
| e1b4639 | 2020/09/02 | MapR [SPARK-778] Delete ssl keys for spark job during spark package remove |
| 4999284 | 2020/09/02 | MapR [SPARK-759] Allow PAM and MapR-Sasl to be enabled at the same time |
| 729a9a5 | 2020/09/07 | MapR [SPARK-796] java.lang.ClassNotFoundException: Class com.mapr.fs.MapRFileSystem not found for spark-hive integration jobs |
| 1d4ab93 | 2020/09/12 | MapR [SPARK-797] Do not overwrite existing warden configuration files |
| ed0be85 | 2020/09/17 | MapR [SPARK-795] - extended logging for Spark Streaming and Structured Streaming |
| 547b943 | 2020/10/07 | MapR [K8S-1882] Error on Spark SSL key generation |
| 35f3ff5 | 2020/10/21 | MapR [SPARK-805] Spark standalone worker can't start with enabled SSL for UI |
| 3f7a240 | 2020/10/26 | MapR [SPARK-703] Fix org.codehaus.jackson vulnerability |
| 5660d09 | 2020/11/03 | MapR [SPARK-804] sparkContext.loadFromMapRDB throwing RuntimeException in scala code |
| 7373f82 | 2020/11/19 | MapR [SPARK-812] Update Ant version to avoid CVE-2020-1945 |
| 2e2127a | 2020/11/22 | MapR [SPARK-819] CVE-2019-0205: Update Thrift version to v0.13.0 |
| c409736 | 2020/11/30 | [SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20 |
| 4507ee8 | 2020/11/30 | MapR [SPARK-822] WS-2019-0379: update commons-codec jar for spark |
| c3e72fd | 2020/12/09 | MapR [SPARK-824] CVE-2020-11612: Netty vulnerabilities |
| 5394b4a | 2020/12/29 | MapR [SPARK-832] Multiple Vulnerabilities including Spark and hive jars found |
| 187e7f2 | 2021/01/05 | MapR [SPARK-833] Spark thriftserver can't start after the latest commit |
| 3dd6575 | 2021/01/08 | MapR [SPARK-812] Update Ant version to 1.10.9 to avoid CVE-2020-1945 |
| 816db59 | 2021/01/14 | MapR [SPARK-776] IllegalArgumentException: Null user is thrown when using some methods from SparkContext |
| c3e4963 | 2021/01/15 | MapR [SPARK-839] Unstable problem with security keys for job on yarn-cluster mode |
Known Issues
- None.
Resolved Issues
- None.