Spark 3.1.2.0 - 2110 (EEP 8.0.0) Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 3.1.2.0.
The notes below relate specifically to the HPE Ezmeral Data Fabric Distribution for Apache Hadoop. For more information, you may also want to consult the open-source Spark 3.1.2 Release Notes.
These release notes contain only HPE-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.
| Spark Version | 3.1.2.0 |
| Release Date | October 2021 |
| MapR Version Interoperability | See Component Versions for Released EEPs and EEP Components and OS Support. |
| Source on GitHub | https://github.com/mapr/spark |
| GitHub Release Tag |
3.1.2.0-eep-800 |
| Maven Artifacts | https://repository.mapr.com/maven/ |
| Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
IMPORTANT
- Beginning with EEP 6.0.0, the KeyStore and
TrustStore password can be removed from
spark-defaults.confand set in/opt/mapr/conf/ssl-client.xml. - Beginning with EEP 6.0.0, after an upgrade,
the previous version's configuration files are saved in the
/opt/mapr/sparkdirectory. - MapR 6.1.0 with EEP 6.0.0 and later support simplified security. If you enable security on your data-fabric cluster, HPE scripts automatically configure Spark security features.
- Beginning with Core 6.2 and EEP 7.0, Spark supports SSL for WebUI.
Hive Support
- Starting from Spark 3.1.2, Spark supports Hive 2.3.
Delta Lake Support
Starting from EEP 8.0.0, Delta Lake support is available for Apache Spark 3.1.2 on HPE Ezmeral Data Fabric. See Apache Spark Feature Support.
New in This Release
- For a complete list of new features, see the open-source Spark 3.1.2 Release Notes.
Fixes
This HPE release includes the following new fixes since the latest Spark release. For details, refer to the commit log for this project in GitHub.
| GitHub Commit Number | Date (YYYY/MM/DD) | HPE Fix Number and Description |
|---|---|---|
| 22b57ce | 2021/07/29 | MapR [SPARK-811] Updating to Spark 3.x |
| bfd5a1a | 2021/07/29 | MapR [SPARK-839] Unstable problem with security keys for job on yarn-cluster mode |
| 9045b8f | 2021/07/29 | [SPARK-32723][WEBUI] Upgrade to jQuery 3.5.1 |
| c4c0808 | 2021/07/29 | MapR [SPARK-843] Improve logging for SSL certs generation |
| 3cf3522 | 2021/07/29 | MapR [SPARK-847] Spark can't read data from symlink |
| 2f6cbe1 | 2021/07/29 | MapR [SPARK-846] Add service verifier to Spark package |
| f11cc5a | 2021/07/29 | MapR [SPARK-861] Error logs in spark-historyserver |
| 711eb00 | 2021/07/29 | MapR [SPARK-863] Interaction with HBase via hbase spark connector fails |
| bd65fe6 | 2021/07/29 | MapR [SPARK-841] Backport SPARK-32723 |
| 213226d | 2021/07/29 | MapR [SPARK-851] Spark SQL transient FS error handling when writing output |
| 6f33abe | 2021/07/29 | MapR [SPARK-869] fix logging location |
| 41ac719 | 2021/07/29 | MapR [SPARK-867] Spark Hive Example fails from simple user |
| b234c78 | 2021/07/29 | MapR [SPARK-870] Can't download event logs from SHS twice |
| df7dbfe | 2021/07/29 | MapR [SPARK-871] Spark job fails from mapr-client |
| 165644d | 2021/07/29 | MapR [SPARK-846] Add service verifier to Spark package - moving to proper directory |
| 92e3441 | 2021/07/29 | MapR [SPARK-877] Update Jenkins file to build Spark-3.x from MEP-8.0.0 private package branch |
| 42b5f36 | 2021/07/29 | MapR [SPARK-881] fix duplicate heade |
| 9c75ccb | 2021/07/29 | MapR [SPARK-879] Add changes to examples module to build for Spark-3.x |
| 0282515 | 2021/07/29 | MapR [SPARK-883] Spark-3.1.1 job submition |
| a87034c | 2021/07/29 | MapR [SPARK-885] Spark-submit fails on 8.2 and 8.3 centos |
| 9d7de9e | 2021/07/29 | MapR [SPARK-888] Collect of selected result from orc table fails |
| b80ea8e | 2021/07/29 | MapR [SPARK-893] Clean deprecated kafka08 and kafka09 from pyspark code |
| c26d17c | 2021/07/29 | MapR [SPARK-891] Thrift server start fails due to unsupported Hive Metastore version |
| 28a492a | 2021/07/29 | MapR [SPARK-897] Spar-3.1.1 doesn't suppot MapRSASL for Thriftserver |
| 3b7064d | 2021/07/29 | MapR [SPARK-901] Spark-3.1.1 doesn't start by warden |
| 37bc0ac | 2021/07/29 | MapR [SPARK-903] spark.loadFromMapRDB(tableName, schema) using v2 api fail |
| 568f406 | 2021/07/29 | MapR [SPARK-817] Update kafka client to v2.6.X |
| 8b29cfa | 2021/07/29 | MapR [SPARK-886] MapRDB table loading fails via spark session + java api |
| e86c244 | 2021/07/29 | MapR [SPARK-907] Update hadoop dependency for Spark-3.1.1 |
| 9ed986b | 2021/07/29 | MapR [SPARK-905] Can't connect to spark thriftserver via beeline with MAPRSASL |
| 1422b69 | 2021/07/29 | MapR [SPARK-904] Implement inferSchema method for MaprDBDataSource |
| c4a1990 | 2021/07/29 | [EZSPA-212] Move creating of spark-env.sh script to Spark |
| 078555a | 2021/07/29 | [SPARK-22769] Do not log rpc post message error when sparkEnv is already stopped |
| a8cb292 | 2021/07/29 | [EZSPA-213] Add Spark-3.x to dockerfiles project |
| 5e1ddd3 | 2021/08/02 | MapR [SPARK-917] Porting Spark-3.1.2 to MapR |
| 2c5d494 | 2021/08/09 | MapR [SPARK-922] Move latest Spark commits to 3.1.2 branch |
| 2f6b259 | 2021/08/10 | MapR [SPARK-882] Making netcat to work on Ubuntu too |
| 42328a6 | 2021/08/12 | MapR [SPARK-878] remove redundant filter setting |
| 13f8e12 | 2021/08/17 | MapR [SPARK-919] Errors in spark DEBUG logs |
| 605d290 | 2021/08/18 | MapR [SPARK-927] Update Hive in Spark-3.1.2 |
| 1b188e3 | 2021/08/19 | MapR [SPARK-923] Update Avro to 1.10.1 in Spark |
| 6f8abad | 2021/08/19 | MapR [SPARK-915] CVE-2020-13956,WS-2017-3734 vulnerabilities in http-client |
| 75d47c5 | 2021/08/24 | MapR [SPARK-906] Spark streaming (structured and unstructured) fails with kafka 2.6.1.0 |
| 09f6deb | 2021/08/27 | MapR [SPARK-911] STS HA doesn't work |
| 8252a91 | 2021/09/01 | MapR [SPARK-928] Can't connect to spark thriftserver on kerberos cluster |
| dba8119 | 2021/09/03 | MapR [SPARK-929] Spark 3.1.2 requires password when you try to remove packages |
| c942759 | 2021/09/06 | MapR [SPARK-882] [Installer] Add verification scripts for spark-thrifserver and spark-historyserver |
| 31f4b86 | 2021/09/06 | MapR [SPARK-936] Investigate Spark warning on start of application |
| 050309c | 2021/09/06 | MapR [SPARK-938] Run-example doesn't work |
| 334bc98 | 2021/09/07 | MapR [SPARK-919] Errors in spark DEBUG logs |
| 354abf87 | 2021/09/09 | [EZSPA-270] adopt mapr spark feature to work in non-mapr env |
| 718fa44 | 2021/09/10 | MapR [SPARK-939] Replace "mapr" to "eep" in Spark package |
| ac7871c | 2021/09/10 | MapR [SPARK-934] Spark and Livy jobs fail on core 7.0.0 with encrypted ssl password |
| a6b59d0 | 2021/09/20 | MapR [SPARK-943] Spark 3 and S3 integration fails |
| 4a04f88 | 2021/09/20 | MapR [SPARK-945] Components can't read keyPassword |
| a5cb0b8 | 2021/09/21 | MapR [SPARK-894] Hadoop artifacts should be taken from the cluster |
| e8c8667 | 2021/09/22 | MapR [SPARK-946] Excessive warning messages in spark-shell |
| 715edb7 | 2021/09/22 | MapR [SPARK-947] Error when using Spark SQL with derby db |
Known Issues
-
If you are using Spark SQL with Derby database without Hive or Hive Metastore installation, you will see the Java Runtime Exception. See Apache Spark Feature Support for workaround. Spark 3.1.2 does not support
log4j1.2logging on HPE Ezmeral Data Fabric. - HPE Ezmeral Data Fabric does not support GPU aware scheduling feature on Spark 3.1.2. See Apache Spark Feature Support.
Resolved Issues
- None.