Spark 2.4.4-2009 (EEP 6.3.1) Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 2.4.4.
The notes below relate specifically to the MapR Distribution for Apache Hadoop. For more information, you may also wish to consult the open-source Spark 2.4.4 Release Notes.
These release notes contain only MapR-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.
| Spark Version | 2.4.4 |
| Release Date | September 2020 |
| MapR Version Interoperability | See Component Versions for Released EEPs and EEP Components and OS Support. |
| Source on GitHub | https://github.com/mapr/spark |
| GitHub Release Tag | 2.4.4-mapr-2009 |
| Maven Artifacts | https://repository.mapr.com/maven/ |
| Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
IMPORTANT
- Beginning with EEP 6.0.0, the keyStore and
trustStore password can be removed from
spark-defaults.confand set in/opt/mapr/conf/ssl-client.xml. - Beginning with EEP 6.0.0, after an upgrade,
the previous version's configuration files are saved in the
/opt/mapr/sparkdirectory. - MapR 6.1.0 with EEP 6.0.0 and later support simplified security. If you enable security on your MapR cluster, MapR scripts automatically configure Spark security features.
- Beginning with EEP 6.3.0, the Spark MapRDB JSON connector supports secondary indexes.
- Beginning with EEP 6.3.0, Spark supports configurable HTTP security headers.
- Beginning with MapR 6.2 and EEP 7.0, Spark supports SSL for WebUI.
Hive Support
This version of Spark supports integration with Hive, but has the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but is not fully compatible with Hive. See the Apache Spark documentation and the MapR Spark documentation for details.
New in This Release
- For a complete list of new features, see the open-source Spark 2.4.4 Release Notes.
Fixes
This MapR release includes the following new fixes since the latest MapR Spark release. For details, refer to the commit log for this project in GitHub.
| GitHub Commit | Date (YYYY-MM-DD) | Comment |
|---|---|---|
| e5a8ff7 | 2020/01/09 | MapR [SPARK-642] Improve work with security headers |
| 13584a9 | 2020/01/13 | MapR [SPARK-642] Improve work with security headers |
| f6f14db | 2020/01/15 | MapR [SPARK-654] Update protobuf version to 3.X.X for Spark |
| db6c15f | 2020/01/21 | MapR [SPARK-585] Sertificates generation for cluster mode |
| 4e5d6fd | 2020/02/10 | [SPARK-27992][SPARK-28881][PYTHON][2.4] Allow Python to join with connection thread to propagate errors |
| 79366d5 | 2020/02/11 | [SPARK-29410][BUILD] Update commons-beanutils to 1.9.4 |
| b05a4ae | 2020/02/18 | MapR [SPARK-662] Spark job is failed with updated protobuf version |
| 2c7467b | 2020/02/18 | MapR [SPARK-664] commons-beanutils as dependency for mapr-spark |
| d95867e | 2020/03/03 | MapR [SPARK-677] No TLS protocol logging for Spark thrift server & spark history server |
| 5a9b5f8 | 2020/04/07 | [SPARK-30238][SQL][2.4] hive partition pruning can only support string and integral types |
| add846c | 2020/04/08 | MapR [SPARK-691] maprdb-spark jar MetaTableImpl class reference in DBOlderClientImpl |
| 8259253 | 2020/04/08 | MapR [SPARK-705] Spark streaming fail with NoSuchMethodError: org.apache.kafka.clients.producer.ProducerConfig.getBoolean(Ljava/lang/String;)Ljava/lang/Boolean; |
| 8c4e21c | 2020/04/19 | MapR [SPARK-717] Spark Master start fail after installation due to javax.security.auth.login.LoginException |
| 61b7376 | 2020/04/27 | MapR [SPARK-724] Application startup fail with java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3 |
| eec85e1 | 2020/04/28 | MapR [SPARK-715] Driver https port bind fail when application starts under non-mapr user |
| e28302b | 2020/04/30 | [SPARK-29445][CORE] Bump netty-all from 4.1.39.Final to 4.1.42.Final |
| 3f8b017 | 2020/05/19 | MapR [SPARK-740] UI shows "Kafka 0.9 direct stream" even if 0.10 is used |
| f014001 | 2020/05/20 | MapR [SPARK-737] Calling poll(1000) from paranoidPoll causes batch scheduling delay |
| ccda955 | 2020/05/22 | MapR [SPARK-738] CLONE - Check if jackson-databind-2.9.4.jar will work with Spark 2.4.X in MEP 6 and MEP7 |
| 5e30e36 | 2020/05/22 | [SPARK-26966][ML] Update to JPMML 1.4.8 |
| fbc0634 | 2020/05/22 | MapR [SPARK-714] Update bcprov-jdk15on to fix CVE issue |
| 3348810 | 2020/05/27 | [K8S-1458] Spark 2.4.4 encounters error and does not start executors |
| 2b903c6 | 2020/06/12 | MapR [SPARK-746] Secondary index not picked |
| 8154afe | 2020/06/16 | MapR [SPARK-751] Spark Jetty servers send webserver details to client |
| e8afcde | 2020/06/25 | [CORE-459] MapRDB spark connector shows incorrect results when Secondary Index enabled on fields |
| e74a28e | 2020/07/01 | [SPARK-29055][CORE] Update driver/executors' storage memory when block is removed from BlockManager |
| 79912a9 | 2020/07/17 | MapR [SPARK-767] Backport SPARK-29444 into Spark 2.4.4 |
| 432729b | 2020/08/19 | MapR [SPARK-781] SSL for spark UI services |
| ee0107a | 2020/08/28 | MapR [SPARK-759] Allow PAM and MapR-Sasl to be enabled at the same time |
Known Issues
- None.
Resolved Issues
- None.