Spark 2.4.5-2009 (EEP 7.0.0) Release Notes
This section provides reference information, including new features, patches, and known issues for Spark 2.4.5.
The notes below relate specifically to the MapR Distribution for Apache Hadoop. For more information, you may also wish to consult the open-source Spark 2.4.5 Release Notes.
These release notes contain only MapR-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.
Spark Version | 2.4.5 |
Release Date | September 2020 |
MapR Version Interoperability | See Component Versions for Released EEPs and EEP Components and OS Support. |
Source on GitHub | https://github.com/mapr/spark |
GitHub Release Tag | 2.4.5-mapr-2009 |
Maven Artifacts | https://repository.mapr.com/maven/ |
Package Names | Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names. |
IMPORTANT
- Beginning with EEP 6.0.0, the keyStore and
trustStore password can be removed from
spark-defaults.conf
and set in/opt/mapr/conf/ssl-client.xml
. - Beginning with EEP 6.0.0, after an upgrade,
the previous version's configuration files are saved in the
/opt/mapr/spark
directory. - MapR 6.1.0 with EEP 6.0.0 and later support simplified security. If you enable security on your MapR cluster, MapR scripts automatically configure Spark security features.
- Beginning with EEP 6.3.0, the Spark MapRDB JSON connector supports secondary indexes.
- Beginning with EEP 6.3.0, Spark supports configurable HTTP security headers.
- Beginning with MapR 6.2 and EEP 7.0, Spark supports SSL for WebUI.
Hive Support
This version of Spark supports integration with Hive, but has the following exceptions:
- Hive-on-Spark is not supported.
- Spark-SQL is supported, but is not fully compatible with Hive. See the Apache Spark documentation and the MapR Spark documentation for details.
New in This Release
- For a complete list of new features, see the open-source Spark 2.4.5 Release Notes.
Fixes
This MapR release includes the following new fixes since the latest MapR Spark release. For details, refer to the commit log for this project in GitHub.
GitHub Commit | Date (YYYY-MM-DD) | Comment |
---|---|---|
b5bc89b | 2020/04/01 | MapR [SPARK-695] Spark 2.4.5 porting to MapR |
7c3c09b | 2020/04/07 | MapR [SPARK-701] CVE-2019-17571 security vulnerability for log4j-1.2.17.jar |
5d35d4c | 2020/04/08 | MapR [SPARK-705] Spark streaming fail with NoSuchMethodError: org.apache.kafka.clients.producer.ProducerConfig.getBoolean(Ljava/lang/String;)Ljava/lang/Boolean; |
7ce734b | 2020/04/08 | MapR [SPARK-706] HBase example execution fail with NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes |
78935fe | 2020/04/13 | [SPARK-26966][ML] Update to JPMML 1.4.8 |
8123d15 | 2020/04/14 | MapR [SPARK-714] Update bcprov-jdk15on to fix CVE issue |
1f5fb4a | 2020/04/28 | MapR [SPARK-715] Driver https port bind fail when application starts under non-mapr user |
79d29d4 | 2020/04/30 | MapR [SPARK-733] Update OJAI version in Spark |
0b8b1a9 | 2020/04/30 | MapR [SPARK-732] Update Hive version to 2.3.7 |
2bdc162 | 2020/05/07 | MapR [SPARK-735] Spark master doesn't start after installation |
45b682c | 2020/05/14 | MapR [SPARK-719] ZK updates to v3.5.6 at MEP 7.0.0 |
d67eafb | 2020/05/17 | MapR [SPARK-739] Upgrade curator version to 4.2.0 |
03c7195 | 2020/05/19 | MapR [SPARK-740] UI shows "Kafka 0.9 direct stream" even if 0.10 is used |
e75abc7 | 2020/05/20 | MapR [SPARK-737] Calling poll(1000) from paranoidPoll causes batch scheduling delay |
a253f8a | 2020/05/22 | MapR [SPARK-738] CLONE - Check if jackson-databind-2.9.4.jar will work with Spark 2.4.X in MEP 6 and MEP7 |
0309667 | 2020/05/27 | [K8S-1458] Spark 2.4.4 encounters error and does not start executors |
035f4e4 | 2020/06/03 | MapR [SPARK-688] Spark Web UI has broken styles and works wrong |
8c180b4 | 2020/06/10 | [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cleaner in JDK11 |
8d63d69 | 2020/06/10 | [SPARK-25946][BUILD] Upgrade ASM to 7.x to support JDK11 |
cf0992f | 2020/06/10 | [SPARK-25984][CORE][SQL][STREAMING] Remove deprecated .newInstance(), primitive box class constructor calls |
bc64754 | 2020/06/10 | [SPARK-26507][CORE] Fix core tests for Java 11 |
278fc64 | 2020/06/10 | [SPARK-26536][BUILD][TEST] Upgrade Mockito to 2.23.4 |
90c0cc1 | 2020/06/10 | [SPARK-26839][SQL] Work around classloader changes in Java 9 for Hive isolation |
2263dba | 2020/06/10 | [SPARK-26963][MLLIB] SizeEstimator can't make some JDK fields accessible in Java 9+ |
f632a69 | 2020/06/10 | [SPARK-26986][ML] Add JAXB reference impl to build for Java 9+ |
50664cd | 2020/06/10 | [SPARK-27121][REPL] Resolve Scala compiler failure for Java 9+ in REPL |
27d5882 | 2020/06/10 | [SPARK-27981][CORE] Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+ |
758a3f9 | 2020/06/10 | [SPARK-28072][SQL] Fix IncompatibleClassChangeError in `FromUnixTime` codegen on JDK9+ |
5e968b8 | 2020/06/10 | [SPARK-28736][SPARK-28735][PYTHON][ML] Fix PySpark ML tests to pass in JDK 11 |
4ad4116 | 2020/06/10 | [SPARK-28737][CORE] Update Jersey to 2.29 |
8863fcc | 2020/06/10 | [SPARK-28755][R][TESTS] Increase tolerance in 'spark.mlp' SparkR test for JDK 11 |
4abbd53 | 2020/06/10 | [SPARK-28756][R] Fix checkJavaVersion to accept JDK8+ |
30ee1cb | 2020/06/10 | [SPARK-29674][CORE] Update dropwizard metrics to 4.1.x for JDK 9+ |
97c94ec | 2020/06/12 | MapR [SPARK-746] Secondary index not picked |
64ca771 | 2020/06/12 | [SPARK-27467][BUILD] Upgrade Maven to 3.6.1 |
a976dc6 | 2020/06/15 | MapR [SPARK-753] Update maven plugins to build with JDK11 |
62549fa | 2020/06/15 | MapR [SPARK-752] Spark 2.4.5 build hangs with JDK11 |
a0c6004 | 2020/06/16 | MapR [SPARK-754] Kubernetes module build failed after changes to build with JDK11 |
e50347b | 2020/06/16 | MapR [SPARK-755] Spark build failed because of using banned dependencies |
22218c8 | 2020/06/16 | MapR [SPARK-756] skip-kafka-0-8 needs to be used for build Spark with Scala-12 |
60b6cde | 2020/06/16 | MapR [SPARK-751] Spark Jetty servers send webserver details to client |
1ebaada | 2020/06/18 | MapR [SPARK-758] Change Core version to 4-digits |
f83a849 | 2020/06/24 | MapR [SPARK-760] DataSourceV2 ClassCastException |
Ab60c7f | 2020/06/25 | [CORE-459] MapRDB spark connector shows incorrect results when Secondary Index enabled on fields |
a3728cf | 2020/07/08 | MapR [SPARK-763] Spark and Hive integration fails for spark-submit |
4204015 | 2020/07/17 | MapR [SPARK-679] Spark application fail with "Cannot recover key" |
49d1fa9 | 2020/07/17 | MapR [SPARK-767] Backport SPARK-29444 into Spark 2.4.5 |
5b2ddba | 2020/07/29 | MapR [SPARK-775] Update json4s library |
3b74fbb | 2020/08/14 | MapR [SPARK-785] Investigate Spark-2.4.5 SLF4J messages on job start |
54cec38 | 2020/08/17 | MapR [SPARK-780] Can't download event logs from SHS |
d12e0db | 2020/08/18 | MapR [SPARK-778] Delete ssl keys for spark job during spark package remove |
b206526 | 2020/08/27 | MapR [SPARK-786] Spark thrift server fails to read from hive-maprdb json table |
e13fc5a | 2020/08/28 | MapR [SPARK-791] Spark does not work on non-secure cluster |
c932822 | 2020/09/01 | MapR [SPARK-793] Update Zookeeper to 4 digits version |
Known Issues
- MapR [SPARK-688] - The Spark Web UI does not display properly using Spark 2.4.5 in
EEP 7.0.0. Workaround: Display the
Spark job through the YARN ResourceManager UI by setting the
spark.ui.reverseProxy
property totrue
. Note that when the property is set totrue
, the Spark worker and application UI is not accessible directly. You can only access the UI through the Spark Master or proxy public URL. This behavior is described in the Apache Spark documentation. - MapR [SPARK-742] - On a secure cluster, if
spark.ssl.ui.enabled
is not set or is set totrue
, you cannot access the Spark job through the YARN ResourceManager UI. You can access the Spark job using the URL (https://<node.name>:4440
). Workaround: Setspark.ssl.ui.enabled
tofalse
. Whenspark.ssl.ui.enabled
is set tofalse
, you can access the Spark UI directly by using the URL (http://<node.name>:4040
) or by using the YARN ResourceManager UI.
Resolved Issues
- None.