Spark 2.4.5-2009 (EEP 7.0.0) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.4.5.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. For more information, you may also wish to consult the open-source Spark 2.4.5 Release Notes.

These release notes contain only MapR-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Spark Version 2.4.5
Release Date September 2020
MapR Version Interoperability See Component Versions for Released EEPs and EEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.4.5-mapr-2009
Maven Artifacts https://repository.mapr.com/maven/
Package Names Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names.
IMPORTANT
  • Beginning with EEP 6.0.0, the keyStore and trustStore password can be removed from spark-defaults.conf and set in /opt/mapr/conf/ssl-client.xml.
  • Beginning with EEP 6.0.0, after an upgrade, the previous version's configuration files are saved in the /opt/mapr/spark directory.
  • MapR 6.1.0 with EEP 6.0.0 and later support simplified security. If you enable security on your MapR cluster, MapR scripts automatically configure Spark security features.
  • Beginning with EEP 6.3.0, the Spark MapRDB JSON connector supports secondary indexes.
  • Beginning with EEP 6.3.0, Spark supports configurable HTTP security headers.
  • Beginning with MapR 6.2 and EEP 7.0, Spark supports SSL for WebUI.

Hive Support

This version of Spark supports integration with Hive, but has the following exceptions:

New in This Release

Fixes

This MapR release includes the following new fixes since the latest MapR Spark release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
b5bc89b 2020/04/01 MapR [SPARK-695] Spark 2.4.5 porting to MapR
7c3c09b 2020/04/07 MapR [SPARK-701] CVE-2019-17571 security vulnerability for log4j-1.2.17.jar
5d35d4c 2020/04/08 MapR [SPARK-705] Spark streaming fail with NoSuchMethodError: org.apache.kafka.clients.producer.ProducerConfig.getBoolean(Ljava/lang/String;)Ljava/lang/Boolean;
7ce734b 2020/04/08 MapR [SPARK-706] HBase example execution fail with NoClassDefFoundError: org/apache/hadoop/hbase/util/Bytes
78935fe 2020/04/13 [SPARK-26966][ML] Update to JPMML 1.4.8
8123d15 2020/04/14 MapR [SPARK-714] Update bcprov-jdk15on to fix CVE issue
1f5fb4a 2020/04/28 MapR [SPARK-715] Driver https port bind fail when application starts under non-mapr user
79d29d4 2020/04/30 MapR [SPARK-733] Update OJAI version in Spark
0b8b1a9 2020/04/30 MapR [SPARK-732] Update Hive version to 2.3.7
2bdc162 2020/05/07 MapR [SPARK-735] Spark master doesn't start after installation
45b682c 2020/05/14 MapR [SPARK-719] ZK updates to v3.5.6 at MEP 7.0.0
d67eafb 2020/05/17 MapR [SPARK-739] Upgrade curator version to 4.2.0
03c7195 2020/05/19 MapR [SPARK-740] UI shows "Kafka 0.9 direct stream" even if 0.10 is used
e75abc7 2020/05/20 MapR [SPARK-737] Calling poll(1000) from paranoidPoll causes batch scheduling delay
a253f8a 2020/05/22 MapR [SPARK-738] CLONE - Check if jackson-databind-2.9.4.jar will work with Spark 2.4.X in MEP 6 and MEP7
0309667 2020/05/27 [K8S-1458] Spark 2.4.4 encounters error and does not start executors
035f4e4 2020/06/03 MapR [SPARK-688] Spark Web UI has broken styles and works wrong
8c180b4 2020/06/10 [SPARK-24421][BUILD][CORE] Accessing sun.misc.Cleaner in JDK11
8d63d69 2020/06/10 [SPARK-25946][BUILD] Upgrade ASM to 7.x to support JDK11
cf0992f 2020/06/10 [SPARK-25984][CORE][SQL][STREAMING] Remove deprecated .newInstance(), primitive box class constructor calls
bc64754 2020/06/10 [SPARK-26507][CORE] Fix core tests for Java 11
278fc64 2020/06/10 [SPARK-26536][BUILD][TEST] Upgrade Mockito to 2.23.4
90c0cc1 2020/06/10 [SPARK-26839][SQL] Work around classloader changes in Java 9 for Hive isolation
2263dba 2020/06/10 [SPARK-26963][MLLIB] SizeEstimator can't make some JDK fields accessible in Java 9+
f632a69 2020/06/10 [SPARK-26986][ML] Add JAXB reference impl to build for Java 9+
50664cd 2020/06/10 [SPARK-27121][REPL] Resolve Scala compiler failure for Java 9+ in REPL
27d5882 2020/06/10 [SPARK-27981][CORE] Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+
758a3f9 2020/06/10 [SPARK-28072][SQL] Fix IncompatibleClassChangeError in `FromUnixTime` codegen on JDK9+
5e968b8 2020/06/10 [SPARK-28736][SPARK-28735][PYTHON][ML] Fix PySpark ML tests to pass in JDK 11
4ad4116 2020/06/10 [SPARK-28737][CORE] Update Jersey to 2.29
8863fcc 2020/06/10 [SPARK-28755][R][TESTS] Increase tolerance in 'spark.mlp' SparkR test for JDK 11
4abbd53 2020/06/10 [SPARK-28756][R] Fix checkJavaVersion to accept JDK8+
30ee1cb 2020/06/10 [SPARK-29674][CORE] Update dropwizard metrics to 4.1.x for JDK 9+
97c94ec 2020/06/12 MapR [SPARK-746] Secondary index not picked
64ca771 2020/06/12 [SPARK-27467][BUILD] Upgrade Maven to 3.6.1
a976dc6 2020/06/15 MapR [SPARK-753] Update maven plugins to build with JDK11
62549fa 2020/06/15 MapR [SPARK-752] Spark 2.4.5 build hangs with JDK11
a0c6004 2020/06/16 MapR [SPARK-754] Kubernetes module build failed after changes to build with JDK11
e50347b 2020/06/16 MapR [SPARK-755] Spark build failed because of using banned dependencies
22218c8 2020/06/16 MapR [SPARK-756] skip-kafka-0-8 needs to be used for build Spark with Scala-12
60b6cde 2020/06/16 MapR [SPARK-751] Spark Jetty servers send webserver details to client
1ebaada 2020/06/18 MapR [SPARK-758] Change Core version to 4-digits
f83a849 2020/06/24 MapR [SPARK-760] DataSourceV2 ClassCastException
Ab60c7f 2020/06/25 [CORE-459] MapRDB spark connector shows incorrect results when Secondary Index enabled on fields
a3728cf 2020/07/08 MapR [SPARK-763] Spark and Hive integration fails for spark-submit
4204015 2020/07/17 MapR [SPARK-679] Spark application fail with "Cannot recover key"
49d1fa9 2020/07/17 MapR [SPARK-767] Backport SPARK-29444 into Spark 2.4.5
5b2ddba 2020/07/29 MapR [SPARK-775] Update json4s library
3b74fbb 2020/08/14 MapR [SPARK-785] Investigate Spark-2.4.5 SLF4J messages on job start
54cec38 2020/08/17 MapR [SPARK-780] Can't download event logs from SHS
d12e0db 2020/08/18 MapR [SPARK-778] Delete ssl keys for spark job during spark package remove
b206526 2020/08/27 MapR [SPARK-786] Spark thrift server fails to read from hive-maprdb json table
e13fc5a 2020/08/28 MapR [SPARK-791] Spark does not work on non-secure cluster
c932822 2020/09/01 MapR [SPARK-793] Update Zookeeper to 4 digits version

Known Issues

  • MapR [SPARK-688] - The Spark Web UI does not display properly using Spark 2.4.5 in EEP 7.0.0. Workaround: Display the Spark job through the YARN ResourceManager UI by setting the spark.ui.reverseProxy property to true. Note that when the property is set to true, the Spark worker and application UI is not accessible directly. You can only access the UI through the Spark Master or proxy public URL. This behavior is described in the Apache Spark documentation.
  • MapR [SPARK-742] - On a secure cluster, if spark.ssl.ui.enabled is not set or is set to true, you cannot access the Spark job through the YARN ResourceManager UI. You can access the Spark job using the URL (https://<node.name>:4440). Workaround: Set spark.ssl.ui.enabled to false. When spark.ssl.ui.enabled is set to false, you can access the Spark UI directly by using the URL (http://<node.name>:4040) or by using the YARN ResourceManager UI.

Resolved Issues

  • None.