Spark 2.3.3.0-1904 (EEP 6.1.1 and EEP 6.0.2) Release Notes

This section provides reference information, including new features, patches, and known issues for Spark 2.3.3.

The notes below relate specifically to the MapR Distribution for Apache Hadoop. This release of Spark has backward-compatibility changes, see the open-source Spark 2.3.3 Release Notes for more information.

These release notes contain only MapR-specific information and are not necessarily cumulative in nature. For information about how to use the release notes, see Ecosystem Component Release Notes.

Spark Version 2.3.3
Release Date April 2019
MapR Version Interoperability See EEP Components and OS Support.
Source on GitHub https://github.com/mapr/spark
GitHub Release Tag 2.3.3-mapr-1904
Maven Artifacts https://repository.mapr.com/maven/
Package Names Navigate to https://package.ezmeral.hpe.com/releases/MEP/ and select your EEP and OS to view the list of package names.
IMPORTANT
  • Starting with EEP 6.0.0, keyStore and trustStore passwords can be removed from the spark-defaults.conf file and can be set in the /opt/mapr/conf/ssl-client.xml file.
  • Starting with EEP 6.0.0, after an upgrade, configuration files of previous versions are saved in the /opt/mapr/spark directory.
  • The MapR 6.1 and EEP 6.0.0 release introduces "Simplified Security". If you are using these versions and enable security on your MapR cluster, MapR scripts automatically configure Spark security features.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

New in This Release

Fixes

This MapR release includes the following new fixes since the latest MapR Spark 2.3.1 release. For details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comment
63acd83 2019/01/23 MapR [SPARK-374] Spark Hive example fails when we submit job from another(simple) cluster user
9a634a7 2019/01/23 MapR [SPARK-396] Interface change of sendToKafka
2b2c934 2019/01/25 MapR [SPARK-357] consumer groups are prepeneded with a "service_" prefix
0ab4b07 2019/01/30 MapR [SPARK-429] Changes in maprdb connector are the cause of broken backward compatibility
9421abc 2019/02/13 MapR [SPARK-417] impersonation fixes for spark executor. Impersonation is moved from HadoopRDD.compute() method to org.apache.spark.executor.Executor.run() method
684235c 2019/02/15 [SPARK-449] Kafka offset commit issue fixed
524de52 2019/02/27 MapR [SPARK-465] Error messages after update of spark 2.4
9d04094 2019/03/07 CVE-2018-1320 vulnerability in Apache Thrift
f3296aa 2019/03/08 MapR [SPARK-482] Spark streaming app fails to start by UnknownTopicOrPartitionException with checkpoint
39f7af7 2019/03/12 MapR [SPARK-445] Messages loss fixed by reverting [MAPR-32290]
ffda9a5 2019/03/13 [SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error when using Python 3.6 and Pandas 0.23
78ec0b2 2019/03/14 MapR [SPARK-499] version is changed to 2.3.3
54e66b3 2019/03/19 MapR [SPARK-463] MAPR_MAVEN_REPO variable for specifying mapR repository
2b37b41 2019/03/26 MapR [SPARK-510] nonmapr "admin" users not able to view other user logs in SHS
3bf2145 2019/03/26 [SPARK-508] MapR-DB OJAI Connector for Spark isNull condition returns incorrect result
f0745b8 2019/03/26 MapR [SPARK-482] Spark streaming app fails to start by UnknownTopicOrPartitionException with checkpoint
648fb33

2019/03/28

 MapR [SPARK-462] Spark and SparkHistoryServer allow week ciphers, which can allow man in the middle attack
50abc9d

2019/03/28

 MapR [SPARK-461] Stop graph after jobs completion to prevent 'java.lang.IllegalStateException: No active subscriptions'
0bb97c7

2019/04/04

 MapR [SPARK-516] Spark jobs failure using yarn mode on kerberos fixed
a8e8a19

2019/04/11

 MapR [SPARK-444] Fix of hive version for spark dev branches
e2e9ea0

2019/04/19

 SPARK-539 Workaround for absent MapRDBJsonSplit class
2bdf60b

2019/04/22

 MapR [SPARK-536] PySpark streaming package for kafka-0-10 added
65ad08a

2019/05/17

 MapR [SPARK-541] Avoid duplication of the first unexpired record

Known Issues

  • None.

Resolved Issues

  • None.