Spark 2.0.1-1611 Release Notes
The notes below relate specifically to the MapR Distribution for Apache Hadoop. You may also be interested in the open-source Spark 2.0.1 Release Notes.
| Spark Version | 2.0.1 |
| Release Date | December 9, 2016 |
| MapR Version Interoperability | See EEP Components and OS Support. |
| Source on GitHub | https://github.com/mapr/spark |
| GitHub Release Tag | 2.0.1-mapr-1611 |
| Maven Artifacts | https://repository.mapr.com/maven/ |
| Package Names | See Package Names for Ezmeral Ecosystem Packs (EEPs) |
New in This Release
This version of Spark supports integration with Hive. However, note the following exceptions:
- Hive on Spark is not supported.
- Spark-SQL is supported, but it is not fully compatible with Hive; for details, see the Apache Spark documentation and the MapR Spark documentation.
NOTE For the API changes in this version, see Spark 2.0.1 API.
Fixes
This MapR release includes the following new fixes on top of the Apache Spark 2.0.1 release. For details, refer to the commit log for this project in GitHub.
| GitHub Commit Number | Data (YYYY-MM-DD) | MapR Fix Number and Description |
|---|---|---|
| bed0ba9 | 2016-11-14 | [MINOR] Fixed the spark-jars.sh script. |
| 287a9b2 | 2016-11-09 | [MINOR] Removed the unused config template. |
| b05ddad | 2016-11-09 | [MAPR-24603] Could not launch the Beeline shell after starting the Spark Thrift Server. |
| b64908b | 2016-11-07 | Fixed a syntax error in V09DirectKafkaWordCount example (#75). |
| b6788d0 | 2016-11-01 | Spark 2.0.1 MapR-Streams Python API (#73). |
| b1dff21 | 2016-10-31 | [MAPR-24415] SPARK_JAVA_OPTS was deprecated (#71). |
| 361effd | 2016-10-26 | [MAPR-24863] Added spark-defaults template for installer (#69). |
| a1e0492 | 2016-10-20 | [MAPR-25002] FileNotFoundException during SparkHiveExample (#68). |
| 7357889 | 2016-10-12 | Added the Kafka streaming producer (#66). |
| d0c05d4 | 2016-10-07 | [SPARK-17707][WEBUI] The Web UI prevented the spark-submit application from finishing. |
| 23f9305 | 2016-09-08 | [SPARK-15487][WEBUI] Spark Master UI to reverse proxy Application and Workers UI. |
| d99c601 | 2016-10-05 | Fixed Scala style for SparkHiveExample. |
| fb327ad | 2016-10-05 | Changed the Kafka 0.9 version to 2.0.1. |
| cd374ce | 2016-10-05 | Changed version to 2.0.1-mapr-SNAPSHOT. |
| 73a298a | 2016-09-20 | [MAPR-24491] The HBase classpath might contain Hive libraries. |
| f010164 | 2016-09-14 | Minor fix for previous commit. |
| f3ec15a | 2016-09-14 | Added script for MAPR-24374. |
| 4292173 | 2016-09-13 | Some minor changes to spark-defaults.conf. |
| 36e847d | 2016-09-13 | Changed the default HBase version to 1.1.1 in compatibility.version. |
| d721e09 | 2016-09-12 | Refactored a streaming example. |
| 9aba5a1 | 2016-09-12 | [MAPR-24470] HiveFromSpark test failed in yarn-cluster mode. |
| 70809c9 | 2016-09-08 | Changed the Hive execution version to 1.2.0. |
| acabece | 2016-09-01 | Added spark streaming integration with Kafka 0.9 and Mapr-Streams. |
| 641faa3 | 2016-08-31 | Added MapR Repo. |
| 1a2e25e | 2016-08-22 | [MAPR-23559] Spark PID in /opt/mapr/pid. |
| 690788c | 2016-05-25 | [MAPR-22940] Failed to connect Spark Beeline (after Spark Thrift Server is started) on Kerberos cluster. |
| fe2c1ce | 2016-05-03 | Removed Hive jars from generated classpath for Hive. |
| 85c45d1 | 2016-05-03 | Fixed hardcoded Hive library path. |
| 1124353 | 2016-04-29 | [MAPR-23203] Removed derby jars from the generated Hive classpath. |
| 3bc0cd4 | 2016-04-13 | [MAPR-23068] Spark samples failed in Hue 3.9. |
| 506bce9 | 2016-03-16 | [MAPR-18865] Unable to submit Spark apps from Windows client. |
| 9c2fda0 | 2015-10-20 | Skip maven clean task on the parent module. |
| b699ada | 2015-07-30 | New: Issue with running Hive commands in Spark. |
| 49d106a | 2015-07-28 | Spark warden.services.conf should have a dependency on the CLDB. |
| d7425bd | 2015-05-20 | Removed DFS shuffle settings. |
| 5ed864c | 2015-04-19 | Fixed bugs in the logic to avoid SSH for localhost. |
| ad12fcf | 2015-04-14 | Copied every file in the conf directory into the distribution package. |
| 4b15fb8 | 2015-04-10 | Created spark-defaults.conf for MapR. |
| e9e4ab1 | 2015-04-08 | Avoided SSH to localhost when stopping secondary instances. |
| c8405cd | 2015-02-25 | Added htrace jar to Spark classpath for HBase 0.98. |
| c502093 | 2015-02-25 | Added the Scala library. |
| 254aecf | 2015-02-25 | Supported HBase classpath computation in util script. |
| f6319e4 | 2015-02-24 | Created ext-util. |
| c850a32 | 2015-02-24 | Added external conf and scripts. |
| 93279f6 | 2015-02-06 | Enabled SPARK_HIVE mode while building. |
| c851ac8 | 2015-10-12 | Built Spark on MapR. |
| 7e1b86c | 2015-11-25 | Spark Master failed to start in HA mode. |
| bcf7665 | 2015-02-04 | Updated the datanucleus jar in Spark for Bug 21228. |
| fbaf081 | 2015-08-22 | Changed the dependencies for MapR and increased the Hadoop version from 2.2.0 to 2.7.0. |
Known Issues and Limitations
Known issues:
- MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
- MAPR-25052:The Spark Thrift Server does not start on clusters secured by MapR-SASL.
- Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability
Limitations:
- Spark 2.0.1 Standalone mode is supported only on clusters in MRv2 (YARN) mode.
- Full support of MapR Event Store For Apache Kafka is available only on clusters with MapR 5.2 and later.
Resolved Issues
None.