Spark 1.5.2-1608 Release Notes

The notes below relate specifically to the MapR Converged Data Platform. You may also be interested in the open source Spark 1.5.2 Release Notes.

Spark Version 1.5.2
Release Date September 1, 2016
MapR Version Interoperability See the Ecosystem Support Matrix (Pre-5.2 releases) and Spark Support Matrix.
Source on GitHub https://github.com/mapr/spark/tree/1.5.2-mapr-1608
Package Names The following packages are associated with this release:
  • mapr-spark-master-1.5.2.201608302219-1.noarch.rpm
  • mapr-spark-master_1.5.2.201608302219_all.deb

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

Important Notes

This release of Apache Spark for MapR includes the following shared library dependencies:
  • Mesos version: 0.21.1
  • HBase version: 0.98.12-mapr-1506
  • Flume version 1.6.0
  • Zookeeper version: 3.4.5-mapr-1503
  • Hive version: 1.2.0-mapr-1603
  • Hive parquet version: 1.6.0
  • Kafka version: 0.9.0.0
  • Kafka clients version: 0.9.0.0-mapr-1602

Fixes

This release by MapR includes the following fixes on the base Apache release. For complete details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comments
ac11283 2016-08-02 MAPR-24129: Backported SPARK-16796 so that passwords are now masked on the Environment page of the Spark History Server UI.
0bda0b9 2016-08-18 MARP-24264: The following fixes have been backported to resolve the Spark vulnerability to code injection in the toCommentSafeString method:
  • [SPARK-11352][SQL][BRANCH-1.5] Escape */ in the generated comments.
  • [SPARK-12138][SQL] Escape \u in the generated comments of codegen
  • [SPARK-15165][SQL] Codegen can break because toCommentSafeString is not actually safe
For more information, see Spark Vulnerability with the toCommentSafeString Method.
fc72384 2016-08-22 MAPR-24318: Backported SPARK -1191 and SPARK-11311 so that Spark is now able to create UDF's using the Hive Thrift Service.
681dcf4 2016-08-24 MAPR-24263: The Spark assembly and example JAR files no longer include a dependency on MapR-FS libraries.

Known Issues

  • MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
  • MAPR-19761: On a secure cluster, MapR does not support the Spark SQL Thrift JDBC server. When the cluster is secure, the Spark Thrift server will not start.
  • MAPR-22940/SPARK-11851: On clusters that use Kerberos authentication, Spark Thrift Server is unable to connect to beeline.
  • Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability