Spark 1.6.1-1608 Release Notes

The notes below relate specifically to the MapR Converged Data Platform. You may also be interested in the open source Spark 1.6.1 Release Notes

Spark Version 1.6.1
Release Date September 1, 2016
MapR Version Interoperability See the Interoperability Matrices, Ecosystem Support Matrix (Pre-5.2 releases), and Spark Support Matrix.
Source on GitHub https://github.com/mapr/spark/tree/1.6.1-mapr-1608
Package Names The following packages are associated with this release:
  • mapr-spark-master-1.6.1.201608302253-1.noarch.rpm
  • mapr-spark-master_1.6.1.201608302253_all.deb

Important Notes

If you want to integrate Spark 1.6.1-1608 with HPE Ezmeral Data Fabric Streams, you must install the Kafka 0.9.0-1607 package.

Hive Support

This version of Spark supports integration with Hive. However, note the following exceptions:

Fixes

This release by MapR includes the following fixes on the base Apache release. For complete details, refer to the commit log for this project in GitHub.

GitHub Commit Date (YYYY-MM-DD) Comments
3024b71 2016-08-01 MAPR-24129: Backported SPARK-16796 so that passwords are now masked on the Environment page of the Spark History Server UI.
7755848 2016-08-16 [MARP-24264] The following fixes have been backported to resolve the Spark vulnerability to code injection in the toCommentSafeString method:
  • [SPARK-11352][SQL][BRANCH-1.5] Escape */ in the generated comments.
  • [SPARK-12138][SQL] Escape \u in the generated comments of codegen
  • [SPARK-15165][SQL] Codegen can break because toCommentSafeString is not actually safe
For more information, see Spark Vulnerability with the toCommentSafeString Method.
04c451a 2016-08-18 [MAPR-24292] HPE Ezmeral Data Fabric Streams consumer no longer fails to read the first message when the message contains a single string.
12df59b 2016-08-25 [MAPR-24263] The Spark assembly and example JAR files no longer include a dependency on MapR-FS libraries.

Known Issues

  • MAPR-17271: On secure clusters, the MapR Control System (MCS) does not display links for Spark-Master and Spark-HistoryServer.
  • MAPR-19761: On a secure cluster, MapR does not support the Spark SQL Thrift JDBC server. When the cluster is secure, the Spark Thrift server will not start.
  • Spark versions up to and including 2.3.0 have the following security vulnerability: CVE-2018-1334 Apache Spark local privilege escalation vulnerability