Spark Feature Support

MapR supports most Spark features. However, there a few exceptions.

Spark R

SparkR is supported as of the Spark 1.5.2-1512 release. It is not supported in MapR's Spark 1.4.1 release.

Spark Thrift JDBC/ODBC Server Support

Running the Spark Thrift JDBC/ODBC Server on a secure cluster is not supported.

As of Spark 1.5.2, you can run the Spark Thrift JDBC/ODBC Server to enable connections to Hive 1.2.1 using beeline; however, you cannot connect to other versions of Hive using beeline.

NOTE: As of Spark 1.6.1-1607, when HiveServer2 uses Kerberos authentication, the Spark Thrift Server must run on the same node as HiveServer2. Otherwise, beeline will not be able to connect to the Spark Thirft Server.

Spark SQL and Hive Support

Spark SQL is supported but it is not fully compatible with Hive; see the Apache Spark documentation for details. Also, as of Spark 1.5.2, the following Spark SQL operations support the following Hive table formats:

	Hive 0.13 Table Format
Spark SQL Operation	AVRO	ORC	Parquet	RC	default
create	Yes	Yes	Yes	Yes	Yes
drop	Yes	Yes	Yes	Yes	Yes
insert into	Yes	Yes	Yes	Yes	Yes
insert overwrite	No	No	Yes	No	Yes
load data	Yes	Yes	Yes	Yes	Yes
select	Yes	Yes	Yes	Yes	Yes

	Hive 1.0 Table Format
Spark SQL Operation	AVRO	ORC	Parquet	RC	default
create	Yes	Yes	Yes	Yes	Yes
drop	Yes	Yes	Yes	Yes	Yes
insert into	Yes	Yes	Yes	Yes	Yes
insert overwrite	No	No	Yes	No	Yes
select	Yes	Yes	Yes	Yes	Yes
load data	Yes	Yes	Yes	Yes	Yes

	Hive 1.2 Table Format
Spark SQL Operations	AVRO	ORC	Parquet	RC	default
create	Yes	Yes	Yes	Yes	Yes
drop	Yes	Yes	Yes	Yes	Yes
insert into	Yes	Yes	Yes	Yes	Yes
insert overwrite	Yes	Yes	Yes	Yes	Yes
select	Yes	Yes	Yes	Yes	Yes
load data	Yes	Yes	Yes	Yes	Yes