Hive 2.3.9.0 - 2201 (EEP 8.1.0) Release Notes

The following notes relate specifically to the HPE Ezmeral Data Fabric Distribution for Apache Hive. You may also be interested in the Apache Hive-2.3.9 Release Notes and the Apache Hive homepage.
Hive Version 2.3.9.0
Release Date January 2022
HPE Version Interoperability See Hive and HCatalog Support Matrix and Ecosystem Support Matrix and EEP Components and OS Support.
Source on GitHub https://github.com/mapr/hive
GitHub Release Tag 2.3.9.0-eep-810
Maven Artifacts https://repository.mapr.com/maven/
Package Names Navigate to http://package.ezmeral.hpe.com/releases/MEP/, and select your EEP(MEP) and OS to view the list of package names.
ODBC/JDBC Drivers
Hive 2.3.9 works with the following HPE Hive drivers:

For additional driver information, see Connecting to HiveServer2.

Feature support

The following list describes support of various components and functionality with Hive 2.3.9 - 2201:

  • Supports Hive-2.3.9 on Tez-0.9.2 For more information, see Tez 0.9.2 - 2201 (EEP 8.1.0) Release Notes.

  • Does not support Hive on Spark. You cannot use Spark as a query engine for Hive. However, you can run Hive and Spark on the same cluster. You can also use Spark SQL and Drill to query Hive tables.

  • Does not support HDFS encryption in Hive tables.

  • Does not support LLAP with Hive-2.3.9 because Apache Slider is not an HPE supported ecosystem component.

  • Starting from Hive 2.1, Hive must run the schematool command as an initialization step.

Changes in default security configuration

The following list describes changes in default security for Hive 2.3.9 - 2201:

  • Starting from EEP-8.1.0, Hive supports SCRAM token and SCRAM-SHA-256 authentication in HPE Ezmeral Data Fabric.

    Table 1. Description of hive.delegation.token.authentication
    # Property Data Type Default value Description
    1 hive.delegation.token.authentication String DIGEST Delegation token authentication method. Possible values are DIGEST, SCRAM
    To configure SCRAM token and SCRAM-SHA-256 authentication, set the following property on HIVE_HOME/conf/hive-site.xml file:
    <property>
    <name>hive.delegation.token.authentication</name>    
    <value>SCRAM</value>  
</property>

    Execute MAPR_HOME/server/configure.sh -R script on a newly installed and Data-Fabric SASL or KERBEROS secured cluster to automatically configure the following authentications:

    1. For a FIPS enabled cluster, Hive configures hive.delegation.token.authentication=SCRAM authentication.
    2. For a non-FIPS cluster if you configure Hadoop with hadoop.security.token.authentication.method=SCRAM authentication, Hive configures the SCRAM authentication.
    3. For other clusters, Hive configures hive.delegation.token.authentication=DIGEST authentication.

    For non-secure clusters, Hive configures hive.delegation.token.authentication=DIGEST authentication.

    When you upgrade Hive, the upgrade does not update the value of the set hive.delegation.token.authentication property.

    Manually set the value of hive.delegation.token.authentication property when you change the cluster settings from FIPS to non-FIPS or from non-FIPS to FIPS.

New in This Release

Hive 2.3.9.0 - 2201 introduces the following enhancements or HPE platform-specific behavior changes:

  • Starting from EEP-8.1.0, Hive supports FIPS and SCRAM SASL.
  • Beginning with EEP 8.1.0, JAR artifacts for Hive use four digits instead of three digits. For example:
    hive-service-rpc-2.3.9.0-mapr-SNAPSHOT.jar
hive-llap-ext-client-2.3.9.0-mapr-SNAPSHOT.jar
hive-exec-2.3.9.0-mapr-SNAPSHOT.jar
hive-beeline-2.3.9.0-mapr-SNAPSHOT.jar
    If your application includes a Hive dependency in the pom.xml file, you must update the JAR artifact before using Hive 2.3.9 - 2201.

    The Hive package name on package.ezmeral.hpe.com continues to use two digits, and the Hive root folder continues to use three digits.

  • Improved Describe table operator in terms of fetching statistics of partitions. Starting from EEP 8.0.0, you can fetch the partition information using the describe command with formatted or extended statements.
    Configure the hive.describe.partitionedtable.ignore.stats property to change the behaviour of fetching statistics of partitions. It is set to the default value of false.
    <property>
    <name>hive.describe.partitionedtable.ignore.stats</name>
    <value>false</value>
    <description>Enables partitioned table stats collection for 'DESCRIBE FORMATTED' or 'DESCRIBE EXTENDED' commands</description>
</property>
    <property>
    <name>hive.describe.partitionedtable.ignore.stats</name>
    <value>true</value>
    <description>Disables partitioned table stats collection for 'DESCRIBE FORMATTED' or 'DESCRIBE EXTENDED' commands</description>
 </property
  • Hive supports symbolic links on file system. See Hive Features in HPE Ezmeral Data Fabric.

Fixes

This HPE release includes the following fixes on the base release:
GitHub Commit Number Data (YYYY-MM-DD) HPE Fix Number and Description
a6d17888e2 2022-01-31 EEP-HIVE-1153 : FAILURE! - in org.apache.hive.conftool.SslDefaultTest
5d4dcca0d9 2022-01-31 EEP-HIVE-1151 : Build Hive ECO EEP 8.1.0 components with DF v 6.2.0
be5e7ce3a7 2022-01-27 EEP-HIVE-1148 : Cannot run HiveMetastore service on non-fips cluster for EEP-8.1.0
bc913e56b1 2022-01-27 EEP-HIVE-1147 : JDBC connection failed if working with different connections on FIPS enabled clusters
9dc2d25f98 2022-01-24 EEP-HIVE-1141 : Hive services cannot communicate between FIPS / non-FIPS nodes
e527acd2dd 2022-01-18 EEP-HIVE-1135: HiveServer2 will fail if default provider will not support FIPS
3498c725a6 2022-01-06 EEP-HIVE-1065: CVE-2021-37136, CVE-2021-37137, WS-2020-0408, CVE-2021-21290: netty-*-4.1.55.Final.jar
e82cca2f60 2022-01-05 EEP-HIVE-1062: CVE-2016-5007, CVE-2016-9878 ,CVE-2018-1271, CVE-2018-1272, CVE-2020-5421: spring-*-3.2.16.RELEASE.jar
e2ae1f021e 2022-01-05 EEP-HIVE-1068: CVE-2020-9480: spark-network-common_2.11-2.3.0.jar, CVE-2018-17190: spark-core_2.11-2.3.0.jar
110f14340f 2022-01-04 EEP-HIVE-1117: Update log4j v2 to the latest available (to 2.17+)
5cd9fcc2d9 2022-01-04 EEP-HIVE-1116: Hive returns an incorrect number of columns
ae3e3d49e8 2022-01-04 Revert "MAPR-HIVE-930: Cannot run join with Order by and Limit clause specified at the same time"
8b01f203da 2021-12-30 EEP-HIVE-1119: com.fasterxml.jackson.annotation.JsonFormat.empty()Lcom/fasterxml/jackson/annotation/JsonFormat
250a524cbe 2021-12-27 EEP-HIVE-1099 : [FIPS] HS2 connection PAM + SSL doesn't work with SBD configuration when FIPS enabled
da9feb7041 2021-12-24 EEP-HIVE-1064: CVE-2021-30639 ; CVE-2021-33037: tomcat-coyote-10.0.4.jar
3288fbbea0 2021-12-24 EEP-HIVE-1059: CVE-2019-10172, CVE-2019-10202: jackson-mapper-asl-1.9.13.jar, jackson-mapper-asl-1.9.2.jar
70be9aeb2c 2021-12-24 EEP-HIVE-1056: CVE-2021-35515, CVE-2021-35516, CVE-2021-35517, CVE-2021-36090: commons-compress-1.20.jar
2a7a346e3e 2021-12-24 EEP-HIVE-1055: CVE fixes of bcprov-jdk15on-1.52.jar
2e9596dc8a 2021-12-24 EEP-HIVE-1054: WS-2021-0419: gson-2.2.4.jar
63a06a47d7 2021-12-22 EEP-HIVE-1115 : Unrecognized VM option UseGCLogFileRotation
3dd5c4f710 2021-12-21 EEP-HIVE-1088: [Hive-2.3.9] Hive on Tez engine + native S3/OPAL Unable to load AWS credentials from any provider in the chain
6e06d411a3 2021-12-21 EEP-HIVE-1097: CVE-2021-44228 - Log4j vulnerability
facbb602ee 2021-12-21 EEP-HIVE-1091 : Add SCRAM-SASL to Hive
158d663d48 2021-12-07 EEP-HIVE-1087: CAST gives NULL values during insert when vectorization enabled.
1e74e4cccb 2021-12-02 MAPR-HIVE-1090 : TLSv1.2 SSLContext not available
904758aef6 2021-11-25 MAPR-HIVE-1086 : Upgrade Jetty to 9.4.44.v20210927
04c60792f4 2021-11-24 MAPR-HIVE-1074 : Beeline fails to connect to HiveServer2 with java.lang.NoSuchFieldError: BCFKS error on Core 6.2.0/EEP 8.1.0
2851a75533 2021-11-22 MAPR-HIVE-1071 : Fix SslDefaultTest
c6e21ea94d 2021-11-22 MAPR-HIVE-1069 : HIVE-1069 Update hbase version to 1.4.13.0-eep-810-SNAPSHOT
62c6ca2825 2021-11-18 MAPR-HIVE-1053 : Relative path in absolute URI: slider reads hdfs-site.xml from hadoop-hdfs.jar
35f8178503 2021-11-18 MAPR-HIVE-1036 : java.lang.NoClassDefFoundError: org/apache/commons/digester/Digester
e7b42cb8b1 2021-11-16 MAPR-HIVE-1026 : Migrate to python 3 in LLAP package.py file
b925b5c99d 2021-11-16 MAPR-HIVE-1025 : LLAP server expects tez.tar.gz archive in MapR FS
7fa0949cde 2021-11-16 MAPR-HIVE-1033 : Add MapR slider dependency to Hive
b7cf317ce4 2021-11-11 MAPR-HIVE-1031 : logError: command not found if any error happens during configuring Hive
c2c8453683 2021-11-11 MAPR-HIVE-1024 : Replace deprecated AuthMethod.DIGEST with AuthMethod.TOKEN in HadoopThriftAuthBridge25Sasl
806f29ca93 2021-11-10 DFDEVOPS-2081 : Configure Jenkins job for hive-2.3.9 mep-8.1.0
4364d7d1c4 2021-11-10 MAPR-HIVE-1022 : Update Hive version to 2.3.9.0-eep-810-SNAPSHOT
3212a89673 2021-11-09 MAPR-HIVE-1021 : Upgrade mapr-core version to 7.0.0.0-mapr-SNAPSHOT
d300f64a22 2021-11-04 MAPR-HIVE-831 : Move Hive to 4 digits in jar artefacts
bca47dbbea 2021-11-02 MAPR-HIVE-975: Customer request to investigate temporary hive session files cleanup improvements
304849fc9a 2021-11-01 MAPR-HIVE-1018 : Update hadoop version to 2.7.6.200-eep-810-SNAPSHOT
7c8fe3cc80 2021-11-01 MAPR-HIVE-1012 : Hiveserver2 could not start on cluster with enabled FIPS
1ccc9d7b5a 2021-10-13 MAPR-HIVE-1002 : Hive-2.3 does not remove old compressed logs
c26ff7e273 2021-10-11 MAPR-HIVE-1016 : Update Conjars repository URL to secure
4bf0c7aadc 2021-10-11 MAPR-HIVE-1015 : Configure repositories for Jenkins job
69d55d6809 2021-10-11 MAPR-HIVE-997 : ConfigureShInsecureTest hangs up
92a25a685c 2021-10-11 MAPR-HIVE-1013 : Update calcite version to 1.10.0-eep
16c0ba4efd 2021-10-11 MAPR-HIVE-1014 : Could not transfer artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
This release from HPE also includes the following back-ported issues. For complete details, refer to the commit log for this project in GitHub.
GitHub Commit Number Data (YYYY-MM-DD) HPE Fix Number and Description
95ae2bebc9 2021-12-21 HIVE-17774: compaction may start with 0 splits and fail
ed4b16c0fa 2021-10-08 HIVE-16820 : TezTask may not shut down correctly before submit (Sergey Shelukhin, reviewed by Siddharth Seth)
794c971152 2021-10-28 HIVE-20072 : Write access being requested when performing select on a table

Known Issues and Limitations

  • HIVE-1089: Hive on MapReduce engine does not support data insertion into Versioned buckets.You must use Unversioned buckets on MaprReduce engine in S3 file system.

    Hewlett Packard Enterprise recommends using Hive on Tez engine for full S3 file system support.

  • HIVE-19502: Unable to insert values into table stored by JdbcStorageHandler

  • HIVE-19286: NPE in MERGE operator on MR mode

  • HIVE-760: [Hive-2.3] Could not start hive-metastore on Centos 8 MetaException(message:Version information not found in metastore)

    Starting in EEP 7.0.0, use the MySQL driver with MariaDB.
    <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
 </property>
 <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
 </property>
    Some SELECT queries can be converted to a single FETCH task minimizing latency. Currently, the query should be single sourced and should not have a subquery or any aggregations or distincts (which incurs RS), lateral views and joins:
    none : disable hive.fetch.task.conversion
minimal : SELECT star, filter on partition columns, LIMIT only
more :  SELECT, filter, LIMIT only (support TABLESAMPLE and virtual columns)

Resolved Issues