Drill 1.16.1.500-2305 (EEP 8.1.1) Release Notes

The notes below relate specifically to the HPE Ezmeral Data Fabric Distribution for Apache Drill. You may also be interested in the Apache Drill homepage and the Apache Drill release notes:

Version 1.16.1.500
Release Date May 2023
HPE Version Interoperability See Component Versions for Released EEPs and EEP Components and OS Support.
Package Names See Package Names for Ecosystem Packs (EEPs).

New in This Release

Drill 1.16.1.500-2305 introduces the following enhancements or HPE platform-specific behavior changes:

Fixes

This HPE release includes the following fixes on the base release:
Commit Date (YYYY-MM-DD) Comment
38c9c314a9 2023-05-05 MD-6438: Update version of shaded guava to 31.1-jre
df338e7671 2023-05-03 MD-6439: Update jackson version to 2.12.7 and jackson databind 2.12.7.1
2572e8a8e2 2023-05-03 MD-6437: Update protobuf-java to 3.21.12
33da89f90f 2023-04-28 MD-6426: Fix endless retrying zk set data for a large query
2cfc983fbb 2023-04-20 MD-6435: Update jetty to 9.4.51.v20230217
c7bac76d3f 2023-04-19 MD-6429: remove conjars repo
8a69b8c69b 2023-03-10 MD-6412: Update commons-text to 1.10.0 due to CVE
b1abff4981 2022-09-09 MD-6371: Enable mapr maven profile by default
2531a80da4 2022-09-08 MD-6370: Exclude Xalan dependency
e20c4ca438 2022-05-04 MD-6272: Update Drill to use Antlr4 version 4.9.3
53181b0213 2022-05-02 MD-6312: Hide warning about an illegal reflective access operation while connecting via sqlline
88bfe5171c 2022-04-21 DRILL-8122: Change kafka metadata obtaining due to KAFKA-5697 (#2456)
7f7fe93ce3 2022-04-12 MD-6304: add requested file from DRILL-7204
64b78f7f04 2022-04-12 DRILL-7619: Fixed link to the metrics endpoint
8933f37e56 2022-04-12 DRILL-7582: Moved Drillbits REST API communication to the back end layer
2b4db26be7 2022-04-11 MD-6264: Cannot verify certificate when ssl enabled on DOY (#581)
c9eef515ca 2022-04-11 MD-6271: drill couldn't start when FIPS enabled on Rocky linux (#580)
59c28b849f 2022-04-07 MD-6306: Remove spring transitive dependencies (#583)

Known Issues

  • During the Drill-on-YARN installation, the system fails to upload the Drill archive because the /user/drill directory does not exist. If you install and try to start Drill-on-YARN (version 1.16.1.400 in EEP-8.1.0 or 1.16.1.500 in EEP-8.1.1 on Core 6.2.0 or 7.0.0) using the mapr-drill-yarn package, the system returns the following messages:
    /opt/mapr/drill/drill-1.16.1/bin/drill-on-yarn.sh start
    
    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by javassist.util.proxy.SecurityActions (file:/opt/mapr/drill/drill-1.16.1/jars/3rdparty/javassist-3.24.0-GA.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
    WARNING: Please consider reporting this to the maintainers of javassist.util.proxy.SecurityActions
    WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
    Connecting to DFS... Connected.
    2023-06-20 02:28:59,1379 ERROR JniCommon fc/jni_MapRClient.cc:816 Thread: 1438881 Mismatch found for java and native libraries java build version 6.2.0.0.20200915234957.GA, native build version 6.2.0.0.20200909000740.GA java patch version $Id: mapr-version: 6.2.0.0.20200915234957.GA ccd6754df227770285, native patch version $Id: mapr-version: 6.2.0.0.20200909000740.GA a40a31acab7f5e88e1
    Uploading /opt/mapr/drill/drill-1.16.1/drill.tar.gz to /user/drill/drill.tar.gz ... Failed.
    Failed to upload Drill archive
      Caused by: Failed to create DFS directory: /user/drill
      Caused by: Could not create FileClient err: 0
      Caused by: Could not create FileClient err: 0
    To resolve this issue, complete the following steps:
    1. Install Drill-on-YARN, as described in Installing Drill to Run Under YARN, but do not configure or start Drill-on-YARN.
    2. Create a file named recreate_archive.sh with the following information:
      TIP You can create and run this file in any location you choose.
      read -p "This operation will recreate drill.tar.gz in drill home directory. Continue? " -n 1 -r
      echo
      if [[ ! $REPLY =~ ^[Yy]$ ]]
      then
              exit 1
      fi
      
      drillHome="/opt/mapr/drill/drill-$(cat /opt/mapr/drill/drillversion)"
      hadoopHome="/opt/mapr/hadoop/hadoop-$(cat /opt/mapr/hadoop/hadoopversion)"
      
      hbaseJar="$(ls /opt/mapr/lib/mapr-hbase-*-mapr.jar)"
      maprWebJar="$(ls /opt/mapr/lib/mapr-security*-mapr.jar)"
      maprdbJar="$(ls /opt/mapr/lib/maprdb-[0-9].[0-9].[0-9].[0-9]-mapr.jar)"
      mapredJar="$(ls /opt/mapr/lib/maprdb-mapreduce-*-mapr.jar)"
      maprfsJar="$(ls /opt/mapr/lib/maprfs-[0-9].[0-9].[0-9].[0-9]-mapr.jar)"
      jerseyClientJar="$(ls ${hadoopHome}/share/hadoop/yarn/lib/jersey-client-*.jar)"
      jerseyCoreJar="$(ls ${hadoopHome}/share/hadoop/yarn/lib/jersey-core-*.jar)"
      
      echo "Drop old mapr jars from ${drillHome}/jars/3rdparty/"
      rm -f ${drillHome}/jars/3rdparty/mapr-hbase*
      rm -f ${drillHome}/jars/3rdparty/mapr-security-web-*
      rm -f ${drillHome}/jars/3rdparty/maprdb-*
      rm -f ${drillHome}/jars/3rdparty/maprfs-*
      
      echo "Copy new jars from /opt/mapr/lib/"
      cp ${hbaseJar} ${drillHome}/jars/3rdparty/
      [ $? == 0 ] && echo "${hbaseJar} has been copied"
      cp ${maprWebJar} ${drillHome}/jars/3rdparty/
      [ $? == 0 ] && echo "${maprWebJar} has been copied"
      cp ${maprdbJar} ${drillHome}/jars/3rdparty/
      [ $? == 0 ] && echo "${maprdbJar} has been copied"
      cp ${mapredJar} ${drillHome}/jars/3rdparty/
      [ $? == 0 ] && echo "${mapredJar} has been copied"
      cp ${maprfsJar} ${drillHome}/jars/3rdparty/
      [ $? == 0 ] && echo "${maprfsJar} has been copied"
      
      echo "Copy jersey jars from hadoop for timeline client"
      cp ${jerseyClientJar} ${drillHome}/jars/3rdparty/
      [ $? == 0 ] && echo "${jerseyClientJar} has been copied"
      cp ${jerseyCoreJar} ${drillHome}/jars/3rdparty/
      [ $? == 0 ] && echo "${jerseyCoreJar} has been copied"
      
      if [ -f ${drillHome}/drill.tar.gz ]; then
      	rm -f ${drillHome}/drill.tar.gz
      	echo "${drillHome}/drill.tar.gz has been dropped"
      fi
      
      tempDir=drill-$(date +%s)
      
      mkdir /tmp/$tempDir
      [ $? == 0 ] && echo "Created temporary directory ${tempDir}"
      
      cd /tmp/$tempDir
      
      mkdir drill
      [ $? == 0 ] && echo "Created drill directory"
      
      cp -r ${drillHome}/* ./drill/
      echo "${drillHome} copied to drill directory"
      
      tar -czf drill.tar.gz ./drill
      echo "Created new drill archive"
      
      cp drill.tar.gz ${drillHome}/
      echo "drill.tar.gz copied to ${drillHome}"
      
      rm -rf /tmp/$tempDir
      echo -e "\033[0;32mDONE.\033[0m"
    3. Issue the following command to run the script:
      sh recreate_archive.sh
      When the script runs, a prompt appears:
      This operation will recreate drill.tar.gz in drill home directory. Continue?

      Reply with y to allow the script to update the JAR files in Drill-on-YARN.

      You should see the following output:
      Drop old mapr jars from /opt/mapr/drill/drill-1.16.1/jars/3rdparty/
      Copy new jars from /opt/mapr/lib/
      /opt/mapr/lib/mapr-hbase-6.2.0.0-mapr.jar has been copied
      /opt/mapr/lib/mapr-security-web-6.2.0.0-mapr.jar has been copied
      /opt/mapr/lib/maprdb-6.2.0.0-mapr.jar has been copied
      /opt/mapr/lib/maprdb-mapreduce-6.2.0.0-mapr.jar has been copied
      /opt/mapr/lib/maprfs-6.2.0.0-mapr.jar has been copied
      Copy jersey jars from hadoop for timeline client
      /opt/mapr/hadoop/hadoop-2.7.6/share/hadoop/yarn/lib/jersey-client-1.19.jar has been copied
      /opt/mapr/hadoop/hadoop-2.7.6/share/hadoop/yarn/lib/jersey-core-1.19.jar has been copied
      /opt/mapr/drill/drill-1.16.1/drill.tar.gz has been dropped
      Created temporary directory drill-1656424210
      Created drill directory
      /opt/mapr/drill/drill-1.16.1 copied to drill directory
      Created new drill archive
      drill.tar.gz copied to /opt/mapr/drill/drill-1.16.1
      DONE.
    4. Configure Drill to run under YARN and start Drill-on-YARN, as described in Configuring Drill to Run Under YARN.
  • Due to Drill version changes (3-digit to 4-digit), you cannot upgrade from Drill in EEP 7.0.0 (Drill 1.16.1) to Drill in EEP 7.0.1 (Drill 1.16.1.5) or later. You must perform a new installation of Drill. Alternatively, if you are running Drill on CentOS or RHEL, you can issue the following command as a workaround to upgrade Drill:
    rpm -Uv --<old-package> <path/to/packages>/*.rpm

    See Upgrading Drill.

Limitations

  • Older versions of Drill, such as Drill 1.10.0, supported the HBase plug-in, but Drill versions 1.11.0 through 1.16.0.x do not support queries on HBase tables.