Hive Features in HPE Ezmeral Data Fabric

Describes HPE Ezmeral Data Fabric-specific features in Hive.

Removing Temporary Hive Files

Starting from EEP 8.1.0, EEP 7.1.2, and EEP 6.3.6, to remove the temporary Hive files created during the Hive session, set the value of hive.scratchdir.lock property to true on hive-site.xml file.

<property> 
   <name>hive.scratchdir.lock</name> 
   <value>true</value> 
</property>

For the previous EEP versions, manually remove the temporary Hive files that are not used by the active Hive sessions.

You have two different situations:

If you have configured the HiveServer2 in a node, set hive.scratchdir.lock property on the hive-site.xml file to automatically remove the temporary Hive files.
If you have not configured the HiveServer2 in a node, set the hive.scratchdir.lock property and run the following command to remove the temporary Hive files.
```
hive --service cleardanglingscratchdir
```

Symbolic Link Support in Hive

Starting from EEP 7.1.0, all hadoop fs commands support operations on symlinks (symbolic links). Hive supports symlinks in EEP 8.0.0 onwards. You can create symlinks through the command line interface or file system API (MapRFileSystem.java).

Symlink creation via CLI has the following requirements:

NFS installed
NFS mounted (mount hadoop fs to the local file system)

Creating Symlinks

The following examples demonstrate how to create symbolic links via CLI and MapRFileSystem API:

Create a relative symlink via CLI:

ln -rs /mountPoint/path/to/file /mountPoint/path/to/symlink

Create an absolute symlink via CLI:

ln -s /mountPoint/path/to/file /mountPoint/path/to/symlink

Create a symlink via MapRFileSystem API:

MapRFileSystem maprFS = MapRFileSystem.get(new Configuration()); 
maprFS.createSymlink(pathToTarget, pathToLink, createParentFlag);

Using Symlinks for Hive Operations

Once a symlink is created, you can use the symlink for Hive operations, such as table location and data file, as demonstrated in the following steps:

Create a table directory:

mkdir /mapr/my.cluster.com/user/hive/warehouse/ext_tbl_symlink

Create a symlink from a data source to a table location:

ln -s /mapr/my.cluster.com/user/mapr/source_files/data.txt /mapr/my.cluster.com/user/hive/warehouse/ext_tbl_mh120/data_link.txt

Create an external Hive table in the ext_tbl_symlink directory (created in step 1):

CREATE EXTERNAL TABLE file_link_table (...) ROW FORMAT DELIMITED FIELDS TERMINATED BY "," STORED AS TEXTFILE LOCATION '/user/hive/warehouse/ext_tbl_symlink';

The Hive table has a symbolically linked text file as the data source. Data can be processed as if it is a regular data file.