com.mapr.db.mapreduce

Class TotalOrderPartitioner<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.Partitioner<K,V>

com.mapr.db.mapreduce.TotalOrderPartitioner<K,V>

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

@InterfaceAudience.Public
 @InterfaceStability.Stable
public class TotalOrderPartitioner<K,V>
extends org.apache.hadoop.mapreduce.Partitioner<K,V>
implements org.apache.hadoop.conf.Configurable

Partitioner effecting a total order by reading split points from an externally generated source.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_PATH
static java.lang.String	MAX_TRIE_DEPTH
static java.lang.String	NATURAL_ORDER
static java.lang.String	PARTITIONER_PATH

Constructor Summary

Constructors

Constructor and Description
TotalOrderPartitioner()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.conf.Configuration	getConf()
int	getPartition(K key, V value, int numPartitions)
static java.lang.String	getPartitionFile(org.apache.hadoop.conf.Configuration conf) Get the path to the SequenceFile storing the sorted partition keyset.
void	setConf(org.apache.hadoop.conf.Configuration conf) Read in the partition file and build indexing data structures.
static void	setPartitionFile(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path p) Set the path to the SequenceFile storing the sorted partition keyset.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_PATH

public static final java.lang.String DEFAULT_PATH

See Also:

Constant Field Values

PARTITIONER_PATH

public static final java.lang.String PARTITIONER_PATH

See Also:

Constant Field Values

MAX_TRIE_DEPTH

public static final java.lang.String MAX_TRIE_DEPTH

See Also:

Constant Field Values

NATURAL_ORDER

public static final java.lang.String NATURAL_ORDER

See Also:

Constant Field Values

Constructor Detail

TotalOrderPartitioner

public TotalOrderPartitioner()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Read in the partition file and build indexing data structures. If the keytype is BinaryComparable and total.order.partitioner.natural.order is not false, a trie of the first total.order.partitioner.max.trie.depth(2) + 1 bytes will be built. Otherwise, keys will be located using a binary search of the partition keyset using the RawComparator defined for this job. The input file must be sorted with the same comparator and contain JobContextImpl.getNumReduceTasks() - 1 keys.

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

getPartition

public int getPartition(K key,
                        V value,
                        int numPartitions)

Specified by:

getPartition in class org.apache.hadoop.mapreduce.Partitioner<K,V>

setPartitionFile

public static void setPartitionFile(org.apache.hadoop.conf.Configuration conf,
                                    org.apache.hadoop.fs.Path p)

Set the path to the SequenceFile storing the sorted partition keyset. It must be the case that for R reduces, there are R-1 keys in the SequenceFile.

getPartitionFile

public static java.lang.String getPartitionFile(org.apache.hadoop.conf.Configuration conf)

Get the path to the SequenceFile storing the sorted partition keyset.

See Also:

setPartitionFile(Configuration, Path)