org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners

Class SkewedPartitioner

java.lang.Object

org.apache.hadoop.mapreduce.Partitioner<PigNullableWritable,org.apache.hadoop.io.Writable>

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.SkewedPartitioner

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

SkewedPartitionerTez

public class SkewedPartitioner
extends org.apache.hadoop.mapreduce.Partitioner<PigNullableWritable,org.apache.hadoop.io.Writable>
implements org.apache.hadoop.conf.Configurable

This class is used by skewed join. For the partitioned table, the skewedpartitioner reads the key distribution data from the sampler file and returns the reducer index in a round robin fashion. For ex: if the key distribution file contains (k1, 5, 3) as an entry, reducers from 5 to 3 are returned in a round robin manner.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	inited
protected java.util.Map<Tuple,Pair<java.lang.Integer,java.lang.Integer>>	reducerMap
protected static TupleFactory	tf
protected java.lang.Integer	totalReducers

Constructor Summary

Constructors

Constructor and Description
SkewedPartitioner()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.conf.Configuration	getConf()
int	getPartition(PigNullableWritable wrappedKey, org.apache.hadoop.io.Writable value, int numPartitions)
protected void	init()
void	setConf(org.apache.hadoop.conf.Configuration job)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tf

protected static final TupleFactory tf

reducerMap

protected java.util.Map<Tuple,Pair<java.lang.Integer,java.lang.Integer>> reducerMap

totalReducers

protected java.lang.Integer totalReducers

inited

protected boolean inited

Constructor Detail

SkewedPartitioner

public SkewedPartitioner()

Method Detail

getPartition

public int getPartition(PigNullableWritable wrappedKey,
               org.apache.hadoop.io.Writable value,
               int numPartitions)

Specified by:

getPartition in class org.apache.hadoop.mapreduce.Partitioner<PigNullableWritable,org.apache.hadoop.io.Writable>

setConf

public void setConf(org.apache.hadoop.conf.Configuration job)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

init

protected void init()