org.apache.pig.backend.hadoop.executionengine.spark

Class MapReducePartitionerWrapper

java.lang.Object

Partitioner

org.apache.pig.backend.hadoop.executionengine.spark.MapReducePartitionerWrapper

public class MapReducePartitionerWrapper
extends Partitioner

Spark Partitioner that wraps a custom partitioner that implements org.apache.hadoop.mapreduce.Partitioner interface. Since Spark's shuffle API takes a different parititioner class (@see org.apache.spark.Partitioner) compared to MapReduce, we need to wrap custom partitioners written for MapReduce inside this Spark Partitioner. MR Custom partitioners are expected to implement getPartition() with specific arguments: public int getPartition(PigNullableWritable key, Writable value, int numPartitions) For an example of such a partitioner,

See Also:

org.apache.pig.test.utils.SimpleCustomPartitioner

Constructor Summary

Constructors

Constructor and Description
MapReducePartitionerWrapper(String partitionerName, int numPartitions)

Method Summary

Modifier and Type	Method and Description
int	getPartition(Object key)
int	numPartitions()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MapReducePartitionerWrapper

public MapReducePartitionerWrapper(String partitionerName,
                                   int numPartitions)

Method Detail

numPartitions

public int numPartitions()

getPartition

public int getPartition(Object key)