org.apache.pig.backend.hadoop.executionengine.mapReduceLayer

Class PigGenericMapReduce

java.lang.Object

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce

Direct Known Subclasses:

PigMapReduce

public class PigGenericMapReduce
extends Object

This class is the static Mapper & Reducer classes that are used by Pig to execute Pig Map Reduce jobs. Since there is a reduce phase, the leaf is bound to be a POLocalRearrange. So the map phase has to separate the key and tuple and collect it into the output collector. The shuffle and sort phase sorts these keys & tuples and creates key, List<Tuple> and passes the key and iterator to the list. The deserialized POPackage operator is used to package the key, List<Tuple> into pigKey, Bag<Tuple> where pigKey is of the appropriate pig type and then the result of the package is attached to the reduce plan which is executed if its not empty. Either the result of the reduce plan or the package res is collected into the output collector. The index of the tuple (that is, which bag it should be placed in by the package) is packed into the key. This is done so that hadoop sorts the keys in order of index for join. This class is the base class for PigMapReduce, which has slightly difference among different versions of hadoop. PigMapReduce implementation is located in $PIG_HOME/shims.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PigGenericMapReduce.Map
static class	PigGenericMapReduce.MapWithComparator This "specialized" map class is ONLY to be used in pig queries with order by a udf.
static class	PigGenericMapReduce.MapWithPartitionIndex Used by Skewed Join
static class	PigGenericMapReduce.Reduce
static class	PigGenericMapReduce.ReduceWithComparator This "specialized" reduce class is ONLY to be used in pig queries with order by a udf.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.conf.Configuration	sJobConf Deprecated. Use UDFContext instead in the following way to get the job's Configuration: UdfContext.getUdfContext().getJobConf()
static ThreadLocal<org.apache.hadoop.conf.Configuration>	sJobConfInternal
static org.apache.hadoop.mapreduce.JobContext	sJobContext

Constructor Summary

Constructors

Constructor and Description
PigGenericMapReduce()

Method Summary

Modifier and Type	Method and Description
static void	staticDataCleanup()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sJobContext

public static org.apache.hadoop.mapreduce.JobContext sJobContext

sJobConf

@Deprecated
public static org.apache.hadoop.conf.Configuration sJobConf

Deprecated. Use UDFContext instead in the following way to get the job's Configuration:

UdfContext.getUdfContext().getJobConf()

sJobConfInternal

public static ThreadLocal<org.apache.hadoop.conf.Configuration> sJobConfInternal

Constructor Detail

PigGenericMapReduce

public PigGenericMapReduce()

Method Detail

staticDataCleanup

public static void staticDataCleanup()