org.apache.pig.backend.hadoop.executionengine.mapReduceLayer

Class PigGenericMapReduce.Reduce

java.lang.Object

org.apache.hadoop.mapreduce.Reducer<PigNullableWritable,NullableTuple,PigNullableWritable,org.apache.hadoop.io.Writable>

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce.Reduce

Direct Known Subclasses:

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce.Reduce

Enclosing class:

PigGenericMapReduce

public abstract static class PigGenericMapReduce.Reduce
extends org.apache.hadoop.mapreduce.Reducer<PigNullableWritable,NullableTuple,PigNullableWritable,org.apache.hadoop.io.Writable>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Reducer

org.apache.hadoop.mapreduce.Reducer.Context

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	errorInReduce
protected boolean	initialized
protected org.apache.commons.logging.Log	log
protected org.apache.hadoop.mapreduce.Reducer.Context	outputCollector
protected POPackage	pack
protected PhysicalPlan	rp
protected List<POStore>	stores

Constructor Summary

Constructors

Constructor and Description
PigGenericMapReduce.Reduce()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	cleanup(org.apache.hadoop.mapreduce.Reducer.Context context) Will be called once all the intermediate keys and values are processed.
abstract org.apache.hadoop.mapreduce.Reducer.Context	getIllustratorContext(org.apache.hadoop.mapred.jobcontrol.Job job, List<Pair<PigNullableWritable,org.apache.hadoop.io.Writable>> input, POPackage pkg) Get reducer's illustrator context
abstract POPackage	getPack(org.apache.hadoop.mapreduce.Reducer.Context context)
abstract boolean	inIllustrator(org.apache.hadoop.mapreduce.Reducer.Context context)
boolean	processOnePackageOutput(org.apache.hadoop.mapreduce.Reducer.Context oc)
protected void	reduce(PigNullableWritable key, Iterable<NullableTuple> tupIter, org.apache.hadoop.mapreduce.Reducer.Context context) The reduce function which packages the key and List<Tuple> into key, Bag<Tuple> after converting Hadoop type key into Pig type.
protected void	runPipeline(PhysicalOperator leaf)
void	setReducePlan(PhysicalPlan plan) Set the reduce plan: to be used by local runner for illustrator
protected void	setup(org.apache.hadoop.mapreduce.Reducer.Context context) Configures the Reduce plan, the POPackage operator and the reporter thread

Methods inherited from class org.apache.hadoop.mapreduce.Reducer

run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected final org.apache.commons.logging.Log log

rp

protected PhysicalPlan rp

stores

protected List<POStore> stores

pack

protected POPackage pack

outputCollector

protected org.apache.hadoop.mapreduce.Reducer.Context outputCollector

errorInReduce

protected boolean errorInReduce

initialized

protected volatile boolean initialized

Constructor Detail

PigGenericMapReduce.Reduce

public PigGenericMapReduce.Reduce()

Method Detail

setReducePlan

public void setReducePlan(PhysicalPlan plan)

Set the reduce plan: to be used by local runner for illustrator

Parameters:

plan - Reduce plan

setup

protected void setup(org.apache.hadoop.mapreduce.Reducer.Context context)
              throws IOException,
                     InterruptedException

Configures the Reduce plan, the POPackage operator and the reporter thread

Overrides:

setup in class org.apache.hadoop.mapreduce.Reducer<PigNullableWritable,NullableTuple,PigNullableWritable,org.apache.hadoop.io.Writable>

Throws:

IOException

InterruptedException

reduce

protected void reduce(PigNullableWritable key,
          Iterable<NullableTuple> tupIter,
          org.apache.hadoop.mapreduce.Reducer.Context context)
               throws IOException,
                      InterruptedException

The reduce function which packages the key and List<Tuple> into key, Bag<Tuple> after converting Hadoop type key into Pig type. The package result is either collected as is, if the reduce plan is empty or after passing through the reduce plan.

Overrides:

reduce in class org.apache.hadoop.mapreduce.Reducer<PigNullableWritable,NullableTuple,PigNullableWritable,org.apache.hadoop.io.Writable>

Throws:

IOException

InterruptedException

processOnePackageOutput

public boolean processOnePackageOutput(org.apache.hadoop.mapreduce.Reducer.Context oc)
                                throws IOException,
                                       InterruptedException

Throws:

IOException

InterruptedException

runPipeline

protected void runPipeline(PhysicalOperator leaf)
                    throws InterruptedException,
                           IOException

Parameters:

leaf -

Throws:

InterruptedException

IOException

cleanup

protected void cleanup(org.apache.hadoop.mapreduce.Reducer.Context context)
                throws IOException,
                       InterruptedException

Will be called once all the intermediate keys and values are processed. So right place to stop the reporter thread.

Overrides:

cleanup in class org.apache.hadoop.mapreduce.Reducer<PigNullableWritable,NullableTuple,PigNullableWritable,org.apache.hadoop.io.Writable>

Throws:

IOException

InterruptedException

getIllustratorContext

public abstract org.apache.hadoop.mapreduce.Reducer.Context getIllustratorContext(org.apache.hadoop.mapred.jobcontrol.Job job,
                                                                List<Pair<PigNullableWritable,org.apache.hadoop.io.Writable>> input,
                                                                POPackage pkg)
                                                                           throws IOException,
                                                                                  InterruptedException

Get reducer's illustrator context

Parameters:

input - Input buffer as output by maps

pkg - package

Returns:

reducer's illustrator context

Throws:

IOException

InterruptedException

inIllustrator

public abstract boolean inIllustrator(org.apache.hadoop.mapreduce.Reducer.Context context)

getPack

public abstract POPackage getPack(org.apache.hadoop.mapreduce.Reducer.Context context)