org.apache.pig.backend.hadoop.executionengine.mapReduceLayer
Class PigGenericMapReduce.Reduce

java.lang.Object
  extended by org.apache.hadoop.mapreduce.Reducer<PigNullableWritable,NullableTuple,PigNullableWritable,org.apache.hadoop.io.Writable>
      extended by org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce.Reduce
Enclosing class:
PigGenericMapReduce

public abstract static class PigGenericMapReduce.Reduce
extends org.apache.hadoop.mapreduce.Reducer<PigNullableWritable,NullableTuple,PigNullableWritable,org.apache.hadoop.io.Writable>


Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Reducer
org.apache.hadoop.mapreduce.Reducer.Context
 
Field Summary
protected  boolean errorInReduce
           
protected  boolean initialized
           
protected  org.apache.commons.logging.Log log
           
protected  org.apache.hadoop.mapreduce.Reducer.Context outputCollector
           
protected  POPackage pack
           
protected  PhysicalPlan rp
           
protected  List<POStore> stores
           
 
Constructor Summary
PigGenericMapReduce.Reduce()
           
 
Method Summary
protected  void cleanup(org.apache.hadoop.mapreduce.Reducer.Context context)
          Will be called once all the intermediate keys and values are processed.
abstract  org.apache.hadoop.mapreduce.Reducer.Context getIllustratorContext(org.apache.hadoop.mapred.jobcontrol.Job job, List<Pair<PigNullableWritable,org.apache.hadoop.io.Writable>> input, POPackage pkg)
          Get reducer's illustrator context
abstract  POPackage getPack(org.apache.hadoop.mapreduce.Reducer.Context context)
           
abstract  boolean inIllustrator(org.apache.hadoop.mapreduce.Reducer.Context context)
           
 boolean processOnePackageOutput(org.apache.hadoop.mapreduce.Reducer.Context oc)
           
protected  void reduce(PigNullableWritable key, Iterable<NullableTuple> tupIter, org.apache.hadoop.mapreduce.Reducer.Context context)
          The reduce function which packages the key and List<Tuple> into key, Bag<Tuple> after converting Hadoop type key into Pig type.
protected  void runPipeline(PhysicalOperator leaf)
           
 void setReducePlan(PhysicalPlan plan)
          Set the reduce plan: to be used by local runner for illustrator
protected  void setup(org.apache.hadoop.mapreduce.Reducer.Context context)
          Configures the Reduce plan, the POPackage operator and the reporter thread
 
Methods inherited from class org.apache.hadoop.mapreduce.Reducer
run
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

log

protected final org.apache.commons.logging.Log log

rp

protected PhysicalPlan rp

stores

protected List<POStore> stores

pack

protected POPackage pack

outputCollector

protected org.apache.hadoop.mapreduce.Reducer.Context outputCollector

errorInReduce

protected boolean errorInReduce

initialized

protected volatile boolean initialized
Constructor Detail

PigGenericMapReduce.Reduce

public PigGenericMapReduce.Reduce()
Method Detail

setReducePlan

public void setReducePlan(PhysicalPlan plan)
Set the reduce plan: to be used by local runner for illustrator

Parameters:
plan - Reduce plan

setup

protected void setup(org.apache.hadoop.mapreduce.Reducer.Context context)
              throws IOException,
                     InterruptedException
Configures the Reduce plan, the POPackage operator and the reporter thread

Overrides:
setup in class org.apache.hadoop.mapreduce.Reducer<PigNullableWritable,NullableTuple,PigNullableWritable,org.apache.hadoop.io.Writable>
Throws:
IOException
InterruptedException

reduce

protected void reduce(PigNullableWritable key,
                      Iterable<NullableTuple> tupIter,
                      org.apache.hadoop.mapreduce.Reducer.Context context)
               throws IOException,
                      InterruptedException
The reduce function which packages the key and List<Tuple> into key, Bag<Tuple> after converting Hadoop type key into Pig type. The package result is either collected as is, if the reduce plan is empty or after passing through the reduce plan.

Overrides:
reduce in class org.apache.hadoop.mapreduce.Reducer<PigNullableWritable,NullableTuple,PigNullableWritable,org.apache.hadoop.io.Writable>
Throws:
IOException
InterruptedException

processOnePackageOutput

public boolean processOnePackageOutput(org.apache.hadoop.mapreduce.Reducer.Context oc)
                                throws IOException,
                                       InterruptedException
Throws:
IOException
InterruptedException

runPipeline

protected void runPipeline(PhysicalOperator leaf)
                    throws InterruptedException,
                           IOException
Parameters:
leaf -
Throws:
InterruptedException
IOException

cleanup

protected void cleanup(org.apache.hadoop.mapreduce.Reducer.Context context)
                throws IOException,
                       InterruptedException
Will be called once all the intermediate keys and values are processed. So right place to stop the reporter thread.

Overrides:
cleanup in class org.apache.hadoop.mapreduce.Reducer<PigNullableWritable,NullableTuple,PigNullableWritable,org.apache.hadoop.io.Writable>
Throws:
IOException
InterruptedException

getIllustratorContext

public abstract org.apache.hadoop.mapreduce.Reducer.Context getIllustratorContext(org.apache.hadoop.mapred.jobcontrol.Job job,
                                                                                  List<Pair<PigNullableWritable,org.apache.hadoop.io.Writable>> input,
                                                                                  POPackage pkg)
                                                                           throws IOException,
                                                                                  InterruptedException
Get reducer's illustrator context

Parameters:
input - Input buffer as output by maps
pkg - package
Returns:
reducer's illustrator context
Throws:
IOException
InterruptedException

inIllustrator

public abstract boolean inIllustrator(org.apache.hadoop.mapreduce.Reducer.Context context)

getPack

public abstract POPackage getPack(org.apache.hadoop.mapreduce.Reducer.Context context)


Copyright © ${year} The Apache Software Foundation