org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators

Class POPartialAgg

java.lang.Object

org.apache.pig.impl.plan.Operator<PhyPlanVisitor>

org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg

All Implemented Interfaces:

Serializable, Cloneable, Comparable<Operator>, GroupingSpillable, Spillable, Illustrable

public class POPartialAgg
extends PhysicalOperator
implements Spillable, GroupingSpillable

Do partial aggregation in map plan. Inputs are buffered up in a hashmap until a threshold is reached; then the combiner functions are fed these buffered up inputs, and results stored in a secondary map. Once that map fills up or all input has been seen, results are piped out into the next operator (caller of getNext()).

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

PhysicalOperator.OriginalLocation

Field Summary

Fields inherited from class org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

alias, illustrator, input, inputAttached, inputs, lineageTracer, mBagFactory, mTupleFactory, outputs, parentPlan, pigLogger, reporter, requestedParallelism, res, RESULT_EMPTY, RESULT_EOP, resultType

Fields inherited from class org.apache.pig.impl.plan.Operator

mKey

Constructor Summary

Constructors

Constructor and Description
POPartialAgg(OperatorKey k)
POPartialAgg(OperatorKey k, boolean isGroupAll)

Method Summary

Methods

Modifier and Type	Method and Description
PhysicalOperator	clone() Make a copy of this operator.
PhysicalPlan	getKeyPlan()
long	getMemorySize() Requests that an object return an estimate of its in memory size.
Result	getNextTuple()
List<PhysicalPlan>	getValuePlans()
Tuple	illustratorMarkup(Object in, Object out, int eqClassIndex) input tuple mark up to be illustrate-able
String	name()
void	setKeyPlan(PhysicalPlan keyPlan)
void	setValuePlans(List<PhysicalPlan> valuePlans)
long	spill() Instructs an object to spill whatever it can to disk and release references to any data structures it spills.
boolean	supportsMultipleInputs() Indicates whether this operator supports multiple inputs.
boolean	supportsMultipleOutputs() Indicates whether this operator supports multiple outputs.
void	visit(PhyPlanVisitor v) Visit this node with the provided visitor.

Methods inherited from class org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

addOriginalLocation, addOriginalLocation, attachInput, cloneHelper, clonePlans, copyAliasFrom, detachInput, getAlias, getAliasString, getIllustrator, getInputs, getLogger, getNext, getNextBigDecimal, getNextBigInteger, getNextBoolean, getNextDataBag, getNextDataByteArray, getNextDateTime, getNextDouble, getNextFloat, getNextInteger, getNextLong, getNextMap, getNextString, getOriginalLocations, getPigLogger, getReporter, getRequestedParallelism, getResultType, isAccumStarted, isAccumulative, isBlocking, isInputAttached, processInput, reset, setAccumEnd, setAccumStart, setAccumulative, setIllustrator, setInputs, setParentPlan, setPigLogger, setReporter, setRequestedParallelism, setResultType, staticDataCleanup

Methods inherited from class org.apache.pig.impl.plan.Operator

compareTo, equals, getOperatorKey, getProjectionMap, hashCode, regenerateProjectionMap, rewire, toString, unsetProjectionMap

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

POPartialAgg

public POPartialAgg(OperatorKey k)

POPartialAgg

public POPartialAgg(OperatorKey k,
            boolean isGroupAll)

Method Detail

getNextTuple

public Result getNextTuple()
                    throws ExecException

Overrides:

getNextTuple in class PhysicalOperator

Throws:

ExecException

illustratorMarkup

public Tuple illustratorMarkup(Object in,
                      Object out,
                      int eqClassIndex)

Description copied from interface: Illustrable

input tuple mark up to be illustrate-able

Specified by:

illustratorMarkup in interface Illustrable

Parameters:

in - input tuple

out - output tuple before wrapped in ExampleTuple

eqClassIndex - index into equivalence classes in illustrator

Returns:

tuple

visit

public void visit(PhyPlanVisitor v)
           throws VisitorException

Description copied from class: Operator

Visit this node with the provided visitor. This should only be called by the visitor class itself, never directly.

Specified by:

visit in class PhysicalOperator

Parameters:

v - Visitor to visit with.

Throws:

VisitorException - if the visitor has a problem.

supportsMultipleInputs

public boolean supportsMultipleInputs()

Description copied from class: Operator

Indicates whether this operator supports multiple inputs.

Specified by:

supportsMultipleInputs in class Operator<PhyPlanVisitor>

Returns:

true if it does, otherwise false.

supportsMultipleOutputs

public boolean supportsMultipleOutputs()

Description copied from class: Operator

Indicates whether this operator supports multiple outputs.

Specified by:

supportsMultipleOutputs in class Operator<PhyPlanVisitor>

Returns:

true if it does, otherwise false.

name

public String name()

Specified by:

name in class Operator<PhyPlanVisitor>

getKeyPlan

public PhysicalPlan getKeyPlan()

setKeyPlan

public void setKeyPlan(PhysicalPlan keyPlan)

getValuePlans

public List<PhysicalPlan> getValuePlans()

setValuePlans

public void setValuePlans(List<PhysicalPlan> valuePlans)

spill

public long spill()

Description copied from interface: Spillable

Instructs an object to spill whatever it can to disk and release references to any data structures it spills.

Specified by:

spill in interface Spillable

Returns:

number of objects spilled.

getMemorySize

public long getMemorySize()

Description copied from interface: Spillable

Requests that an object return an estimate of its in memory size.

Specified by:

getMemorySize in interface Spillable

Returns:

estimated in memory size.

clone

public PhysicalOperator clone()
                       throws CloneNotSupportedException

Description copied from class: PhysicalOperator

Make a copy of this operator. This function is blank, however, we should leave a place holder so that the subclasses can clone to make deep copy as this one creates a shallow copy of non-primitive types (objects, arrays and lists)

Overrides:

clone in class PhysicalOperator

Throws:

CloneNotSupportedException

See Also:

Do not use the clone method directly. Operators are cloned when logical plans are cloned using {@link LogicalPlanCloner}