org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators

Class POFRJoin

java.lang.Object

org.apache.pig.impl.plan.Operator<PhyPlanVisitor>

org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, java.lang.Comparable<Operator>, Illustrable

Direct Known Subclasses:

POFRJoinSpark, POFRJoinTez

public class POFRJoin
extends PhysicalOperator

The operator models the join keys using the Local Rearrange operators which are configured with the plan specified by the user. It also sets up one Hashtable per replicated input which maps the Key(k) stored as a Tuple to a DataBag which holds all the values in the input having the same key(k) The getNext() reads an input from its predecessor and separates them into key & value. It configures a foreach operator with the databags obtained from each Hashtable for the key and also with the value for the fragment input. It then returns tuples returned by this foreach operator.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	POFRJoin.TupleToMapKey

Nested classes/interfaces inherited from class org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

PhysicalOperator.OriginalLocation

Field Summary

Fields

Modifier and Type	Field and Description
protected ConstantExpression[]	constExps
protected Tuple	dumTup
protected POForEach	fe
protected int	fragment
protected Schema[]	inputSchemas
protected boolean	isLeftOuterJoin
protected Schema[]	keySchemas
protected java.util.List<java.util.List<java.lang.Byte>>	keyTypes
protected POLocalRearrange[]	LRs
protected DataBag	nullBag
protected java.util.List<java.util.List<PhysicalPlan>>	phyPlanLists
protected boolean	processingPlan
protected FileSpec[]	replFiles
protected java.util.List<java.util.Map<? extends java.lang.Object,? extends java.util.List<Tuple>>>	replicates
protected boolean	setUp

Fields inherited from class org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

alias, illustrator, input, inputAttached, inputs, lineageTracer, mBagFactory, mTupleFactory, outputs, parentPlan, pigLogger, reporter, requestedParallelism, res, RESULT_EMPTY, RESULT_EOP, resultType

Fields inherited from class org.apache.pig.impl.plan.Operator

mKey

Constructor Summary

Constructors

Constructor and Description
POFRJoin(OperatorKey k, int rp, java.util.List<PhysicalOperator> inp, java.util.List<java.util.List<PhysicalPlan>> ppLists, java.util.List<java.util.List<java.lang.Byte>> keyTypes, FileSpec[] replFiles, int fragment, boolean isLeftOuter, Tuple nullTuple)
POFRJoin(OperatorKey k, int rp, java.util.List<PhysicalOperator> inp, java.util.List<java.util.List<PhysicalPlan>> ppLists, java.util.List<java.util.List<java.lang.Byte>> keyTypes, FileSpec[] replFiles, int fragment, boolean isLeftOuter, Tuple nullTuple, Schema[] inputSchemas, Schema[] keySchemas)
POFRJoin(POFRJoin copy)

Method Summary

Modifier and Type	Method and Description
POFRJoin	clone() Make a copy of this operator.
int	getFragment()
java.util.List<java.util.List<PhysicalPlan>>	getJoinPlans()
POLocalRearrange[]	getLRs()
Result	getNextTuple()
FileSpec[]	getReplFiles()
protected Tuple	getValueTuple(POLocalRearrange lr, Tuple tuple)
Tuple	illustratorMarkup(java.lang.Object in, java.lang.Object out, int eqClassIndex) input tuple mark up to be illustrate-able
protected boolean	isKeyNull(java.lang.Object key)
boolean	isLeftOuterJoin()
java.lang.String	name()
void	setFragment(int fragment)
void	setReplFiles(FileSpec[] replFiles)
protected void	setUpHashMap() Builds the HashMaps by reading each replicated input from the DFS using a Load operator
boolean	supportsMultipleInputs() Indicates whether this operator supports multiple inputs.
boolean	supportsMultipleOutputs() Indicates whether this operator supports multiple outputs.
void	visit(PhyPlanVisitor v) Visit this node with the provided visitor.

Methods inherited from class org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator

addOriginalLocation, addOriginalLocation, attachInput, cloneHelper, clonePlans, copyAliasFrom, detachInput, getAlias, getAliasString, getIllustrator, getInputs, getLogger, getNext, getNextBigDecimal, getNextBigInteger, getNextBoolean, getNextDataBag, getNextDataByteArray, getNextDateTime, getNextDouble, getNextFloat, getNextInteger, getNextLong, getNextMap, getNextString, getOriginalLocations, getParentPlan, getPigLogger, getReporter, getRequestedParallelism, getResultType, isAccumStarted, isAccumulative, isBlocking, isEndOfAllInput, isInputAttached, processInput, reset, setAccumEnd, setAccumStart, setAccumulative, setIllustrator, setInputs, setParentPlan, setPigLogger, setReporter, setRequestedParallelism, setResultType, staticDataCleanup

Methods inherited from class org.apache.pig.impl.plan.Operator

compareTo, equals, getOperatorKey, getProjectionMap, hashCode, regenerateProjectionMap, rewire, toString, unsetProjectionMap

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

fragment

protected int fragment

phyPlanLists

protected java.util.List<java.util.List<PhysicalPlan>> phyPlanLists

keyTypes

protected java.util.List<java.util.List<java.lang.Byte>> keyTypes

LRs

protected POLocalRearrange[] LRs

replFiles

protected FileSpec[] replFiles

constExps

protected ConstantExpression[] constExps

fe

protected POForEach fe

isLeftOuterJoin

protected boolean isLeftOuterJoin

nullBag

protected DataBag nullBag

inputSchemas

protected Schema[] inputSchemas

keySchemas

protected Schema[] keySchemas

replicates

protected transient java.util.List<java.util.Map<? extends java.lang.Object,? extends java.util.List<Tuple>>> replicates

processingPlan

protected transient boolean processingPlan

dumTup

protected transient Tuple dumTup

setUp

protected transient boolean setUp

Constructor Detail

POFRJoin

public POFRJoin(OperatorKey k,
                int rp,
                java.util.List<PhysicalOperator> inp,
                java.util.List<java.util.List<PhysicalPlan>> ppLists,
                java.util.List<java.util.List<java.lang.Byte>> keyTypes,
                FileSpec[] replFiles,
                int fragment,
                boolean isLeftOuter,
                Tuple nullTuple)
         throws ExecException

Throws:

ExecException

POFRJoin

public POFRJoin(OperatorKey k,
                int rp,
                java.util.List<PhysicalOperator> inp,
                java.util.List<java.util.List<PhysicalPlan>> ppLists,
                java.util.List<java.util.List<java.lang.Byte>> keyTypes,
                FileSpec[] replFiles,
                int fragment,
                boolean isLeftOuter,
                Tuple nullTuple,
                Schema[] inputSchemas,
                Schema[] keySchemas)
         throws ExecException

Throws:

ExecException

POFRJoin

public POFRJoin(POFRJoin copy)
         throws ExecException

Throws:

ExecException

Method Detail

visit

public void visit(PhyPlanVisitor v)
           throws VisitorException

Description copied from class: Operator

Visit this node with the provided visitor. This should only be called by the visitor class itself, never directly.

Specified by:

visit in class PhysicalOperator

Parameters:

v - Visitor to visit with.

Throws:

VisitorException - if the visitor has a problem.

name

public java.lang.String name()

Specified by:

name in class Operator<PhyPlanVisitor>

supportsMultipleInputs

public boolean supportsMultipleInputs()

Description copied from class: Operator

Indicates whether this operator supports multiple inputs.

Specified by:

supportsMultipleInputs in class Operator<PhyPlanVisitor>

Returns:

true if it does, otherwise false.

supportsMultipleOutputs

public boolean supportsMultipleOutputs()

Description copied from class: Operator

Indicates whether this operator supports multiple outputs.

Specified by:

supportsMultipleOutputs in class Operator<PhyPlanVisitor>

Returns:

true if it does, otherwise false.

getNextTuple

public Result getNextTuple()
                    throws ExecException

Overrides:

getNextTuple in class PhysicalOperator

Throws:

ExecException

setUpHashMap

protected void setUpHashMap()
                     throws ExecException

Builds the HashMaps by reading each replicated input from the DFS using a Load operator

Throws:

ExecException

isKeyNull

protected boolean isKeyNull(java.lang.Object key)
                     throws ExecException

Throws:

ExecException

getValueTuple

protected Tuple getValueTuple(POLocalRearrange lr,
                              Tuple tuple)
                       throws ExecException

Throws:

ExecException

getJoinPlans

public java.util.List<java.util.List<PhysicalPlan>> getJoinPlans()

getLRs

public POLocalRearrange[] getLRs()

isLeftOuterJoin

public boolean isLeftOuterJoin()

getFragment

public int getFragment()

setFragment

public void setFragment(int fragment)

getReplFiles

public FileSpec[] getReplFiles()

setReplFiles

public void setReplFiles(FileSpec[] replFiles)

illustratorMarkup

public Tuple illustratorMarkup(java.lang.Object in,
                               java.lang.Object out,
                               int eqClassIndex)

Description copied from interface: Illustrable

input tuple mark up to be illustrate-able

Parameters:

in - input tuple

out - output tuple before wrapped in ExampleTuple

eqClassIndex - index into equivalence classes in illustrator

Returns:

tuple

clone

public POFRJoin clone()
               throws java.lang.CloneNotSupportedException

Description copied from class: PhysicalOperator

Make a copy of this operator. This function is blank, however, we should leave a place holder so that the subclasses can clone to make deep copy as this one creates a shallow copy of non-primitive types (objects, arrays and lists)

Overrides:

clone in class PhysicalOperator

Throws:

java.lang.CloneNotSupportedException

See Also:

Do not use the clone method directly. Operators are cloned when logical plans are cloned using {@link LogicalPlanCloner}