org.apache.pig.backend.hadoop.executionengine.tez.plan

Class TezCompiler

java.lang.Object

org.apache.pig.impl.plan.PlanVisitor<PhysicalOperator,PhysicalPlan>

org.apache.pig.backend.hadoop.executionengine.physicalLayer.plans.PhyPlanVisitor

org.apache.pig.backend.hadoop.executionengine.tez.plan.TezCompiler

public class TezCompiler
extends PhyPlanVisitor

The compiler that compiles a given physical plan into a DAG of Tez operators which can then be converted into the JobControl structure.

Field Summary

Fields

Modifier and Type	Field and Description
static String	FILE_CONCATENATION_THRESHOLD
static String	OPTIMISTIC_FILE_CONCATENATION
static String	USER_COMPARATOR_MARKER

Fields inherited from class org.apache.pig.impl.plan.PlanVisitor

mCurrentWalker, mPlan

Constructor Summary

Constructors

Constructor and Description
TezCompiler(PhysicalPlan plan, PigContext pigContext)

Method Summary

Modifier and Type	Method and Description
TezOperPlan	compile() The front-end method that the user calls to compile the plan.
TezPlanContainer	getPlanContainer()
static Pair<POProject,Byte>[]	getSortCols(List<PhysicalPlan> plans)
static FileSpec	getTempFileSpec(PigContext pigContext) Returns a temporary DFS Path
TezOperPlan	getTezPlan()
void	visitCollectedGroup(POCollectedGroup op)
void	visitCounter(POCounter op)
void	visitCross(POCross op)
void	visitDistinct(PODistinct op)
void	visitFilter(POFilter op)
void	visitFRJoin(POFRJoin op)
void	visitGlobalRearrange(POGlobalRearrange op)
void	visitLimit(POLimit op)
void	visitLoad(POLoad op)
void	visitLocalRearrange(POLocalRearrange op)
void	visitMergeCoGroup(POMergeCogroup poCoGrp)
void	visitMergeJoin(POMergeJoin joinOp) Since merge-join works on two inputs there are exactly two TezOper predecessors identified as left and right.
void	visitNative(PONative op)
void	visitPackage(POPackage op)
void	visitPOForEach(POForEach op)
void	visitRank(PORank op)
void	visitSkewedJoin(POSkewedJoin op)
void	visitSort(POSort op)
void	visitSplit(POSplit op)
void	visitStore(POStore op)
void	visitStream(POStream op)
void	visitUnion(POUnion op)

Methods inherited from class org.apache.pig.backend.hadoop.executionengine.physicalLayer.plans.PhyPlanVisitor

visit, visitAdd, visitAnd, visitBinCond, visitBroadcastSpark, visitCast, visitComparisonFunc, visitConstant, visitDemux, visitDivide, visitEqualTo, visitGreaterThan, visitGTOrEqual, visitIsNull, visitLessThan, visitLTOrEqual, visitMapLookUp, visitMod, visitMultiply, visitNegative, visitNot, visitNotEqualTo, visitOr, visitPartialAgg, visitPartitionRearrange, visitPoissonSample, visitPOOptimizedForEach, visitPreCombinerLocalRearrange, visitProject, visitRegexp, visitReservoirSample, visitSubtract, visitUserFunc

Methods inherited from class org.apache.pig.impl.plan.PlanVisitor

getPlan, popWalker, pushWalker, visit

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

USER_COMPARATOR_MARKER

public static final String USER_COMPARATOR_MARKER

See Also:

Constant Field Values

FILE_CONCATENATION_THRESHOLD

public static final String FILE_CONCATENATION_THRESHOLD

See Also:

Constant Field Values

OPTIMISTIC_FILE_CONCATENATION

public static final String OPTIMISTIC_FILE_CONCATENATION

See Also:

Constant Field Values

Constructor Detail

TezCompiler

public TezCompiler(PhysicalPlan plan,
                   PigContext pigContext)
            throws TezCompilerException

Throws:

TezCompilerException

Method Detail

getTezPlan

public TezOperPlan getTezPlan()

getPlanContainer

public TezPlanContainer getPlanContainer()
                                  throws PlanException

Throws:

PlanException

compile

public TezOperPlan compile()
                    throws IOException,
                           PlanException,
                           VisitorException

The front-end method that the user calls to compile the plan. Assumes that all submitted plans have a Store operators as the leaf.

Returns:

A Tez plan

Throws:

IOException

PlanException

VisitorException

visitCollectedGroup

public void visitCollectedGroup(POCollectedGroup op)
                         throws VisitorException

Overrides:

visitCollectedGroup in class PhyPlanVisitor

Throws:

VisitorException

visitCounter

public void visitCounter(POCounter op)
                  throws VisitorException

Overrides:

visitCounter in class PhyPlanVisitor

Throws:

VisitorException

visitCross

public void visitCross(POCross op)
                throws VisitorException

Overrides:

visitCross in class PhyPlanVisitor

Throws:

VisitorException

visitDistinct

public void visitDistinct(PODistinct op)
                   throws VisitorException

Overrides:

visitDistinct in class PhyPlanVisitor

Throws:

VisitorException

visitFilter

public void visitFilter(POFilter op)
                 throws VisitorException

Overrides:

visitFilter in class PhyPlanVisitor

Throws:

VisitorException

visitFRJoin

public void visitFRJoin(POFRJoin op)
                 throws VisitorException

Overrides:

visitFRJoin in class PhyPlanVisitor

Throws:

VisitorException

visitLimit

public void visitLimit(POLimit op)
                throws VisitorException

Overrides:

visitLimit in class PhyPlanVisitor

Throws:

VisitorException

visitLoad

public void visitLoad(POLoad op)
               throws VisitorException

Overrides:

visitLoad in class PhyPlanVisitor

Throws:

VisitorException

visitLocalRearrange

public void visitLocalRearrange(POLocalRearrange op)
                         throws VisitorException

Overrides:

visitLocalRearrange in class PhyPlanVisitor

Throws:

VisitorException

visitGlobalRearrange

public void visitGlobalRearrange(POGlobalRearrange op)
                          throws VisitorException

Overrides:

visitGlobalRearrange in class PhyPlanVisitor

Throws:

VisitorException

visitMergeCoGroup

public void visitMergeCoGroup(POMergeCogroup poCoGrp)
                       throws VisitorException

Overrides:

visitMergeCoGroup in class PhyPlanVisitor

Throws:

VisitorException

visitMergeJoin

public void visitMergeJoin(POMergeJoin joinOp)
                    throws VisitorException

Since merge-join works on two inputs there are exactly two TezOper predecessors identified as left and right. Right input generates index on-the-fly. This consists of two Tez vertexes. The first vertex generates index, and the second vertex sort them. Left input contains POMergeJoin which do the actual join. First right Tez oper is identified as rightTezOpr, second is identified as rightTezOpr2 Left Tez oper is identified as curTezOper. 1) RightTezOpr: It can be preceded only by POLoad. If there is anything else in physical plan, that is yanked and set as inner plans of joinOp. 2) LeftTezOper: add the Join operator in it. We also need to segment the DAG into two, because POMergeJoin depends on the index file which loads with DefaultIndexableLoader. It is possible to convert the index as a broadcast input, but that is costly because a lot of logic is built into DefaultIndexableLoader. We can revisit it later.

Overrides:

visitMergeJoin in class PhyPlanVisitor

Throws:

VisitorException

visitNative

public void visitNative(PONative op)
                 throws VisitorException

Overrides:

visitNative in class PhyPlanVisitor

Throws:

VisitorException

visitPackage

public void visitPackage(POPackage op)
                  throws VisitorException

Overrides:

visitPackage in class PhyPlanVisitor

Throws:

VisitorException

visitPOForEach

public void visitPOForEach(POForEach op)
                    throws VisitorException

Overrides:

visitPOForEach in class PhyPlanVisitor

Throws:

VisitorException

visitRank

public void visitRank(PORank op)
               throws VisitorException

Overrides:

visitRank in class PhyPlanVisitor

Throws:

VisitorException

visitSkewedJoin

public void visitSkewedJoin(POSkewedJoin op)
                     throws VisitorException

Overrides:

visitSkewedJoin in class PhyPlanVisitor

Throws:

VisitorException

getTempFileSpec

public static FileSpec getTempFileSpec(PigContext pigContext)
                                throws IOException

Returns a temporary DFS Path

Returns:

Throws:

IOException

getSortCols

public static Pair<POProject,Byte>[] getSortCols(List<PhysicalPlan> plans)
                                          throws PlanException

Throws:

PlanException

visitSort

public void visitSort(POSort op)
               throws VisitorException

Overrides:

visitSort in class PhyPlanVisitor

Throws:

VisitorException

visitSplit

public void visitSplit(POSplit op)
                throws VisitorException

Overrides:

visitSplit in class PhyPlanVisitor

Throws:

VisitorException

visitStore

public void visitStore(POStore op)
                throws VisitorException

Overrides:

visitStore in class PhyPlanVisitor

Throws:

VisitorException

visitStream

public void visitStream(POStream op)
                 throws VisitorException

Overrides:

visitStream in class PhyPlanVisitor

Throws:

VisitorException

visitUnion

public void visitUnion(POUnion op)
                throws VisitorException

Overrides:

visitUnion in class PhyPlanVisitor

Throws:

VisitorException