org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer

Class UnionOptimizer

java.lang.Object

org.apache.pig.impl.plan.PlanVisitor<TezOperator,TezOperPlan>

org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOpPlanVisitor

org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.UnionOptimizer

public class UnionOptimizer
extends TezOpPlanVisitor

Optimizes union by removing the intermediate union vertex and making the successor get input from the predecessor vertices directly using VertexGroup. This should be run after MultiQueryOptimizer so that it handles cases like union followed by split and then store. For eg: 1) Union followed by store Vertex 1 (Load), Vertex 2 (Load) -> Vertex 3 (Union + Store) will be optimized to Vertex 1 (Load + Store), Vertex 2 (Load + Store). Both the vertices will be writing output to same store location directly which is supported by Tez. 2) Union followed by groupby Vertex 1 (Load), Vertex 2 (Load) -> Vertex 3 (Union + POLocalRearrange) -> Vertex 4 (Group by) will be optimized to Vertex 1 (Load + POLR), Vertex 2 (Load + POLR) -> Vertex 4 (Group by)

Field Summary

Fields inherited from class org.apache.pig.impl.plan.PlanVisitor

mCurrentWalker, mPlan

Constructor Summary

Constructors

Constructor and Description
UnionOptimizer(TezOperPlan plan, List<String> supportedStoreFuncs, List<String> unsupportedStoreFuncs)

Method Summary

Methods

Modifier and Type	Method and Description
void	connectPredecessorsToVertexGroups(TezOperator unionOp, TezOperator pred, PhysicalPlan predClonedUnionPlan, TezOperator[] storeVertexGroupOps, TezOperator[] outputVertexGroupOps) Connects the unionOp predecessor to the store vertex groups and the output vertex groups and disconnects it from the unionOp.
static PhysicalPlan	getUnionPredPlanFromSplit(PhysicalPlan plan, String unionOpKey)
static boolean	isOptimizable(TezOperator tezOp)
static boolean	isOptimizableStoreFunc(TezOperator tezOp, List<String> supportedStoreFuncs, List<String> unsupportedStoreFuncs)
void	visitTezOp(TezOperator tezOp)

Methods inherited from class org.apache.pig.impl.plan.PlanVisitor

getPlan, popWalker, pushWalker, visit

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

UnionOptimizer

public UnionOptimizer(TezOperPlan plan,
              List<String> supportedStoreFuncs,
              List<String> unsupportedStoreFuncs)

Method Detail

isOptimizable

public static boolean isOptimizable(TezOperator tezOp)
                             throws VisitorException

Throws:

VisitorException

isOptimizableStoreFunc

public static boolean isOptimizableStoreFunc(TezOperator tezOp,
                             List<String> supportedStoreFuncs,
                             List<String> unsupportedStoreFuncs)
                                      throws VisitorException

Throws:

VisitorException

visitTezOp

public void visitTezOp(TezOperator tezOp)
                throws VisitorException

Overrides:

visitTezOp in class TezOpPlanVisitor

Throws:

VisitorException

connectPredecessorsToVertexGroups

public void connectPredecessorsToVertexGroups(TezOperator unionOp,
                                     TezOperator pred,
                                     PhysicalPlan predClonedUnionPlan,
                                     TezOperator[] storeVertexGroupOps,
                                     TezOperator[] outputVertexGroupOps)
                                       throws VisitorException,
                                              PlanException

Connects the unionOp predecessor to the store vertex groups and the output vertex groups and disconnects it from the unionOp.

Parameters:

pred - Predecessor of union which will be made part of the vertex group

unionOp - Union operator

predClonedUnionPlan - Cloned plan of the union merged to the predecessor

storeVertexGroupOps - Store vertex groups to connect to

outputVertexGroupOps - Tez LogicalOutput vertex groups to connect to

Throws:

VisitorException

PlanException

getUnionPredPlanFromSplit

public static PhysicalPlan getUnionPredPlanFromSplit(PhysicalPlan plan,
                                     String unionOpKey)
                                              throws VisitorException

Throws:

VisitorException