org.apache.pig.backend.hadoop.executionengine.util

Class CombinerOptimizerUtil

java.lang.Object

org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil

@InterfaceAudience.Private
public class CombinerOptimizerUtil
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CombinerOptimizerUtil.DistinctPatcher A visitor to replace Project[bag][*] | |---PODistinct[bag] with POUserFunc(org.apache.pig.builtin.Distinct)[DataBag]
static class	CombinerOptimizerUtil.fixMapProjects

Method Summary

Modifier and Type	Method and Description
static void	addAlgebraicFuncToCombineFE(POForEach cfe, Map<PhysicalOperator,Integer> op2newpos) add algebraic functions with appropriate projection to new foreach in combiner
static void	addCombiner(PhysicalPlan mapPlan, PhysicalPlan reducePlan, PhysicalPlan combinePlan, CompilationMessageCollector messageCollector, boolean doMapAgg) Algebraic functions and distinct in nested plan of a foreach are partially computed in the map and combine phase.
static void	changeFunc(POForEach fe, byte type) Change the algebriac function type for algebraic functions in map and combine In map and combine the algebraic functions will be leaf of the plan
static POForEach	createForEachWithGrpProj(POForEach foreach, byte keyType) Create a new foreach with same scope,alias as given foreach add an inner plan that projects the group column, which is going to be the first input
static OperatorKey	createOperatorKey(String scope)
static PhysicalPlan	createPlanWithPredecessors(PhysicalOperator algeOp, PhysicalPlan pplan) Create new plan and add to it the clones of operator algeOp and its predecessors from the physical plan pplan .
static List<Pair<PhysicalOperator,PhysicalPlan>>	findAlgebraicOps(List<PhysicalPlan> feInners) find algebraic operators and also check if the foreach statement is suitable for combiner use
static POLocalRearrange	getNewRearrange(POLocalRearrange rearrange) create new Local rearrange by cloning existing rearrange and add plan for projecting the key
static POPreCombinerLocalRearrange	getPreCombinerLR(POLocalRearrange rearrange)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

addCombiner

public static void addCombiner(PhysicalPlan mapPlan,
                               PhysicalPlan reducePlan,
                               PhysicalPlan combinePlan,
                               CompilationMessageCollector messageCollector,
                               boolean doMapAgg)
                        throws VisitorException

Algebraic functions and distinct in nested plan of a foreach are partially computed in the map and combine phase. A new foreach statement with initial and intermediate forms of algebraic functions are added to map and combine plans respectively. If bag portion of group-by result is projected or a non algebraic expression/udf has bag as input, combiner will not be used. This is because the use of combiner in such case is likely to degrade performance as there will not be much reduction in data size in combine stage to offset the cost of the additional number of times (de)serialization is done. Major areas for enhancement: 1. use of combiner in cogroup 2. queries with order-by, limit or sort in a nested foreach after group-by 3. case where group-by is followed by filter that has algebraic expression

Throws:

VisitorException

findAlgebraicOps

public static List<Pair<PhysicalOperator,PhysicalPlan>> findAlgebraicOps(List<PhysicalPlan> feInners)
                                                                  throws VisitorException

find algebraic operators and also check if the foreach statement is suitable for combiner use

Parameters:

feInners - inner plans of foreach

Returns:

null if plan is not combinable, otherwise list of combinable operators

Throws:

VisitorException

createForEachWithGrpProj

public static POForEach createForEachWithGrpProj(POForEach foreach,
                                                 byte keyType)

Create a new foreach with same scope,alias as given foreach add an inner plan that projects the group column, which is going to be the first input

Parameters:

foreach - source foreach

keyType - type for group-by key

Returns:

new POForeach

createPlanWithPredecessors

public static PhysicalPlan createPlanWithPredecessors(PhysicalOperator algeOp,
                                                      PhysicalPlan pplan)
                                               throws CloneNotSupportedException,
                                                      PlanException

Create new plan and add to it the clones of operator algeOp and its predecessors from the physical plan pplan .

Parameters:

algeOp - algebraic operator

pplan - physical plan that has algeOp

Returns:

new plan

Throws:

CloneNotSupportedException

PlanException

addAlgebraicFuncToCombineFE

public static void addAlgebraicFuncToCombineFE(POForEach cfe,
                                               Map<PhysicalOperator,Integer> op2newpos)
                                        throws CloneNotSupportedException,
                                               PlanException

add algebraic functions with appropriate projection to new foreach in combiner

Parameters:

cfe - - the new foreach in combiner

op2newpos - - mapping of physical operator to position in input

Throws:

CloneNotSupportedException

PlanException

getPreCombinerLR

public static POPreCombinerLocalRearrange getPreCombinerLR(POLocalRearrange rearrange)

Parameters:

rearrange -

Returns:

createOperatorKey

public static OperatorKey createOperatorKey(String scope)

changeFunc

public static void changeFunc(POForEach fe,
                              byte type)
                       throws PlanException

Change the algebriac function type for algebraic functions in map and combine In map and combine the algebraic functions will be leaf of the plan

Parameters:

fe -

type -

Throws:

PlanException

getNewRearrange

public static POLocalRearrange getNewRearrange(POLocalRearrange rearrange)
                                        throws PlanException,
                                               CloneNotSupportedException

create new Local rearrange by cloning existing rearrange and add plan for projecting the key

Parameters:

rearrange -

Returns:

Throws:

PlanException

CloneNotSupportedException