org.apache.pig.backend.hadoop.executionengine.spark.optimizer

Class SecondaryKeyOptimizerSpark

java.lang.Object

org.apache.pig.impl.plan.PlanVisitor<SparkOperator,SparkOperPlan>

org.apache.pig.backend.hadoop.executionengine.spark.plan.SparkOpPlanVisitor

org.apache.pig.backend.hadoop.executionengine.spark.optimizer.SecondaryKeyOptimizerSpark

All Implemented Interfaces:

SecondaryKeyOptimizer

public class SecondaryKeyOptimizerSpark
extends SparkOpPlanVisitor
implements SecondaryKeyOptimizer

Secondary key sort optimization for spark mode

Field Summary

Fields inherited from class org.apache.pig.impl.plan.PlanVisitor

mCurrentWalker, mPlan

Constructor Summary

Constructors

Constructor and Description
SecondaryKeyOptimizerSpark(SparkOperPlan plan)

Method Summary

Modifier and Type	Method and Description
int	getNumDistinctChanged()
int	getNumSortRemoved()
int	getNumUseSecondaryKey()
void	visitSparkOp(SparkOperator sparkOperator) Secondary key sort optimization is enabled in group + foreach nested situation, like TestAccumlator#testAccumWithSort After calling SecondaryKeyOptimizerUtil.applySecondaryKeySort, the POSort in the POForeach will be deleted in the spark plan.

Methods inherited from class org.apache.pig.impl.plan.PlanVisitor

getPlan, popWalker, pushWalker, visit

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SecondaryKeyOptimizerSpark

public SecondaryKeyOptimizerSpark(SparkOperPlan plan)

Method Detail

visitSparkOp

public void visitSparkOp(SparkOperator sparkOperator)
                  throws VisitorException

Secondary key sort optimization is enabled in group + foreach nested situation, like TestAccumlator#testAccumWithSort After calling SecondaryKeyOptimizerUtil.applySecondaryKeySort, the POSort in the POForeach will be deleted in the spark plan. Sort function can be implemented in secondary key sort even though POSort is deleted in the spark plan.

Overrides:

visitSparkOp in class SparkOpPlanVisitor

Parameters:

sparkOperator -

Throws:

VisitorException

getNumSortRemoved

public int getNumSortRemoved()

Specified by:

getNumSortRemoved in interface SecondaryKeyOptimizer

getNumDistinctChanged

public int getNumDistinctChanged()

Specified by:

getNumDistinctChanged in interface SecondaryKeyOptimizer

getNumUseSecondaryKey

public int getNumUseSecondaryKey()

Specified by:

getNumUseSecondaryKey in interface SecondaryKeyOptimizer