org.apache.pig.backend.hadoop.executionengine.mapReduceLayer

Class MapReduceOper

java.lang.Object

org.apache.pig.impl.plan.Operator<MROpPlanVisitor>

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceOper

All Implemented Interfaces:

Serializable, Cloneable, Comparable<Operator>

Direct Known Subclasses:

NativeMapReduceOper

public class MapReduceOper
extends Operator<MROpPlanVisitor>

An operator model for a Map Reduce job. Acts as a host to the plans that will execute in map, reduce and optionally combine phases. These will be embedded in the MROperPlan in order to capture the dependencies amongst jobs.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
PhysicalPlan	combinePlan
byte	mapKeyType
byte[]	mapKeyTypeOfSplittees
PhysicalPlan	mapPlan
MultiMap<PhysicalOperator,PhysicalOperator>	phyToMRMap
PhysicalPlan	reducePlan
Set<PhysicalOperator>	scalars
Set<String>	UDFs

Fields inherited from class org.apache.pig.impl.plan.Operator

mKey

Constructor Summary

Constructors

Constructor and Description
MapReduceOper(OperatorKey k)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	combineSmallSplits()
String	getCustomPartitioner()
String	getOperationID()
String	getQuantFile()
ArrayList<String>	getRankOperationId()
int	getRequestedParallelism()
boolean[]	getSecondarySortOrder()
boolean	getSkewedJoin()
String	getSkewedJoinPartitionFile()
boolean[]	getSortOrder()
boolean	getUseSecondaryKey()
boolean	isCogroup()
boolean	isCounterOperation()
boolean	isEndOfAllInputSetInMap()
boolean	isEndOfAllInputSetInReduce()
boolean	isGlobalSort()
boolean	isGroupBy()
boolean	isIndexer()
boolean	isLimitAfterSort()
boolean	isLimitOnly()
boolean	isMapDone()
boolean	isRankOperation()
boolean	isReduceDone()
boolean	isRegularJoin()
boolean	isRowNumber()
boolean	isSampler()
boolean	isSkewedJoin()
boolean	isSplitter()
void	markCogroup()
void	markGroupBy()
void	markIndexer()
void	markRegularJoin()
void	markSampler()
String	name() Uses the string representation of the component plans to identify itself.
boolean	needsDistinctCombiner()
protected void	noCombineSmallSplits()
void	setEndOfAllInputInMap(boolean endOfAllInputInMap)
void	setEndOfAllInputInReduce(boolean endOfAllInputInReduce)
void	setGlobalSort(boolean globalSort)
void	setLimitAfterSort(boolean las)
void	setLimitOnly(boolean limitOnly)
void	setMapDone(boolean mapDone)
void	setMapDoneMultiple(boolean mapDone)
void	setMapDoneSingle(boolean mapDone)
void	setNeedsDistinctCombiner(boolean nic)
void	setQuantFile(String quantFile)
void	setReduceDone(boolean reduceDone)
void	setSecondarySortOrder(boolean[] secondarySortOrder)
void	setSkewedJoin(boolean skJoin)
void	setSkewedJoinPartitionFile(String file)
void	setSortOrder(boolean[] sortOrder)
void	setSplitter(boolean spl)
void	setUseSecondaryKey(boolean useSecondaryKey)
boolean	supportsMultipleInputs() Indicates whether this operator supports multiple inputs.
boolean	supportsMultipleOutputs() Indicates whether this operator supports multiple outputs.
protected void	useTypedComparator(boolean useTypedComparator)
protected boolean	usingTypedComparator()
void	visit(MROpPlanVisitor v) Visit this node with the provided visitor.

Methods inherited from class org.apache.pig.impl.plan.Operator

clone, compareTo, equals, getOperatorKey, getProjectionMap, hashCode, regenerateProjectionMap, rewire, toString, unsetProjectionMap

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

mapPlan

public PhysicalPlan mapPlan

reducePlan

public PhysicalPlan reducePlan

combinePlan

public PhysicalPlan combinePlan

mapKeyType

public byte mapKeyType

mapKeyTypeOfSplittees

public byte[] mapKeyTypeOfSplittees

UDFs

public Set<String> UDFs

scalars

public Set<PhysicalOperator> scalars

phyToMRMap

public MultiMap<PhysicalOperator,PhysicalOperator> phyToMRMap

Constructor Detail

MapReduceOper

public MapReduceOper(OperatorKey k)

Method Detail

name

public String name()

Uses the string representation of the component plans to identify itself.

Specified by:

name in class Operator<MROpPlanVisitor>

supportsMultipleInputs

public boolean supportsMultipleInputs()

Description copied from class: Operator

Indicates whether this operator supports multiple inputs.

Specified by:

supportsMultipleInputs in class Operator<MROpPlanVisitor>

Returns:

true if it does, otherwise false.

supportsMultipleOutputs

public boolean supportsMultipleOutputs()

Description copied from class: Operator

Indicates whether this operator supports multiple outputs.

Specified by:

supportsMultipleOutputs in class Operator<MROpPlanVisitor>

Returns:

true if it does, otherwise false.

visit

public void visit(MROpPlanVisitor v)
           throws VisitorException

Description copied from class: Operator

Visit this node with the provided visitor. This should only be called by the visitor class itself, never directly.

Specified by:

visit in class Operator<MROpPlanVisitor>

Parameters:

v - Visitor to visit with.

Throws:

VisitorException - if the visitor has a problem.

isMapDone

public boolean isMapDone()

setMapDone

public void setMapDone(boolean mapDone)

setMapDoneSingle

public void setMapDoneSingle(boolean mapDone)
                      throws PlanException

Throws:

PlanException

setMapDoneMultiple

public void setMapDoneMultiple(boolean mapDone)
                        throws PlanException

Throws:

PlanException

isReduceDone

public boolean isReduceDone()

setReduceDone

public void setReduceDone(boolean reduceDone)

isGlobalSort

public boolean isGlobalSort()

isSkewedJoin

public boolean isSkewedJoin()

setSkewedJoinPartitionFile

public void setSkewedJoinPartitionFile(String file)

getSkewedJoinPartitionFile

public String getSkewedJoinPartitionFile()

setSkewedJoin

public void setSkewedJoin(boolean skJoin)

getSkewedJoin

public boolean getSkewedJoin()

setGlobalSort

public void setGlobalSort(boolean globalSort)

isLimitAfterSort

public boolean isLimitAfterSort()

setLimitAfterSort

public void setLimitAfterSort(boolean las)

isLimitOnly

public boolean isLimitOnly()

setLimitOnly

public void setLimitOnly(boolean limitOnly)

isIndexer

public boolean isIndexer()

markIndexer

public void markIndexer()

isSampler

public boolean isSampler()

markSampler

public void markSampler()

isGroupBy

public boolean isGroupBy()

markGroupBy

public void markGroupBy()

isCogroup

public boolean isCogroup()

markCogroup

public void markCogroup()

isRegularJoin

public boolean isRegularJoin()

markRegularJoin

public void markRegularJoin()

needsDistinctCombiner

public boolean needsDistinctCombiner()

setNeedsDistinctCombiner

public void setNeedsDistinctCombiner(boolean nic)

getQuantFile

public String getQuantFile()

setQuantFile

public void setQuantFile(String quantFile)

setSortOrder

public void setSortOrder(boolean[] sortOrder)

setSecondarySortOrder

public void setSecondarySortOrder(boolean[] secondarySortOrder)

getSortOrder

public boolean[] getSortOrder()

getSecondarySortOrder

public boolean[] getSecondarySortOrder()

isEndOfAllInputSetInMap

public boolean isEndOfAllInputSetInMap()

Returns:

whether end of all input is set in the map plan

setEndOfAllInputInMap

public void setEndOfAllInputInMap(boolean endOfAllInputInMap)

Parameters:

endOfAllInputInMap - the streamInMap to set

isEndOfAllInputSetInReduce

public boolean isEndOfAllInputSetInReduce()

Returns:

whether end of all input is set in the reduce plan

setEndOfAllInputInReduce

public void setEndOfAllInputInReduce(boolean endOfAllInputInReduce)

Parameters:

endOfAllInputInReduce - the streamInReduce to set

getRequestedParallelism

public int getRequestedParallelism()

getCustomPartitioner

public String getCustomPartitioner()

setSplitter

public void setSplitter(boolean spl)

isSplitter

public boolean isSplitter()

getUseSecondaryKey

public boolean getUseSecondaryKey()

setUseSecondaryKey

public void setUseSecondaryKey(boolean useSecondaryKey)

usingTypedComparator

protected boolean usingTypedComparator()

useTypedComparator

protected void useTypedComparator(boolean useTypedComparator)

noCombineSmallSplits

protected void noCombineSmallSplits()

combineSmallSplits

public boolean combineSmallSplits()

isRankOperation

public boolean isRankOperation()

getRankOperationId

public ArrayList<String> getRankOperationId()

isCounterOperation

public boolean isCounterOperation()

isRowNumber

public boolean isRowNumber()

getOperationID

public String getOperationID()