org.apache.pig.tools.pigstats

Class JobStats

java.lang.Object

org.apache.pig.newplan.Operator

org.apache.pig.tools.pigstats.JobStats

Direct Known Subclasses:

MRJobStats, SparkJobStats, TezDAGStats, TezVertexStats

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public abstract class JobStats
extends Operator

This class encapsulates the runtime statistics of a MapReduce job. Job statistics is collected when job is completed.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	JobStats.JobState

Field Summary

Fields

Modifier and Type	Field and Description
static String	ALIAS
static String	ALIAS_LOCATION
protected org.apache.hadoop.conf.Configuration	conf
static String	FAILURE_HEADER
static String	FEATURE
protected long	hdfsBytesRead
protected long	hdfsBytesWritten
protected ArrayList<InputStats>	inputs
protected ArrayList<OutputStats>	outputs
protected JobStats.JobState	state
static String	SUCCESS_HEADER

Fields inherited from class org.apache.pig.newplan.Operator

annotations, hashPrime, location, name, plan

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	JobStats(String name, PigStats.JobGraph plan)

Method Summary

Modifier and Type	Method and Description
abstract void	accept(PlanVisitor v) Accept a visitor at this node in the graph.
protected long	calculateMedianValue(List<Long> durations) Calculate the median value from the given array
String	getAlias()
String	getAliasLocation()
abstract long	getAvgMapTime() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getAvgMapTime() instead.
abstract long	getAvgREduceTime() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getAvgREduceTime() instead.
long	getBytesWritten() Returns the total bytes written to user specified HDFS locations of this job.
abstract String	getDisplayString()
String	getErrorMessage()
Exception	getException()
String	getFeature()
abstract org.apache.hadoop.mapred.Counters	getHadoopCounters() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getHadoopCounters() instead.
long	getHdfsBytesRead()
long	getHdfsBytesWritten()
List<InputStats>	getInputs()
abstract String	getJobId()
abstract long	getMapInputRecords() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMapInputRecords() instead.
abstract long	getMapOutputRecords() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMapOutputRecords() instead.
abstract long	getMaxMapTime() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMaxMapTime() instead.
abstract long	getMaxReduceTime() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMaxReduceTime() instead.
abstract long	getMinMapTime() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMinMapTime() instead.
abstract long	getMinReduceTime() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMinReduceTime() instead.
abstract Map<String,Long>	getMultiInputCounters() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMultiInputCounters() instead.
abstract Map<String,Long>	getMultiStoreCounters() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMultiStoreCounters() instead.
abstract int	getNumberMaps() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getNumberMaps() instead.
abstract int	getNumberReduces() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getNumberReduces() instead.
List<OutputStats>	getOutputs()
static long	getOutputSize(POStore sto, org.apache.hadoop.conf.Configuration conf) Looks up the output size reader from OUTPUT_SIZE_READER_KEY and invokes it to get the size of output.
abstract long	getProactiveSpillCountObjects() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getProactiveSpillCountObjects() instead.
abstract long	getProactiveSpillCountRecs() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getProactiveSpillCountRecs() instead.
long	getRecordWrittern() Returns the total number of records in user specified output locations of this job.
abstract long	getReduceInputRecords() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getReduceInputRecords() instead.
abstract long	getReduceOutputRecords() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getReduceOutputRecords() instead.
abstract long	getSMMSpillCount() Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getSMMSpillCount() instead.
JobStats.JobState	getState()
boolean	isEqual(Operator operator) This is like a shallow equals comparison.
boolean	isIndexer()
boolean	isSampler()
boolean	isSuccessful()
void	setBackendException(Exception e)
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	setErrorMsg(String errorMsg)
void	setSuccessful(boolean isSuccessful)

Methods inherited from class org.apache.pig.newplan.Operator

annotate, getAnnotation, getLocation, getName, getPlan, removeAnnotation, setLocation, setPlan

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ALIAS

public static final String ALIAS

See Also:

Constant Field Values

ALIAS_LOCATION

public static final String ALIAS_LOCATION

See Also:

Constant Field Values

FEATURE

public static final String FEATURE

See Also:

Constant Field Values

SUCCESS_HEADER

public static final String SUCCESS_HEADER

FAILURE_HEADER

public static final String FAILURE_HEADER

state

protected JobStats.JobState state

outputs

protected ArrayList<OutputStats> outputs

inputs

protected ArrayList<InputStats> inputs

conf

protected org.apache.hadoop.conf.Configuration conf

hdfsBytesRead

protected long hdfsBytesRead

hdfsBytesWritten

protected long hdfsBytesWritten

Constructor Detail

JobStats

protected JobStats(String name,
                   PigStats.JobGraph plan)

Method Detail

getJobId

public abstract String getJobId()

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

getState

public JobStats.JobState getState()

isSuccessful

public boolean isSuccessful()

setSuccessful

public void setSuccessful(boolean isSuccessful)

getErrorMessage

public String getErrorMessage()

getException

public Exception getException()

getOutputs

public List<OutputStats> getOutputs()

getInputs

public List<InputStats> getInputs()

getAlias

public String getAlias()

getAliasLocation

public String getAliasLocation()

getFeature

public String getFeature()

getHdfsBytesRead

public long getHdfsBytesRead()

getHdfsBytesWritten

public long getHdfsBytesWritten()

getBytesWritten

public long getBytesWritten()

Returns the total bytes written to user specified HDFS locations of this job.

getRecordWrittern

public long getRecordWrittern()

Returns the total number of records in user specified output locations of this job.

accept

public abstract void accept(PlanVisitor v)
                     throws FrontendException

Description copied from class: Operator

Accept a visitor at this node in the graph.

Specified by:

accept in class Operator

Parameters:

v - Visitor to accept.

Throws:

FrontendException

isEqual

public boolean isEqual(Operator operator)

Description copied from class: Operator

This is like a shallow equals comparison. It returns true if two operators have equivalent properties even if they are different objects. Here properties mean equivalent plan and equivalent name.

Specified by:

isEqual in class Operator

Returns:

true if two object have equivalent properties, else false

setErrorMsg

public void setErrorMsg(String errorMsg)

setBackendException

public void setBackendException(Exception e)

getDisplayString

public abstract String getDisplayString()

calculateMedianValue

protected long calculateMedianValue(List<Long> durations)

Calculate the median value from the given array

Parameters:

durations -

Returns:

median value

isSampler

public boolean isSampler()

isIndexer

public boolean isIndexer()

getNumberMaps

@Deprecated
public abstract int getNumberMaps()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getNumberMaps() instead.

getNumberReduces

@Deprecated
public abstract int getNumberReduces()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getNumberReduces() instead.

getMaxMapTime

@Deprecated
public abstract long getMaxMapTime()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMaxMapTime() instead.

getMinMapTime

@Deprecated
public abstract long getMinMapTime()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMinMapTime() instead.

getAvgMapTime

@Deprecated
public abstract long getAvgMapTime()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getAvgMapTime() instead.

getMaxReduceTime

@Deprecated
public abstract long getMaxReduceTime()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMaxReduceTime() instead.

getMinReduceTime

@Deprecated
public abstract long getMinReduceTime()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMinReduceTime() instead.

getAvgREduceTime

@Deprecated
public abstract long getAvgREduceTime()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getAvgREduceTime() instead.

getMapInputRecords

@Deprecated
public abstract long getMapInputRecords()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMapInputRecords() instead.

getMapOutputRecords

@Deprecated
public abstract long getMapOutputRecords()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMapOutputRecords() instead.

getReduceInputRecords

@Deprecated
public abstract long getReduceInputRecords()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getReduceInputRecords() instead.

getReduceOutputRecords

@Deprecated
public abstract long getReduceOutputRecords()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getReduceOutputRecords() instead.

getSMMSpillCount

@Deprecated
public abstract long getSMMSpillCount()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getSMMSpillCount() instead.

getProactiveSpillCountObjects

@Deprecated
public abstract long getProactiveSpillCountObjects()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getProactiveSpillCountObjects() instead.

getProactiveSpillCountRecs

@Deprecated
public abstract long getProactiveSpillCountRecs()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getProactiveSpillCountRecs() instead.

getHadoopCounters

@Deprecated
public abstract org.apache.hadoop.mapred.Counters getHadoopCounters()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getHadoopCounters() instead.

getMultiStoreCounters

@Deprecated
public abstract Map<String,Long> getMultiStoreCounters()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMultiStoreCounters() instead.

getMultiInputCounters

@Deprecated
public abstract Map<String,Long> getMultiInputCounters()

Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMultiInputCounters() instead.

getOutputSize

public static long getOutputSize(POStore sto,
                                 org.apache.hadoop.conf.Configuration conf)

Looks up the output size reader from OUTPUT_SIZE_READER_KEY and invokes it to get the size of output. If OUTPUT_SIZE_READER_KEY is not set, defaults to FileBasedOutputSizeReader.

Parameters:

sto - POStore

conf - configuration