org.apache.pig.tools.pigstats
Class JobStats

java.lang.Object
  extended by org.apache.pig.newplan.Operator
      extended by org.apache.pig.tools.pigstats.JobStats
Direct Known Subclasses:
MRJobStats

@InterfaceAudience.Public
@InterfaceStability.Evolving
public abstract class JobStats
extends Operator

This class encapsulates the runtime statistics of a MapReduce job. Job statistics is collected when job is completed.


Nested Class Summary
static class JobStats.JobState
           
 
Field Summary
static String ALIAS
           
static String ALIAS_LOCATION
           
protected  org.apache.hadoop.conf.Configuration conf
           
static String FAILURE_HEADER
           
static String FEATURE
           
protected  long hdfsBytesRead
           
protected  long hdfsBytesWritten
           
protected  ArrayList<InputStats> inputs
           
protected  ArrayList<OutputStats> outputs
           
protected  JobStats.JobState state
           
static String SUCCESS_HEADER
           
 
Fields inherited from class org.apache.pig.newplan.Operator
annotations, hashPrime, location, name, plan
 
Constructor Summary
protected JobStats(String name, PigStats.JobGraph plan)
           
 
Method Summary
abstract  void accept(PlanVisitor v)
          Accept a visitor at this node in the graph.
protected  long calculateMedianValue(long[] durations)
          Calculate the median value from the given array
 String getAlias()
           
 String getAliasLocation()
           
abstract  long getAvgMapTime()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getAvgMapTime() instead.
abstract  long getAvgREduceTime()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getAvgREduceTime() instead.
 long getBytesWritten()
          Returns the total bytes written to user specified HDFS locations of this job.
abstract  String getDisplayString()
           
 String getErrorMessage()
           
 Exception getException()
           
 String getFeature()
           
abstract  org.apache.hadoop.mapred.Counters getHadoopCounters()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getHadoopCounters() instead.
 long getHdfsBytesRead()
           
 long getHdfsBytesWritten()
           
 List<InputStats> getInputs()
           
abstract  String getJobId()
           
abstract  long getMapInputRecords()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMapInputRecords() instead.
abstract  long getMapOutputRecords()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMapOutputRecords() instead.
abstract  long getMaxMapTime()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMaxMapTime() instead.
abstract  long getMaxReduceTime()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMaxReduceTime() instead.
abstract  long getMinMapTime()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMinMapTime() instead.
abstract  long getMinReduceTime()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMinReduceTime() instead.
abstract  Map<String,Long> getMultiInputCounters()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMultiInputCounters() instead.
abstract  Map<String,Long> getMultiStoreCounters()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMultiStoreCounters() instead.
abstract  int getNumberMaps()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getNumberMaps() instead.
abstract  int getNumberReduces()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getNumberReduces() instead.
 List<OutputStats> getOutputs()
           
static long getOutputSize(POStore sto, org.apache.hadoop.conf.Configuration conf)
          Looks up the output size reader from OUTPUT_SIZE_READER_KEY and invokes it to get the size of output.
abstract  long getProactiveSpillCountObjects()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getProactiveSpillCountObjects() instead.
abstract  long getProactiveSpillCountRecs()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getProactiveSpillCountRecs() instead.
 long getRecordWrittern()
          Returns the total number of records in user specified output locations of this job.
abstract  long getReduceInputRecords()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getReduceInputRecords() instead.
abstract  long getReduceOutputRecords()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getReduceOutputRecords() instead.
abstract  long getSMMSpillCount()
          Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getSMMSpillCount() instead.
 JobStats.JobState getState()
           
 boolean isEqual(Operator operator)
          This is like a shallow equals comparison.
 boolean isIndexer()
           
 boolean isSampler()
           
 boolean isSuccessful()
           
 void setBackendException(Exception e)
           
 void setConf(org.apache.hadoop.conf.Configuration conf)
           
 void setErrorMsg(String errorMsg)
           
 void setSuccessful(boolean isSuccessful)
           
 
Methods inherited from class org.apache.pig.newplan.Operator
annotate, getAnnotation, getLocation, getName, getPlan, removeAnnotation, setLocation, setPlan
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

ALIAS

public static final String ALIAS
See Also:
Constant Field Values

ALIAS_LOCATION

public static final String ALIAS_LOCATION
See Also:
Constant Field Values

FEATURE

public static final String FEATURE
See Also:
Constant Field Values

SUCCESS_HEADER

public static final String SUCCESS_HEADER

FAILURE_HEADER

public static final String FAILURE_HEADER

state

protected JobStats.JobState state

outputs

protected ArrayList<OutputStats> outputs

inputs

protected ArrayList<InputStats> inputs

conf

protected org.apache.hadoop.conf.Configuration conf

hdfsBytesRead

protected long hdfsBytesRead

hdfsBytesWritten

protected long hdfsBytesWritten
Constructor Detail

JobStats

protected JobStats(String name,
                   PigStats.JobGraph plan)
Method Detail

getJobId

public abstract String getJobId()

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

getState

public JobStats.JobState getState()

isSuccessful

public boolean isSuccessful()

setSuccessful

public void setSuccessful(boolean isSuccessful)

getErrorMessage

public String getErrorMessage()

getException

public Exception getException()

getOutputs

public List<OutputStats> getOutputs()

getInputs

public List<InputStats> getInputs()

getAlias

public String getAlias()

getAliasLocation

public String getAliasLocation()

getFeature

public String getFeature()

getHdfsBytesRead

public long getHdfsBytesRead()

getHdfsBytesWritten

public long getHdfsBytesWritten()

getBytesWritten

public long getBytesWritten()
Returns the total bytes written to user specified HDFS locations of this job.


getRecordWrittern

public long getRecordWrittern()
Returns the total number of records in user specified output locations of this job.


accept

public abstract void accept(PlanVisitor v)
                     throws FrontendException
Description copied from class: Operator
Accept a visitor at this node in the graph.

Specified by:
accept in class Operator
Parameters:
v - Visitor to accept.
Throws:
FrontendException

isEqual

public boolean isEqual(Operator operator)
Description copied from class: Operator
This is like a shallow equals comparison. It returns true if two operators have equivalent properties even if they are different objects. Here properties mean equivalent plan and equivalent name.

Specified by:
isEqual in class Operator
Returns:
true if two object have equivalent properties, else false

setErrorMsg

public void setErrorMsg(String errorMsg)

setBackendException

public void setBackendException(Exception e)

getDisplayString

public abstract String getDisplayString()

calculateMedianValue

protected long calculateMedianValue(long[] durations)
Calculate the median value from the given array

Parameters:
durations -
Returns:
median value

isSampler

public boolean isSampler()

isIndexer

public boolean isIndexer()

getNumberMaps

@Deprecated
public abstract int getNumberMaps()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getNumberMaps() instead.


getNumberReduces

@Deprecated
public abstract int getNumberReduces()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getNumberReduces() instead.


getMaxMapTime

@Deprecated
public abstract long getMaxMapTime()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMaxMapTime() instead.


getMinMapTime

@Deprecated
public abstract long getMinMapTime()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMinMapTime() instead.


getAvgMapTime

@Deprecated
public abstract long getAvgMapTime()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getAvgMapTime() instead.


getMaxReduceTime

@Deprecated
public abstract long getMaxReduceTime()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMaxReduceTime() instead.


getMinReduceTime

@Deprecated
public abstract long getMinReduceTime()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMinReduceTime() instead.


getAvgREduceTime

@Deprecated
public abstract long getAvgREduceTime()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getAvgREduceTime() instead.


getMapInputRecords

@Deprecated
public abstract long getMapInputRecords()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMapInputRecords() instead.


getMapOutputRecords

@Deprecated
public abstract long getMapOutputRecords()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMapOutputRecords() instead.


getReduceInputRecords

@Deprecated
public abstract long getReduceInputRecords()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getReduceInputRecords() instead.


getReduceOutputRecords

@Deprecated
public abstract long getReduceOutputRecords()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getReduceOutputRecords() instead.


getSMMSpillCount

@Deprecated
public abstract long getSMMSpillCount()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getSMMSpillCount() instead.


getProactiveSpillCountObjects

@Deprecated
public abstract long getProactiveSpillCountObjects()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getProactiveSpillCountObjects() instead.


getProactiveSpillCountRecs

@Deprecated
public abstract long getProactiveSpillCountRecs()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getProactiveSpillCountRecs() instead.


getHadoopCounters

@Deprecated
public abstract org.apache.hadoop.mapred.Counters getHadoopCounters()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getHadoopCounters() instead.


getMultiStoreCounters

@Deprecated
public abstract Map<String,Long> getMultiStoreCounters()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMultiStoreCounters() instead.


getMultiInputCounters

@Deprecated
public abstract Map<String,Long> getMultiInputCounters()
Deprecated. If you are using mapreduce, please cast JobStats to org.apache.pig.tools.pigstats.mapreduce.MRJobStats, then use MRJobStats.getMultiInputCounters() instead.


getOutputSize

public static long getOutputSize(POStore sto,
                                 org.apache.hadoop.conf.Configuration conf)
Looks up the output size reader from OUTPUT_SIZE_READER_KEY and invokes it to get the size of output. If OUTPUT_SIZE_READER_KEY is not set, defaults to FileBasedOutputSizeReader.

Parameters:
sto - POStore
conf - configuration


Copyright © 2007-2012 The Apache Software Foundation