public class MRPigStatsUtil extends PigStatsUtil
Modifier and Type | Field and Description |
---|---|
static String |
FS_COUNTER_GROUP |
static String |
TASK_COUNTER_GROUP |
FILE_BYTES_READ, FILE_BYTES_WRITTEN, HDFS_BYTES_READ, HDFS_BYTES_WRITTEN, MAP_INPUT_RECORDS, MAP_OUTPUT_RECORDS, MULTI_INPUTS_COUNTER_GROUP, MULTI_INPUTS_RECORD_COUNTER, MULTI_STORE_COUNTER_GROUP, MULTI_STORE_RECORD_COUNTER, REDUCE_INPUT_RECORDS, REDUCE_OUTPUT_RECORDS
Constructor and Description |
---|
MRPigStatsUtil() |
Modifier and Type | Method and Description |
---|---|
static void |
accumulateStats(org.apache.hadoop.mapred.jobcontrol.JobControl jc)
Updates the statistics after a patch of jobs is done
|
static JobStats |
addJobStats(org.apache.hadoop.mapred.jobcontrol.Job job)
Add stats for a new Job, which doesn't yet need to be completed.
|
static MRJobStats |
addNativeJobStats(PigStats ps,
NativeMapReduceOper mr,
boolean success) |
static MRJobStats |
addNativeJobStats(PigStats ps,
NativeMapReduceOper mr,
boolean success,
Exception e) |
static void |
displayStatistics()
Logs the statistics in the Pig log file at INFO level
|
static long |
getMultiStoreCount(org.apache.hadoop.mapred.jobcontrol.Job job,
org.apache.hadoop.mapred.JobClient jobClient,
String counterName)
Returns the count for the given counter name in the counter group
'MultiStoreCounters'
|
static void |
setBackendException(org.apache.hadoop.mapred.jobcontrol.Job job,
Exception e) |
static void |
startCollection(PigContext pc,
org.apache.hadoop.mapred.JobClient client,
JobControlCompiler jcc,
MROperPlan plan)
Starts collecting statistics for the given MR plan
|
static void |
stopCollection(boolean display)
Stops collecting statistics for a MR plan
|
static void |
updateJobMroMap(Map<org.apache.hadoop.mapred.jobcontrol.Job,MapReduceOper> jobMroMap)
Updates the
PigStats.JobGraph of the PigStats . |
getEmptyPigStats, getMultiInputsCounterName, getMultiStoreCounterName, getPigStats, getShortName, isTempFile, setErrorCode, setErrorMessage, setErrorThrowable, setStatsMap
public static final String TASK_COUNTER_GROUP
public static final String FS_COUNTER_GROUP
public static long getMultiStoreCount(org.apache.hadoop.mapred.jobcontrol.Job job, org.apache.hadoop.mapred.JobClient jobClient, String counterName)
job
- the MR jobjobClient
- the Hadoop job clientcounterName
- the counter namepublic static void startCollection(PigContext pc, org.apache.hadoop.mapred.JobClient client, JobControlCompiler jcc, MROperPlan plan)
pc
- the Pig contextclient
- the Hadoop job clientjcc
- the job compilerplan
- the MR planpublic static void stopCollection(boolean display)
display
- if true, log collected statistics in the Pig log
file at INFO levelpublic static JobStats addJobStats(org.apache.hadoop.mapred.jobcontrol.Job job)
job
- the job being runpublic static void displayStatistics()
public static void updateJobMroMap(Map<org.apache.hadoop.mapred.jobcontrol.Job,MapReduceOper> jobMroMap)
PigStats.JobGraph
of the PigStats
. The initial
PigStats.JobGraph
is created without job ids using MROperPlan
,
before any job is submitted for execution. The PigStats.JobGraph
then
is updated with job ids after jobs are executed.jobMroMap
- the map that maps Job
s to MapReduceOper
spublic static void accumulateStats(org.apache.hadoop.mapred.jobcontrol.JobControl jc)
jc
- the job control@InterfaceAudience.Private public static void setBackendException(org.apache.hadoop.mapred.jobcontrol.Job job, Exception e)
public static MRJobStats addNativeJobStats(PigStats ps, NativeMapReduceOper mr, boolean success)
public static MRJobStats addNativeJobStats(PigStats ps, NativeMapReduceOper mr, boolean success, Exception e)
Copyright © 2007-2017 The Apache Software Foundation