org.apache.pig.tools.pigstats.spark

Class SparkPigStats

java.lang.Object

org.apache.pig.tools.pigstats.PigStats

org.apache.pig.tools.pigstats.spark.SparkPigStats

public class SparkPigStats
extends PigStats

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.pig.tools.pigstats.PigStats

PigStats.JobGraph, PigStats.JobGraphPrinter

Field Summary

Fields inherited from class org.apache.pig.tools.pigstats.PigStats

aliasOuputMap, DATE_FORMAT, endTime, errorCode, errorMessage, errorThrowable, jobPlan, pigContext, returnCode, startTime, userId

Constructor Summary

Constructors

Constructor and Description
SparkPigStats()

Method Summary

Modifier and Type	Method and Description
void	addFailJobStats(POStore poStore, SparkOperator sparkOperator, String jobId, JobMetricsListener jobMetricsListener, JavaSparkContext sparkContext, Exception e)
void	addJobStats(POStore poStore, SparkOperator sparkOperator, int jobId, JobMetricsListener jobMetricsListener, JavaSparkContext sparkContext)
void	addNativeJobStats(NativeSparkOperator sparkOperator, String jobId, boolean isSuccess, Exception e)
void	finish()
List<String>	getAllErrorMessages()
Map<String,List<PigStats>>	getAllStats()
String	getDisplayString() Returns the display message in pig grunt
org.apache.hadoop.mapred.JobClient	getJobClient()
int	getNumberJobs() Returns the number of jobs for this script
long	getProactiveSpillCountObjects() Returns the total number of bags that spilled proactively
long	getProactiveSpillCountRecords() Returns the total number of records that spilled proactively
long	getSMMSpillCount() Returns the total spill counts from SpillableMemoryManager.
void	initialize(PigContext pigContext, SparkOperPlan sparkPlan, org.apache.hadoop.conf.Configuration conf)
boolean	isEmbedded()

Methods inherited from class org.apache.pig.tools.pigstats.PigStats

get, getBytesWritten, getDuration, getErrorCode, getErrorMessage, getErrorThrowable, getFeatures, getFileName, getHadoopVersion, getInputStats, getJobGraph, getNumberBytes, getNumberFailedJobs, getNumberRecords, getNumberSuccessfulJobs, getOutputAlias, getOutputLocations, getOutputNames, getOutputStats, getPigContext, getPigProperties, getPigVersion, getRecordWritten, getReturnCode, getScript, getScriptId, isSuccessful, result, setBackendException, setErrorCode, setErrorMessage, setErrorThrowable, setReturnCode, start, start, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkPigStats

public SparkPigStats()

Method Detail

initialize

public void initialize(PigContext pigContext,
                       SparkOperPlan sparkPlan,
                       org.apache.hadoop.conf.Configuration conf)

addJobStats

public void addJobStats(POStore poStore,
                        SparkOperator sparkOperator,
                        int jobId,
                        JobMetricsListener jobMetricsListener,
                        JavaSparkContext sparkContext)

addFailJobStats

public void addFailJobStats(POStore poStore,
                            SparkOperator sparkOperator,
                            String jobId,
                            JobMetricsListener jobMetricsListener,
                            JavaSparkContext sparkContext,
                            Exception e)

addNativeJobStats

public void addNativeJobStats(NativeSparkOperator sparkOperator,
                              String jobId,
                              boolean isSuccess,
                              Exception e)

finish

public void finish()

getDisplayString

public String getDisplayString()

Description copied from class: PigStats

Returns the display message in pig grunt

Specified by:

getDisplayString in class PigStats

getJobClient

public org.apache.hadoop.mapred.JobClient getJobClient()

Specified by:

getJobClient in class PigStats

isEmbedded

public boolean isEmbedded()

Specified by:

isEmbedded in class PigStats

getAllStats

public Map<String,List<PigStats>> getAllStats()

Specified by:

getAllStats in class PigStats

getAllErrorMessages

public List<String> getAllErrorMessages()

Specified by:

getAllErrorMessages in class PigStats

getSMMSpillCount

public long getSMMSpillCount()

Description copied from class: PigStats

Returns the total spill counts from SpillableMemoryManager.

Specified by:

getSMMSpillCount in class PigStats

getProactiveSpillCountObjects

public long getProactiveSpillCountObjects()

Description copied from class: PigStats

Returns the total number of bags that spilled proactively

Specified by:

getProactiveSpillCountObjects in class PigStats

getProactiveSpillCountRecords

public long getProactiveSpillCountRecords()

Description copied from class: PigStats

Returns the total number of records that spilled proactively

Specified by:

getProactiveSpillCountRecords in class PigStats

getNumberJobs

public int getNumberJobs()

Description copied from class: PigStats

Returns the number of jobs for this script

Overrides:

getNumberJobs in class PigStats