org.apache.pig.backend.hadoop.executionengine

Class Launcher

java.lang.Object

org.apache.pig.backend.hadoop.executionengine.Launcher

Direct Known Subclasses:

MapReduceLauncher, SparkLauncher, TezLauncher

public abstract class Launcher
extends Object

Provides core processing implementation for the backend of Pig if ExecutionEngine chosen decides to delegate it's work to this class. Also contains set of utility methods, including ones centered around Hadoop.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	Launcher.HangingJobKiller
class	Launcher.JobControlThreadExceptionHandler An exception handler class to handle exceptions thrown by the job controller thread Its a local class.

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<FileSpec,Exception>	failureMap
protected org.apache.hadoop.mapred.jobcontrol.JobControl	jc
protected Exception	jobControlException
protected String	jobControlExceptionStackTrace
protected long	totalHadoopTimeSpent

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	Launcher()

Method Summary

Modifier and Type	Method and Description
protected double	calculateProgress(org.apache.hadoop.mapred.jobcontrol.JobControl jc) Compute the progress of the current job submitted through the JobControl object jc to the JobClient jobClient
protected long	computeTimeSpent(Iterator<org.apache.hadoop.mapreduce.TaskReport> taskReports)
void	destroy()
abstract void	explain(PhysicalPlan pp, PigContext pc, PrintStream ps, String format, boolean verbose) Explain how a pig job will be executed on the underlying infrastructure.
protected void	getErrorMessages(Iterator<org.apache.hadoop.mapreduce.TaskReport> reports, String type, boolean errNotDbg, PigContext pigContext)
Exception	getExceptionFromString(String stackTrace)
StackTraceElement	getStackTraceElement(String line)
long	getTotalHadoopTimeSpent()
protected boolean	isComplete(double prog)
abstract void	kill()
abstract void	killJob(String jobID, org.apache.hadoop.conf.Configuration conf)
abstract PigStats	launchPig(PhysicalPlan php, String grpName, PigContext pc) Method to launch pig for hadoop either for a cluster's job tracker or for a local job runner.
void	reset() Resets the state after a launch
protected void	setJobException(Throwable throwable)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

jobControlExceptionStackTrace

protected String jobControlExceptionStackTrace

jobControlException

protected Exception jobControlException

totalHadoopTimeSpent

protected long totalHadoopTimeSpent

failureMap

protected Map<FileSpec,Exception> failureMap

jc

protected org.apache.hadoop.mapred.jobcontrol.JobControl jc

Constructor Detail

Launcher

protected Launcher()

Method Detail

reset

public void reset()

Resets the state after a launch

launchPig

public abstract PigStats launchPig(PhysicalPlan php,
                                   String grpName,
                                   PigContext pc)
                            throws Exception

Method to launch pig for hadoop either for a cluster's job tracker or for a local job runner. THe only difference between the two is the job client. Depending on the pig context the job client will be initialize to one of the two. Launchers for other frameworks can overide these methods. Given an input PhysicalPlan, it compiles it to get a MapReduce Plan. The MapReduce plan which has multiple MapReduce operators each one of which has to be run as a map reduce job with dependency information stored in the plan. It compiles the MROperPlan into a JobControl object. Each Map Reduce operator is converted into a Job and added to the JobControl object. Each Job also has a set of dependent Jobs that are created using the MROperPlan. The JobControl object is obtained from the JobControlCompiler Then a new thread is spawned that submits these jobs while respecting the dependency information. The parent thread monitors the submitted jobs' progress and after it is complete, stops the JobControl thread.

Parameters:

php -

grpName -

pc -

Throws:

Exception

explain

public abstract void explain(PhysicalPlan pp,
                             PigContext pc,
                             PrintStream ps,
                             String format,
                             boolean verbose)
                      throws PlanException,
                             VisitorException,
                             IOException

Explain how a pig job will be executed on the underlying infrastructure.

Parameters:

pp - PhysicalPlan to explain

pc - PigContext to use for configuration

ps - PrintStream to write output on.

format - Format to write in

verbose - Amount of information to print

Throws:

VisitorException

IOException

PlanException

kill

public abstract void kill()
                   throws BackendException

Throws:

BackendException

killJob

public abstract void killJob(String jobID,
                             org.apache.hadoop.conf.Configuration conf)
                      throws BackendException

Throws:

BackendException

isComplete

protected boolean isComplete(double prog)

computeTimeSpent

protected long computeTimeSpent(Iterator<org.apache.hadoop.mapreduce.TaskReport> taskReports)

getErrorMessages

protected void getErrorMessages(Iterator<org.apache.hadoop.mapreduce.TaskReport> reports,
                                String type,
                                boolean errNotDbg,
                                PigContext pigContext)
                         throws Exception

Throws:

Exception

calculateProgress

protected double calculateProgress(org.apache.hadoop.mapred.jobcontrol.JobControl jc)
                            throws IOException

Compute the progress of the current job submitted through the JobControl object jc to the JobClient jobClient

Parameters:

jc - - The JobControl object that has been submitted

jobClient - - The JobClient to which it has been submitted

Returns:

The progress as a precentage in double format

Throws:

IOException

getTotalHadoopTimeSpent

public long getTotalHadoopTimeSpent()

setJobException

protected void setJobException(Throwable throwable)

getExceptionFromString

public Exception getExceptionFromString(String stackTrace)
                                 throws Exception

Parameters:

stackTrace - The string representation of printStackTrace Handles internal PigException and its subclasses that override the toString method

Returns:

An exception object whose string representation of printStackTrace is the input stackTrace

Throws:

Exception

getStackTraceElement

public StackTraceElement getStackTraceElement(String line)
                                       throws Exception

Parameters:

line - the string representation of a stack trace returned by printStackTrace

Returns:

the StackTraceElement object representing the stack trace

Throws:

Exception

destroy

public void destroy()