org.apache.pig.backend.hadoop.executionengine.spark

Class SparkLauncher

java.lang.Object

org.apache.pig.backend.hadoop.executionengine.Launcher

org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher

public class SparkLauncher
extends Launcher

Main class that launches pig for Spark

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SparkLauncher.ResourceType

Nested classes/interfaces inherited from class org.apache.pig.backend.hadoop.executionengine.Launcher

Launcher.HangingJobKiller, Launcher.JobControlThreadExceptionHandler

Field Summary

Fields inherited from class org.apache.pig.backend.hadoop.executionengine.Launcher

failureMap, jc, jobControlException, jobControlExceptionStackTrace, totalHadoopTimeSpent

Constructor Summary

Constructors

Constructor and Description
SparkLauncher()

Method Summary

Modifier and Type	Method and Description
SparkOperPlan	compile(PhysicalPlan physicalPlan, PigContext pigContext)
void	explain(PhysicalPlan pp, PigContext pc, PrintStream ps, String format, boolean verbose) Explain how a pig job will be executed on the underlying infrastructure.
void	kill()
void	killJob(String jobID, org.apache.hadoop.conf.Configuration conf)
PigStats	launchPig(PhysicalPlan physicalPlan, String grpName, PigContext pigContext) Method to launch pig for hadoop either for a cluster's job tracker or for a local job runner.

Methods inherited from class org.apache.pig.backend.hadoop.executionengine.Launcher

calculateProgress, computeTimeSpent, destroy, getErrorMessages, getExceptionFromString, getStackTraceElement, getTotalHadoopTimeSpent, isComplete, reset, setJobException

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkLauncher

public SparkLauncher()

Method Detail

launchPig

public PigStats launchPig(PhysicalPlan physicalPlan,
                          String grpName,
                          PigContext pigContext)
                   throws Exception

Description copied from class: Launcher

Method to launch pig for hadoop either for a cluster's job tracker or for a local job runner. THe only difference between the two is the job client. Depending on the pig context the job client will be initialize to one of the two. Launchers for other frameworks can overide these methods. Given an input PhysicalPlan, it compiles it to get a MapReduce Plan. The MapReduce plan which has multiple MapReduce operators each one of which has to be run as a map reduce job with dependency information stored in the plan. It compiles the MROperPlan into a JobControl object. Each Map Reduce operator is converted into a Job and added to the JobControl object. Each Job also has a set of dependent Jobs that are created using the MROperPlan. The JobControl object is obtained from the JobControlCompiler Then a new thread is spawned that submits these jobs while respecting the dependency information. The parent thread monitors the submitted jobs' progress and after it is complete, stops the JobControl thread.

Specified by:

launchPig in class Launcher

Throws:

Exception

compile

public SparkOperPlan compile(PhysicalPlan physicalPlan,
                             PigContext pigContext)
                      throws PlanException,
                             IOException,
                             VisitorException

Throws:

PlanException

IOException

VisitorException

explain

public void explain(PhysicalPlan pp,
                    PigContext pc,
                    PrintStream ps,
                    String format,
                    boolean verbose)
             throws IOException

Description copied from class: Launcher

Explain how a pig job will be executed on the underlying infrastructure.

Specified by:

explain in class Launcher

Parameters:

pp - PhysicalPlan to explain

pc - PigContext to use for configuration

ps - PrintStream to write output on.

format - Format to write in

verbose - Amount of information to print

Throws:

VisitorException

IOException

kill

public void kill()
          throws BackendException

Specified by:

kill in class Launcher

Throws:

BackendException

killJob

public void killJob(String jobID,
                    org.apache.hadoop.conf.Configuration conf)
             throws BackendException

Specified by:

killJob in class Launcher

Throws:

BackendException