org.apache.pig.backend.executionengine

Interface ExecutionEngine

All Known Implementing Classes:

HExecutionEngine, MRExecutionEngine, SparkExecutionEngine, TezExecutionEngine

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public interface ExecutionEngine

The main interface bridging the front end and back end of Pig. This allows Pig to be ran on multiple Execution Engines, and not being limited to only Hadoop MapReduce. The ExecutionEngines must support the following methods as these are all the access points for the Pig frontend for processing. Traditionally there is one ExecutionEngine created per processing job, but this is not necessary. The ExecutionEngine instance comes from the ExecType, and it can choose to reuse ExecutionEngine instances. All specifications for methods are listed below as well as expected behavior, and the ExecutionEngine must conform to these.

Method Summary

Modifier and Type	Method and Description
void	destroy() Perform any cleanup operation
void	explain(LogicalPlan lp, PigContext pc, PrintStream ps, String format, boolean verbose, File dir, String suffix) This method handles the backend processing of the Explain command.
Properties	getConfiguration() Returns the Properties representation of the ExecutionEngine configuration.
DataStorage	getDataStorage() Returns the DataStorage the ExecutionEngine is using.
ExecutableManager	getExecutableManager() Returns the ExecutableManager to be used in Pig Streaming.
void	init() This method is responsible for the initialization of the ExecutionEngine.
PigStats	instantiatePigStats() Creates a PigStats object which will be accessible as a ThreadLocal variable inside the PigStats class.
ScriptState	instantiateScriptState() Creates a ScriptState object which will be accessible as a ThreadLocal variable inside the ScriptState class.
void	kill() This method is called when user requests to kill all jobs associated with the execution engine
void	killJob(String jobID) This method is called when a user requests to kill a job associated with the given job id.
PigStats	launchPig(LogicalPlan lp, String grpName, PigContext pc) This method is responsible for the actual execution of a LogicalPlan.
void	setConfiguration(Properties newConfiguration) Responsible for updating the properties for the ExecutionEngine.
void	setProperty(String property, String value) Responsible for setting a specific property and value.

Method Detail

init

void init()
   throws ExecException

This method is responsible for the initialization of the ExecutionEngine. All necessary setup tasks and configuration should execute in the init() method. This method will be called via the PigContext object.

Throws:

ExecException

setConfiguration

void setConfiguration(Properties newConfiguration)
               throws ExecException

Responsible for updating the properties for the ExecutionEngine. The update may require reinitialization of the engine, perhaps through another call to init() if appropriate. This decision is delegated to the ExecutionEngine -- that is, the caller will not call init() after updating the properties. The Properties passed in should replace any configuration that occurred from previous Properties object. The Properties object should also be updated to reflect the deprecation/modifications that the ExecutionEngine may trigger.

Parameters:

newConfiguration - -- Properties object holding all configuration vals

Throws:

ExecException

setProperty

void setProperty(String property,
                 String value)

Responsible for setting a specific property and value. This method may be called as a result of a user "SET" command in the script or elsewhere in Pig to set certain properties. The properties object of the PigContext should be updated with the property and value with deprecation/other configuration done by the ExecutionEngine reflected. The ExecutionEngine should also update its internal configuration view as well.

Parameters:

property - to update

value - to set for property

getConfiguration

Properties getConfiguration()

Returns the Properties representation of the ExecutionEngine configuration. The Properties object returned does not have to be the same object between distinct calls to getConfiguration(). The ExecutionEngine may create a new Properties object populated with all the properties each time.

launchPig

PigStats launchPig(LogicalPlan lp,
                   String grpName,
                   PigContext pc)
            throws FrontendException,
                   ExecException

This method is responsible for the actual execution of a LogicalPlan. No assumptions is made about the architecture of the ExecutionEngine, except that it is capable of executing the LogicalPlan representation of a script. The ExecutionEngine should take care of all cleanup after executing the logical plan and returns an implementation of PigStats that contains the relevant information/statistics of the execution of the script.

Parameters:

lp - -- plan to compile

grpName - -- group name for submission

pc - -- context for execution

Throws:

ExecException

FrontendException

explain

void explain(LogicalPlan lp,
             PigContext pc,
             PrintStream ps,
             String format,
             boolean verbose,
             File dir,
             String suffix)
      throws PlanException,
             VisitorException,
             IOException

This method handles the backend processing of the Explain command. Once again, no assumptions is made about the architecture of the ExecutionEngine, except that it is capable of "explaining" the LogicalPlan representation of a script. The ExecutionEngine should print all of it's explain statements in the PrintStream provided UNLESS the File object is NOT null. In that case, the file is the directory for which the ExecutionEngine must write out the explain statements into semantically distinct files. For example, if the ExecutionEngine operates on a PhysicalPlan and an ExecutionPlan then there would be two separate files detailing each. The suffix param indicates the suffix of each of these file names.

Parameters:

lp - -- plan to explain

pc - -- context for explain processing

ps - -- print stream to write all output to (if dir param is null)

format - -- format to print explain

verbose -

dir - -- directory to write output to. if not null, write to files

suffix - -- if writing to files, suffix to be used for each file

Throws:

PlanException

VisitorException

IOException

getDataStorage

DataStorage getDataStorage()

Returns the DataStorage the ExecutionEngine is using.

Returns:

DataStorage the ExecutionEngine is using.

instantiateScriptState

ScriptState instantiateScriptState()

Creates a ScriptState object which will be accessible as a ThreadLocal variable inside the ScriptState class. This method is called when first initializing the ScriptState as to delegate to the ExecutionEngine the version of ScriptState to use to manage the execution at hand.

Returns:

ScriptState object to manage execution of the script

instantiatePigStats

PigStats instantiatePigStats()

Creates a PigStats object which will be accessible as a ThreadLocal variable inside the PigStats class. This method is called when first initializing the PigStats.

Returns:

PigStats object.

getExecutableManager

ExecutableManager getExecutableManager()

Returns the ExecutableManager to be used in Pig Streaming.

Returns:

ExecutableManager to be used in Pig Streaming.

kill

void kill()
   throws BackendException

This method is called when user requests to kill all jobs associated with the execution engine

Throws:

BackendException

killJob

void killJob(String jobID)
      throws BackendException

This method is called when a user requests to kill a job associated with the given job id. If it is not possible for a user to kill a job, throw a exception. It is imperative for the job id's being displayed to be unique such that the correct jobs are being killed when the user supplies the id.

Throws:

BackendException

destroy

void destroy()

Perform any cleanup operation