org.apache.pig.backend.hadoop.executionengine
Class HExecutionEngine

java.lang.Object
  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine

All Implemented Interfaces:

ExecutionEngine

public class HExecutionEngine
extends Object
implements ExecutionEngine

Field Summary

protected DataStorage	ds
static String	JOB_TRACKER_LOCATION
protected org.apache.hadoop.mapred.JobConf	jobConf
static String	LOCAL
protected Map<OperatorKey,OperatorKey>	logicalToPhysicalKeys
protected Map<OperatorKey,MapRedResult>	materializedResults
protected Map<OperatorKey,ExecPhysicalOperator>	physicalOpTable
protected PigContext	pigContext

Constructor Summary

HExecutionEngine(PigContext pigContext)

Method Summary

Collection<String>	activeScopes() List scopes that are active in the back-end
void	close() Clean-up and releasing of resources.
PhysicalPlan	compile(LogicalPlan plan, Properties properties) Compiles a logical plan into a physical plan, given a set of configuration properties that apply at the plan-level.
List<ExecJob>	execute(PhysicalPlan plan, String jobName) Execute the physical plan in blocking mode.
void	explain(PhysicalPlan plan, PrintStream stream, String format, boolean verbose) Explain executor specific information.
Properties	getConfiguration() Provides configuration information about the execution engine itself.
DataStorage	getDataStorage()
org.apache.hadoop.mapred.JobConf	getJobConf()
Map<OperatorKey,MapRedResult>	getMaterializedResults()
Map<OperatorKey,ExecPhysicalOperator>	getPhysicalOpTable()
Map<String,Object>	getStatistics() Provides statistics on the Execution Engine: number of nodes, node failure rates, average load, average job wait time...
void	init() Place holder for possible initialization activities.
void	init(Properties properties)
void	reclaimScope(String scope) A mechanism to communicate to the back-end that a set of logical plans go out of scope
Collection<ExecJob>	runningJobs(Properties properties) Return currently running jobs (can be useful for admin purposes)
List<ExecJob>	submit(PhysicalPlan plan, String jobName) Execute the physical plan in non-blocking mode
void	updateConfiguration(Properties newConfiguration) Provides a way to dynamically change configuration parameters at the Execution Engine level.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

JOB_TRACKER_LOCATION

public static final String JOB_TRACKER_LOCATION

See Also:

Constant Field Values

LOCAL

public static final String LOCAL

See Also:

Constant Field Values

pigContext

protected PigContext pigContext

ds

protected DataStorage ds

jobConf

protected org.apache.hadoop.mapred.JobConf jobConf

logicalToPhysicalKeys

protected Map<OperatorKey,OperatorKey> logicalToPhysicalKeys

physicalOpTable

protected Map<OperatorKey,ExecPhysicalOperator> physicalOpTable

materializedResults

protected Map<OperatorKey,MapRedResult> materializedResults

Constructor Detail

HExecutionEngine

public HExecutionEngine(PigContext pigContext)

Method Detail

getJobConf

public org.apache.hadoop.mapred.JobConf getJobConf()

getMaterializedResults

public Map<OperatorKey,MapRedResult> getMaterializedResults()

getPhysicalOpTable

public Map<OperatorKey,ExecPhysicalOperator> getPhysicalOpTable()

getDataStorage

public DataStorage getDataStorage()

Specified by:

getDataStorage in interface ExecutionEngine

init

public void init()
          throws ExecException

Description copied from interface: ExecutionEngine

Place holder for possible initialization activities.

Specified by:

init in interface ExecutionEngine

Throws:

ExecException

init

public void init(Properties properties)
          throws ExecException

Throws:

ExecException

getConfiguration

public Properties getConfiguration()
                            throws ExecException

Description copied from interface: ExecutionEngine

Provides configuration information about the execution engine itself.

Specified by:

getConfiguration in interface ExecutionEngine

Returns:

- information about the configuration used to connect to execution engine

Throws:

ExecException

updateConfiguration

public void updateConfiguration(Properties newConfiguration)
                         throws ExecException

Description copied from interface: ExecutionEngine

Provides a way to dynamically change configuration parameters at the Execution Engine level.

Specified by:

updateConfiguration in interface ExecutionEngine

Parameters:

newConfiguration - - the new configuration settings

Throws:

ExecException

close

public void close()
           throws ExecException

Description copied from interface: ExecutionEngine

Clean-up and releasing of resources.

Specified by:

close in interface ExecutionEngine

Throws:

ExecException

getStatistics

public Map<String,Object> getStatistics()
                                 throws ExecException

Description copied from interface: ExecutionEngine

Provides statistics on the Execution Engine: number of nodes, node failure rates, average load, average job wait time...

Specified by:

getStatistics in interface ExecutionEngine

Returns:

ExecutionEngineProperties

Throws:

ExecException

compile

public PhysicalPlan compile(LogicalPlan plan,
                            Properties properties)
                     throws ExecException

Description copied from interface: ExecutionEngine

Compiles a logical plan into a physical plan, given a set of configuration properties that apply at the plan-level. For instance desired degree of parallelism for this plan, which could be different from the "default" one set at the execution engine level.

Specified by:

compile in interface ExecutionEngine

Parameters:

plan - logical plan to compile

Returns:

physical plan

Throws:

ExecException

execute

public List<ExecJob> execute(PhysicalPlan plan,
                             String jobName)
                      throws ExecException

Description copied from interface: ExecutionEngine

Execute the physical plan in blocking mode.

Specified by:

execute in interface ExecutionEngine

Parameters:

plan - PhysicalPlan to execute.

jobName - Name of this plan, will be used to identify the plan

Throws:

ExecException

submit

public List<ExecJob> submit(PhysicalPlan plan,
                            String jobName)
                     throws ExecException

Description copied from interface: ExecutionEngine

Execute the physical plan in non-blocking mode

Specified by:

submit in interface ExecutionEngine

Parameters:

plan - PhysicalPlan to submit.

jobName - Name of this plan, will be used to identify the plan

Throws:

ExecException

explain

public void explain(PhysicalPlan plan,
                    PrintStream stream,
                    String format,
                    boolean verbose)

Description copied from interface: ExecutionEngine

Explain executor specific information.

Specified by:

explain in interface ExecutionEngine

Parameters:

plan - PhysicalPlan to explain

stream - Stream to print output to

format - Format to print in

verbose - Amount of information to print

runningJobs

public Collection<ExecJob> runningJobs(Properties properties)
                                throws ExecException

Description copied from interface: ExecutionEngine

Return currently running jobs (can be useful for admin purposes)

Specified by:

runningJobs in interface ExecutionEngine

Returns:

All jobs that are currently active in hte execution engine.

Throws:

ExecException

activeScopes

public Collection<String> activeScopes()
                                throws ExecException

Description copied from interface: ExecutionEngine

List scopes that are active in the back-end

Specified by:

activeScopes in interface ExecutionEngine

Returns:

All scopes that are currently active in the execution engine.

Throws:

ExecException

reclaimScope

public void reclaimScope(String scope)
                  throws ExecException

Description copied from interface: ExecutionEngine

A mechanism to communicate to the back-end that a set of logical plans go out of scope

Specified by:

reclaimScope in interface ExecutionEngine

Throws:

ExecException