org.apache.pig.backend.hadoop.streaming

Class HadoopExecutableManager

java.lang.Object

org.apache.pig.impl.streaming.ExecutableManager

org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager

Direct Known Subclasses:

SparkExecutableManager

public class HadoopExecutableManager
extends ExecutableManager

HadoopExecutableManager is a specialization of ExecutableManager and provides HDFS-specific support for secondary outputs, task-logs etc. HadoopExecutableManager provides support for secondary outputs of the managed process and also persists the logs of the tasks on HDFS.

Field Summary

Fields inherited from class org.apache.pig.impl.streaming.ExecutableManager

command, exitCode, inputBytes, inputRecords, outerrThreadsError, outputBytes, outputRecords, stdin

Constructor Summary

Constructors

Constructor and Description
HadoopExecutableManager()

Method Summary

Modifier and Type	Method and Description
void	close() Close and cleanup the ExecutableManager.
void	configure(POStream stream) Configure and initialize the ExecutableManager.
protected void	exec() Start execution of the external process.
protected void	processError(String error) Workhorse to process the stderr stream of the managed process.
protected boolean	writeErrorToHDFS(int limit, String taskId) Should the stderr data of this task be persisted on HDFS?

Methods inherited from class org.apache.pig.impl.streaming.ExecutableManager

run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopExecutableManager

public HadoopExecutableManager()

Method Detail

configure

public void configure(POStream stream)
               throws IOException,
                      ExecException

Description copied from class: ExecutableManager

Configure and initialize the ExecutableManager.

Overrides:

configure in class ExecutableManager

Parameters:

stream - POStream operator

Throws:

IOException

ExecException

exec

protected void exec()
             throws IOException

Description copied from class: ExecutableManager

Start execution of the external process. This takes care of setting up the environment of the process and also starts ProcessErrorThread to process the stderr of the managed process.

Overrides:

exec in class ExecutableManager

Throws:

IOException

close

public void close()
           throws IOException

Description copied from class: ExecutableManager

Close and cleanup the ExecutableManager.

Overrides:

close in class ExecutableManager

Throws:

IOException

writeErrorToHDFS

protected boolean writeErrorToHDFS(int limit,
                                   String taskId)

Should the stderr data of this task be persisted on HDFS?

Parameters:

limit - maximum number of tasks whose stderr log-files are persisted

taskId - id of the task

Returns:

true if stderr data of task should be persisted on HDFS, false otherwise

processError

protected void processError(String error)

Description copied from class: ExecutableManager

Workhorse to process the stderr stream of the managed process. By default ExecuatbleManager just sends out the received error message to the stderr of itself.

Overrides:

processError in class ExecutableManager

Parameters:

error - error message from the managed process.