org.apache.pig.backend.hadoop.executionengine.mapReduceLayer

Class PigOutputFormat

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.WritableComparable,Tuple>

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat

Direct Known Subclasses:

PigOutputFormatTez

public class PigOutputFormat
extends org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.WritableComparable,Tuple>

The better half of PigInputFormat which is responsible for the Store functionality. It is the exact mirror image of PigInputFormat having RecordWriter instead of a RecordReader.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PigOutputFormat.PigRecordWriter Wrapper class which will delegate calls to the actual RecordWriter - this should only get called in the single store case.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	currentConf
protected List<POStore>	mapStores
static String	PIG_MAPRED_OUTPUT_DIR the temporary directory for the multi store
static String	PIG_TMP_PATH the relative path that can be used to build a temporary place to store the output from a number of map-reduce tasks
protected List<POStore>	reduceStores

Constructor Summary

Constructors

Constructor and Description
PigOutputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext jobcontext)
org.apache.hadoop.mapreduce.OutputCommitter	getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext taskattemptcontext)
org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.WritableComparable,Tuple>	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext taskattemptcontext)
static void	setLocation(org.apache.hadoop.mapreduce.JobContext jobContext, POStore store) Before delegating calls to underlying OutputFormat or OutputCommitter Pig needs to ensure the Configuration in the JobContext contains the output location and StoreFunc for the specific store - so set these up in the context for this specific store
protected void	setupUdfEnvAndStores(org.apache.hadoop.mapreduce.JobContext jobcontext)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PIG_MAPRED_OUTPUT_DIR

public static final String PIG_MAPRED_OUTPUT_DIR

the temporary directory for the multi store

See Also:

Constant Field Values

PIG_TMP_PATH

public static final String PIG_TMP_PATH

the relative path that can be used to build a temporary place to store the output from a number of map-reduce tasks

See Also:

Constant Field Values

reduceStores

protected List<POStore> reduceStores

mapStores

protected List<POStore> mapStores

currentConf

protected org.apache.hadoop.conf.Configuration currentConf

Constructor Detail

PigOutputFormat

public PigOutputFormat()

Method Detail

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.WritableComparable,Tuple> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext taskattemptcontext)
                                                                                                        throws IOException,
                                                                                                               InterruptedException

Specified by:

getRecordWriter in class org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.WritableComparable,Tuple>

Throws:

IOException

InterruptedException

setLocation

public static void setLocation(org.apache.hadoop.mapreduce.JobContext jobContext,
               POStore store)
                        throws IOException

Before delegating calls to underlying OutputFormat or OutputCommitter Pig needs to ensure the Configuration in the JobContext contains the output location and StoreFunc for the specific store - so set these up in the context for this specific store

Parameters:

jobContext - the JobContext

store - the POStore

Throws:

IOException - on failure

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext jobcontext)
                      throws IOException,
                             InterruptedException

Specified by:

checkOutputSpecs in class org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.WritableComparable,Tuple>

Throws:

IOException

InterruptedException

setupUdfEnvAndStores

protected void setupUdfEnvAndStores(org.apache.hadoop.mapreduce.JobContext jobcontext)
                             throws IOException

Throws:

IOException

getOutputCommitter

public org.apache.hadoop.mapreduce.OutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext taskattemptcontext)
                                                               throws IOException,
                                                                      InterruptedException

Specified by:

getOutputCommitter in class org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.WritableComparable,Tuple>

Throws:

IOException

InterruptedException