org.apache.pig.backend.hadoop.executionengine.tez.util

Class TezInputHelper

java.lang.Object

org.apache.pig.backend.hadoop.executionengine.tez.util.TezInputHelper

public class TezInputHelper
extends java.lang.Object

Constructor Summary

Constructors

Constructor and Description
TezInputHelper()

Method Summary

Modifier and Type	Method and Description
static Pair<java.lang.Long,java.lang.Boolean>	createSplitsProto(org.apache.tez.mapreduce.hadoop.InputSplitInfo inputSplitInfo, org.apache.hadoop.conf.Configuration conf, org.apache.tez.mapreduce.protos.MRRuntimeProtos.MRSplitsProto.Builder splitsBuilder, long spillThreshold) Creates MRSplitsProto from inputSplitInfo and adds into splitsBuilder.
static org.apache.tez.mapreduce.hadoop.InputSplitInfoMem	generateInputSplitsToMem(org.apache.hadoop.conf.Configuration conf) This method creates input splits similar to MRInputHelpers.generateInputSplitsToMem(org.apache.hadoop.conf.Configuration, boolean, int) but only does it for mapreduce API and does not do grouping of splits or create MRRuntimeProtos.MRSplitsProto which is an expensive operation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TezInputHelper

public TezInputHelper()

Method Detail

generateInputSplitsToMem

public static org.apache.tez.mapreduce.hadoop.InputSplitInfoMem generateInputSplitsToMem(org.apache.hadoop.conf.Configuration conf)
                                                                                  throws java.io.IOException,
                                                                                         java.lang.ClassNotFoundException,
                                                                                         java.lang.InterruptedException

This method creates input splits similar to MRInputHelpers.generateInputSplitsToMem(org.apache.hadoop.conf.Configuration, boolean, int) but only does it for mapreduce API and does not do grouping of splits or create MRRuntimeProtos.MRSplitsProto which is an expensive operation.

Parameters:

conf - an instance of Configuration. This Configuration instance should contain adequate information to be able to generate splits - like the InputFormat being used and related configuration.

Returns:

an instance of InputSplitInfoMem which supports a subset of the APIs defined on InputSplitInfo

Throws:

java.io.IOException

java.lang.ClassNotFoundException

java.lang.InterruptedException

createSplitsProto

public static Pair<java.lang.Long,java.lang.Boolean> createSplitsProto(org.apache.tez.mapreduce.hadoop.InputSplitInfo inputSplitInfo,
                                                                       org.apache.hadoop.conf.Configuration conf,
                                                                       org.apache.tez.mapreduce.protos.MRRuntimeProtos.MRSplitsProto.Builder splitsBuilder,
                                                                       long spillThreshold)

Creates MRSplitsProto from inputSplitInfo and adds into splitsBuilder.

Parameters:

inputSplitInfo -

conf -

splitsBuilder -

spillThreshold -

Returns:

Pair containing first element, a long, as serialized size and second element, a boolean, as true if all splits are serialized. Second element will be false, if only some of the splits are serialized because we reached to spillThreshold.