org.apache.pig.backend.hadoop.executionengine.spark

Interface SparkPigSplit

All Superinterfaces:

org.apache.hadoop.conf.Configurable, java.io.Serializable, org.apache.hadoop.io.Writable

All Known Implementing Classes:

SparkPigSplit.FileSparkPigSplit, SparkPigSplit.GenericSparkPigSplit

public interface SparkPigSplit
extends org.apache.hadoop.io.Writable, org.apache.hadoop.conf.Configurable, java.io.Serializable

Wrapper class for PigSplits in Spark mode Spark only counts HDFS bytes read if we provide a FileSplit, so we have to wrap PigSplits and have the wrapper class extend FileSplit

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	SparkPigSplit.FileSparkPigSplit
static class	SparkPigSplit.GenericSparkPigSplit
static class	SparkPigSplit.SparkPigSplitsUtils

Method Summary

Modifier and Type	Method and Description
boolean	disableCounter()
long	getLength(int idx)
org.apache.hadoop.mapred.SplitLocationInfo[]	getLocationInfo()
int	getNumPaths()
int	getSplitIndex()
PigSplit	getWrappedPigSplit()
org.apache.hadoop.mapreduce.InputSplit	getWrappedSplit()
org.apache.hadoop.mapreduce.InputSplit	getWrappedSplit(int idx)
boolean	isMultiInputs()
void	setCurrentIdx(int idx)
void	setDisableCounter(boolean disableCounter)
void	setMultiInputs(boolean b)

Methods inherited from interface org.apache.hadoop.io.Writable

readFields, write

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Method Detail

getWrappedSplit

org.apache.hadoop.mapreduce.InputSplit getWrappedSplit()

getWrappedSplit

org.apache.hadoop.mapreduce.InputSplit getWrappedSplit(int idx)

getLocationInfo

org.apache.hadoop.mapred.SplitLocationInfo[] getLocationInfo()
                                                      throws java.io.IOException

Throws:

java.io.IOException

getLength

long getLength(int idx)
        throws java.io.IOException,
               java.lang.InterruptedException

Throws:

java.io.IOException

java.lang.InterruptedException

getSplitIndex

int getSplitIndex()

setMultiInputs

void setMultiInputs(boolean b)

isMultiInputs

boolean isMultiInputs()

getNumPaths

int getNumPaths()

setDisableCounter

void setDisableCounter(boolean disableCounter)

disableCounter

boolean disableCounter()

setCurrentIdx

void setCurrentIdx(int idx)

getWrappedPigSplit

PigSplit getWrappedPigSplit()