org.apache.pig.backend.hadoop.executionengine.mapReduceLayer

Interface PigReducerEstimator

All Known Implementing Classes:

InputSizeReducerEstimator

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public interface PigReducerEstimator

Interface to implement when you want to use a custom approach to estimating the number of reducers for a job.

See Also:

InputSizeReducerEstimator

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	BYTES_PER_REDUCER_PARAM
static long	DEFAULT_BYTES_PER_REDUCER
static int	DEFAULT_MAX_REDUCER_COUNT_PARAM
static java.lang.String	MAX_REDUCER_COUNT_PARAM

Method Summary

Modifier and Type	Method and Description
int	estimateNumberOfReducers(org.apache.hadoop.mapreduce.Job job, MapReduceOper mapReduceOper) Estimate the number of reducers for a given job based on the collection of load funcs passed.

Field Detail

BYTES_PER_REDUCER_PARAM

static final java.lang.String BYTES_PER_REDUCER_PARAM

See Also:

Constant Field Values

MAX_REDUCER_COUNT_PARAM

static final java.lang.String MAX_REDUCER_COUNT_PARAM

See Also:

Constant Field Values

DEFAULT_BYTES_PER_REDUCER

static final long DEFAULT_BYTES_PER_REDUCER

See Also:

Constant Field Values

DEFAULT_MAX_REDUCER_COUNT_PARAM

static final int DEFAULT_MAX_REDUCER_COUNT_PARAM

See Also:

Constant Field Values

Method Detail

estimateNumberOfReducers

int estimateNumberOfReducers(org.apache.hadoop.mapreduce.Job job,
                             MapReduceOper mapReduceOper)
                      throws java.io.IOException

Estimate the number of reducers for a given job based on the collection of load funcs passed.

Parameters:

job - job instance

mapReduceOper - map reducer operator of the job

Returns:

the number of reducers to use, or -1 if the count couldn't be estimated

Throws:

java.io.IOException