org.apache.pig

Class PigConfiguration

java.lang.Object

org.apache.pig.PigConfiguration

public class PigConfiguration
extends java.lang.Object

Container for static configuration strings, defaults, etc. This is intended just for keys that can be set by users, not for keys that are generally used within pig.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	INSERT_ENABLED Deprecated. use PIG_SCRIPT_INFO_ENABLED instead. Will be removed in Pig 0.16
static java.lang.String	MAX_SCRIPT_SIZE Deprecated. use PIG_SCRIPT_MAX_SIZE instead. Will be removed in Pig 0.16
static java.lang.String	OPT_FETCH Deprecated. use PIG_OPT_FETCH instead. Will be removed in Pig 0.16
static java.lang.String	PARTAGG_MINREDUCTION Deprecated. use PIG_EXEC_MAP_PARTAGG_MINREDUCTION instead. Will be removed in Pig 0.16
static java.lang.String	PIG_ACCUMULATIVE_BATCHSIZE
static java.lang.String	PIG_AUTO_LOCAL_ENABLED This key is to turn on auto local mode feature
static java.lang.String	PIG_AUTO_LOCAL_INPUT_MAXBYTES Controls the max threshold size to convert jobs to run in local mode
static java.lang.String	PIG_BLACKLIST Comma-delimited entries of commands/operators that must be disallowed.
static java.lang.String	PIG_CACHEDBAG_DISTINCT_TYPE
static java.lang.String	PIG_CACHEDBAG_MEMUSAGE Controls the fraction of total memory that is allowed to be used by cached bags.
static java.lang.String	PIG_CACHEDBAG_SORT_TYPE
static java.lang.String	PIG_CACHEDBAG_TYPE Configurations for specifying alternate implementations for cached bags.
static java.lang.String	PIG_DEFAULT_LOAD_FUNC This key is used to define the default load func.
static java.lang.String	PIG_DEFAULT_STORE_FUNC This key is used to define the default store func.
static java.lang.String	PIG_DELETE_TEMP_FILE This key is used to define whether to delete intermediate files of Hadoop jobs.
static java.lang.String	PIG_ENABLE_TEMP_FILE_COMPRESSION This key is used to define whether to have intermediate file compressed
static java.lang.String	PIG_EXEC_MAP_PARTAGG Boolean value to enable or disable partial aggregation in map.
static java.lang.String	PIG_EXEC_MAP_PARTAGG_MINREDUCTION Controls the minimum reduction in-mapper Partial Aggregation should achieve in order to stay on.
static java.lang.String	PIG_EXEC_NO_COMBINER Boolean value to enable or disable use of combiners in MapReduce jobs.
static java.lang.String	PIG_EXEC_NO_SECONDARY_KEY This key controls whether secondary sort key is used for optimization in case of nested distinct or sort
static java.lang.String	PIG_EXEC_REDUCER_ESTIMATOR
static java.lang.String	PIG_EXEC_REDUCER_ESTIMATOR_CONSTRUCTOR_ARG_KEY
static java.lang.String	PIG_JOIN_REPLICATED_MAX_BYTES This key used to control the maximum size loaded into the distributed cache when doing fragment-replicated join
static java.lang.String	PIG_MAX_COMBINED_SPLIT_SIZE Specifies the size, in bytes, of data to be processed by a single map.
static java.lang.String	PIG_NO_SPLIT_COMBINATION Whether turns combine split files off.
static java.lang.String	PIG_NO_TASK_REPORT This key is used to turns off use of task reports in job statistics.
static java.lang.String	PIG_OPT_ACCUMULATOR Boolean value used to enable or disable accumulator optimization.
static java.lang.String	PIG_OPT_FETCH Boolean value used to enable or disable fetching without a mapreduce job for DUMP.
static java.lang.String	PIG_OPT_MULTIQUERY Boolean value used to enable or disable multiquery optimization.
static java.lang.String	PIG_OUTPUT_COMMITTER_RECOVERY This key is used to define whether to support recovery to handle the application master getting restarted.
static java.lang.String	PIG_OUTPUT_LAZY This key is used to define whether PigOutputFormat will be wrapped with LazyOutputFormat so that jobs won't write empty part files if no output is generated
static java.lang.String	PIG_POISSON_SAMPLER_SAMPLE_RATE For a given mean and a confidence, a sample rate is obtained from a poisson udf
static java.lang.String	PIG_RANDOM_SAMPLER_SAMPLE_SIZE This key used to control the sample size of RandomeSampleLoader for order-by.
static java.lang.String	PIG_SCHEMA_TUPLE_ALLOW_FORCE
static java.lang.String	PIG_SCHEMA_TUPLE_ENABLED This key must be set to true by the user for code generation to be used.
static java.lang.String	PIG_SCHEMA_TUPLE_USE_IN_FOREACH
static java.lang.String	PIG_SCHEMA_TUPLE_USE_IN_FRJOIN
static java.lang.String	PIG_SCHEMA_TUPLE_USE_IN_MERGEJOIN
static java.lang.String	PIG_SCHEMA_TUPLE_USE_IN_UDF
static java.lang.String	PIG_SCRIPT_INFO_ENABLED This key is used to turn off the inclusion of settings in the jobs.
static java.lang.String	PIG_SCRIPT_MAX_SIZE Controls the size of Pig script stored in job xml.
static java.lang.String	PIG_SKEWEDJOIN_REDUCE_MEMUSAGE % of memory available for the input data.
static java.lang.String	PIG_SPLIT_COMBINATION Turns combine split files on or off
static java.lang.String	PIG_STREAMING_ENVIRONMENT This key can be used to defined what properties will be set in the streaming environment.
static java.lang.String	PIG_STREAMING_UDF_PYTHON_COMMAND This key can be used to configure the python command for python streaming udf.
static java.lang.String	PIG_TEMP_DIR Location where pig stores temporary files for job setup
static java.lang.String	PIG_TEMP_FILE_COMPRESSION_CODEC Compression codec used by intermediate storage TFileStorage only support gzip and lzo.
static java.lang.String	PIG_TEMP_FILE_COMPRESSION_STORAGE This key is used to set the storage type used by intermediate file storage If pig.tmpfilecompression, default storage used is TFileStorage.
static java.lang.String	PIG_TEZ_AUTO_PARALLELISM This key is used to configure auto parallelism in tez.
static java.lang.String	PIG_TEZ_DAG_STATUS_REPORT_INTERVAL This key is used to configure the interval of dag status report in seconds.
static java.lang.String	PIG_TEZ_OPT_UNION This key is used to enable or disable union optimization in tez.
static java.lang.String	PIG_TEZ_SESSION_REUSE This key is used to define whether to reuse AM in Tez jobs.
static java.lang.String	PIG_UDF_PROFILE Controls whether execution time of Pig UDFs should be tracked.
static java.lang.String	PIG_UDF_PROFILE_FREQUENCY
static java.lang.String	PIG_USER_CACHE_ENABLED This key is turn on the user level cache
static java.lang.String	PIG_USER_CACHE_LOCATION Location where additional jars are cached for the user Additional jar will be cached under PIG_USER_CACHE_LOCATION/${user.name}/.pigcache and will be re-used across the jobs run by the user if the jar has not changed
static java.lang.String	PIG_WHITELIST Comma-delimited entries of commands/operators that must be allowed.
static java.lang.String	PROP_CACHEDBAG_MEMUSAGE Deprecated. use PIG_CACHEDBAG_MEMUSAGE instead. Will be removed in Pig 0.16
static java.lang.String	PROP_EXEC_MAP_PARTAGG Deprecated. use PIG_EXEC_MAP_PARTAGG instead. Will be removed in Pig 0.16
static java.lang.String	PROP_NO_COMBINER Deprecated. use #PROP_NO_COMBINER1 instead. Will be removed in Pig 0.16
static java.lang.String	SCHEMA_TUPLE_SHOULD_ALLOW_FORCE Deprecated.
static java.lang.String	SCHEMA_TUPLE_SHOULD_USE_IN_FOREACH Deprecated.
static java.lang.String	SCHEMA_TUPLE_SHOULD_USE_IN_FRJOIN Deprecated.
static java.lang.String	SCHEMA_TUPLE_SHOULD_USE_IN_MERGEJOIN Deprecated.
static java.lang.String	SCHEMA_TUPLE_SHOULD_USE_IN_UDF Deprecated.
static java.lang.String	SHOULD_USE_SCHEMA_TUPLE Deprecated.

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PIG_AUTO_LOCAL_ENABLED

public static final java.lang.String PIG_AUTO_LOCAL_ENABLED

This key is to turn on auto local mode feature

See Also:

Constant Field Values

PIG_AUTO_LOCAL_INPUT_MAXBYTES

public static final java.lang.String PIG_AUTO_LOCAL_INPUT_MAXBYTES

Controls the max threshold size to convert jobs to run in local mode

See Also:

Constant Field Values

PIG_OPT_FETCH

public static final java.lang.String PIG_OPT_FETCH

Boolean value used to enable or disable fetching without a mapreduce job for DUMP. True by default

See Also:

Constant Field Values

PIG_OPT_MULTIQUERY

public static final java.lang.String PIG_OPT_MULTIQUERY

Boolean value used to enable or disable multiquery optimization. True by default

See Also:

Constant Field Values

PIG_OPT_ACCUMULATOR

public static final java.lang.String PIG_OPT_ACCUMULATOR

Boolean value used to enable or disable accumulator optimization. True by default

See Also:

Constant Field Values

PIG_ACCUMULATIVE_BATCHSIZE

public static final java.lang.String PIG_ACCUMULATIVE_BATCHSIZE

See Also:

Constant Field Values

PIG_TEZ_OPT_UNION

public static final java.lang.String PIG_TEZ_OPT_UNION

This key is used to enable or disable union optimization in tez. True by default

See Also:

Constant Field Values

PIG_EXEC_MAP_PARTAGG

public static final java.lang.String PIG_EXEC_MAP_PARTAGG

Boolean value to enable or disable partial aggregation in map. Disabled by default

See Also:

Constant Field Values

PIG_EXEC_MAP_PARTAGG_MINREDUCTION

public static final java.lang.String PIG_EXEC_MAP_PARTAGG_MINREDUCTION

Controls the minimum reduction in-mapper Partial Aggregation should achieve in order to stay on. If after a period of observation this reduction is not achieved, in-mapper aggregation will be turned off and a message logged to that effect.

See Also:

Constant Field Values

PIG_EXEC_NO_COMBINER

public static final java.lang.String PIG_EXEC_NO_COMBINER

Boolean value to enable or disable use of combiners in MapReduce jobs. Enabled by default

See Also:

Constant Field Values

PIG_EXEC_NO_SECONDARY_KEY

public static final java.lang.String PIG_EXEC_NO_SECONDARY_KEY

This key controls whether secondary sort key is used for optimization in case of nested distinct or sort

See Also:

Constant Field Values

PIG_CACHEDBAG_MEMUSAGE

public static final java.lang.String PIG_CACHEDBAG_MEMUSAGE

Controls the fraction of total memory that is allowed to be used by cached bags. Default is 0.2.

See Also:

Constant Field Values

PIG_SKEWEDJOIN_REDUCE_MEMUSAGE

public static final java.lang.String PIG_SKEWEDJOIN_REDUCE_MEMUSAGE

% of memory available for the input data. This is currently equal to the memory available for the skewed join

See Also:

Constant Field Values

PIG_JOIN_REPLICATED_MAX_BYTES

public static final java.lang.String PIG_JOIN_REPLICATED_MAX_BYTES

This key used to control the maximum size loaded into the distributed cache when doing fragment-replicated join

See Also:

Constant Field Values

PIG_CACHEDBAG_TYPE

public static final java.lang.String PIG_CACHEDBAG_TYPE

Configurations for specifying alternate implementations for cached bags. Rarely used

See Also:

Constant Field Values

PIG_CACHEDBAG_DISTINCT_TYPE

public static final java.lang.String PIG_CACHEDBAG_DISTINCT_TYPE

See Also:

Constant Field Values

PIG_CACHEDBAG_SORT_TYPE

public static final java.lang.String PIG_CACHEDBAG_SORT_TYPE

See Also:

Constant Field Values

PIG_EXEC_REDUCER_ESTIMATOR

public static final java.lang.String PIG_EXEC_REDUCER_ESTIMATOR

See Also:

Constant Field Values

PIG_EXEC_REDUCER_ESTIMATOR_CONSTRUCTOR_ARG_KEY

public static final java.lang.String PIG_EXEC_REDUCER_ESTIMATOR_CONSTRUCTOR_ARG_KEY

See Also:

Constant Field Values

PIG_TEZ_AUTO_PARALLELISM

public static final java.lang.String PIG_TEZ_AUTO_PARALLELISM

This key is used to configure auto parallelism in tez. Default is true.

See Also:

Constant Field Values

PIG_UDF_PROFILE

public static final java.lang.String PIG_UDF_PROFILE

Controls whether execution time of Pig UDFs should be tracked. This feature uses counters; use judiciously.

See Also:

Constant Field Values

PIG_UDF_PROFILE_FREQUENCY

public static final java.lang.String PIG_UDF_PROFILE_FREQUENCY

See Also:

Constant Field Values

PIG_SCHEMA_TUPLE_ENABLED

public static final java.lang.String PIG_SCHEMA_TUPLE_ENABLED

This key must be set to true by the user for code generation to be used. In the future, it may be turned on by default (at least in certain cases), but for now it is too experimental.

See Also:

Constant Field Values

PIG_SCHEMA_TUPLE_USE_IN_UDF

public static final java.lang.String PIG_SCHEMA_TUPLE_USE_IN_UDF

See Also:

Constant Field Values

PIG_SCHEMA_TUPLE_USE_IN_FOREACH

public static final java.lang.String PIG_SCHEMA_TUPLE_USE_IN_FOREACH

See Also:

Constant Field Values

PIG_SCHEMA_TUPLE_USE_IN_FRJOIN

public static final java.lang.String PIG_SCHEMA_TUPLE_USE_IN_FRJOIN

See Also:

Constant Field Values

PIG_SCHEMA_TUPLE_USE_IN_MERGEJOIN

public static final java.lang.String PIG_SCHEMA_TUPLE_USE_IN_MERGEJOIN

See Also:

Constant Field Values

PIG_SCHEMA_TUPLE_ALLOW_FORCE

public static final java.lang.String PIG_SCHEMA_TUPLE_ALLOW_FORCE

See Also:

Constant Field Values

PIG_STREAMING_ENVIRONMENT

public static final java.lang.String PIG_STREAMING_ENVIRONMENT

This key can be used to defined what properties will be set in the streaming environment. Just set this property to a comma-delimited list of properties to set, and those properties will be set in the environment.

See Also:

Constant Field Values

PIG_STREAMING_UDF_PYTHON_COMMAND

public static final java.lang.String PIG_STREAMING_UDF_PYTHON_COMMAND

This key can be used to configure the python command for python streaming udf. For eg, python2.7.

See Also:

Constant Field Values

PIG_SPLIT_COMBINATION

public static final java.lang.String PIG_SPLIT_COMBINATION

Turns combine split files on or off

See Also:

Constant Field Values

PIG_NO_SPLIT_COMBINATION

public static final java.lang.String PIG_NO_SPLIT_COMBINATION

Whether turns combine split files off. This is for internal use only

See Also:

Constant Field Values

PIG_MAX_COMBINED_SPLIT_SIZE

public static final java.lang.String PIG_MAX_COMBINED_SPLIT_SIZE

Specifies the size, in bytes, of data to be processed by a single map. Smaller files are combined untill this size is reached.

See Also:

Constant Field Values

PIG_OUTPUT_LAZY

public static final java.lang.String PIG_OUTPUT_LAZY

This key is used to define whether PigOutputFormat will be wrapped with LazyOutputFormat so that jobs won't write empty part files if no output is generated

See Also:

Constant Field Values

PIG_OUTPUT_COMMITTER_RECOVERY

public static final java.lang.String PIG_OUTPUT_COMMITTER_RECOVERY

This key is used to define whether to support recovery to handle the application master getting restarted.

See Also:

Constant Field Values

PIG_TEMP_DIR

public static final java.lang.String PIG_TEMP_DIR

Location where pig stores temporary files for job setup

See Also:

Constant Field Values

PIG_ENABLE_TEMP_FILE_COMPRESSION

public static final java.lang.String PIG_ENABLE_TEMP_FILE_COMPRESSION

This key is used to define whether to have intermediate file compressed

See Also:

Constant Field Values

PIG_TEMP_FILE_COMPRESSION_STORAGE

public static final java.lang.String PIG_TEMP_FILE_COMPRESSION_STORAGE

This key is used to set the storage type used by intermediate file storage If pig.tmpfilecompression, default storage used is TFileStorage. This can be overriden to use SequenceFileInterStorage by setting following property to "seqfile".

See Also:

Constant Field Values

PIG_TEMP_FILE_COMPRESSION_CODEC

public static final java.lang.String PIG_TEMP_FILE_COMPRESSION_CODEC

Compression codec used by intermediate storage TFileStorage only support gzip and lzo.

See Also:

Constant Field Values

PIG_DELETE_TEMP_FILE

public static final java.lang.String PIG_DELETE_TEMP_FILE

This key is used to define whether to delete intermediate files of Hadoop jobs.

See Also:

Constant Field Values

PIG_POISSON_SAMPLER_SAMPLE_RATE

public static final java.lang.String PIG_POISSON_SAMPLER_SAMPLE_RATE

For a given mean and a confidence, a sample rate is obtained from a poisson udf

See Also:

Constant Field Values

PIG_RANDOM_SAMPLER_SAMPLE_SIZE

public static final java.lang.String PIG_RANDOM_SAMPLER_SAMPLE_SIZE

This key used to control the sample size of RandomeSampleLoader for order-by. The default value is 100 rows per task.

See Also:

Constant Field Values

PIG_DEFAULT_LOAD_FUNC

public static final java.lang.String PIG_DEFAULT_LOAD_FUNC

This key is used to define the default load func. Pig will fallback on PigStorage as default in case this is undefined.

See Also:

Constant Field Values

PIG_DEFAULT_STORE_FUNC

public static final java.lang.String PIG_DEFAULT_STORE_FUNC

This key is used to define the default store func. Pig will fallback on PigStorage as default in case this is undefined.

See Also:

Constant Field Values

PIG_SCRIPT_INFO_ENABLED

public static final java.lang.String PIG_SCRIPT_INFO_ENABLED

This key is used to turn off the inclusion of settings in the jobs.

See Also:

Constant Field Values

PIG_SCRIPT_MAX_SIZE

public static final java.lang.String PIG_SCRIPT_MAX_SIZE

Controls the size of Pig script stored in job xml.

See Also:

Constant Field Values

PIG_USER_CACHE_ENABLED

public static final java.lang.String PIG_USER_CACHE_ENABLED

This key is turn on the user level cache

See Also:

Constant Field Values

PIG_USER_CACHE_LOCATION

public static final java.lang.String PIG_USER_CACHE_LOCATION

Location where additional jars are cached for the user Additional jar will be cached under PIG_USER_CACHE_LOCATION/${user.name}/.pigcache and will be re-used across the jobs run by the user if the jar has not changed

See Also:

Constant Field Values

PIG_BLACKLIST

public static final java.lang.String PIG_BLACKLIST

Comma-delimited entries of commands/operators that must be disallowed. This is a security feature to be used by administrators to block use of commands by users. For eg, an admin might like to block all filesystem commands and setting configs in pig script. In which case, the entry would be "pig.blacklist=fs,set"

See Also:

Constant Field Values

PIG_WHITELIST

public static final java.lang.String PIG_WHITELIST

Comma-delimited entries of commands/operators that must be allowed. This is a security feature to be used by administrators to block use of commands by users that are not a part of the whitelist. For eg, an admin might like to allow only LOAD, STORE, FILTER, GROUP in pig script. In which case, the entry would be "pig.whitelist=load,store,filter,group"

See Also:

Constant Field Values

PIG_NO_TASK_REPORT

public static final java.lang.String PIG_NO_TASK_REPORT

This key is used to turns off use of task reports in job statistics.

See Also:

Constant Field Values

PIG_TEZ_SESSION_REUSE

public static final java.lang.String PIG_TEZ_SESSION_REUSE

This key is used to define whether to reuse AM in Tez jobs.

See Also:

Constant Field Values

PIG_TEZ_DAG_STATUS_REPORT_INTERVAL

public static final java.lang.String PIG_TEZ_DAG_STATUS_REPORT_INTERVAL

This key is used to configure the interval of dag status report in seconds. Default is 20

See Also:

Constant Field Values

OPT_FETCH

@Deprecated
public static final java.lang.String OPT_FETCH

Deprecated. use PIG_OPT_FETCH instead. Will be removed in Pig 0.16

See Also:

Constant Field Values

PROP_CACHEDBAG_MEMUSAGE

@Deprecated
public static final java.lang.String PROP_CACHEDBAG_MEMUSAGE

Deprecated. use PIG_CACHEDBAG_MEMUSAGE instead. Will be removed in Pig 0.16

See Also:

Constant Field Values

PROP_EXEC_MAP_PARTAGG

@Deprecated
public static final java.lang.String PROP_EXEC_MAP_PARTAGG

Deprecated. use PIG_EXEC_MAP_PARTAGG instead. Will be removed in Pig 0.16

See Also:

Constant Field Values

PARTAGG_MINREDUCTION

@Deprecated
public static final java.lang.String PARTAGG_MINREDUCTION

Deprecated. use PIG_EXEC_MAP_PARTAGG_MINREDUCTION instead. Will be removed in Pig 0.16

See Also:

Constant Field Values

PROP_NO_COMBINER

@Deprecated
public static final java.lang.String PROP_NO_COMBINER

Deprecated. use #PROP_NO_COMBINER1 instead. Will be removed in Pig 0.16

See Also:

Constant Field Values

SHOULD_USE_SCHEMA_TUPLE

@Deprecated
public static final java.lang.String SHOULD_USE_SCHEMA_TUPLE

Deprecated.

See Also:

Constant Field Values

SCHEMA_TUPLE_SHOULD_USE_IN_UDF

@Deprecated
public static final java.lang.String SCHEMA_TUPLE_SHOULD_USE_IN_UDF

Deprecated.

See Also:

Constant Field Values

SCHEMA_TUPLE_SHOULD_USE_IN_FOREACH

@Deprecated
public static final java.lang.String SCHEMA_TUPLE_SHOULD_USE_IN_FOREACH

Deprecated.

See Also:

Constant Field Values

SCHEMA_TUPLE_SHOULD_USE_IN_FRJOIN

@Deprecated
public static final java.lang.String SCHEMA_TUPLE_SHOULD_USE_IN_FRJOIN

Deprecated.

See Also:

Constant Field Values

SCHEMA_TUPLE_SHOULD_USE_IN_MERGEJOIN

@Deprecated
public static final java.lang.String SCHEMA_TUPLE_SHOULD_USE_IN_MERGEJOIN

Deprecated.

See Also:

Constant Field Values

SCHEMA_TUPLE_SHOULD_ALLOW_FORCE

@Deprecated
public static final java.lang.String SCHEMA_TUPLE_SHOULD_ALLOW_FORCE

Deprecated.

See Also:

Constant Field Values

INSERT_ENABLED

@Deprecated
public static final java.lang.String INSERT_ENABLED

Deprecated. use PIG_SCRIPT_INFO_ENABLED instead. Will be removed in Pig 0.16

See Also:

Constant Field Values

MAX_SCRIPT_SIZE

@Deprecated
public static final java.lang.String MAX_SCRIPT_SIZE

Deprecated. use PIG_SCRIPT_MAX_SIZE instead. Will be removed in Pig 0.16

See Also:

Constant Field Values