org.apache.pig.backend.hadoop.executionengine.spark

Class SparkPigSplit.SparkPigSplitsUtils

java.lang.Object

org.apache.pig.backend.hadoop.executionengine.spark.SparkPigSplit.SparkPigSplitsUtils

Enclosing interface:

SparkPigSplit

public static class SparkPigSplit.SparkPigSplitsUtils
extends java.lang.Object

Constructor Summary

Constructors

Constructor and Description
SparkPigSplitsUtils()

Method Summary

Modifier and Type	Method and Description
static void	writeConfigForPigSplits(org.apache.hadoop.conf.Configuration originalConf, java.io.DataOutput os) Writes a subset of the originalConf into the output stream os.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkPigSplitsUtils

public SparkPigSplitsUtils()

Method Detail

writeConfigForPigSplits

public static void writeConfigForPigSplits(org.apache.hadoop.conf.Configuration originalConf,
                                           java.io.DataOutput os)
                                    throws java.io.IOException

Writes a subset of the originalConf into the output stream os. Only keys in PIG_SPLIT_CONFIG_KEYS are considered due to optimization purposes. During deseralization on a Spark executor we need to take care of setting the configuration manually because Spark only sets an empty Configuration instance on the PigSplit.

Parameters:

originalConf -

os -

Throws:

java.io.IOException