- All Implemented Interfaces:
- Serializable
public class SparkEngineConf
extends Object
implements Serializable
The object of SparkEngineConf is to solve the initialization problem of PigContext.properties.get("udf.import.list"),
UDFContext#udfConfs, UDFContext#clientSysProps in spark mode. These variables can not be
serialized because they are ThreadLocal variables. In MR mode, they are serialized in JobConfiguration
in JobControlCompiler#getJob and deserialized by JobConfiguration in PigGenericMapBase#setup. But there is no
setup() in spark like what in mr, so these variables can be correctly deserialized before spark programs call them.
Here we use following solution to solve:
these variables are saved in SparkEngineConf#writeObject and available and then initialized
in SparkEngineConf#readObject in spark executor thread.
- See Also:
- Serialized Form