org.apache.pig.backend.hadoop.executionengine.tez.util

Class MRToTezHelper

java.lang.Object

org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper

@InterfaceAudience.Private
public class MRToTezHelper
extends java.lang.Object

Method Summary

Modifier and Type	Method and Description
static void	processMRSettings(org.apache.hadoop.conf.Configuration tezConf, org.apache.hadoop.conf.Configuration mrConf) Process the mapreduce configuration settings and - copy as is the still required ones (like those used by FileInputFormat/FileOutputFormat) - convert and set equivalent tez runtime settings - handle compression related settings
static void	setVertexConfig(org.apache.tez.dag.api.Vertex vertex, boolean isMapVertex, org.apache.hadoop.conf.Configuration conf) Set config with Scope.Vertex in TezConfiguration on the vertex
static void	translateMRSettingsForTezAM(org.apache.tez.dag.api.TezConfiguration dagAMConf)
static void	updateLocalResourcesForInputSplits(org.apache.hadoop.fs.FileSystem fs, org.apache.tez.mapreduce.hadoop.InputSplitInfo inputSplitInfo, java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> localResources) Exact copy of private method from from org.apache.tez.mapreduce.hadoop.MRInputHelpers Update provided localResources collection with the required local resources needed by MapReduce tasks with respect to Input splits.
static org.apache.tez.mapreduce.hadoop.InputSplitInfoDisk	writeInputSplitInfoToDisk(org.apache.tez.mapreduce.hadoop.InputSplitInfoMem infoMem, org.apache.hadoop.fs.Path inputSplitsDir, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.fs.FileSystem fs, org.apache.tez.mapreduce.protos.MRRuntimeProtos.MRSplitsProto splitsProto) Write input splits (job.split and job.splitmetainfo) to disk.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

translateMRSettingsForTezAM

public static void translateMRSettingsForTezAM(org.apache.tez.dag.api.TezConfiguration dagAMConf)

setVertexConfig

public static void setVertexConfig(org.apache.tez.dag.api.Vertex vertex,
                                   boolean isMapVertex,
                                   org.apache.hadoop.conf.Configuration conf)

Set config with Scope.Vertex in TezConfiguration on the vertex

Parameters:

vertex - Vertex on which config is to be set

isMapVertex - Whether map or reduce vertex. i.e root or intermediate/leaf vertex

conf - Config that contains the tez or equivalent mapreduce settings.

processMRSettings

public static void processMRSettings(org.apache.hadoop.conf.Configuration tezConf,
                                     org.apache.hadoop.conf.Configuration mrConf)

Process the mapreduce configuration settings and - copy as is the still required ones (like those used by FileInputFormat/FileOutputFormat) - convert and set equivalent tez runtime settings - handle compression related settings

Parameters:

tezConf - Configuration on which the mapreduce settings will have to be transferred

mrConf - Configuration that contains mapreduce settings

writeInputSplitInfoToDisk

public static org.apache.tez.mapreduce.hadoop.InputSplitInfoDisk writeInputSplitInfoToDisk(org.apache.tez.mapreduce.hadoop.InputSplitInfoMem infoMem,
                                                                                           org.apache.hadoop.fs.Path inputSplitsDir,
                                                                                           org.apache.hadoop.mapred.JobConf jobConf,
                                                                                           org.apache.hadoop.fs.FileSystem fs,
                                                                                           org.apache.tez.mapreduce.protos.MRRuntimeProtos.MRSplitsProto splitsProto)
                                                                                    throws java.io.IOException,
                                                                                           java.lang.InterruptedException

Write input splits (job.split and job.splitmetainfo) to disk. It uses already serialized splits from given MRSplitsProto

Parameters:

infoMem -

inputSplitsDir -

jobConf -

fs -

splitsProto - MRSplitsProto containing already serialized splits

Returns:

Throws:

java.io.IOException

java.lang.InterruptedException

updateLocalResourcesForInputSplits

public static void updateLocalResourcesForInputSplits(org.apache.hadoop.fs.FileSystem fs,
                                                      org.apache.tez.mapreduce.hadoop.InputSplitInfo inputSplitInfo,
                                                      java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
                                               throws java.io.IOException

Exact copy of private method from from org.apache.tez.mapreduce.hadoop.MRInputHelpers Update provided localResources collection with the required local resources needed by MapReduce tasks with respect to Input splits.

Parameters:

fs - Filesystem instance to access status of splits related files

inputSplitInfo - Information on location of split files

localResources - LocalResources collection to be updated

Throws:

java.io.IOException