@InterfaceAudience.Private public class MRToTezHelper extends java.lang.Object
Modifier and Type | Method and Description |
---|---|
static void |
processMRSettings(org.apache.hadoop.conf.Configuration tezConf,
org.apache.hadoop.conf.Configuration mrConf)
Process the mapreduce configuration settings and
- copy as is the still required ones (like those used by FileInputFormat/FileOutputFormat)
- convert and set equivalent tez runtime settings
- handle compression related settings
|
static void |
setVertexConfig(org.apache.tez.dag.api.Vertex vertex,
boolean isMapVertex,
org.apache.hadoop.conf.Configuration conf)
Set config with Scope.Vertex in TezConfiguration on the vertex
|
static void |
translateMRSettingsForTezAM(org.apache.tez.dag.api.TezConfiguration dagAMConf) |
static void |
updateLocalResourcesForInputSplits(org.apache.hadoop.fs.FileSystem fs,
org.apache.tez.mapreduce.hadoop.InputSplitInfo inputSplitInfo,
java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
Exact copy of private method from from org.apache.tez.mapreduce.hadoop.MRInputHelpers
Update provided localResources collection with the required local
resources needed by MapReduce tasks with respect to Input splits.
|
static org.apache.tez.mapreduce.hadoop.InputSplitInfoDisk |
writeInputSplitInfoToDisk(org.apache.tez.mapreduce.hadoop.InputSplitInfoMem infoMem,
org.apache.hadoop.fs.Path inputSplitsDir,
org.apache.hadoop.mapred.JobConf jobConf,
org.apache.hadoop.fs.FileSystem fs,
org.apache.tez.mapreduce.protos.MRRuntimeProtos.MRSplitsProto splitsProto)
Write input splits (job.split and job.splitmetainfo) to disk.
|
public static void translateMRSettingsForTezAM(org.apache.tez.dag.api.TezConfiguration dagAMConf)
public static void setVertexConfig(org.apache.tez.dag.api.Vertex vertex, boolean isMapVertex, org.apache.hadoop.conf.Configuration conf)
vertex
- Vertex on which config is to be setisMapVertex
- Whether map or reduce vertex. i.e root or intermediate/leaf vertexconf
- Config that contains the tez or equivalent mapreduce settings.public static void processMRSettings(org.apache.hadoop.conf.Configuration tezConf, org.apache.hadoop.conf.Configuration mrConf)
tezConf
- Configuration on which the mapreduce settings will have to be transferredmrConf
- Configuration that contains mapreduce settingspublic static org.apache.tez.mapreduce.hadoop.InputSplitInfoDisk writeInputSplitInfoToDisk(org.apache.tez.mapreduce.hadoop.InputSplitInfoMem infoMem, org.apache.hadoop.fs.Path inputSplitsDir, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.fs.FileSystem fs, org.apache.tez.mapreduce.protos.MRRuntimeProtos.MRSplitsProto splitsProto) throws java.io.IOException, java.lang.InterruptedException
infoMem
- inputSplitsDir
- jobConf
- fs
- splitsProto
- MRSplitsProto containing already serialized splitsjava.io.IOException
java.lang.InterruptedException
public static void updateLocalResourcesForInputSplits(org.apache.hadoop.fs.FileSystem fs, org.apache.tez.mapreduce.hadoop.InputSplitInfo inputSplitInfo, java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> localResources) throws java.io.IOException
fs
- Filesystem instance to access status of splits related filesinputSplitInfo
- Information on location of split fileslocalResources
- LocalResources collection to be updatedjava.io.IOException
Copyright © 2007-2025 The Apache Software Foundation