public class HiveColumnarLoader extends FileInputLoadFunc implements LoadMetadata, LoadPushDown
Hive Type | Pig Type from DataType |
---|---|
string | CHARARRAY |
int | INTEGER |
bigint or long | LONG |
float | float |
double | DOUBLE |
boolean | BOOLEAN |
byte | BYTE |
array | TUPLE |
map | MAP |
/user/hive/warehouse/mytable /year=2010/month=02/day=01The mytable schema is (id int,name string).
a = LOAD 'file' USING HiveColumnarLoader("uid bigint, ts long, arr array, m map");
-- to reference the fields
b = FOREACH GENERATE a.uid, a.ts, a.arr, a.m;
Usage 2:
To load a hive table: uid bigint, ts long, arr ARRAY
a = LOAD 'file' USING HiveColumnarLoader("uid bigint, ts long, arr array, m map", "2009-10-01:2009-10-02");
-- to reference the fields
b = FOREACH GENERATE a.uid, a.ts, a.arr, a.m;
a = LOAD 'file' USING HiveColumnarLoader("uid bigint, ts long, arr array, m map");
f = FILTER a BY daydate>='2009-10-01' AND daydate >='2009-10-02';
Usage 3:
To load a hive table: uid bigint, ts long, arr ARRAY
a = LOAD 'file' USING HiveColumnarLoader("uid bigint, ts long, arr array, m map");
f = FILTER a BY daydate>='2009-10-01' AND daydate >='2009-10-02';
-- to reference the fields
b = FOREACH a GENERATE uid, ts, arr, m;
Issues
Table schema definitionmytable/hour=00 mytable/day=01/hour=00
LoadPushDown.OperatorSet, LoadPushDown.RequiredField, LoadPushDown.RequiredFieldList, LoadPushDown.RequiredFieldResponse
Modifier and Type | Field and Description |
---|---|
static String |
DATE_RANGE |
protected static org.apache.commons.logging.Log |
LOG |
protected static Pattern |
pcols
Regex to filter out column names
|
static String |
PROJECTION_ID |
protected TupleFactory |
tupleFactory |
Constructor and Description |
---|
HiveColumnarLoader(String table_schema)
Table schema should be a space and comma separated string describing the
Hive schema.
For example uid BIGINT, pid long, means 1 column of uid type BIGINT and one column of pid type LONG. The types are not case sensitive. |
HiveColumnarLoader(String table_schema,
String dateRange)
This constructor is for backward compatibility.
|
HiveColumnarLoader(String table_schema,
String dateRange,
String columns)
This constructor is for backward compatibility.
|
Modifier and Type | Method and Description |
---|---|
List<LoadPushDown.OperatorSet> |
getFeatures()
Determine the operators that can be pushed to the loader.
|
org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.hive.serde2.columnar.BytesRefArrayWritable> |
getInputFormat()
This will be called during planning on the front end.
|
Tuple |
getNext()
Retrieves the next tuple to be processed.
|
String[] |
getPartitionKeys(String location,
org.apache.hadoop.mapreduce.Job job)
Find what columns are partition keys for this input.
|
ResourceSchema |
getSchema(String location,
org.apache.hadoop.mapreduce.Job job)
Get a schema for the data to be loaded.
|
ResourceStatistics |
getStatistics(String location,
org.apache.hadoop.mapreduce.Job job)
Get statistics about the data to be loaded.
|
void |
prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader,
PigSplit split)
Initializes LoadFunc for reading data.
|
LoadPushDown.RequiredFieldResponse |
pushProjection(LoadPushDown.RequiredFieldList requiredFieldList)
Indicate to the loader fields that will be needed.
|
void |
setLocation(String location,
org.apache.hadoop.mapreduce.Job job)
Communicate to the loader the location of the object(s) being loaded.
|
void |
setPartitionFilter(Expression partitionFilter)
Set the filter for partitioning.
|
void |
setUDFContextSignature(String signature)
This method will be called by Pig both in the front end and back end to
pass a unique signature to the
LoadFunc . |
getSplitComparable
getAbsolutePath, getCacheFiles, getLoadCaster, getPathStrings, getShipFiles, join, relativeToAbsolutePath, warn
public static final String PROJECTION_ID
public static final String DATE_RANGE
protected static final Pattern pcols
protected static final org.apache.commons.logging.Log LOG
protected TupleFactory tupleFactory
public HiveColumnarLoader(String table_schema)
table_schema
- This property cannot be nullpublic HiveColumnarLoader(String table_schema, String dateRange, String columns)
table_schema
- This property cannot be nulldateRange
- Stringcolumns
- String not used any morepublic HiveColumnarLoader(String table_schema, String dateRange)
table_schema
- This property cannot be nulldateRange
- Stringpublic org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.hive.serde2.columnar.BytesRefArrayWritable> getInputFormat() throws IOException
LoadFunc
getInputFormat
in class LoadFunc
IOException
- if there is an exception during InputFormat
constructionpublic Tuple getNext() throws IOException
LoadFunc
getNext
in class LoadFunc
IOException
- if there is an exception while retrieving the next
tuplepublic void prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader, PigSplit split) throws IOException
LoadFunc
prepareToRead
in class LoadFunc
reader
- RecordReader
to be used by this instance of the LoadFuncsplit
- The input PigSplit
to processIOException
- if there is an exception during initializationpublic void setLocation(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadFunc
LoadFunc.relativeToAbsolutePath(String, Path)
. Implementations
should use this method to communicate the location (and any other information)
to its underlying InputFormat through the Job object.
This method will be called in the frontend and backend multiple times. Implementations
should bear in mind that this method is called multiple times and should
ensure there are no inconsistent side effects due to the multiple calls.setLocation
in class LoadFunc
location
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, Path)
job
- the Job
object
store or retrieve earlier stored information from the UDFContext
IOException
- if the location is not valid.public String[] getPartitionKeys(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadMetadata
getPartitionKeys
in interface LoadMetadata
location
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)
job
- The Job
object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration()
and not to set/query
any runtime job information.IOException
- if an exception occurs while retrieving partition keyspublic ResourceSchema getSchema(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadMetadata
getSchema
in interface LoadMetadata
location
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)
job
- The Job
object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration()
and not to set/query
any runtime job information.IOException
- if an exception occurs while determining the schemapublic ResourceStatistics getStatistics(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadMetadata
LoadFunc
, then LoadFunc.setLocation(String, org.apache.hadoop.mapreduce.Job)
is guaranteed to be called before this method.getStatistics
in interface LoadMetadata
location
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)
job
- The Job
object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration()
and not to set/query
any runtime job information.IOException
- if an exception occurs while retrieving statisticspublic void setPartitionFilter(Expression partitionFilter) throws IOException
LoadMetadata
LoadMetadata.getPartitionKeys(String, Job)
, then this method is not
called by Pig runtime. This method is also not called by the Pig runtime
if there are no partition filter conditions.setPartitionFilter
in interface LoadMetadata
partitionFilter
- that describes filter for partitioningIOException
- if the filter is not compatible with the storage
mechanism or contains non-partition fields.public List<LoadPushDown.OperatorSet> getFeatures()
LoadPushDown
getFeatures
in interface LoadPushDown
public LoadPushDown.RequiredFieldResponse pushProjection(LoadPushDown.RequiredFieldList requiredFieldList) throws FrontendException
LoadPushDown
pushProjection
in interface LoadPushDown
requiredFieldList
- RequiredFieldList indicating which columns will be needed.
This structure is read only. User cannot make change to it inside pushProjection.FrontendException
public void setUDFContextSignature(String signature)
LoadFunc
LoadFunc
. The signature can be used
to store into the UDFContext
any information which the
LoadFunc
needs to store between various method invocations in the
front end and back end. A use case is to store LoadPushDown.RequiredFieldList
passed to it in LoadPushDown.pushProjection(RequiredFieldList)
for
use in the back end before returning tuples in LoadFunc.getNext()
.
This method will be call before other methods in LoadFunc
setUDFContextSignature
in class LoadFunc
signature
- a unique signature to identify this LoadFuncCopyright © 2007-2017 The Apache Software Foundation