public class AvroStorage extends LoadFunc implements StoreFuncInterface, LoadMetadata, LoadPushDown, StoreResources
LoadPushDown.OperatorSet, LoadPushDown.RequiredField, LoadPushDown.RequiredFieldList, LoadPushDown.RequiredFieldResponse
Modifier and Type | Field and Description |
---|---|
protected boolean |
allowRecursive |
protected boolean |
doubleColonsToDoubleUnderscores |
static String |
INPUT_AVRO_SCHEMA
Pig property name for the input avro schema.
|
protected org.apache.commons.logging.Log |
log |
static String |
OUTPUT_AVRO_SCHEMA
Pig property name for the output avro schema.
|
protected LoadPushDown.RequiredFieldList |
requiredFieldList
List of required fields passed by pig in a push down projection.
|
protected org.apache.avro.Schema |
schema |
protected String |
udfContextSignature
Context signature for this UDF instance.
|
Constructor and Description |
---|
AvroStorage()
Creates new instance of Pig Storage function, without specifying
the schema.
|
AvroStorage(String sn)
Creates new instance of Pig Storage function.
|
AvroStorage(String sn,
String opts)
Creates new instance of AvroStorage function, specifying output schema
properties.
|
Modifier and Type | Method and Description |
---|---|
void |
checkSchema(ResourceSchema rs)
Set the schema for data to be stored.
|
void |
cleanupOnFailure(String location,
org.apache.hadoop.mapreduce.Job job)
This method will be called by Pig if the job which contains this store
fails.
|
void |
cleanupOnSuccess(String location,
org.apache.hadoop.mapreduce.Job job)
This method will be called by Pig if the job which contains this store
is successful, and some cleanup of intermediate resources is required.
|
org.apache.avro.Schema |
getAvroSchema(org.apache.hadoop.fs.Path[] p,
org.apache.hadoop.mapreduce.Job job)
Reads the avro schemas at the specified location.
|
protected org.apache.avro.Schema |
getAvroSchema(String location,
org.apache.hadoop.mapreduce.Job job)
Reads the avro schema at the specified location.
|
List<LoadPushDown.OperatorSet> |
getFeatures()
Determine the operators that can be pushed to the loader.
|
org.apache.avro.Schema |
getInputAvroSchema()
Helper function reads the input avro schema from the UDF
Properties.
|
org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.avro.generic.GenericData.Record> |
getInputFormat()
This will be called during planning on the front end.
|
Tuple |
getNext()
Retrieves the next tuple to be processed.
|
protected org.apache.avro.Schema |
getOutputAvroSchema()
Utility function that gets the output schema from the udf
properties for this instance of the store function.
|
org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.NullWritable,Object> |
getOutputFormat()
Return the OutputFormat associated with StoreFuncInterface.
|
String[] |
getPartitionKeys(String location,
org.apache.hadoop.mapreduce.Job job)
Find what columns are partition keys for this input.
|
protected Properties |
getProperties()
Internal function for getting the Properties object associated with
this UDF instance.
|
protected Properties |
getProperties(Class c,
String signature)
Internal function for getting the Properties object associated with
this UDF instance.
|
ResourceSchema |
getSchema(String location,
org.apache.hadoop.mapreduce.Job job)
Get a schema for the data to be loaded.
|
List<String> |
getShipFiles()
Allow a LoadFunc to specify a list of files located locally and would like to ship to backend
(through distributed cache).
|
ResourceStatistics |
getStatistics(String location,
org.apache.hadoop.mapreduce.Job job)
Get statistics about the data to be loaded.
|
void |
prepareToRead(org.apache.hadoop.mapreduce.RecordReader r,
PigSplit s)
Initializes LoadFunc for reading data.
|
void |
prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter w)
Initialize StoreFuncInterface to write data.
|
LoadPushDown.RequiredFieldResponse |
pushProjection(LoadPushDown.RequiredFieldList rfl)
Indicate to the loader fields that will be needed.
|
void |
putNext(Tuple t)
Write a tuple to the data store.
|
String |
relToAbsPathForStoreLocation(String location,
org.apache.hadoop.fs.Path curDir)
This method is called by the Pig runtime in the front end to convert the
output location to an absolute path if the location is relative.
|
protected void |
setInputAvroSchema(org.apache.avro.Schema s)
Sets the input avro schema to .
|
void |
setLocation(String location,
org.apache.hadoop.mapreduce.Job job)
Communicate to the loader the location of the object(s) being loaded.
|
protected void |
setOutputAvroSchema(org.apache.avro.Schema s)
Sets the output avro schema to .
|
void |
setPartitionFilter(Expression partitionFilter)
Set the filter for partitioning.
|
void |
setStoreFuncUDFContextSignature(String signature)
This method will be called by Pig both in the front end and back end to
pass a unique signature to the
StoreFuncInterface which it can use to store
information in the UDFContext which it needs to store between
various method invocations in the front end and back end. |
void |
setStoreLocation(String location,
org.apache.hadoop.mapreduce.Job job)
Communicate to the storer the location where the data needs to be stored.
|
void |
setUDFContextSignature(String signature)
This method will be called by Pig both in the front end and back end to
pass a unique signature to the
LoadFunc . |
getAbsolutePath, getCacheFiles, getLoadCaster, getPathStrings, join, relativeToAbsolutePath, warn
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getCacheFiles
protected boolean allowRecursive
protected boolean doubleColonsToDoubleUnderscores
protected org.apache.avro.Schema schema
protected final org.apache.commons.logging.Log log
protected String udfContextSignature
public static final String OUTPUT_AVRO_SCHEMA
public static final String INPUT_AVRO_SCHEMA
protected LoadPushDown.RequiredFieldList requiredFieldList
public AvroStorage()
public AvroStorage(String sn)
sn
- Specifies the input/output schema or record name.public AvroStorage(String sn, String opts)
sn
- Specifies the input/output schema or record name.opts
- Options for AvroStorage:
-namespace
Namespace for an automatically generated
output schema.-schemafile
Specifies URL for avro schema file
from which to read the input schema (can be local file, hdfs,
url, etc).-schemaclass
Specifies fully qualified class name for avro
class in your classpath which implements GenericContainer.-examplefile
Specifies URL for avro data file from
which to copy the input schema (can be local file, hdfs, url, etc).-allowrecursive
Option to allow recursive schema
definitions (default is false).-doublecolons
Option to translate Pig schema names
with double colons to names with double underscores (default is false).public final void setUDFContextSignature(String signature)
LoadFunc
LoadFunc
. The signature can be used
to store into the UDFContext
any information which the
LoadFunc
needs to store between various method invocations in the
front end and back end. A use case is to store LoadPushDown.RequiredFieldList
passed to it in LoadPushDown.pushProjection(RequiredFieldList)
for
use in the back end before returning tuples in LoadFunc.getNext()
.
This method will be call before other methods in LoadFunc
setUDFContextSignature
in class LoadFunc
signature
- a unique signature to identify this LoadFuncprotected final Properties getProperties()
protected final Properties getProperties(Class c, String signature)
c
- Class of this UDFsignature
- Signature stringpublic final ResourceSchema getSchema(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadMetadata
getSchema
in interface LoadMetadata
location
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)
job
- The Job
object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration()
and not to set/query
any runtime job information.IOException
- if an exception occurs while determining the schemaprotected final org.apache.avro.Schema getAvroSchema(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
location
- Location of filejob
- Hadoop job objectIOException
public org.apache.avro.Schema getAvroSchema(org.apache.hadoop.fs.Path[] p, org.apache.hadoop.mapreduce.Job job) throws IOException
p
- Location of filejob
- Hadoop job objectIOException
public final ResourceStatistics getStatistics(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadMetadata
LoadFunc
, then LoadFunc.setLocation(String, org.apache.hadoop.mapreduce.Job)
is guaranteed to be called before this method.getStatistics
in interface LoadMetadata
location
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)
job
- The Job
object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration()
and not to set/query
any runtime job information.IOException
- if an exception occurs while retrieving statisticspublic final String[] getPartitionKeys(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadMetadata
getPartitionKeys
in interface LoadMetadata
location
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)
job
- The Job
object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration()
and not to set/query
any runtime job information.IOException
- if an exception occurs while retrieving partition keyspublic void setPartitionFilter(Expression partitionFilter) throws IOException
LoadMetadata
LoadMetadata.getPartitionKeys(String, Job)
, then this method is not
called by Pig runtime. This method is also not called by the Pig runtime
if there are no partition filter conditions.setPartitionFilter
in interface LoadMetadata
partitionFilter
- that describes filter for partitioningIOException
- if the filter is not compatible with the storage
mechanism or contains non-partition fields.public final String relToAbsPathForStoreLocation(String location, org.apache.hadoop.fs.Path curDir) throws IOException
StoreFuncInterface
LoadFunc.getAbsolutePath(java.lang.String, org.apache.hadoop.fs.Path)
provides a default
implementation for hdfs and hadoop local file system and it can be used
to implement this method.relToAbsPathForStoreLocation
in interface StoreFuncInterface
location
- location as provided in the "store" statement of the scriptcurDir
- the current working direction based on any "cd" statements
in the script before the "store" statement. If there are no "cd" statements
in the script, this would be the home directory -
/user/
IOException
- if the conversion is not possiblepublic org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.NullWritable,Object> getOutputFormat() throws IOException
StoreFuncInterface
getOutputFormat
in interface StoreFuncInterface
OutputFormat
associated with StoreFuncInterfaceIOException
- if an exception occurs while constructing the
OutputFormatpublic final void setStoreLocation(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreFuncInterface
StoreFuncInterface
here is the
return value of StoreFuncInterface.relToAbsPathForStoreLocation(String, Path)
This method will be called in the frontend and backend multiple times. Implementations
should bear in mind that this method is called multiple times and should
ensure there are no inconsistent side effects due to the multiple calls.
StoreFuncInterface.checkSchema(ResourceSchema)
will be called before any call to
StoreFuncInterface.setStoreLocation(String, Job)
.setStoreLocation
in interface StoreFuncInterface
location
- Location returned by
StoreFuncInterface.relToAbsPathForStoreLocation(String, Path)
job
- The Job
objectIOException
- if the location is not valid.public final void checkSchema(ResourceSchema rs) throws IOException
StoreFuncInterface
checkSchema
in interface StoreFuncInterface
rs
- to be checkedIOException
- if this schema is not acceptable. It should include
a detailed error message indicating what is wrong with the schema.protected final void setOutputAvroSchema(org.apache.avro.Schema s)
s
- An Avro schemaprotected final org.apache.avro.Schema getOutputAvroSchema()
public final void prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter w) throws IOException
StoreFuncInterface
prepareToWrite
in interface StoreFuncInterface
w
- RecordWriter to use.IOException
- if an exception occurs during initializationpublic final void putNext(Tuple t) throws IOException
StoreFuncInterface
putNext
in interface StoreFuncInterface
t
- the tuple to store.IOException
- if an exception occurs during the writepublic final void setStoreFuncUDFContextSignature(String signature)
StoreFuncInterface
StoreFuncInterface
which it can use to store
information in the UDFContext
which it needs to store between
various method invocations in the front end and back end. This is necessary
because in a Pig Latin script with multiple stores, the different
instances of store functions need to be able to find their (and only their)
data in the UDFContext object.setStoreFuncUDFContextSignature
in interface StoreFuncInterface
signature
- a unique signature to identify this StoreFuncInterfacepublic final void cleanupOnFailure(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreFuncInterface
cleanupOnFailure
in interface StoreFuncInterface
location
- Location returned by
StoreFuncInterface.relToAbsPathForStoreLocation(String, Path)
job
- The Job
object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration()
and not to set/query
any runtime job information.IOException
public void setLocation(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadFunc
LoadFunc.relativeToAbsolutePath(String, Path)
. Implementations
should use this method to communicate the location (and any other information)
to its underlying InputFormat through the Job object.
This method will be called in the frontend and backend multiple times. Implementations
should bear in mind that this method is called multiple times and should
ensure there are no inconsistent side effects due to the multiple calls.setLocation
in class LoadFunc
location
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, Path)
job
- the Job
object
store or retrieve earlier stored information from the UDFContext
IOException
- if the location is not valid.protected final void setInputAvroSchema(org.apache.avro.Schema s)
s
- The specified schemapublic final org.apache.avro.Schema getInputAvroSchema()
public org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.avro.generic.GenericData.Record> getInputFormat() throws IOException
LoadFunc
getInputFormat
in class LoadFunc
IOException
- if there is an exception during InputFormat
constructionLoadFunc.getInputFormat()
public final void prepareToRead(org.apache.hadoop.mapreduce.RecordReader r, PigSplit s) throws IOException
LoadFunc
prepareToRead
in class LoadFunc
r
- RecordReader
to be used by this instance of the LoadFuncs
- The input PigSplit
to processIOException
- if there is an exception during initializationpublic final Tuple getNext() throws IOException
LoadFunc
getNext
in class LoadFunc
IOException
- if there is an exception while retrieving the next
tuplepublic void cleanupOnSuccess(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreFuncInterface
cleanupOnSuccess
in interface StoreFuncInterface
location
- Location returned by
StoreFuncInterface.relToAbsPathForStoreLocation(String, Path)
job
- The Job
object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration()
and not to set/query
any runtime job information.IOException
public List<LoadPushDown.OperatorSet> getFeatures()
LoadPushDown
getFeatures
in interface LoadPushDown
public LoadPushDown.RequiredFieldResponse pushProjection(LoadPushDown.RequiredFieldList rfl) throws FrontendException
LoadPushDown
pushProjection
in interface LoadPushDown
rfl
- RequiredFieldList indicating which columns will be needed.
This structure is read only. User cannot make change to it inside pushProjection.FrontendException
public List<String> getShipFiles()
LoadFunc
FuncUtils
for utility function to facilitate it
The default implementation returns null.getShipFiles
in interface StoreResources
getShipFiles
in class LoadFunc
Copyright © 2007-2017 The Apache Software Foundation