public class JsonStorage extends StoreFunc implements StoreMetadata, StoreResources
Modifier and Type | Field and Description |
---|---|
protected ResourceSchema |
schema |
protected org.apache.hadoop.mapreduce.RecordWriter |
writer |
Constructor and Description |
---|
JsonStorage() |
Modifier and Type | Method and Description |
---|---|
void |
checkSchema(ResourceSchema s)
Set the schema for data to be stored.
|
ResourceSchema |
fixSchema(ResourceSchema s) |
List<String> |
getCacheFiles()
Allow a StoreFunc to specify a list of files it would like placed in the distributed
cache.
|
org.apache.hadoop.mapreduce.OutputFormat |
getOutputFormat()
Return the OutputFormat associated with StoreFunc.
|
List<String> |
getShipFiles()
Allow a StoreFunc to specify a list of files located locally and would like to ship to backend
(through distributed cache).
|
void |
prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)
Initialize StoreFunc to write data.
|
void |
putNext(Tuple t)
Write a tuple to the data store.
|
void |
setStoreFuncUDFContextSignature(String signature)
This method will be called by Pig both in the front end and back end to
pass a unique signature to the
StoreFunc which it can use to store
information in the UDFContext which it needs to store between
various method invocations in the front end and back end. |
void |
setStoreLocation(String location,
org.apache.hadoop.mapreduce.Job job)
Communicate to the storer the location where the data needs to be stored.
|
void |
storeSchema(ResourceSchema schema,
String location,
org.apache.hadoop.mapreduce.Job job)
Store schema of the data being written
|
void |
storeStatistics(ResourceStatistics stats,
String location,
org.apache.hadoop.mapreduce.Job job)
Store statistics about the data being written.
|
cleanupOnFailure, cleanupOnFailureImpl, cleanupOnSuccess, relToAbsPathForStoreLocation, supportsParallelWriteToStoreLocation, warn
protected org.apache.hadoop.mapreduce.RecordWriter writer
protected ResourceSchema schema
public org.apache.hadoop.mapreduce.OutputFormat getOutputFormat() throws IOException
StoreFunc
getOutputFormat
in interface StoreFuncInterface
getOutputFormat
in class StoreFunc
OutputFormat
associated with StoreFuncIOException
- if an exception occurs while constructing the
OutputFormatpublic void setStoreLocation(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreFunc
StoreFunc
here is the
return value of StoreFunc.relToAbsPathForStoreLocation(String, Path)
This method will be called in the frontend and backend multiple times. Implementations
should bear in mind that this method is called multiple times and should
ensure there are no inconsistent side effects due to the multiple calls.
StoreFunc.checkSchema(ResourceSchema)
will be called before any call to
StoreFunc.setStoreLocation(String, Job)
.setStoreLocation
in interface StoreFuncInterface
setStoreLocation
in class StoreFunc
location
- Location returned by
StoreFunc.relToAbsPathForStoreLocation(String, Path)
job
- The Job
objectIOException
- if the location is not valid.public void setStoreFuncUDFContextSignature(String signature)
StoreFunc
StoreFunc
which it can use to store
information in the UDFContext
which it needs to store between
various method invocations in the front end and back end. This method
will be called before other methods in StoreFunc
. This is necessary
because in a Pig Latin script with multiple stores, the different
instances of store functions need to be able to find their (and only their)
data in the UDFContext object. The default implementation is a no-op.setStoreFuncUDFContextSignature
in interface StoreFuncInterface
setStoreFuncUDFContextSignature
in class StoreFunc
signature
- a unique signature to identify this StoreFuncpublic void checkSchema(ResourceSchema s) throws IOException
StoreFunc
checkSchema
in interface StoreFuncInterface
checkSchema
in class StoreFunc
s
- to be checkedIOException
- if this schema is not acceptable. It should include
a detailed error message indicating what is wrong with the schema.public void prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer) throws IOException
StoreFunc
prepareToWrite
in interface StoreFuncInterface
prepareToWrite
in class StoreFunc
writer
- RecordWriter to use.IOException
- if an exception occurs during initializationpublic void putNext(Tuple t) throws IOException
StoreFunc
putNext
in interface StoreFuncInterface
putNext
in class StoreFunc
t
- the tuple to store.IOException
- if an exception occurs during the writepublic void storeStatistics(ResourceStatistics stats, String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreMetadata
storeStatistics
in interface StoreMetadata
stats
- statistics to be recordedlocation
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)
job
- The Job
object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration()
and not to set/query
any runtime job information.IOException
public void storeSchema(ResourceSchema schema, String location, org.apache.hadoop.mapreduce.Job job) throws IOException
StoreMetadata
storeSchema
in interface StoreMetadata
schema
- Schema to be recordedlocation
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, org.apache.hadoop.fs.Path)
job
- The Job
object - this should be used only to obtain
cluster properties through JobContextImpl.getConfiguration()
and not to set/query
any runtime job information.IOException
public ResourceSchema fixSchema(ResourceSchema s)
public List<String> getShipFiles()
StoreResources
FuncUtils
for utility function to facilitate it
The default implementation returns null.getShipFiles
in interface StoreResources
public List<String> getCacheFiles()
StoreResources
getCacheFiles
in interface StoreResources
Copyright © 2007-2017 The Apache Software Foundation