public class CSVExcelStorage extends PigStorage implements StoreFuncInterface, LoadPushDown
Modifier and Type | Class and Description |
---|---|
static class |
CSVExcelStorage.Headers |
static class |
CSVExcelStorage.Linebreaks |
static class |
CSVExcelStorage.Multiline |
LoadPushDown.OperatorSet, LoadPushDown.RequiredField, LoadPushDown.RequiredFieldList, LoadPushDown.RequiredFieldResponse
Modifier and Type | Field and Description |
---|---|
protected static byte |
CARRIAGE_RETURN |
protected static byte |
DOUBLE_QUOTE |
protected org.apache.hadoop.mapreduce.RecordReader |
in |
protected static byte |
LINEFEED |
protected static byte |
RECORD_DEL |
protected ResourceSchema |
schema |
caster, mLog, signature, writer
Constructor and Description |
---|
CSVExcelStorage() |
CSVExcelStorage(String delimiter) |
CSVExcelStorage(String delimiter,
String multilineTreatmentStr) |
CSVExcelStorage(String delimiter,
String multilineTreatmentStr,
String eolTreatmentStr) |
CSVExcelStorage(String delimiter,
String multilineTreatmentStr,
String eolTreatmentStr,
String headerTreatmentStr) |
Modifier and Type | Method and Description |
---|---|
void |
checkSchema(ResourceSchema s)
Set the schema for data to be stored.
|
List<LoadPushDown.OperatorSet> |
getFeatures()
Determine the operators that can be pushed to the loader.
|
org.apache.hadoop.mapreduce.InputFormat |
getInputFormat()
This will be called during planning on the front end.
|
Tuple |
getNext()
Retrieves the next tuple to be processed.
|
void |
prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader,
PigSplit split)
Initializes LoadFunc for reading data.
|
void |
prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)
Initialize StoreFuncInterface to write data.
|
LoadPushDown.RequiredFieldResponse |
pushProjection(LoadPushDown.RequiredFieldList requiredFieldList)
Indicate to the loader fields that will be needed.
|
void |
putNext(Tuple tupleToWrite)
Write a tuple to the data store.
|
void |
setLocation(String location,
org.apache.hadoop.mapreduce.Job job)
Communicate to the loader the location of the object(s) being loaded.
|
void |
setStoreFuncUDFContextSignature(String signature)
This method will be called by Pig both in the front end and back end to
pass a unique signature to the
StoreFuncInterface which it can use to store
information in the UDFContext which it needs to store between
various method invocations in the front end and back end. |
void |
setUDFContextSignature(String signature)
This method will be called by Pig both in the front end and back end to
pass a unique signature to the
LoadFunc . |
cleanupOnFailure, cleanupOnSuccess, cleanupOutput, equals, equals, getOutputFormat, getPartitionKeys, getSchema, getStatistics, hashCode, readField, relToAbsPathForStoreLocation, setPartitionFilter, setStoreLocation, shouldOverwrite, storeSchema, storeStatistics
getSplitComparable
getAbsolutePath, getCacheFiles, getLoadCaster, getPathStrings, getShipFiles, join, relativeToAbsolutePath, warn
clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait
cleanupOnFailure, cleanupOnSuccess, getOutputFormat, relToAbsPathForStoreLocation, setStoreLocation
protected static final byte LINEFEED
protected static final byte CARRIAGE_RETURN
protected static final byte DOUBLE_QUOTE
protected static final byte RECORD_DEL
protected org.apache.hadoop.mapreduce.RecordReader in
protected ResourceSchema schema
public CSVExcelStorage()
public CSVExcelStorage(String delimiter)
public CSVExcelStorage(String delimiter, String multilineTreatmentStr, String eolTreatmentStr)
public void checkSchema(ResourceSchema s) throws IOException
StoreFuncInterface
checkSchema
in interface StoreFuncInterface
checkSchema
in class PigStorage
s
- to be checkedIOException
- if this schema is not acceptable. It should include
a detailed error message indicating what is wrong with the schema.public void prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)
StoreFuncInterface
prepareToWrite
in interface StoreFuncInterface
prepareToWrite
in class PigStorage
writer
- RecordWriter to use.public void putNext(Tuple tupleToWrite) throws IOException
StoreFuncInterface
putNext
in interface StoreFuncInterface
putNext
in class PigStorage
tupleToWrite
- the tuple to store.IOException
- if an exception occurs during the writepublic Tuple getNext() throws IOException
LoadFunc
getNext
in class PigStorage
IOException
- if there is an exception while retrieving the next
tuplepublic void setLocation(String location, org.apache.hadoop.mapreduce.Job job) throws IOException
LoadFunc
LoadFunc.relativeToAbsolutePath(String, Path)
. Implementations
should use this method to communicate the location (and any other information)
to its underlying InputFormat through the Job object.
This method will be called in the frontend and backend multiple times. Implementations
should bear in mind that this method is called multiple times and should
ensure there are no inconsistent side effects due to the multiple calls.setLocation
in class PigStorage
location
- Location as returned by
LoadFunc.relativeToAbsolutePath(String, Path)
job
- the Job
object
store or retrieve earlier stored information from the UDFContext
IOException
- if the location is not valid.public org.apache.hadoop.mapreduce.InputFormat getInputFormat()
LoadFunc
getInputFormat
in class PigStorage
public void prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader, PigSplit split)
LoadFunc
prepareToRead
in class PigStorage
reader
- RecordReader
to be used by this instance of the LoadFuncsplit
- The input PigSplit
to processpublic LoadPushDown.RequiredFieldResponse pushProjection(LoadPushDown.RequiredFieldList requiredFieldList) throws FrontendException
LoadPushDown
pushProjection
in interface LoadPushDown
pushProjection
in class PigStorage
requiredFieldList
- RequiredFieldList indicating which columns will be needed.
This structure is read only. User cannot make change to it inside pushProjection.FrontendException
public void setUDFContextSignature(String signature)
LoadFunc
LoadFunc
. The signature can be used
to store into the UDFContext
any information which the
LoadFunc
needs to store between various method invocations in the
front end and back end. A use case is to store LoadPushDown.RequiredFieldList
passed to it in LoadPushDown.pushProjection(RequiredFieldList)
for
use in the back end before returning tuples in LoadFunc.getNext()
.
This method will be call before other methods in LoadFunc
setUDFContextSignature
in class PigStorage
signature
- a unique signature to identify this LoadFuncpublic void setStoreFuncUDFContextSignature(String signature)
StoreFuncInterface
StoreFuncInterface
which it can use to store
information in the UDFContext
which it needs to store between
various method invocations in the front end and back end. This is necessary
because in a Pig Latin script with multiple stores, the different
instances of store functions need to be able to find their (and only their)
data in the UDFContext object.setStoreFuncUDFContextSignature
in interface StoreFuncInterface
setStoreFuncUDFContextSignature
in class PigStorage
signature
- a unique signature to identify this StoreFuncInterfacepublic List<LoadPushDown.OperatorSet> getFeatures()
LoadPushDown
getFeatures
in interface LoadPushDown
getFeatures
in class PigStorage
Copyright © 2007-2017 The Apache Software Foundation