org.apache.pig.piggybank.storage.avro

Class PigSchema2Avro

java.lang.Object

org.apache.pig.piggybank.storage.avro.PigSchema2Avro

public class PigSchema2Avro
extends java.lang.Object

This class contains functions to convert Pig schema to Avro. It consists of two sets of methods: 1. Convert a Pig schema to Avro schema; 2. Validate whether a Pig schema is compatible with a given Avro schema. Notice that the Avro schema doesn't need to cover all fields in Pig schema, and the missing fields are converted using methods in set 1.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	FIELD_NAME
static java.lang.String	TUPLE_NAME
static int	tupleIndex

Constructor Summary

Constructors

Constructor and Description
PigSchema2Avro()

Method Summary

Methods

Modifier and Type	Method and Description
protected static org.apache.avro.Schema	convert(ResourceSchema.ResourceFieldSchema pigSchema, boolean nullable) Convert a Pig ResourceFieldSchema to avro schema
static org.apache.avro.Schema	convert(ResourceSchema pigSchema, boolean nullable) Convert a pig ResourceSchema to avro schema
protected static org.apache.avro.Schema	convertPrimitiveType(byte pigType) Convert Pig primitive type to Avro type
protected static org.apache.avro.Schema	convertRecord(ResourceSchema.ResourceFieldSchema[] pigFields, boolean nullable) Convert pig data to Avro record
protected static boolean	isCompatible(org.apache.avro.Schema avroSchema, ResourceSchema.ResourceFieldSchema pigSchema) Check whether Avro type is compatible with Pig type
static void	setTupleIndex(int index) This is a painful hack to make unit tests pass.
protected static org.apache.avro.Schema	validateAndConvert(org.apache.avro.Schema avroSchema, ResourceSchema.ResourceFieldSchema pigSchema) Validate whether pigSchema is compatible with avroSchema and convert those Pig fields with to corresponding Avro schemas.
static org.apache.avro.Schema	validateAndConvert(org.apache.avro.Schema avroSchema, ResourceSchema pigSchema) Validate whether pigSchema is compatible with avroSchema
protected static org.apache.avro.Schema	validateAndConvertRecord(org.apache.avro.Schema avroSchema, ResourceSchema.ResourceFieldSchema[] pigFields) Validate a Pig tuple is compatible with Avro record.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TUPLE_NAME

public static final java.lang.String TUPLE_NAME

See Also:

Constant Field Values

FIELD_NAME

public static final java.lang.String FIELD_NAME

See Also:

Constant Field Values

tupleIndex

public static int tupleIndex

Constructor Detail

PigSchema2Avro

public PigSchema2Avro()

Method Detail

convert

public static org.apache.avro.Schema convert(ResourceSchema pigSchema,
                             boolean nullable)
                                      throws java.io.IOException

Convert a pig ResourceSchema to avro schema

Throws:

java.io.IOException

convert

protected static org.apache.avro.Schema convert(ResourceSchema.ResourceFieldSchema pigSchema,
                             boolean nullable)
                                         throws java.io.IOException

Convert a Pig ResourceFieldSchema to avro schema

Throws:

java.io.IOException

convertRecord

protected static org.apache.avro.Schema convertRecord(ResourceSchema.ResourceFieldSchema[] pigFields,
                                   boolean nullable)
                                               throws java.io.IOException

Convert pig data to Avro record

Throws:

java.io.IOException

setTupleIndex

public static void setTupleIndex(int index)

This is a painful hack to make unit tests pass. The static counter holds state between unit tests, so it needs to be reset. Otherwise tests will fail if their order is swapped or a new test is added.

Parameters:

index -

convertPrimitiveType

protected static org.apache.avro.Schema convertPrimitiveType(byte pigType)
                                                      throws java.io.IOException

Convert Pig primitive type to Avro type

Throws:

java.io.IOException

validateAndConvert

public static org.apache.avro.Schema validateAndConvert(org.apache.avro.Schema avroSchema,
                                        ResourceSchema pigSchema)
                                                 throws java.io.IOException

Validate whether pigSchema is compatible with avroSchema

Throws:

java.io.IOException

validateAndConvert

protected static org.apache.avro.Schema validateAndConvert(org.apache.avro.Schema avroSchema,
                                        ResourceSchema.ResourceFieldSchema pigSchema)
                                                    throws java.io.IOException

Validate whether pigSchema is compatible with avroSchema and convert those Pig fields with to corresponding Avro schemas.

Throws:

java.io.IOException

validateAndConvertRecord

protected static org.apache.avro.Schema validateAndConvertRecord(org.apache.avro.Schema avroSchema,
                                              ResourceSchema.ResourceFieldSchema[] pigFields)
                                                          throws java.io.IOException

Validate a Pig tuple is compatible with Avro record. If the Avro schema is not complete (with uncovered fields), then convert those fields using methods in set 1. Notice that users can get rid of Pig tuple wrappers, e.g. an Avro schema "int" is compatible with a Pig schema "T:(int)"

Throws:

java.io.IOException

isCompatible

protected static boolean isCompatible(org.apache.avro.Schema avroSchema,
                   ResourceSchema.ResourceFieldSchema pigSchema)

Check whether Avro type is compatible with Pig type