org.apache.pig.builtin

Class TOP

java.lang.Object

org.apache.pig.EvalFunc<T>

org.apache.pig.AccumulatorEvalFunc<DataBag>

org.apache.pig.builtin.TOP

All Implemented Interfaces:

Accumulator<DataBag>, Algebraic

public class TOP
extends AccumulatorEvalFunc<DataBag>
implements Algebraic

Top UDF accepts a bag of tuples and returns top-n tuples depending upon the tuple field value of type long. Both n and field number needs to be provided to the UDF. The UDF iterates through the input bag and just retains top-n tuples by storing them in a priority queue of size n+1 where priority is the long field. This is efficient as priority queue provides constant time - O(1) removal of the least element and O(log n) time for heap restructuring. The UDF is especially helpful for turning the nested grouping operation inside out and retaining top-n in a nested group. Assumes all tuples in the bag contain an element of the same type in the compared column. Sample usage: DEFINE TOP_ASC TOP("ASC") DEFINE TOP_DESC TOP("DESC") A = LOAD 'test.tsv' as (first: chararray, second: chararray); B = GROUP A BY (first, second); C = FOREACH B generate FLATTEN(group), COUNT(*) as count; D = GROUP C BY first; // again group by first topResults = FOREACH D { result = TOP_ASC(10, 1, C); // and retain top 10 occurrences of 'second' in first GENERATE FLATTEN(result); topDescResults = FOREACH D { result = TOP_DESC(10, 1, C); // and retain top 10 occurrences of 'second' in first GENERATE FLATTEN(result); * }

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TOP.Final
static class	TOP.Initial
static class	TOP.Intermed

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
TOP()
TOP(String arg)

Method Summary

Modifier and Type	Method and Description
void	accumulate(Tuple tuple) Pass tuples to the UDF.
void	cleanup() Called after getValue() to prepare processing for next key.
List<FuncSpec>	getArgToFuncMapping() Allow a UDF to specify type specific implementations of itself.
String	getFinal() Get the final function.
String	getInitial() Get the initial function.
String	getIntermed() Get the intermediate function.
DataBag	getValue() Called when all tuples from current key have been passed to accumulate.
Schema	outputSchema(Schema input) Report the schema of the output of this UDF.
protected static void	updateTop(PriorityQueue<Tuple> store, int limit, DataBag inputBag)

Methods inherited from class org.apache.pig.AccumulatorEvalFunc

exec

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getCacheFiles, getInputSchema, getLoadCaster, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, needEndOfAllInputProcessing, progress, setEndOfAllInput, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TOP

public TOP()

TOP

public TOP(String arg)

Method Detail

accumulate

public void accumulate(Tuple tuple)
                throws IOException

Description copied from interface: Accumulator

Pass tuples to the UDF.

Specified by:

accumulate in interface Accumulator<DataBag>

Specified by:

accumulate in class AccumulatorEvalFunc<DataBag>

Parameters:

tuple - A tuple containing a single field, which is a bag. The bag will contain the set of tuples being passed to the UDF in this iteration.

Throws:

IOException

getValue

public DataBag getValue()

Description copied from interface: Accumulator

Called when all tuples from current key have been passed to accumulate.

Specified by:

getValue in interface Accumulator<DataBag>

Specified by:

getValue in class AccumulatorEvalFunc<DataBag>

Returns:

the value for the UDF for this key.

cleanup

public void cleanup()

Description copied from interface: Accumulator

Called after getValue() to prepare processing for next key.

Specified by:

cleanup in interface Accumulator<DataBag>

Specified by:

cleanup in class AccumulatorEvalFunc<DataBag>

updateTop

protected static void updateTop(PriorityQueue<Tuple> store,
                                int limit,
                                DataBag inputBag)

getArgToFuncMapping

public List<FuncSpec> getArgToFuncMapping()
                                   throws FrontendException

Description copied from class: EvalFunc

Allow a UDF to specify type specific implementations of itself. For example, an implementation of arithmetic sum might have int and float implementations, since integer arithmetic performs much better than floating point arithmetic. Pig's typechecker will call this method and using the returned list plus the schema of the function's input data, decide which implementation of the UDF to use.

Overrides:

getArgToFuncMapping in class EvalFunc<DataBag>

Returns:

A List containing FuncSpec objects representing the EvalFunc class which can handle the inputs corresponding to the schema in the objects. Each FuncSpec should be constructed with a schema that describes the input for that implementation. For example, the sum function above would return two elements in its list:

1. FuncSpec(this.getClass().getName(), new Schema(new Schema.FieldSchema(null, DataType.DOUBLE)))
2. FuncSpec(IntSum.getClass().getName(), new Schema(new Schema.FieldSchema(null, DataType.INTEGER)))

This would indicate that the main implementation is used for doubles, and the special implementation IntSum is used for ints.

Throws:

FrontendException

outputSchema

public Schema outputSchema(Schema input)

Description copied from class: EvalFunc

Report the schema of the output of this UDF. Pig will make use of this in error checking, optimization, and planning. The schema of input data to this UDF is provided.

The default implementation interprets the OutputSchema annotation, if one is present. Otherwise, it returns null (no known output schema).

Overrides:

outputSchema in class EvalFunc<DataBag>

Parameters:

input - Schema of the input

Returns:

Schema of the output

getInitial

public String getInitial()

Description copied from interface: Algebraic

Get the initial function.

Specified by:

getInitial in interface Algebraic

Returns:

A function name of f_init. f_init should be an eval func. The return type of f_init.exec() has to be Tuple

getIntermed

public String getIntermed()

Description copied from interface: Algebraic

Get the intermediate function.

Specified by:

getIntermed in interface Algebraic

Returns:

A function name of f_intermed. f_intermed should be an eval func. The return type of f_intermed.exec() has to be Tuple

getFinal

public String getFinal()

Description copied from interface: Algebraic

Get the final function.

Specified by:

getFinal in interface Algebraic

Returns:

A function name of f_final. f_final should be an eval func parametrized by the same datum as the eval func implementing this interface.