org.apache.pig.builtin

Class Bloom

java.lang.Object

org.apache.pig.EvalFunc<Boolean>

org.apache.pig.FilterFunc

org.apache.pig.builtin.Bloom

public class Bloom
extends FilterFunc

Use a Bloom filter build previously by BuildBloom. You would first build a bloom filter in a group all job. For example: in a group all job. For example: define bb BuildBloom('jenkins', '100', '0.1'); A = load 'foo' as (x, y); B = group A all; C = foreach B generate bb(A.x); store C into 'mybloom'; The bloom filter can be on multiple keys by passing more than one field (or the entire bag) to BuildBloom. The resulting file can then be used in a Bloom filter as: define bloom Bloom(mybloom); A = load 'foo' as (x, y); B = load 'bar' as (z); C = filter B by bloom(z); D = join C by z, A by x; It uses BloomFilter. You can also pass the Bloom filter from BuildBloom directly to Bloom UDF as a scalar instead of storing it to file and loading again. This is simpler if the Bloom filter will not be reused and needs to be discarded after the run of the script. define bb BuildBloom('jenkins', '100', '0.1'); A = load 'foo' as (x, y); B = group A all; C = foreach B generate bb(A.x) as bloomfilter; D = load 'bar' as (z); E = filter D by Bloom(C.bloomfilter, z); F = join E by z, A by x;

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
Bloom()
Bloom(String filename) The filename containing the serialized Bloom filter.

Method Summary

Modifier and Type	Method and Description
Boolean	exec(Tuple input) This callback method must be implemented by all subclasses.
List<String>	getCacheFiles() Allow a UDF to specify a list of hdfs files it would like placed in the distributed cache.
void	setFilter(DataByteArray dba) For testing only, do not use directly.

Methods inherited from class org.apache.pig.FilterFunc

finish

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, getArgToFuncMapping, getInputSchema, getLoadCaster, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, needEndOfAllInputProcessing, outputSchema, progress, setEndOfAllInput, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Bloom

public Bloom()

Bloom

public Bloom(String filename)

The filename containing the serialized Bloom filter. If filename is null or the no-arg constructor is used, then the bloomfilter bytearray which is the output of BuildBloom should be passed as the first argument to the UDF

Parameters:

filename - file containing the serialized Bloom filter

Method Detail

exec

public Boolean exec(Tuple input)
             throws IOException

Description copied from class: EvalFunc

This callback method must be implemented by all subclasses. This is the method that will be invoked on every Tuple of a given dataset. Since the dataset may be divided up in a variety of ways the programmer should not make assumptions about state that is maintained between invocations of this method.

Specified by:

exec in class EvalFunc<Boolean>

Parameters:

input - the Tuple to be processed.

Returns:

result, of type T.

Throws:

IOException

getCacheFiles

public List<String> getCacheFiles()

Description copied from class: EvalFunc

Allow a UDF to specify a list of hdfs files it would like placed in the distributed cache. These files will be put in the cache for every job the UDF is used in. The default implementation returns null.

Overrides:

getCacheFiles in class EvalFunc<Boolean>

Returns:

A list of files

setFilter

public void setFilter(DataByteArray dba)
               throws IOException

For testing only, do not use directly.

Throws:

IOException