org.apache.pig.data

Class DefaultAbstractBag

java.lang.Object

org.apache.pig.data.DefaultAbstractBag

All Implemented Interfaces:

java.io.Serializable, java.lang.Comparable, java.lang.Iterable<Tuple>, org.apache.hadoop.io.Writable, org.apache.hadoop.io.WritableComparable, DataBag, Spillable

Direct Known Subclasses:

DefaultDataBag, DistinctDataBag, SelfSpillBag, SortedDataBag

public abstract class DefaultAbstractBag
extends java.lang.Object
implements DataBag

Default implementation of DataBag. This is the an abstract class used as a parent for all three of the types of data bags.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DefaultAbstractBag.BagDelimiterTuple
static class	DefaultAbstractBag.EndBag
static class	DefaultAbstractBag.StartBag

Field Summary

Fields

Modifier and Type	Field and Description
static Tuple	endBag
protected static int	MAX_SPILL_FILES
protected java.util.Collection<Tuple>	mContents
protected long	mSize
protected FileList	mSpillFiles
static Tuple	startBag

Constructor Summary

Constructors

Constructor and Description
DefaultAbstractBag()

Method Summary

Modifier and Type	Method and Description
void	add(Tuple t) Add a tuple to the bag.
void	addAll(java.util.Collection<Tuple> c)
void	addAll(DataBag b) Add contents of a bag to the bag.
void	addAll(java.lang.Iterable<Tuple> iterable) Add contents of an iterable (a collection or a DataBag)
void	clear() Clear out the contents of the bag, both on disk and in memory.
int	compareTo(java.lang.Object other) This method is potentially very expensive since it may require a sort of the bag; don't call it unless you have to.
boolean	equals(java.lang.Object other)
long	getMemorySize() Return the size of memory usage.
protected java.io.DataOutputStream	getSpillFile() Get a file to spill contents to.
int	hashCode() Returns the hash code value for the bag.
protected void	incSpillCount(java.lang.Enum counter)
protected void	incSpillCount(java.lang.Enum counter, long numRecsSpilled)
protected void	markSpillableIfNecessary() All bag implementations that can get big enough to be spilled should call this method after every time they add an element.
void	markStale(boolean stale) This is used by FuncEvalSpec.FakeDataBag.
void	readFields(java.io.DataInput in) Read a bag from disk.
protected void	reportProgress() Report progress to HDFS.
protected void	sampleContents() Sample every SPILL_SAMPLE_FREQUENCYth tuple until we reach a max of SPILL_SAMPLE_SIZE to get an estimate of the tuple sizes.
long	size() Get the number of elements in the bag, both in memory and on disk.
java.lang.String	toString() Write the bag into a string.
protected void	warn(java.lang.String msg, java.lang.Enum warningEnum, java.lang.Throwable e)
void	write(java.io.DataOutput out) Write a bag's contents to disk.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.pig.data.DataBag

isDistinct, isSorted, iterator

Methods inherited from interface org.apache.pig.impl.util.Spillable

spill

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

mContents

protected java.util.Collection<Tuple> mContents

mSpillFiles

protected FileList mSpillFiles

mSize

protected long mSize

startBag

public static final Tuple startBag

endBag

public static final Tuple endBag

MAX_SPILL_FILES

protected static final int MAX_SPILL_FILES

See Also:

Constant Field Values

Constructor Detail

DefaultAbstractBag

public DefaultAbstractBag()

Method Detail

size

public long size()

Get the number of elements in the bag, both in memory and on disk.

Specified by:

size in interface DataBag

Returns:

number of elements in the bag

sampleContents

protected void sampleContents()

Sample every SPILL_SAMPLE_FREQUENCYth tuple until we reach a max of SPILL_SAMPLE_SIZE to get an estimate of the tuple sizes.

add

public void add(Tuple t)

Add a tuple to the bag.

Specified by:

add in interface DataBag

Parameters:

t - tuple to add.

markSpillableIfNecessary

protected void markSpillableIfNecessary()

All bag implementations that can get big enough to be spilled should call this method after every time they add an element.

addAll

public void addAll(DataBag b)

Description copied from interface: DataBag

Add contents of a bag to the bag.

Specified by:

addAll in interface DataBag

Parameters:

b - bag to add contents of.

addAll

public void addAll(java.util.Collection<Tuple> c)

addAll

public void addAll(java.lang.Iterable<Tuple> iterable)

Add contents of an iterable (a collection or a DataBag)

Parameters:

iterable - a Collection or DataBag to add contents of

getMemorySize

public long getMemorySize()

Return the size of memory usage.

Specified by:

getMemorySize in interface Spillable

Returns:

estimated in memory size.

clear

public void clear()

Clear out the contents of the bag, both on disk and in memory. Any attempts to read after this is called will produce undefined results.

Specified by:

clear in interface DataBag

compareTo

public int compareTo(java.lang.Object other)

This method is potentially very expensive since it may require a sort of the bag; don't call it unless you have to.

Specified by:

compareTo in interface java.lang.Comparable

equals

public boolean equals(java.lang.Object other)

Overrides:

equals in class java.lang.Object

write

public void write(java.io.DataOutput out)
           throws java.io.IOException

Write a bag's contents to disk.

Specified by:

write in interface org.apache.hadoop.io.Writable

Parameters:

out - DataOutput to write data to.

Throws:

java.io.IOException - (passes it on from underlying calls).

readFields

public void readFields(java.io.DataInput in)
                throws java.io.IOException

Read a bag from disk.

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Parameters:

in - DataInput to read data from.

Throws:

java.io.IOException - (passes it on from underlying calls).

markStale

public void markStale(boolean stale)

This is used by FuncEvalSpec.FakeDataBag.

Specified by:

markStale in interface DataBag

Parameters:

stale - Set stale state.

toString

public java.lang.String toString()

Write the bag into a string.

Overrides:

toString in class java.lang.Object

hashCode

public int hashCode()

Description copied from interface: DataBag

Returns the hash code value for the bag. The hash code of a bag is defined to be the sum of the hash codes of each tuple. This ensures that b1.equals(b2) implies b1.hashCode() == b2.hashCode()

Specified by:

hashCode in interface DataBag

Overrides:

hashCode in class java.lang.Object

Returns:

the hash code value for this bag

getSpillFile

protected java.io.DataOutputStream getSpillFile()
                                         throws java.io.IOException

Get a file to spill contents to. The file will be registered in the mSpillFiles array.

Returns:

stream to write tuples to.

Throws:

java.io.IOException

reportProgress

protected void reportProgress()

Report progress to HDFS.

warn

protected void warn(java.lang.String msg,
                    java.lang.Enum warningEnum,
                    java.lang.Throwable e)

incSpillCount

protected void incSpillCount(java.lang.Enum counter)

incSpillCount

protected void incSpillCount(java.lang.Enum counter,
                             long numRecsSpilled)