org.apache.pig.piggybank.storage.apachelog

Class LogFormatLoader

java.lang.Object

org.apache.pig.LoadFunc

nl.basjes.pig.input.apachehttpdlog.Loader

org.apache.pig.piggybank.storage.apachelog.LogFormatLoader

All Implemented Interfaces:

LoadMetadata, LoadPushDown

public class LogFormatLoader
extends nl.basjes.pig.input.apachehttpdlog.Loader

This is a pig loader that can load Apache HTTPD access logs written in (almost) any Apache HTTPD LogFormat.
Basic usage:
Simply feed the loader your (custom) logformat specification and it will tell you which fields can be extracted from this logformat.
For example:

 -- Specify any existing file as long as it exists.
 -- It won't be read by the loader when no fields are requested.
 Example =
     LOAD 'test.pig'
     USING org.apache.pig.piggybank.storage.apachelog.LogFormatLoader(
       '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"'
     );
 DUMP Example;
 

The output of this command is a (huge) example (yes actual pig code) which demonstrates how all possible fields can be extracted. In normal use cases this example will be trimmed down to request only the fields your application really needs. This loader implements pushdown projection so there is no need to worry too much about the fields you leave in. This loader supports extracting things like an individual cookie or query string parameter regardless of the position it has in the actual log line. In addition to the logformat specification used in your custom config this parser also understands the standard formats:

    common
    combined
    combinedio
    referer
    agent
 

So this works also:

 Example =
     LOAD 'test.pig'
     USING org.apache.pig.piggybank.storage.apachelog.LogFormatLoader('common');
 DUMP Example;
 

This class is simply a wrapper around https://github.com/nielsbasjes/logparser so more detailed documentation can be found there.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.pig.LoadPushDown

LoadPushDown.OperatorSet, LoadPushDown.RequiredField, LoadPushDown.RequiredFieldList, LoadPushDown.RequiredFieldResponse

Constructor Summary

Constructors

Constructor and Description
LogFormatLoader(String... parameters)

Method Summary

Methods inherited from class nl.basjes.pig.input.apachehttpdlog.Loader

getAdditionalDissectors, getFeatures, getInputFormat, getLogformat, getNext, getPartitionKeys, getRequestedFields, getSchema, getStatistics, getTypeRemappings, prepareToRead, pushProjection, setLocation, setPartitionFilter, setUDFContextSignature

Methods inherited from class org.apache.pig.LoadFunc

getAbsolutePath, getCacheFiles, getLoadCaster, getPathStrings, getShipFiles, join, relativeToAbsolutePath, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LogFormatLoader

public LogFormatLoader(String... parameters)