org.apache.pig.backend.hadoop.executionengine.mapReduceLayer

Class MapReduceLauncher

java.lang.Object

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher

public class MapReduceLauncher
extends Launcher

Main class that launches pig for Map Reduce

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	SUCCEEDED_FILE_NAME
static java.lang.String	SUCCESSFUL_JOB_OUTPUT_DIR_MARKER

Constructor Summary

Constructors

Constructor and Description
MapReduceLauncher()

Method Summary

Methods

Modifier and Type	Method and Description
MROperPlan	compile(PhysicalPlan php, PigContext pc)
void	explain(PhysicalPlan php, PigContext pc, java.io.PrintStream ps, java.lang.String format, boolean verbose) Explain how a pig job will be executed on the underlying infrastructure.
java.lang.Exception	getError(FileSpec spec) Get the exception that caused a failure on the backend for a store location (if any).
PigStats	launchPig(PhysicalPlan php, java.lang.String grpName, PigContext pc) Method to launch pig for hadoop either for a cluster's job tracker or for a local job runner.
void	reset() Resets the state after a launch

Methods inherited from class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher

calculateProgress, computeTimeSpent, getErrorMessages, getStackTraceElement, getStats, getTotalHadoopTimeSpent, isComplete, progressOfRunningJob

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SUCCEEDED_FILE_NAME

public static final java.lang.String SUCCEEDED_FILE_NAME

See Also:

Constant Field Values

SUCCESSFUL_JOB_OUTPUT_DIR_MARKER

public static final java.lang.String SUCCESSFUL_JOB_OUTPUT_DIR_MARKER

See Also:

Constant Field Values

Constructor Detail

MapReduceLauncher

public MapReduceLauncher()

Method Detail

getError

public java.lang.Exception getError(FileSpec spec)

Get the exception that caused a failure on the backend for a store location (if any).

reset

public void reset()

Description copied from class: Launcher

Resets the state after a launch

Overrides:

reset in class Launcher

launchPig

public PigStats launchPig(PhysicalPlan php,
                 java.lang.String grpName,
                 PigContext pc)
                   throws PlanException,
                          VisitorException,
                          java.io.IOException,
                          ExecException,
                          JobCreationException,
                          java.lang.Exception

Description copied from class: Launcher

Method to launch pig for hadoop either for a cluster's job tracker or for a local job runner. THe only difference between the two is the job client. Depending on the pig context the job client will be initialize to one of the two. Launchers for other frameworks can overide these methods. Given an input PhysicalPlan, it compiles it to get a MapReduce Plan. The MapReduce plan which has multiple MapReduce operators each one of which has to be run as a map reduce job with dependency information stored in the plan. It compiles the MROperPlan into a JobControl object. Each Map Reduce operator is converted into a Job and added to the JobControl object. Each Job also has a set of dependent Jobs that are created using the MROperPlan. The JobControl object is obtained from the JobControlCompiler Then a new thread is spawned that submits these jobs while respecting the dependency information. The parent thread monitors the submitted jobs' progress and after it is complete, stops the JobControl thread.

Specified by:

launchPig in class Launcher

Throws:

PlanException

VisitorException

java.io.IOException

ExecException

JobCreationException

java.lang.Exception

explain

public void explain(PhysicalPlan php,
           PigContext pc,
           java.io.PrintStream ps,
           java.lang.String format,
           boolean verbose)
             throws PlanException,
                    VisitorException,
                    java.io.IOException

Description copied from class: Launcher

Explain how a pig job will be executed on the underlying infrastructure.

Specified by:

explain in class Launcher

Parameters:

php - PhysicalPlan to explain

pc - PigContext to use for configuration

ps - PrintStream to write output on.

format - Format to write in

verbose - Amount of information to print

Throws:

VisitorException

java.io.IOException

PlanException

compile

public MROperPlan compile(PhysicalPlan php,
                 PigContext pc)
                   throws PlanException,
                          java.io.IOException,
                          VisitorException

Throws:

PlanException

java.io.IOException

VisitorException