org.apache.oozie.action.hadoop

Class JavaActionExecutor

java.lang.Object

org.apache.oozie.action.ActionExecutor

org.apache.oozie.action.hadoop.JavaActionExecutor

Direct Known Subclasses:

DistcpActionExecutor, MapReduceActionExecutor, ScriptLanguageActionExecutor, ShellActionExecutor, SparkActionExecutor, SqoopActionExecutor

public class JavaActionExecutor
extends ActionExecutor

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.oozie.action.ActionExecutor

ActionExecutor.Context, ActionExecutor.RETRYPOLICY

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	ACL_MODIFY_JOB
static java.lang.String	ACL_VIEW_JOB
static java.lang.String	CONF_HADOOP_YARN_UBER_MODE
static java.lang.String	HADOOP_CHILD_JAVA_ENV
static java.lang.String	HADOOP_CHILD_JAVA_OPTS
static java.lang.String	HADOOP_JOB_CLASSLOADER
static java.lang.String	HADOOP_JOB_TRACKER
static java.lang.String	HADOOP_JOB_TRACKER_2
static java.lang.String	HADOOP_MAP_JAVA_ENV
static java.lang.String	HADOOP_MAP_JAVA_OPTS
static java.lang.String	HADOOP_MAP_MEMORY_MB
static java.lang.String	HADOOP_NAME_NODE
static java.lang.String	HADOOP_REDUCE_JAVA_OPTS
static java.lang.String	HADOOP_USER_CLASSPATH_FIRST
static java.lang.String	HADOOP_YARN_KILL_CHILD_JOBS_ON_AMRESTART
static java.lang.String	HADOOP_YARN_RM
static java.lang.String	HADOOP_YARN_UBER_MODE
protected XLog	LOG
static java.lang.String	MAX_EXTERNAL_STATS_SIZE
static java.lang.String	OOZIE_ACTION_LAUNCHER_PREFIX
static java.lang.String	OOZIE_COMMON_LIBDIR
static java.lang.String	OOZIE_CREDENTIALS_SKIP
XConfiguration	workflowConf
static java.lang.String	YARN_AM_COMMAND_OPTS
static java.lang.String	YARN_AM_ENV
static java.lang.String	YARN_AM_RESOURCE_MB
static int	YARN_MEMORY_MB_MIN

Fields inherited from class org.apache.oozie.action.ActionExecutor

ACTION_RETRY_INTERVAL, ACTION_RETRY_POLICY, CONF_PREFIX, ERROR_OTHER, MAX_RETRIES, OOZIE_ACTION_YARN_TAG, RETRY_INTERVAL

Constructor Summary

Constructors

Modifier	Constructor and Description
	JavaActionExecutor()
protected	JavaActionExecutor(java.lang.String type)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	addActionLibs(org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf)
protected void	addShareLib(org.apache.hadoop.conf.Configuration conf, java.lang.String[] actionShareLibNames)
protected void	addSystemShareLibForAction(org.apache.hadoop.conf.Configuration conf)
void	check(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action) Check if an action has completed.
org.apache.hadoop.mapred.JobConf	createBaseHadoopConf(ActionExecutor.Context context, org.jdom.Element actionXml)
protected org.apache.hadoop.mapred.JobClient	createJobClient(ActionExecutor.Context context, org.apache.hadoop.mapred.JobConf jobConf) Create job client object
void	end(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action) End an action after it has executed.
int	extractHeapSizeMB(java.lang.String input)
protected java.util.HashMap<java.lang.String,CredentialsProperties>	getActionCredentialsProperties(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action)
protected void	getActionData(org.apache.hadoop.fs.FileSystem actionFs, org.apache.hadoop.mapred.RunningJob runningJob, org.apache.oozie.client.WorkflowAction action, ActionExecutor.Context context) Get the output data of an action.
protected java.lang.String	getActualExternalId(org.apache.oozie.client.WorkflowAction action) Useful for overriding in actions that do subsequent job runs such as the MapReduce Action, where the launcher job is not the actual job that then gets monitored.
protected boolean	getCaptureOutput(org.apache.oozie.client.WorkflowAction action)
static java.util.List<java.lang.Class>	getCommonLauncherClasses()
protected CredentialsProperties	getCredProperties(ActionExecutor.Context context, java.lang.String credName)
protected java.lang.String	getDefaultShareLibName(org.jdom.Element actionXml) Returns the default sharelib name for the action if any.
java.util.List<java.lang.Class>	getLauncherClasses()
protected java.lang.String	getLauncherMain(org.apache.hadoop.conf.Configuration launcherConf, org.jdom.Element actionXml)
static int	getMaxExternalStatsSize() Get the maximum allowed size of stats
protected static int	getMaxOutputData(org.apache.hadoop.conf.Configuration actionConf)
protected org.apache.hadoop.mapred.RunningJob	getRunningJob(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action, org.apache.hadoop.mapred.JobClient jobClient)
java.lang.String[]	getShareLibFilesForActionConf()
protected java.lang.String[]	getShareLibNames(ActionExecutor.Context context, org.jdom.Element actionXml, org.apache.hadoop.conf.Configuration conf) Return the sharelib names for the action.
protected static java.lang.String	getTrimmedEncodedPath(java.lang.String path)
void	initActionType() Invoked once at system initialization time.
boolean	isCompleted(java.lang.String externalStatus) Return if the external status indicates that the action has completed.
void	kill(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action) Kill an action.
static void	parseJobXmlAndConfiguration(ActionExecutor.Context context, org.jdom.Element element, org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf)
static void	parseJobXmlAndConfiguration(ActionExecutor.Context context, org.jdom.Element element, org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf, boolean isLauncher)
void	prepareActionDir(org.apache.hadoop.fs.FileSystem actionFs, ActionExecutor.Context context)
boolean	requiresNameNodeJobTracker() Returns true if this action type requires a NameNode and JobTracker.
protected void	setActionCompletionData(ActionExecutor.Context context, org.apache.hadoop.fs.FileSystem actionFs) Sets some data for the action on completion
protected java.util.HashMap<java.lang.String,CredentialsProperties>	setCredentialPropertyToActionConf(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action, org.apache.hadoop.conf.Configuration actionConf)
protected void	setCredentialTokens(org.apache.hadoop.mapred.JobConf jobconf, ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action, java.util.HashMap<java.lang.String,CredentialsProperties> credPropertiesMap)
void	setLibFilesArchives(ActionExecutor.Context context, org.jdom.Element actionXml, org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf)
void	start(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action) Start an action.
void	submitLauncher(org.apache.hadoop.fs.FileSystem actionFs, ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action)
boolean	supportsConfigurationJobXML() Returns true if this action type supports a Configuration and JobXML.

Methods inherited from class org.apache.oozie.action.ActionExecutor

convertException, disableInit, enableInit, getActionDir, getActionDirPath, getActionSignal, getActionYarnTag, getMaxRetries, getOozieConf, getOozieRuntimeDir, getOozieSystemId, getRetryInterval, getRetryPolicy, getType, registerError, resetInitInfo, setMaxRetries, setRetryInterval, setRetryPolicy

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HADOOP_JOB_TRACKER

public static final java.lang.String HADOOP_JOB_TRACKER

See Also:

Constant Field Values

HADOOP_JOB_TRACKER_2

public static final java.lang.String HADOOP_JOB_TRACKER_2

See Also:

Constant Field Values

HADOOP_YARN_RM

public static final java.lang.String HADOOP_YARN_RM

See Also:

Constant Field Values

HADOOP_NAME_NODE

public static final java.lang.String HADOOP_NAME_NODE

See Also:

Constant Field Values

OOZIE_COMMON_LIBDIR

public static final java.lang.String OOZIE_COMMON_LIBDIR

See Also:

Constant Field Values

MAX_EXTERNAL_STATS_SIZE

public static final java.lang.String MAX_EXTERNAL_STATS_SIZE

See Also:

Constant Field Values

ACL_VIEW_JOB

public static final java.lang.String ACL_VIEW_JOB

See Also:

Constant Field Values

ACL_MODIFY_JOB

public static final java.lang.String ACL_MODIFY_JOB

See Also:

Constant Field Values

HADOOP_YARN_UBER_MODE

public static final java.lang.String HADOOP_YARN_UBER_MODE

See Also:

Constant Field Values

OOZIE_ACTION_LAUNCHER_PREFIX

public static final java.lang.String OOZIE_ACTION_LAUNCHER_PREFIX

See Also:

Constant Field Values

HADOOP_YARN_KILL_CHILD_JOBS_ON_AMRESTART

public static final java.lang.String HADOOP_YARN_KILL_CHILD_JOBS_ON_AMRESTART

See Also:

Constant Field Values

HADOOP_MAP_MEMORY_MB

public static final java.lang.String HADOOP_MAP_MEMORY_MB

See Also:

Constant Field Values

HADOOP_CHILD_JAVA_OPTS

public static final java.lang.String HADOOP_CHILD_JAVA_OPTS

See Also:

Constant Field Values

HADOOP_MAP_JAVA_OPTS

public static final java.lang.String HADOOP_MAP_JAVA_OPTS

See Also:

Constant Field Values

HADOOP_REDUCE_JAVA_OPTS

public static final java.lang.String HADOOP_REDUCE_JAVA_OPTS

See Also:

Constant Field Values

HADOOP_CHILD_JAVA_ENV

public static final java.lang.String HADOOP_CHILD_JAVA_ENV

See Also:

Constant Field Values

HADOOP_MAP_JAVA_ENV

public static final java.lang.String HADOOP_MAP_JAVA_ENV

See Also:

Constant Field Values

YARN_AM_RESOURCE_MB

public static final java.lang.String YARN_AM_RESOURCE_MB

See Also:

Constant Field Values

YARN_AM_COMMAND_OPTS

public static final java.lang.String YARN_AM_COMMAND_OPTS

See Also:

Constant Field Values

YARN_AM_ENV

public static final java.lang.String YARN_AM_ENV

See Also:

Constant Field Values

YARN_MEMORY_MB_MIN

public static final int YARN_MEMORY_MB_MIN

See Also:

Constant Field Values

LOG

protected XLog LOG

CONF_HADOOP_YARN_UBER_MODE

public static final java.lang.String CONF_HADOOP_YARN_UBER_MODE

See Also:

Constant Field Values

HADOOP_JOB_CLASSLOADER

public static final java.lang.String HADOOP_JOB_CLASSLOADER

See Also:

Constant Field Values

HADOOP_USER_CLASSPATH_FIRST

public static final java.lang.String HADOOP_USER_CLASSPATH_FIRST

See Also:

Constant Field Values

OOZIE_CREDENTIALS_SKIP

public static final java.lang.String OOZIE_CREDENTIALS_SKIP

See Also:

Constant Field Values

workflowConf

public XConfiguration workflowConf

Constructor Detail

JavaActionExecutor

public JavaActionExecutor()

JavaActionExecutor

protected JavaActionExecutor(java.lang.String type)

Method Detail

getCommonLauncherClasses

public static java.util.List<java.lang.Class> getCommonLauncherClasses()

getLauncherClasses

public java.util.List<java.lang.Class> getLauncherClasses()

initActionType

public void initActionType()

Description copied from class: ActionExecutor

Invoked once at system initialization time.

It can be used to register error information for the expected exceptions. Exceptions should be register from subclasses to superclasses to ensure proper detection, same thing that it is done in a normal catch.

This method should invoke the ActionExecutor.registerError(java.lang.String, org.apache.oozie.action.ActionExecutorException.ErrorType, java.lang.String) method to register all its possible errors.

Subclasses overriding must invoke super.

Overrides:

initActionType in class ActionExecutor

getMaxExternalStatsSize

public static int getMaxExternalStatsSize()

Get the maximum allowed size of stats

Returns:

maximum size of stats

createBaseHadoopConf

public org.apache.hadoop.mapred.JobConf createBaseHadoopConf(ActionExecutor.Context context,
                                                    org.jdom.Element actionXml)

extractHeapSizeMB

public int extractHeapSizeMB(java.lang.String input)

parseJobXmlAndConfiguration

public static void parseJobXmlAndConfiguration(ActionExecutor.Context context,
                               org.jdom.Element element,
                               org.apache.hadoop.fs.Path appPath,
                               org.apache.hadoop.conf.Configuration conf)
                                        throws java.io.IOException,
                                               ActionExecutorException,
                                               HadoopAccessorException,
                                               java.net.URISyntaxException

Throws:

java.io.IOException

ActionExecutorException

HadoopAccessorException

java.net.URISyntaxException

parseJobXmlAndConfiguration

public static void parseJobXmlAndConfiguration(ActionExecutor.Context context,
                               org.jdom.Element element,
                               org.apache.hadoop.fs.Path appPath,
                               org.apache.hadoop.conf.Configuration conf,
                               boolean isLauncher)
                                        throws java.io.IOException,
                                               ActionExecutorException,
                                               HadoopAccessorException,
                                               java.net.URISyntaxException

Throws:

java.io.IOException

ActionExecutorException

HadoopAccessorException

java.net.URISyntaxException

prepareActionDir

public void prepareActionDir(org.apache.hadoop.fs.FileSystem actionFs,
                    ActionExecutor.Context context)
                      throws ActionExecutorException

Throws:

ActionExecutorException

addShareLib

protected void addShareLib(org.apache.hadoop.conf.Configuration conf,
               java.lang.String[] actionShareLibNames)
                    throws ActionExecutorException

Throws:

ActionExecutorException

addSystemShareLibForAction

protected void addSystemShareLibForAction(org.apache.hadoop.conf.Configuration conf)
                                   throws ActionExecutorException

Throws:

ActionExecutorException

addActionLibs

protected void addActionLibs(org.apache.hadoop.fs.Path appPath,
                 org.apache.hadoop.conf.Configuration conf)
                      throws ActionExecutorException

Throws:

ActionExecutorException

setLibFilesArchives

public void setLibFilesArchives(ActionExecutor.Context context,
                       org.jdom.Element actionXml,
                       org.apache.hadoop.fs.Path appPath,
                       org.apache.hadoop.conf.Configuration conf)
                         throws ActionExecutorException

Throws:

ActionExecutorException

getTrimmedEncodedPath

protected static java.lang.String getTrimmedEncodedPath(java.lang.String path)

getLauncherMain

protected java.lang.String getLauncherMain(org.apache.hadoop.conf.Configuration launcherConf,
                               org.jdom.Element actionXml)

getMaxOutputData

protected static int getMaxOutputData(org.apache.hadoop.conf.Configuration actionConf)

submitLauncher

public void submitLauncher(org.apache.hadoop.fs.FileSystem actionFs,
                  ActionExecutor.Context context,
                  org.apache.oozie.client.WorkflowAction action)
                    throws ActionExecutorException

Throws:

ActionExecutorException

setCredentialPropertyToActionConf

protected java.util.HashMap<java.lang.String,CredentialsProperties> setCredentialPropertyToActionConf(ActionExecutor.Context context,
                                                                                          org.apache.oozie.client.WorkflowAction action,
                                                                                          org.apache.hadoop.conf.Configuration actionConf)
                                                                                               throws java.lang.Exception

Throws:

java.lang.Exception

setCredentialTokens

protected void setCredentialTokens(org.apache.hadoop.mapred.JobConf jobconf,
                       ActionExecutor.Context context,
                       org.apache.oozie.client.WorkflowAction action,
                       java.util.HashMap<java.lang.String,CredentialsProperties> credPropertiesMap)
                            throws java.lang.Exception

Throws:

java.lang.Exception

getActionCredentialsProperties

protected java.util.HashMap<java.lang.String,CredentialsProperties> getActionCredentialsProperties(ActionExecutor.Context context,
                                                                                       org.apache.oozie.client.WorkflowAction action)
                                                                                            throws java.lang.Exception

Throws:

java.lang.Exception

getCredProperties

protected CredentialsProperties getCredProperties(ActionExecutor.Context context,
                                      java.lang.String credName)
                                           throws java.lang.Exception

Throws:

java.lang.Exception

start

public void start(ActionExecutor.Context context,
         org.apache.oozie.client.WorkflowAction action)
           throws ActionExecutorException

Description copied from class: ActionExecutor

Start an action.

The ActionExecutor.Context.setStartData(java.lang.String, java.lang.String, java.lang.String) method must be called within this method.

If the action has completed, the ActionExecutor.Context.setExecutionData(java.lang.String, java.util.Properties) method must be called within this method.

Specified by:

start in class ActionExecutor

Parameters:

context - executor context.

action - the action to start.

Throws:

ActionExecutorException - thrown if the action could not start.

end

public void end(ActionExecutor.Context context,
       org.apache.oozie.client.WorkflowAction action)
         throws ActionExecutorException

Description copied from class: ActionExecutor

End an action after it has executed.

The ActionExecutor.Context.setEndData(org.apache.oozie.client.WorkflowAction.Status, java.lang.String) method must be called within this method.

Specified by:

end in class ActionExecutor

Parameters:

context - executor context.

action - the action to end.

Throws:

ActionExecutorException - thrown if the action could not end.

createJobClient

protected org.apache.hadoop.mapred.JobClient createJobClient(ActionExecutor.Context context,
                                                 org.apache.hadoop.mapred.JobConf jobConf)
                                                      throws HadoopAccessorException

Create job client object

Parameters:

context -

jobConf -

Returns:

JobClient

Throws:

HadoopAccessorException

getRunningJob

protected org.apache.hadoop.mapred.RunningJob getRunningJob(ActionExecutor.Context context,
                                                org.apache.oozie.client.WorkflowAction action,
                                                org.apache.hadoop.mapred.JobClient jobClient)
                                                     throws java.lang.Exception

Throws:

java.lang.Exception

getActualExternalId

protected java.lang.String getActualExternalId(org.apache.oozie.client.WorkflowAction action)

Useful for overriding in actions that do subsequent job runs such as the MapReduce Action, where the launcher job is not the actual job that then gets monitored.

check

public void check(ActionExecutor.Context context,
         org.apache.oozie.client.WorkflowAction action)
           throws ActionExecutorException

Description copied from class: ActionExecutor

Check if an action has completed. This method must be implemented by Async Action Executors.

If the action has completed, the ActionExecutor.Context.setExecutionData(java.lang.String, java.util.Properties) method must be called within this method.

If the action has not completed, the ActionExecutor.Context.setExternalStatus(java.lang.String) method must be called within this method.

Specified by:

check in class ActionExecutor

Parameters:

context - executor context.

action - the action to end.

Throws:

ActionExecutorException - thrown if the action could not be checked.

getActionData

protected void getActionData(org.apache.hadoop.fs.FileSystem actionFs,
                 org.apache.hadoop.mapred.RunningJob runningJob,
                 org.apache.oozie.client.WorkflowAction action,
                 ActionExecutor.Context context)
                      throws HadoopAccessorException,
                             org.jdom.JDOMException,
                             java.io.IOException,
                             java.net.URISyntaxException

Get the output data of an action. Subclasses should override this method to get action specific output data.

Parameters:

actionFs - the FileSystem object

runningJob - the runningJob

action - the Workflow action

context - executor context

Throws:

HadoopAccessorException

org.jdom.JDOMException

java.io.IOException

java.net.URISyntaxException

getCaptureOutput

protected boolean getCaptureOutput(org.apache.oozie.client.WorkflowAction action)
                            throws org.jdom.JDOMException

Throws:

org.jdom.JDOMException

kill

public void kill(ActionExecutor.Context context,
        org.apache.oozie.client.WorkflowAction action)
          throws ActionExecutorException

Description copied from class: ActionExecutor

Kill an action.

The ActionExecutor.Context.setEndData(org.apache.oozie.client.WorkflowAction.Status, java.lang.String) method must be called within this method.

Specified by:

kill in class ActionExecutor

Parameters:

context - executor context.

action - the action to kill.

Throws:

ActionExecutorException - thrown if the action could not be killed.

isCompleted

public boolean isCompleted(java.lang.String externalStatus)

Description copied from class: ActionExecutor

Return if the external status indicates that the action has completed.

Specified by:

isCompleted in class ActionExecutor

Parameters:

externalStatus - external status to check.

Returns:

if the external status indicates that the action has completed.

getShareLibNames

protected java.lang.String[] getShareLibNames(ActionExecutor.Context context,
                                  org.jdom.Element actionXml,
                                  org.apache.hadoop.conf.Configuration conf)

Return the sharelib names for the action.

If NULL or empty, it means that the action does not use the action sharelib.

If a non-empty string, i.e. foo, it means the action uses the action sharelib sub-directory foo and all JARs in the sharelib foo directory will be in the action classpath. Multiple sharelib sub-directories can be specified as a comma separated list.

The resolution is done using the following precedence order:

• action.sharelib.for.#ACTIONTYPE# in the action configuration
• action.sharelib.for.#ACTIONTYPE# in the job configuration
• action.sharelib.for.#ACTIONTYPE# in the oozie configuration
• Action Executor getDefaultShareLibName() method

Parameters:

context - executor context.

actionXml -

conf - action configuration.

Returns:

the action sharelib names.

getDefaultShareLibName

protected java.lang.String getDefaultShareLibName(org.jdom.Element actionXml)

Returns the default sharelib name for the action if any.

Parameters:

actionXml - the action XML fragment.

Returns:

the sharelib name for the action, NULL if none.

getShareLibFilesForActionConf

public java.lang.String[] getShareLibFilesForActionConf()

setActionCompletionData

protected void setActionCompletionData(ActionExecutor.Context context,
                           org.apache.hadoop.fs.FileSystem actionFs)
                                throws java.io.IOException,
                                       HadoopAccessorException,
                                       java.net.URISyntaxException

Sets some data for the action on completion

Parameters:

context - executor context

actionFs - the FileSystem object

Throws:

java.io.IOException

HadoopAccessorException

java.net.URISyntaxException

requiresNameNodeJobTracker

public boolean requiresNameNodeJobTracker()

Description copied from class: ActionExecutor

Returns true if this action type requires a NameNode and JobTracker. These can either be specified directly in the action via <name-node> and <job-tracker>, from the fields in the global section, or from their default values. If false, Oozie won't ensure (i.e. won't throw an Exception if non-existant) that this action type has these values.

Overrides:

requiresNameNodeJobTracker in class ActionExecutor

Returns:

true if a NameNode and JobTracker are required; false if not

supportsConfigurationJobXML

public boolean supportsConfigurationJobXML()

Description copied from class: ActionExecutor

Returns true if this action type supports a Configuration and JobXML. In this case, Oozie will include the <configuration> and <job-xml> elements from the global section (if provided) with the action. If false, Oozie won't add these.

Overrides:

supportsConfigurationJobXML in class ActionExecutor

Returns:

true if the global section's Configuration and JobXML should be given; false if not