Overview

A new dynamic “extension” subsystem in the Universal Automation Center (UAC) platform allows developers to implement custom solutions that can be integrated into UAC. These “solutions” could be, for example, a task, trigger, or monitor.

The initial phase of this subsystem focuses on tasks.

This feature provides an easy way for Stonebranch, customers, and/or third party developers with special domain knowledge, to develop new solutions that can be seamlessly integrated into the UAC software stack. This is especially beneficial when the Domain knowledge required for a solution does not exist within the Stonebranch development teams and/or justification does not exist to devote the necessary resources.

While the Universal Task addresses this issue to some extent, it does not provide the level of integration and customer experience that can be achieved with the Universal Extension concept.

The main differentiators between Universal Extensions and Universal Tasks are:

• Bidirectional communication between the Extension instance and the Universal Controller. This allows an Extension instance running on an Agent system to dynamically update the associated instance running in the Controller as the state changes. For example, a “Task” type extension could update output fields on a task instance in the Controller (for example, Job ID, Run Status, Distribution Status, and Percentage Complete).

• Dynamic command definitions to support a complete solution (not just a single script execution). This allows an Extension solution to do things such as:

  • Provide commands that can help populate task definition form fields (such as is currently done with the PeopleSoft task type),

  • Provide a relative set of commands to be used on a task instance form that can be used to perform actions related to the unit of work that is specific to that task instance (such as is currently done with the SAP task type: Abort SAP job, Interrupt SAP Process Chain, etc.).

• Dynamic event definition/generation (Universal Events) allow Extension instances to trigger actions in the Controller. Universal Events defined and monitored in the Controller could be raised by an Extension instance on an Agent.

Universal Agent has been enhanced to support this new Universal Extension subsystem.  The Extension subsystem allows various types of functionality to be developed and integrated into the UAC software stack (for example: tasks, triggers, and monitors).

The generic Extension concept supports the development of some primary operation (for example, task, along with an optional set of related secondary/supporting operations (Choice Commands, Dynamic Commands).

The initial phase (and this document) focuses on a “Task” type Extension. However, the architecture is designed with other extension types in mind.

As illustrated in the diagram above:

1. Developer creates Extension implementation and zips up the Extension Template, the Python Extension module and any dependent non-standard Python import modules.

2. The zipped package is uploaded to the marketplace.

3. A user downloads the extension to the Universal Controller.

4. The Universal Controller deploys the Extension to an Agent.

5. The Agent persists the Extension to the file system in the Extension Repository. (after authenticating the deployment with the Controller supplied SHA-256 checksum.

6. The Agent starts a Worker process to execute an Extension instance (supporting bi-directional communication).

7. The Agent authenticates the Extension that is to be run (comparing its SHA-256 checksum with the Controllers reported checksum) and inserts a driver script into the Worker process starts the Extension instance.

Universal Extension Task Overview

A Universal Extension Task is comprised of the following:

• An Extension Task Template definition (defined in the Controller)

  • Input field definitions

  • Output-only field definitions

  • Custom command definitions

• An Extension module developed in Python (delivered/stored to the Agent system for execution)

  • Contains the code that provides the functionality for the Extension’s primary operation (e.g. task)

  • Contains code that provides the functionality for the Extension’s “Dynamic Commands”

Extension modules will be developed in an unspecified development environment.

Fully developed Universal Extensions are distributable (via the Stonebranch Marketplace, in-house Controller-to-Controller bundle promotion, etc.) to target Controller systems.

In a UAC deployment, the Controller serves as the primary repository and distributor of Universal Extensions. The Controller is responsible for pushing extension modules down to Agents as needed.

The deployment of Extension modules from Controller to Agent takes place via OMS messaging.

On the Agent side, Extension modules are stored in a cache repository (referred to as the Extension Repository throughout this document). This cache repository is simply be a directory on the Agent file system where Extensions modules passed from the Controller are stored. Extension modules are stored on the Agent file system as zipimport files (Python import modules stored as Zip archives).

• For Windows systems (System Mode install), the Extension Repository is under: \Program Files\Universal\universal\UAGSrv\extensions

• For *nix systems (System Mode install), the Extension Repository is under: ./var/opt/universal/uag/extensions

• User mode installs use an equivalent path, for the “extension” directories above (for example, under uag in the “unvdata” install path).

The Universal Agent processes Extension tasks in much the same way as any other task type.  A unit of work is executed on behalf of a Controller request.  Artifacts of the unit of work are returned to the Controller (for example: standard output, standard error, return codes).

• Standard output generated from the Extension task instances is directed to *_stdout files in the UAG cache directory.

• Standard error generated from the Extension task instances is directed to *_stderr files in the UAG cache directory.

At a high level, the main difference between an Extension task and a Windows or Unix task is that the Agent is managing (or caching) a repository of modules that contain the functionality for the various “Extension” implementations.

Implementation

UAG has been enhanced in the following ways to support the new Extension subsystem:

• Agent start-up

• Agent registration

• Extension Task-related messages

• Extension Manager component

• UniversalExtension Python Base Class

• Extension Repository

This initial implementation of Universal Extensions uses Python for the language in which Extension modules are developed. 

Extension modules were developed by extending a Stonebranch provided base class (UniversalExtension).  This base class provides the functionality required to integrate, manage, and orchestrate custom functionality running on the Agent system with resources (for example, task instances) running in the Controller.

Agent Start-up

Upon start-up, UAG performs the following Extension specific operations:

• Python Discovery
• Extension Repository Initialization

Python Discovery

Python discovery is the process of locating available Python executables on the system and categorizing them by version. This discovery process is performed once at start-up to prepare for efficient Python Resolution later during the execution of Extension instances.

Universal Extensions may have specific Python version requirements that must be met in order for the Extension to run correctly. These requirements are specified in the “requires_python” metadata for the Extension (specified in extension.yml in the Extension zip module).

To satisfy the Python requirements of an Extension, UAG Extension Manager must know the available Python interpreters installed on the target system and choose one (if any) that meets all specified requirements.

Python discovery is performed by:

1. Checking for a Python distribution installed with the Universal Agent.

2. [Windows specific] Checking the “HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore” registry key for a Python installation.

3. Checking the value specified for the UAG Server EXTENSION_PYTHON_LIST configuration option, which can specify a comma-separated list of locations in which Python is installed.

1. This option has a default value of /usr/bin/python3, /usr/bin/python, and /usr/libexec/platform-python directories/symlinks for Unix agents.
2. For Windows, there is no default value. If this option is left empty, Python discovery stops at step 2.

For each Python executable found, UAG Extension Manager determines the version and record an entry (version and path) in an internal list for later lookup.

Extension Repository Initialization

Upon Agent start-up, UAG scans its Extension Repository for installed extension modules. Extension modules are stored in the repository as zipimport files (Python import modules stored as Zip archives).

For each extension module found in the Extension Repository, UAG collects the following information:

• Extension Name

  The Extension module file name (not including the file extension) is used as the Extension name. This should equate to the Universal Extension Template Name of the related Universal Extension defined in the Controller system to which the Agent connects.

• Extension Checksum

  A SHA-256 checksum of Extension file

The collected Extension information is stored in an internal table for later use with the JSS-HELLO message during Agent registration.

Agent Registration

Universal Extension Deployment

An Agent registering with the Universal Controller has the ability to automatically accept deployment of available Universal Extensions.

However, the owner of an Agent deployment may wish to control which (if any) Universal Extensions are allowed to be installed by the Controller. Therefore, the following configuration options have been added to UAG to control Extension deployment:

• EXTENSION_ACCEPT_LIST
• EXTENSION_DEPLOY_ON_REGISTRATION

An Agent must inform the Controller of which Extensions are currently installed on its system (collected at Agent start-up). Additionally, the Agent must supply the checksum associated with each installed Extension. This is required for the Controller to know which (if any) Extensions need to be installed or replaced on the Agent system. Any Extension on the Agent system with a checksum that does not match the checksum of the associated Extension in the Controller’s repository must be replaced.

Extension Manager

UAG is enhanced with a new internal component (Extension Manager) that manages the execution of Extensions and facilitates the flow of messages between the Controller and the Extension instances.  Extension instances run in a Python interpreter process (the Extension Worker Process) started in the security context of the user specified on the task definition (as do Windows and Unix/Linux tasks).

In general, the Extension Manager is responsible for handling work requests related to Extensions and managing the work resources and resulting message exchanges required to process the work.

Specifically, this involves:

• Starting and stopping Extension Worker processes. 

• Authenticating Extension module with Controller provided checksum prior to import.

• Initiating Extension instances within a Worker process.

• Relaying information from Extension instances running in the Worker process to the Controller.

  • Status updates for associated output fields

  • Universal Events

• Executing Extension Commands (initiated from the Controller) to Worker Process.

• Canceling Extension instance (along with the Worker process) if/when requested by the Controller.

Extension Worker Process

Extension Worker processes are Python interpreter instances that are started by the Extension Manager.  Worker processes exist to run Extension instances. 

The Extension Manager controls the Worker process by formulating a small Python script designed to run a target Extension module and injecting that script into the Worker process over stdin.

The Extension instance running within the Worker process sends messages to the Extension Manager over a control channel (pipe).

Output generated by the Extension instance that is written to stdout and stderr are redirected to cache spool files that are created in the UAG cache directory.

Security Context

Each Worker process is created on demand to process a specific Extension operation.  Each Universal Extension task definition specifies the user credentials that the task should run under.  If no credentials are specified, the task should run under the security context of UAG. Therefore, when a worker process is created, it is created with the security context specified by the Extension task it is intended to process. 

Starting an Extension instance

Extension instances are started when UAG receives a JSS-LAUNCH, JSS-UNVCHOICEREQ, or JSS-UNVCMDREQ message from the Controller.

Upon receiving the JSS-LAUNCH, JSS-UNVCHOICEREQ, or JSS-UNVCMDREQ message, the Extension Manager performs the following operations to start the new Extension instance:

• Authenticate the target Extension module currently residing in the Agents Extension Repository.

  • The checksum of the target extension module must match the checksum provided by the Controller in the initiating JSS-LAUNCH, JSS-UNVCHOICEREQ, or JSS-UNVCMDREQ message.

  • If the checksums do not match, the Extension is not started and a JSS-STATUS(ERROR) message will be returned to the Controller with an ERRDESC value of “Extension checksum mismatch”.

  • This is considered a start failure and no further actions will be taken towards starting the extension.

• Open stdout/stderr spool files in the UAG cache directory - into which stdout and stderr of the Worker process will be redirected (<execid>_stdout and <execid>_stderr).

• Open a pipe to be used as a “message output channel” for the Worker process.

• Start a Worker process using CSK where:

  • The process runs under the security context of the runtime credentials specified in the JSS message.

    • If no credentials are specified in the JSS message, the process runs under the security context of the account the UAG runs under.

  • The stdout and stderr are redirected to the cache files mentioned above (<execid>_stdout and <execid>_stderr).

• Pass a handle to the write end of the “message output channel” into the Worker process.

• Generate a small Python script that will run the target Extension module with the parameters specified in the initiating JSS message

• Close stdin for the Worker process cause the Python interpreter to begin processing the script (and thus the Extension instance).

Generating a SHA-256 checksum for each attempted invocation of an extension instance is obviously ideal in terms of security. However, it could potentially add a significant performance hit to a system that makes heavy use of Extension tasks.

It may be sufficient to only generate a checksum (Agent side) if the cached checksum is older than some threshold. A threshold as small as 1 second could conceivably have significant impact on a burst of short lived Extension executions targeting the same Extension module.

Auto-deployment for “Extension checksum mismatch” start failures

The Controller could silently handle “Extension checksum mismatch” start failures and auto-deploy the target Extension module to the target Agent.

Following a successful deployment, the Controller could re-launch the Extension instance.

After starting the Worker process, the process resources (CSProcess structure, “message output channel” handle, etc.) are stored in an appropriate data structure to be used for monitoring the Worker process over the lifetime of the Extension instance.

Worker Process Management

Once the stdin of the Worker process is closed and the Extension Worker begins executing the Extension instance, no further input is required from the Extension Manager. At that point the management of the Worker process becomes a monitoring operation.

A dedicated monitoring thread will continuously monitor Running Worker processes for messages and completion. Potential messages sent from the Worker process are:

• ESS-STATUS-UPDATE

• ESS-UNVEVENT

• ESS-EXT-COMPLETE

• ESS-CMD-COMPLETE

• ESS-CHOICE-CMD-COMPLETE

Warm Start Processing

Warm Start Processing is a term used to refer to a process UAG goes through upon startup by which all task instances that were active at the time of the last shutdown (intentional or otherwise) are reviewed and proper action is taken based on state and platform.

Two general scenarios exist for tasks that were active at the time of the last UAG shutdown:

1. The process associated with the task instance has completed between the time of UAG shutdown and UAG restart.

2. The process associated with the task instance is still running at the time UAG restarts.

Scenario 1: Process has completed

For scenario 1, where the process has completed, UAG will send a JSS-STATUS(JOB INDOUBT) message back to the Controller to indicate that UAG is unable to determine how things turned out for the process in question. This behavior is consistent between Unix and Windows and no differences exist between Universal Extension tasks and other task types.

Scenario 2: Process is still active

For scenario 2, where the process is still active, there is a difference in behavior between platforms. On Unix platforms, UAG will send a JSS-STATUS(JOB INDOUBT) message back to the Controller just like with scenario 1. No attempt is made to resume monitoring the process. On Windows platforms, UAG will resume monitoring the process. Once monitoring has resumed, the task processing continues as though there was never a termination of UAG.

Warm Start Processing Enhancements for Universal Extension Tasks

Unlike other task types, Universal Extension Worker processes include a message channel (pipe) that allows the Extension instance to send messages back to UAG. If a UAG shutdown occurs while an Extension instance is still active, the message channel between the worker process and UAG is broken. However, all messages sent from Universal Extension instance must be processed in order to consider the execution a success.

Therefore, in order to “reconnect” to a Universal Extension instance during Warm Start Processing, UAG must regain access to the messages in addition to just monitoring the status of the process. To prevent message loss in such a case, Universal Extension instances are provided a message cache (in addition to the message pipe) during the initial instantiation.

The message cache is implemented as a dedicated flat file in the UAG cache directory. If the Extension Instance is unable to to send messages over the pipe (due to a UAG shutdown), the Extension will revert to the message cache. This allows UAG to “reconnect” to the message stream emitted by an Extension instance that is monitored after a Warm start. The process is transparent to the Controller. This brings the Warm Start Processing behavior of Extension tasks in line with other task types (from a user/Controller perspective).

UniversalExtension Base Class Package

The UniversalExtension base class package is a new Agent “component” that is part of the core Agent installation. It is a Python package that provides the functionality required to support the execution of custom Extensions within a Worker process.

Custom Extension implementations must provide a class that derives from the UniversalExtension base class. The UniversalExtension base class provides an interface that the Extension developer must implement. Additionally, it provides methods that allow the Extension instance to propagate state information back to the associated Extension instance running in the Controller.

The UniversalExtension base class package contains the following modules:

For detailed information on this package, see /wiki/spaces/DEV/pages/1312018.

UniversalExtension Interface

def extension_start(self, state)

This method must be overridden by the custom Extension class that derives from the UniversalExtension class. It is called by UniversalExtension base class in response to a JSS-LAUNCH message sent from the Controller. This is essentially the main() function for an Extension implementation and is the starting point for work that will be performed. The state parameter passes in a dictionary of Extension instance fields that were defined in the Extension template.

Communication Methods

def update_extension_status(fields):

This method can be called at any time by the Extension instance. It is used to propagate state changes back to the associated extension instance in the Controller. Essentially, any/all output fields defined in the associated Extension Template can be updated using this method. This method results in an ESS-STATUS-UPDATE message being sent from the Worker process to the UAG Extension Manager, followed by a JSS-STATUS(JOB UPDATE) message being sent from UAG Extension Manager to the Controller (via OMS server).

The fields parameter expects a dictionary of output fields to be sent back to the controller for the associated Extension instance. Field names are implementation dependent and correlate with the Universal Template field names in the Controller's Template definition. For example:

{
  # Output fields to be updated
  "sysid": "BW7",
  "jobid": 81762549
}

def publish_extension_event(event_state):

This method can be called at any time by the Extension instance. It is used to publish a Universal Event to the associated Universal Controller.

Universal Events are optionally defined in the Universal Extension Template as part of the Universal Extension definition.

This method results in an ESS-UNVEVENT message being sent from the Worker process to the UAG Extension Manager, followed by a JSS-UNVEVENT message being sent from UAG Extension Manager to the Universal Controller (via OMS server).

The event_state parameter expects a dictionary of event related fields - including a list of implementation defined “event attributes”. For example:

{
  # The unique Universal Event name defined within the Universal Template.
  "event_name": "job_complete",
  
  # Time to live; how long, in minutes, the Universal Event data is kept (in the Controller).
  "ttl": 30,
  
  # Dictionary of attributes associated with the Universal Event definition.
  # An attribute should be added for each attribute defined in Universal Event Template 
  # definition associated with the specified "event_name". 
  # Keys in the "event_attributes" dictionary correlate with an associated 
  # Universal Event attribute defined in the associated template.
  "event_attributes": 
  {
    "sysid": "BW7",
    "jobname": "JOB-1",
    "jobid": 81762549
  }
}

Universal Extension Definition (module)

Universal Extension modules allow for the development of simple or complex solutions that tightly integrate with the Universal Controller.

The UniversalExtension base class supports two types of custom functionality that can be invoked by the Controller:

• A mandatory singular primary operation (e.g. task implementation)

• An optional collection of Dynamic Commands

Primary Operation

The primary operation is the focus of the Extension and must be implemented. This is done by overriding the extension_start method of the UniversalExtension base class.

Of course, extension_start is just an entry point. A single primary operation could perform any number of variants or sub operations based on values passed in from the Controller.

Secondary Operation - Dynamic Commands

Dynamic commands are intended to support the functionality of the Primary Operation of the Universal Extension, They can be used to return additional information for a task instance or to carry out some related (or unrelated) action on behalf of the task instance. Dynamic commands can pass instance specific field data values from the task instance form to the extension process that executes the command on the Agent.

To implement a Dynamic Command in an Extension module, define a function or method with the following signature:

• def my_command_function_name(self, state):

Decorate the function with a @dynamic_command decorator:

• @dynamic_command(“dynamic_command_name”)

For example:

@dynamic_command(“my_dynamic_command”)
def my_command_function(self, state):
    # Do something 

In the code snippet above, “my_dynamic_command” is the command name given to the Dynamic Command in the Controller’s Extension definition template and my_command_function is the actual python function name from the Extension module that processes the command.

The @dynamic_command decorator provides a means for custom Extension implementations to associate a function defined in the Extension module with a Dynamic Command in the Controller's related Extension template definition. The @dynamic_command decorator is defined in the ./deco/command.py module in the universal_extension.zip package.

When a dynamic command function is called, the state function variable will contain a dictionary populated with dependent fields (as defined in the Dynamic Command template definition) from the command invocation source. The keys of the dictionary will equate to the corresponding template field names and the values will be the values of the fields in the command invocation source (task instance form) at the time of invocation.

Dynamic command functions must return an object of type ExtensionResult. ExtensionResult is a class defined in module ./extension_result.py in the universal_extension.zip base class module.

The ExtensionResult class is used to return the following attributes to the UniversalExtension base class:

Sample Dynamic Command Return Value

Sample 1

  @dynamic_command("sample_1")
  def command_sample_1(self, state):
      """Dynamic command."""
      return ExtensionResult(
          message: 'sample_1 completed successfully',
          output = True,
          output_data = 'Hello from dynamic command sample_1!',
          output_name = 'DYNAMIC_OUTPUT')

Sample 2

    @dynamic_command("sample_2")
    def command_sample_2(self, state):
        """Dynamic command."""
        ...
        if (something_went_wrong):
            return ExtensionResult(rc = 1, message = "Something went wrong")
        return ExtensionResult(
            message: 'sample_2 completed successfully',
            output = True,
            output_data = str(state),
            output_name = 'DYNAMIC_OUTPUT')

Dynamic Command Use Cases

Extension developers can use dynamic commands for many different purposes.

Dynamic Choice Field Commands

Dynamic Choice commands are helper functions for dynamically populating choice fields on the Universal Extension task definition form in the Controller (like with the PeopleSoft task type).

To implement a Dynamic Choice Command in an Extension module, define a function or method with the following signature:

• def my_choice_command_function_name(self, state):

Decorate the function with a @dynamic_choice_command decorator:

• @dynamic_choice_command(“dynamic_choice_command_name”)

For example:

@dynamic_choice_command(“choice_field”)
def my_choice_command_function(self, state):
    # Do something 

In the code snippet above, “my_dynamic_choice_command” is the command name given to the Dynamic Choice_Command in the Controller’s Extension definition template and my_command_function is the actual python function name from the Extension module that processes the command.

The @dynamic_choice_command decorator provide a means for custom Extension implementations to associate a function defined in the Extension module with a Dynamic Choice Field in the Controller's related Extension template definition. The @dynamic_choice_command decorator is defined in the universal_extension module.

When a dynamic choice command is called, the state function variable will contain a dictionary populated with dependent fields (as defined in the Dynamic Choice Field template definition) from the command invocation source (i.e. the task definition form in the Controller). The keys of the dictionary will equate to the corresponding template field names and the values will be the values of the fields in the command invocation source (task definition form, task instance form, etc.) at the time of invocation.

Dynamic Choice command functions must return an object of type ExtensionResult. ExtensionResult is a class defined in the extension_result module in the universal_extension package. The ExtensionResult class is used to return the following attributes to the UniversalExtension base class:

Sample Dynamic Choice Commands

Sample 1

    @dynamic_choice_command("choice_1")
    def choice_2(self, state):
        """Dynamic choice command."""
        self.log.info("Entering choice_2")
        message = 'Message: Hello from dynamic choice command choice_2!'
        values = ['value 1', 'value 2', 'value 3', 'value 4']
        self.log.info("Exiting choice_2")
        return ExtensionResult(message = message, values = values)

Sample 2

   @dynamic_command("sample_2")
    def command_sample_2(self, state):
        """Dynamic command."""
        ...
        if (something_went_wrong):
            return ExtensionResult(rc = 1, message = "Something went wrong.")
        return ExtensionResult(
                message = 'Message: Hello from dynamic command sample_2!', 
                values = ['value 1', 'value 2', 'value 3', 'value 4'])

Dynamic Choice Field Use cases

Case 1: Helper commands on task definition form

In this scenario, commands are used to retrieve data that will be used to populate form fields (like drop-downs). The commands may use input values from dependent fields on the form to pull data that is highly relevant to the task being defined.

This scenario matches the behavior that is seen in the existing PeopleSoft task type. In this case, the command functionality is related to the Extension type but, is unrelated to any specific work that has been executed by a task instance.

Extension Repository

UAG will manage (in cooperation with the Controller) a repository of Extension modules.  This management will include synchronization with the Controller in terms of which extension modules are available or allowed.  The end goal is for the Controller to push Extension solutions down to the Agents as needed.

The extension repository is a path on the Agent file system where "Extension" modules are stored.  The modules themselves are stored as zipimport files (Python import modules stored as Zip archives).

Windows

\Program Files\Universal\universal\UAGSrv\extensions

*nix

./var/opt/universal/uag/extensions

It may be desirable to have this be a configurable value.  Therefore, the following configuration value is proposed for the UAG configuration file:

extension_repository

Extension Repository Security

Extension modules are written to the file system by UAG (after being pushed down from the Controller via OMS messages). Therefore, they will be owned by the account used to execute the Broker.

Extension modules must be world readable to allow Worker processes running under the security context of an unprivileged user to load the modules into the Python interpreter.

Write permission should be limited to the owner (Broker account) to prevent tampering. However, that is not enough. A malicious user with sufficient authority could potentially manipulate extension module code to perform undesirable actions. Therefore, another means is required to guaranty the integrity of the extension modules.

Checksums

The Controller is the central repository/authority with regards to Extension modules. Therefore, the Controller can pass down a known checksum for the Extension module associated with an Extension work request. The Extension manager can then use the provided checksum to verify the integrity of the Extension module residing on the Agent’s file system. If the checksums do not match, the Extension Manager will consider it a pre-start failure and return a JSS-STATUS(ERROR) message to the Controller with an ERRDESC value of “Extension checksum mismatch”. and a EXT_CHECKSUM_MISMATCH value of true.