...

Overview

A new dynamic “extension”  "extension" subsystem in the Universal Automation Center (UAC) platform allows developers to implement custom solutions that can be integrated into UAC. These “solutions” could be, for example, a task, trigger, or monitor.

...

The initial phase (and this document) focuses on a “Task” type Extension. However, the architecture is designed with other extension types in mind.

Image RemovedImage Added

As illustrated in the diagram above:

1. Developer creates Extension implementation and zips up the Extension Template, the Python Extension module and any dependent non-standard Python import modules.

2. The zipped package is uploaded to the marketplace.

3. A user downloads the extension to the Universal Controller.

4. The Universal Controller deploys the Extension to an Agent.

5. The Agent persists the Extension to the file system in the Extension Repository. (after authenticating the deployment with the Controller supplied SHA-256 checksum.

6. The Agent starts a Worker process to execute an Extension instance (supporting bi-directional communication).

7. The Agent authenticates the Extension that is to be run (comparing its SHA-256 checksum with the Controllers reported checksum) and inserts a driver script into the Worker process starts the Extension instance.

Universal Extension Task Overview

A Universal Extension Task is comprised of the following:

...

At a high level, the main difference between an Extension task and a Windows or Unix task is that the Agent is managing (or caching) a repository of modules that contain the functionality for the various “Extension” implementations.

Image RemovedImage Added

Implementation

UAG has been enhanced in the following ways to support the new Extension subsystem:

...

Extension modules were developed by extending a Stonebranch provided base class (UniversalExtension).  This base class provides the functionality required to integrate, manage, and orchestrate custom functionality running on the Agent system with resources (for example, task instances) running in the Controller.

Agent Start-up

Upon start-up, UAG performs the following Extension specific operations:

• Python Discovery
• Extension Repository Initialization

Python Discovery

Python discovery is the process of locating available Python executables on the system and categorizing them by version. This discovery process is performed once at start-up to prepare for efficient Python Resolution later during the execution of Extension instances.

...

For each Python executable found, UAG Extension Manager determines the version and record an entry (version and path) in an internal list for later lookup.

Extension Repository Initialization

Upon Agent start-up, UAG scans its Extension Repository for installed extension modules. Extension modules are stored in the repository as zipimport files (Python import modules stored as Zip archives).

...

The collected Extension information is stored in an internal table for later use with the JSS-HELLO message during Agent registration.

Agent Registration

Universal Extension Deployment

An Agent registering with the Universal Controller has the ability to automatically accept deployment of available Universal Extensions.

However, the owner of an Agent deployment may wish to control which (if any) Universal Extensions are allowed to be installed by the Controller. Therefore, the following configuration options have been added to UAG to control Extension deployment:

• EXTENSION_ACCEPT_LIST
• EXTENSION_DEPLOY_ON_REGISTRATION

An Agent must inform the Controller of which Extensions are currently installed on its system (collected at Agent start-up). Additionally, the Agent must supply the checksum associated with each installed Extension. This is required for the Controller to know which (if any) Extensions need to be installed or replaced on the Agent system. Any Extension on the Agent system with a checksum that does not match the checksum of the associated Extension in the Controller’s repository must be replaced.

Extension Manager

UAG is enhanced with a new internal component (Extension Manager) that manages the execution of Extensions and facilitates the flow of messages between the Controller and the Extension instances.  Extension instances run in a Python interpreter process (the Extension Worker Process) started in the security context of the user specified on the task definition (as do Windows and Unix/Linux tasks).

...

• Starting and stopping Extension Worker processes. 

• Authenticating Extension module with Controller provided checksum prior to import.

• Initiating Extension instances within a Worker process.

• Relaying information from Extension instances running in the Worker process to the Controller.

  • Status updates for associated output fields

  • Universal Events

• Executing Extension Commands (initiated from the Controller) to Worker Process.

• Canceling Extension instance (along with the Worker process) if/when requested by the Controller.

Extension Worker Process

Extension Worker processes are Python interpreter instances that are started by the Extension Manager.  Worker processes exist to run Extension instances. 

...

Output generated by the Extension instance that is written to stdout and stderr are redirected to cache spool files that are created in the UAG cache directory.

Security Context

Each Worker process is created on demand to process a specific Extension operation.  Each Universal Extension task definition specifies the user credentials that the task should run under.  If no credentials are specified, the task should run under the security context of UAG. Therefore, when a worker process is created, it is created with the security context specified by the Extension task it is intended to process. 

Starting an Extension instance

Extension instances are started when UAG receives a JSS-LAUNCH, JSS-UNVCHOICEREQ, or JSS-UNVCMDREQ message from the Controller.

...

After starting the Worker process, the process resources (CSProcess structure, “message output channel” handle, etc.) are stored in an appropriate data structure to be used for monitoring the Worker process over the lifetime of the Extension instance.

Worker Process Management

Once the stdin of the Worker process is closed and the Extension Worker begins executing the Extension instance, no further input is required from the Extension Manager. At that point the management of the Worker process becomes a monitoring operation.

...

• ESS-STATUS-UPDATE

• ESS-UNVEVENT

• ESS-EXT-COMPLETE

• ESS-CMD-COMPLETE

• ESS-CHOICE-CMD-COMPLETE

Warm Start Processing

Warm Start Processing is a term used to refer to a process UAG goes through upon startup by which all task instances that were active at the time of the last shutdown (intentional or otherwise) are reviewed and proper action is taken based on state and platform.

...

1. The process associated with the task instance has completed between the time of UAG shutdown and UAG restart.

2. The process associated with the task instance is still running at the time UAG restarts.

Scenario 1: Process has completed

For scenario 1, where the process has completed, UAG will send a JSS-STATUS(JOB INDOUBT) message back to the Controller to indicate that UAG is unable to determine how things turned out for the process in question. This behavior is consistent between Unix and Windows and no differences exist between Universal Extension tasks and other task types.

Scenario 2: Process is still active

For scenario 2, where the process is still active, there is a difference in behavior between platforms. On Unix platforms, UAG will send a JSS-STATUS(JOB INDOUBT) message back to the Controller just like with scenario 1. No attempt is made to resume monitoring the process. On Windows platforms, UAG will resume monitoring the process. Once monitoring has resumed, the task processing continues as though there was never a termination of UAG.

Warm Start Processing Enhancements for Universal Extension Tasks

Unlike other task types, Universal Extension Worker processes include a message channel (pipe) that allows the Extension instance to send messages back to UAG. If a UAG shutdown occurs while an Extension instance is still active, the message channel between the worker process and UAG is broken. However, all messages sent from Universal Extension instance must be processed in order to consider the execution a success.

...

The message cache is implemented as a dedicated flat file in the UAG cache directory. If the Extension Instance is unable to to send messages over the pipe (due to a UAG shutdown), the Extension will revert to the message cache. This allows UAG to “reconnect” to the message stream emitted by an Extension instance that is monitored after a Warm start. The process is transparent to the Controller. This brings the Warm Start Processing behavior of Extension tasks in line with other task types (from a user/Controller perspective).

UniversalExtension Base Class Package

The UniversalExtension base class package is a new Agent “component” that is part of the core Agent installation. It is a Python package that provides the functionality required to support the execution of custom Extensions within a Worker process.

...

For detailed information on this package, see /wiki/spaces/DEV/pages/1312018.

UniversalExtension Interface

def extension_start(self, state)

This method must be overridden by the custom Extension class that derives from the UniversalExtension class. It is called by UniversalExtension base class in response to a JSS-LAUNCH message sent from the Controller. This is essentially the main() function for an Extension implementation and is the starting point for work that will be performed. The state parameter passes in a dictionary of Extension instance fields that were defined in the Extension template.

Communication Methods

def update_extension_status(fields):

This method can be called at any time by the Extension instance. It is used to propagate state changes back to the associated extension instance in the Controller. Essentially, any/all output fields defined in the associated Extension Template can be updated using this method. This method results in an ESS-STATUS-UPDATE message being sent from the Worker process to the UAG Extension Manager, followed by a JSS-STATUS(JOB UPDATE) message being sent from UAG Extension Manager to the Controller (via OMS server).

...

{
  # Output fields to be updated
  "sysid": "BW7",
  "jobid": 81762549
}

def publish_extension_event(event_state):

This method can be called at any time by the Extension instance. It is used to publish a Universal Event to the associated Universal Controller.

...

{
  # The unique Universal Event name defined within the Universal Template.
  "event_name": "job_complete",
  
  # Time to live; how long, in minutes, the Universal Event data is kept (in the Controller).
  "ttl": 30,
  
  # Dictionary of attributes associated with the Universal Event definition.
  # An attribute should be added for each attribute defined in Universal Event Template 
  # definition associated with the specified "event_name". 
  # Keys in the "event_attributes" dictionary correlate with an associated 
  # Universal Event attribute defined in the associated template.
  "event_attributes": 
  {
    "sysid": "BW7",
    "jobname": "JOB-1",
    "jobid": 81762549
  }
}

Universal Extension Definition (module)

Universal Extension modules allow for the development of simple or complex solutions that tightly integrate with the Universal Controller.

...

• A mandatory singular primary operation (e.g. task implementation)

• An optional collection of Dynamic Commands

Primary Operation

The primary operation is the focus of the Extension and must be implemented. This is done by overriding the extension_start method of the UniversalExtension base class.

Of course, extension_start is just an entry point. A single primary operation could perform any number of variants or sub operations based on values passed in from the Controller.

Secondary Operation - Dynamic Commands

Dynamic commands are intended to support the functionality of the Primary Operation of the Universal Extension, They can be used to return additional information for a task instance or to carry out some related (or unrelated) action on behalf of the task instance. Dynamic commands can pass instance specific field data values from the task instance form to the extension process that executes the command on the Agent.

...

Sample Dynamic Command Return Value

Sample 1

  @dynamic_command("sample_1")
  def command_sample_1(self, state):
      """Dynamic command."""
      return ExtensionResult(
          message: 'sample_1 completed successfully',
          output = True,
          output_data = 'Hello from dynamic command sample_1!',
          output_name = 'DYNAMIC_OUTPUT')

...

    @dynamic_command("sample_2")
    def command_sample_2(self, state):
        """Dynamic command."""
        ...
        if (something_went_wrong):
            return ExtensionResult(rc = 1, message = "Something went wrong")
        return ExtensionResult(
            message: 'sample_2 completed successfully',
            output = True,
            output_data = str(state),
            output_name = 'DYNAMIC_OUTPUT')

Dynamic Command Use Cases

Extension developers can use dynamic commands for many different purposes.

Dynamic Choice Field Commands

Dynamic Choice commands are helper functions for dynamically populating choice fields on the Universal Extension task definition form in the Controller (like with the PeopleSoft task type).

...

Sample Dynamic Choice Commands

Sample 1

    @dynamic_choice_command("choice_1")
    def choice_2(self, state):
        """Dynamic choice command."""
        self.log.info("Entering choice_2")
        message = 'Message: Hello from dynamic choice command choice_2!'
        values = ['value 1', 'value 2', 'value 3', 'value 4']
        self.log.info("Exiting choice_2")
        return ExtensionResult(message = message, values = values)

...

   @dynamic_command("sample_2")
    def command_sample_2(self, state):
        """Dynamic command."""
        ...
        if (something_went_wrong):
            return ExtensionResult(rc = 1, message = "Something went wrong.")
        return ExtensionResult(
                message = 'Message: Hello from dynamic command sample_2!', 
                values = ['value 1', 'value 2', 'value 3', 'value 4'])

Dynamic Choice Field Use cases

Case 1: Helper commands on task definition form

...

This scenario matches the behavior that is seen in the existing PeopleSoft task type. In this case, the command functionality is related to the Extension type but, is unrelated to any specific work that has been executed by a task instance.

Extension Repository

UAG will manage (in cooperation with the Controller) a repository of Extension modules.  This management will include synchronization with the Controller in terms of which extension modules are available or allowed.  The end goal is for the Controller to push Extension solutions down to the Agents as needed.

The extension repository is a path on the Agent file system where "Extension" modules are stored.  The modules themselves are stored as zipimport files (Python import modules stored as Zip archives).

Windows

\Program Files\Universal\universal\UAGSrv\extensions

*nix

./var/opt/universal/uag/extensions

It may be desirable to have this be a configurable value.  Therefore, the following configuration value is proposed for the UAG configuration file:

extension_repository

Extension Repository Security

Extension modules are written to the file system by UAG (after being pushed down from the Controller via OMS messages). Therefore, they will be owned by the account used to execute the Broker.

...

Write permission should be limited to the owner (Broker account) to prevent tampering. However, that is not enough. A malicious user with sufficient authority could potentially manipulate extension module code to perform undesirable actions. Therefore, another means is required to guaranty the integrity of the extension modules.

Checksums

The Controller is the central repository/authority with regards to Extension modules. Therefore, the Controller can pass down a known checksum for the Extension module associated with an Extension work request. The Extension manager can then use the provided checksum to verify the integrity of the Extension module residing on the Agent’s file system. If the checksums do not match, the Extension Manager will consider it a pre-start failure and return a JSS-STATUS(ERROR) message to the Controller with an ERRDESC value of “Extension checksum mismatch”. and a EXT_CHECKSUM_MISMATCH value of true.

...