...
Universal Agent release 7.12.0.0 contains the following high-level features. For a complete list of all the included features and fixes please refer to the Universal Agent 7.12.x Maintenance list.
...
Data Pipeline
Backlog | Title | Description |
B- |
Update SAP LOW and HIGH data types to support up to 255 characters.
| Note | ||
|---|---|---|
| ||
Please make sure you are using the RFC 7.5 Library alongside with the Universal Agent 7.1.0.0 when using this Feature. Any combination other than that might lead to situations where the unavailability of the High option will not be detected and USAP will not identify truncating the field length as SAP enhancements are not present to support the exceeded Length of the Variant Field. |
USAP has been modified to support the enhanced variant structures when available. For NW RFC, this is relatively easy. We can check for the existence of the enhanced structures prior to calling the function module and act accordingly.
For classic RFC, it is more difficult. We have no way to determine ahead of time whether the function module in question has been updated with the enhanced variant structures. Therefore, for classic RFC, we will have to assume the enhanced structures are available on an initial call and automatically downgrade to non-enhanced structures on failure.
If the user specifies a variant HIGH/LOW field that exceeds the maximum size supported by the function module involved in an operation, USAP will terminate with exit code 202.
Error on exceeding size limit will be the expected behavior regardless of the variant structure being used (enhanced or original).
The HIGH/LOW size-limit changes based on function module, and whether enhanced structures are supported (see table below for system/function module/size limit expectations.
Universal Integration Platform
...
Backlog
...
Title
...
Description
...
B-13831
...
UIP: Add support for Cancel command within Universal Extension framework
...
The Universal Integration platform has been enhanced to all cancelling Extension based processes within UIP. The current behavior for cancelling an Extension instance (or any agent based task instance) is as follows:
Cancel is initiated at the Controller.
Controller sends JSS-CANCEL message to UAG and transitions the task instance in the Controller to "Cancel Pending".
UAG receives the JSS-CANCEL message and attempts to terminate process with CSK API function CSProcessCancel().
If the process has not terminated within 10 seconds, UAG calls CSK API function CSProcessTerminate().
If the cancel process completes successfully, UAG sends JSS-STATUS(JOB CANCELLED) back to Controller.
Upon receiving the JSS-STATUS(JOB CANCELLED) message, the Controller transitions the task instance to "Cancelled".
The result is that the Extension instance is terminated without any knowledge of the termination.
In order to give the Extension process a chance to respond to the cancel command, a new extension_cancel() method has been added to the Universal Extension API. This is an optional method that can be implemented by the Extension developer. If implemented, extension_cancel() will be called by the UniversalExtension base class when UAG receives a Cancel command from the Controller.
Within the extension_cancel() method, the Extension developer can perform whatever actions are needed or appropriate before the process is cancelled. However, Extension will not have unlimited time to perform these actions. The specific amount of time available for the cancel operation will be controlled by a UAG configuration option: Cancel Timeout (see below).
With the implementation of this backlog, the cancel behavior for Extension instances changes to the following:
Cancel is initiated at the Controller.
Controller sends a JSS-CANCEL message to UAG and transitions the task instance in the Controller to "Cancel Pending".
UAG receives the JSS-CANCEL message from the Controller and passes an ESS-CANCEL message to the UniversalExtension base class running in the Extension Worker process.
The UniversalExtension base class calls the extension_cancel() method allowing the Extension to perform cleanup.
The cancel operation is initiated with the call to extension_cancel() but, does not necessarily end when extension_cancel() completes.
If the cancel operation does not complete within the available time, the Extension instance will be forcibly cancelled by UAG.
When the cancel process completes, UAG sends JSS-STATUS(JOB CANCELLED) back to Controller.
Upon receiving the JSS-STATUS(JOB CANCELLED) message, the Controller transitions the task instance to "Cancelled".
...
B-13908
...
UIP: Add support for in-process Dynamic Commands
...
This backlog adds the ability to run a Dynamic Command within the running process of the task instance on which the command is called.
Currently, Dynamic Commands always run in a separate process.
Dynamic Commands support custom functionality and it cannot be known what use cases will be implemented. However, there are three general scenarios that must be considered.
Dynamic Command must run in separate process (current functionality).
Dynamic Command must run within the process of the associated Extension task instance.
It does not matter if the Dynamic Command runs in-process or not. Run in-process if available. Otherwise, create new process.
The Dynamic Command functionality was enhanced to support two main execution options that can be selected by the Extension developer:
Dynamic Command runs in a separate dedicated process (current functionality): out-of-process.
Dynamic Command runs within the process of the associated Extension task instance: in-process.
In-process Dynamic Commands also support the option to be run synchronously or asynchronously.
The Extension developer specifies the execution options at the Universal Template level in the Controller.
...
B-13821
...
UIP: Enhance "requires_python" logic to support '*' wildcard in clauses
...
UIP Extensions now allow usage of Wildcard in clause to implement a simpler way of identifying specific Python requirements.
UIP Extensions allow extension developers to specify if a certain version of Python is required using the requires_python option in the extension.yml file. Currently, the option does not support wildcards which results in unnecessary complexity.
For example, to target all Python 2.7.x binaries, the developer would have to specify the following requires_python: ">=2.7.0,<=2.7.9" . With wildcards, it is simple as requires_python: "==2.7.*
The solution allows wildcards for the == and != operators only.
...
B-13860
...
UIP: Refactor logging API
...
The Universal Extension repository contains a ExtensionLogger class that allows users to easily log data. The UniversalExtension class in universal_extension.py internally creates an instance of ExtensionLogger which can be accessed by users using self.log in their implementation of the Extension class. If the user calls a method, foo() from another module called test.py, and wants to log some messages in that method, the only way to do so currently is to pass in an instance of ExtensionLogger.
Instead of passing instances, the user is now able to import the logger in test.py and use it in foo().
The solution allows users to get access to the logger using the following import: from universal_extension import logger. Once imported, the user is able to perform calls such as logger.info('msg'), logger.error('msg') etc.
| Note | ||
|---|---|---|
| ||
To preserve compatibility, passing in an instance of the logger still works. |
...
B-13820
...
UIP: Allow Extension configuration options to be set at install time or via Broker start-up
...
Enhanced UIP to allow certain configuration Options as part of the Install or Broker start-up process.
Three configuration options are available to control Universal Extension behavior:
extension_accept_listextension_python_listextension_deploy_on_registration
Currently, the only way to set these options is by updating the UAG Server configuration file and then refreshing or restarting the Agent. Because a static configuration is not necessarily desirable for an option like this, providing ways to override the configuration on an ad-hoc or dynamic basis is needed. The ways in which such requirements were satisfied for similar features was to allow them to be set at install time and at Agent startup time, via options in the ubrokerd script and/or the ubroker application’s command line interface..
To satisfy the requirement above, the following enhancements are provided in this release:
An ability to set the option’s value is provided using the following methods:
Setting an environment variable that is inherited by the Universal Broker at start-up.
Passing an argument to the Universal Broker script and forwarding that option via the Broker command line.
Setting a property during the Universal Agent installation.
...
B-13822
...
UIP: Allow usermode installs to discover the Python distribution provided by a system mode-installed Agent
...
Enhanced UIP to allow a user Mode Agent to take advantage of an, already installed, Python environment provided by a System-install Agent.
During Python discovery (i.e., the process by which UAG Server looks for a Python interpreter that satisfies a Universal Extension’s version requirements), UAG Server always looks for the Python distribution provided by the system-mode Universal Agent install. UAG Server looks for this Python in a file system location that is relative to UAG Server’s installation directory. Because usermode installs do no provide the Python distribution, any Agent-supplied distribution will not be found.
While this problem exists for Linux/Unix usermode installs a change in the UAG Server discovery implementation is only necessary for Windows. If someone wishes to use the Agent’s Python distribution, they can simply take the tarball from the Agent distribution file and extract it under the usermode installation directory. A similar solution is not available for Windows.
Note that the user can also configure the extension_python_list option to look for an Agent-supplied Python distribution in any location. This means that making the change for Windows is simply a change for convenience.
The system mode Universal Agent install for Windows records has its install location in the Windows system registry, via the installation_directory value under HKLM\SOFTWARE\Stonebranch. That is a highly-reliable value that system-mode and usermode installs may use to locate the Agent’s Python distribution.
...
B-13846
...
UIP: New Credential "Token" Universal Extension Support
...
Implemented a new “Credential token” to better support large tokens provided by certain applications.
Some web/cloud services require authentication tokens that may be 3-4 Kb in length. The requirement is to implement support of token fields in Resolvable Credentials with maximum length at least 4096 bytes. The Credential Token must be passed to Universal Extension and its value must be scrubbed from any visible output of a task instance.
The ‘fields’ dictionary of a Universal Extension Task is populated with the new credential variable “[credential variable name].token” along with previously supported ‘*.user’, ‘*.password’, ‘*.pasphrase’ and ‘*.keyLocation’. Total cumulative length of all encrypted fields passed to the task (password + passphrase + token) cannot exceed 65535 characters. The previous limit was 2052 bytes. The limit if used in the command of a regular task is 4096 characters of command length.
...
B-13819
...
UIP: Expose configuration options that allow custom Python discovery paths
...
A new configuration option for UAG Server was implemented for the UA 7.0.0.0 release. This option, extension_python_list, is currently recognized by UAG Server but no information about its use was made available. This was done to reduce the development and testing complexities around exposing any configurable option. However, providing users with the ability to configure this option is needed for usability and flexibility.
Information about this option must be made public and the ability to configure the option in a flexible, secure way must be provided.
To satisfy the requirement above, the following enhancements are provided:
Information about the option is available in the comments section of the UAG Server configuration file.
Information about the option is available in the UAG Server template file.
Information about the option is available in the user documentation.
An ability to set the option’s value is provided using the following methods:
Directly editing the UAG Server configuration file
Allowing the option to be refreshed via uctl without requiring a restart of the Agent
...
B-13989
...
UIP: Provide access to Extension APIs at install time
...
Simplification of uploading an Extension to the Controller.
So far, with every change, the process of manually building and uploading the Extension has to be repeated.
A tool has been created to make this process faster and convenient.
In addition to building and uploading the Extension, the tool should offer functionality that allows users to:
Quickly prototype Extensions using starter Extension Templates
Pull the latest Universal Template source files from the Controller
Download the full Universal Template package
The Solution is a CLI Utility called uip-cli with a list of parameters associated:
Quickly prototype Extensions using starter Extension templates:
initcommand is used to initialize the selected Extension template. Additionally, the command has logic to accept variables that are used to configure the selected Extension template.template-listcommand is used to list all the available starter Extension templates as well as the variables that can be used to configure a given Extension template.Build and upload Extensions and/or Universal Templates:
buildcommand is used to build the Extension or the full package (Universal Template + Extension) depending on which one is specified.uploadcommand is used to upload the Extension or the full package to the Controller. The Extension/full package must already be built before calling upload.pushcommand is a combination ofbuildandupload. Each time it is called, the Extension or full package is built AND uploaded.
Pull the latest Universal Template source files from the Controller
pullcommand is used to the pull the Universal Template source files from the Controller. The source files includetemplate.json(andtemplate_icon.png, if present). The CLI prints a message indicating what was updated, newly downloaded, or not changed at all.
Download the full Universal Template package
downloadcommand is used to download the full Universal Template package. The CLI prints a message indicating what was updated, newly downloaded, or not changed at all.
...
B-14255
...
UIP: Implement init command support for CLI
Architecture
...
Backlog
...
Title
...
Description
...
B-13768
...
Move stability checks for file monitor tasks to ops_monitor
...
Pre-7.1.0.0 implementation of stability checks for Agent File Monitors required requesting updates about current file information from ops_monitor process by uagsrv every second once a matching file was created. The request was implemented as the STAT command sent to the ops_monitor’s STDIN. Handling the response from ops_monitor involved additional internal message exchange between uagsrv process threads. This design resulted in critical performance degradation in case if there are many matching files get created at the same time and uagsrv initiates file stability monitoring for every files. Depending on the number of files there may be long delays in handling the regular tasks execution. In order to improve performance in such conditions, the file stability check functionality was moved to the ops_monitor process entirely, which eliminated most of the IPC message exchange and decreased load on the LSNR thread. That gives more CPU slots for LSNR for handling regular tasks.
...
B-13885
...
Suppress ICH408I messages in RACF log
...
Even though considered as a Warning message only, this enhancement ensures that the warning message is no longer shown on the RACF Log as it only describes the wanted effect that a user does not have the authority he is not supposed to have.
...
B-13931
...
OMS Java Client - Add support of service timeout
...
B-14049
...
Universal Extension Password Exposed in Agent Cache
Sysplex Failover
...
Backlog
...
Title
...
Description
...
Please make sure that ALL Sysplex agent components (Primary or Secondary) use the SAME UAG NETNAME definition!
...
B-13186
...
Allow a Primary to switch to Secondary
...
Perform the steps necessary to re-configure a Primary into a Secondary without restarting UAG. This ability can be triggered programmatically and by direct user action through a z/OS console command. If a serious error occurs during the switch-over process, UAG will shut down, as the process' state will be unknown.
Command: F <ubroker>,APPL=UAG,SECONDARY
...
B-13184
...
Allow a Secondary to switch to Primary
...
Perform the steps necessary to re-configure a Secondary into a Primary without restarting UAG. This ability can be triggered programmatically and by direct user action through a z/OS console command. If a Primary agent already exists, the command will fail. If a serious error occurs during the switch-over process, UAG will shut down, as the process' state will be unknown.
Command: F <ubroker>,APPL=UAG,PRIMARY
...
B-13282
...
Automatic fail-over
...
The automatic fail-over process is triggered when a Primary disconnects from the XCF structure and fail-over is configured. The active Secondary with the highest priority in the fail-over hierarchy becomes the new Primary. The fail-over hierarchy is known by all members. Whenever a member connects to the UAG XCF group, it broadcasts its relative standing in the hierarchy to all members. If two or more members have the same standing in the hierarchy, the member that connected to the structure the earliest is granted the higher standing.
Currently, the sysplex_role configuration statement in UBRCFG00 controls the Sysplex mode (none, primary, secondary) for an agent.
Since fail-over is a UAG feature, a new configuration statement was added in UAGCFG00. This parameter is used to configure the fail-over behavior of an agent.
The automatic_failover statement has two valid values:
never - Indicates no automatic fail-over will occur. This is the default.
secondary_primary - For Secondary agents only. Indicates this agent is eligible for automatic fail-over.
...
B-13283
...
Add a method to configure the fail-over hierarchy
...
The UAG automatic_failover configuration statement was extended to allow a ranking value to be appended to the secondary_primary value:
secondary_primary - This existing value was enhanced to support appending an integer value between 1 and 32. This value indicates the relative ranking of the agent in the fail-over hierarchy. A lower number indicates a higher ranking. If not specified, the ranking is 1.
Example:
automatic_failover secondary_primary4 # Secondary eligible for fail-over. Ranking is 4.
...
B-13284
...
Method to shut down the broker while preventing fail-over
...
See B-13285 below.
...
B-13285
...
Method to shut down the broker while triggering fail-over
...
The current z/OS STOP command now used to shut down the broker with default (configured) fail-over behavior.
The current z/OS MODIFY command was enhanced with new syntax. The new syntax allows shutdown of a Primary broker while specifying if fail-over is required or not and optionally, which Secondary should take over.
For Secondary and non-Sysplex systems, the modify shutdown command functions like the STOP command. The FAILOVER and NOFAILOVER keywords will be ignored.
e.g.: F <ubroker>,APPL=SHUTDOWN [ [ ,FAILOVER [ ,<sysname> ] ] | ,NOFAILOVER ]
where:
<ubroker> is the name of a running broker.
SHUTDOWN directs the broker to shut down.
FAILOVER, NOFAILOVER specifies whether fail-over is required or not. This will work even if automatic fail-over is not configured. If neither is specified, the command will function like the z/OS STOP command.
<sysname> specifies the name of a system in the Sysplex whose broker should become the new Primary. If no UAG in the UAG Sysplex group is running on the specified system, fail-over will not occur. If not specified, the configured fail-over hierarchy will determine which system will become the broker
...
B-13291
...
A new Primary can receive tracking data for jobs it hasn't submitted or executed
...
A new or backup Primary can receive tracking data for jobs it neither submitted, nor are being executed on the system it is running on. Currently, this would result in an error and the job would not be tracked. Only the system which submitted the job and the system executing it has knowledge of the job.
For instance, System A submits a job and it executes on System B. System A shuts down and System C takes over as Primary. System C will not recognize the job.
System C should consult the UNVJSC dataset for information about jobs which it does not recognize and build the minimum information structures it needs to track the job.
...
B-13964
...
Proper licensing / handover when primary or secondary switch
Data Pipeline
...
Backlog
...
Title
...
Description
...
B-10691
...
UA: Add explicit FTPS (FTPES) support to UFTP
...
Enhanced UFTP to support Explicit FTPS(FTPES).
The Universal FTP client (UFTP) has been modified to be able to connect to an FTP server that has enabled SSL support but does not necessarily require it as a server configured for implicit SSL support would.
A typical use case is one in which a single FTP server, usually – but not necessarily – listening on the default FTP port 21, will support creation of either of the following session types:
An SSL/TLS-encrypted session for clients that support or require it
An unencrypted, plain-text connection for clients that either
have no support for SSL/TLS, or
do not require an encrypted session
To create a secure, encrypted session, the FTP server presents a public/private key pair in X.509 certificate format to the FTP client (UFTP). The client authenticates this information using a certificate authority (CA). If the certificate information presented by the server cannot be authenticated, the connection fails.
The client can set options that changes this behavior. For example, if the client only cares about insecure, plain-text connections, it can choose to bypass certificate (i.e., peer) authentication and accept whatever certificates the server presents. The client can also require host name authentication, where the server’s certificates must contain a host name or IP address that matches the server specified by the client.
Note that server options can also be set that disallow plain-text, unencrypted sessions.
| Note | ||
|---|---|---|
| ||
Further details on set-up and security can be found within the documentation. |
Development
Backlog
Title
Description
B-13316
Upgrade libSSH2 to 1.9.0 to resolve potential sftp transfer issue
Upgraded the version of libSSH2 from 1.8.2 to 1.9.0 for Universal FTP (UFTP) on those platforms where UFTP was using libSSH2 1.8.2 and/or those platforms on which UFTP provides SSH support.
By definition, this removes HP-UX and z/OS from the targeted platforms14036 | UFTP : Establish Connection before processing other options | Establish a connection to the (S)FTP Server before attempting the requested transfer operation. This will allow connection-specific failures to be isolated and reported. To support this change, the following message:
was replaced with:
where curl_rc and curl_msg are the return code and error message, respectively, that identify the problem. In addition, UFTP now displays the entire connection conversation between it and the Server without requiring the -verbose option. |
B-14630 | Add support for new FTPES fields available from UC task form | Universal Agent 7.1.0.0 added FTPES support for UFTP; this adds Universal Agent support for the new Universal Controller FTPES file Transfer Task. |
Sysplex Failover
Backlog | Title | Description |
B-13286 | Allow a Primary to be re-started and become a Secondary | Extends the AUTOMATIC_FAILOVER option to support the primary_secondary value, which will instruct a Universal Agent configured with a SYSPLEX_ROLE of primary to start as a secondary Agent if another primary Agent is already running. |
B-13287 | Allow a Primary to be restarted and become a Primary | Extends the AUTOMATIC_FAILOVER option to support the secondary_primary[n] value, where n is an optional numeric value that sets a priority to determine which secondary Agent (among several active secondary Agents) will take over as primary. |
B-13292 | A new Primary should remain a Primary | Extends the AUTOMATIC_FAILOVER option to support the always_primary value, which will force an Agent whose SYSPLEX_ROLE is primary to always start as a Primary Agent. Any other Agent running as a Primary Agent will become a Secondary Agent when this Agent starts. |
UAG
Backlog | Title | Description | |||||
|---|---|---|---|---|---|---|---|
B-14071 | Wait for process tree to complete after cancel before reporting to Universal Controller | Add the KILL_PROCESS_TREE option to control how UAG Server responds to a Cancel task command from Universal Controller. When yes, the entire process tree is terminated via SIGTERM. Any process within the tree that is still running after 10 seconds is terminated with a SIGKILL. UAG Server will confirm that all processes in the process tree are terminated before it notifies the Controller of the termination. To preserve compatibility with existing behavior, the default value for this option is no, which can prevent the termination of a process in the tree that ignores the SIGTERM signal. This may cause the task's status in the Universal Controller to show Cancelled although the (now orphaned) process continues to run.
|
Universal Integration Platform
Backlog | Title | Description |
B-14270 | Add additional logging level to separate debug and trace level statements | Remove internal DEBUG-level messages that are generated by the Universal Extension base class when the extension task's log level is DEBUG. Only DEBUG messages issued by the extension developer should be shown. |
B-14271 | Add ability to launch extension tasks from UIP-CLI | Adds the following commands/options to the UIP-CLI utility: task launch - launch a Universal Extension task task status - obtain the status of a Universal Extension task task output - obtain the output generated by a Universal Extension task |
B-14720 | Add ability to launch extension tasks from UIP-CLI | Visual Code support for new UIP-CLI commands implemented by B-14271. |
B-14544 | Accept task instance variables for direct access within an extension | Make all all Task Instance variables set by the Controller for a Universal Extension task available to the Extension. An extension can access these variables via the |
B-14724 | Accept new parameters in extension launch message to signal when extension is launched as trigger | To support integration with Universal Monitor tasks, receive information from the Controller that can by used to identify an Extension task launched via a Universal Monitor Trigger. Values received from the Controller are saved in the following properties:
The Universal Extension base class sets the following property when a Trigger ID is received:
|
B-14725 | Add ability for agent to reconnect to extension task upon agent restart | Adds support for Universal Monitor tasks, including the ability to reconnect to Universal Extension tasks (executed on behalf of the Monitor task) that continue to run when an Agent is restarted. |
z/OS Enhancements
Backlog | Title | Description |
B-13288 | Configure which systems are eligible to execute a job | Added the (mutually-exclusive) ZOS_JOB_SYSAFF and ZOS_JOB_SYSTEM configuration options that can manage a z/OS task's execution via a SYSAFF or SYSTEM JOB card parameter, respectively. |