sysupgrade error reference

Reference to warnings and errors generated by sysupgrade.

11 minute read

export command errors and warnings

This section describes example errors and warnings that the sysupgrade export command generates when exporting the data from the old API Gateway installation.

Errors and warnings are logged to the following file:

Axway-7.7/apigateway/export/bin/out/logs/export.log

Check for customizations to jvm.xml and service.xml

Every API Gateway system stores JVM settings in VDISTDIR/system/conf/jvm.xml. Configuration settings for individual API Gateways are stored in VINSTDIR/conf/service.xml (for example, apigateway/groups/group-2/instance-1/conf/service.xml). If you have modified these files in the old installation, the changes must be manually copied to the corresponding files on the new installation. The export step logs an informational message to inform you of this.

Each individual API Gateway instance can also have its own jvm.xml file at VINSTDIR/conf/jvm.xml. This file is not present by default, but can be created by users. Instance-specific jvm.xml files are migrated to the new installation as part of the upgrade process. The export step logs a warning to inform you to review these files and verify if they are still needed on the new installation. If not, remove them from the export directory before proceeding with the upgrade.

The following is an example of output from the export log file:

[System Config] : OK
[INFORMATIONAL] messages from [System Config] : Not migrating the following instance-specific service.xml files:
/opt/Axway-7.4.1/apigateway/groups/group-2/instance-1/conf/service.xml
If you have modified any of these files, you should copy your changes to the new system after upgrade is complete.
Not migrating apigateway/system/conf/jvm.xml. This is specific to the old Gateway version - a new jvm.xml will be present on the new system. If you have customized this file, you should manually copy your modifications to the new system.
[WARNING] messages from [System Config] :
WARN: Migrating the following instance-specific jvm.xml files from the old system:
WARN:     /opt/Axway-7.4.1/apigateway/groups/group-2/instance-1/conf/jvm.xml
WARN:     These jvm.xml files are not part of the standard Gateway installation, and must have been added for customization. Please review these files and verify that they are still needed on the new system. If not, they should be removed from '/home/neil/dev/apigateway/upgrade/bin/out/export/sysconfig' before proceeding with upgrade.

Inconsistent groups

If your old installation contains any inconsistent groups, the export step logs the following message:

Group 'TestGroup' (group-2) is inconsistent. Upgrade requires all groups to be consistent.

To resolve this issue, ensure that all API Gateways in the group have the same configuration deployed.

upgrade command errors and warnings

This section describes example errors and warnings that the sysupgrade upgrade command generates when validating the data exported from the old API Gateway installation.

Errors and warnings are logged to the following file:

Axway-7.7/apigateway/upgrade/bin/out/logs/upgrade.log

Admin Node Manager host name is invalid

If the Admin Node Manager has not been configured correctly, the upgrade step logs the following errors:

ERROR: Host 'XXX' is not a valid host in the topology. Retry with --anm_host <hostname>, where hostname is one of: 'NodeA', 'NodeB
ERROR: Host 'nodec' does not have an Admin Node Manager. Retry with a valid --anm_host <hostname> parameter, where where hostname is one of: 'NodeA', 'NodeB

The solution in all cases is to retry the upgrade command with a valid --anm_host parameter.

Check if API Manager license required

If API Manager is installed on the old API Gateway installation you are upgrading, the upgrade step checks the new API Gateway installation for a valid API Manager license. If the license does not exist, the upgrade step logs a warning message, but continues the upgrade:

WARN: A LicenseException ('feature "apiportal" not licensed') occurred while checking for a API Manager license in directory '/opt/Axway-7.7/apigateway/conf/licenses'.
Please contact support to acquire a new license. Place the new license in directory '/opt/Axway-7.7/apigateway/conf/licenses'.

Check if McAfee license required

If any policy in the old API Gateway installation you are upgrading contains a McAfee Anti-Virus filter, the upgrade step checks the new API Gateway installation for a valid McAfee license. If the license does not exist, the upgrade step logs a warning message, but continues the upgrade:

WARN: A LicenseException ('feature "mcafee" not licensed') occurred while checking for a McAfee License '/opt/Axway-7.7/apigateway/'. Please contact support to acquire a new license. Place the new license in directory '/opt/Axway-7.7/apigateway/'.

Check if RBAC permissions file has changed

Each user who logs in to the API Gateway Manager web console has a specific set of roles for Role-Based Access Control (RBAC). These determine what the user can view and what actions they can perform. The roles are defined in the acl.json file on the Admin Node Manager in the apigateway/conf/ directory.

You can manually edit this RBAC permissions file to create new roles or modify existing ones. However, the upgrade does not migrate these changes to the new installation.

The upgrade step checks for changes in the RBAC permissions file on the old API Gateway installation. If there are changes, the upgrade step logs a warning that the changes need to be manually merged into the acl.json file on the new installation:

WARN: RBAC permissions file has been modified on old system.
WARN: Differences between original file (/opt/Axway-7.7/apigateway/upgrade/scripts/rbac/factory/acl6.json) and active file (/opt/Axway-7.7/apigateway/upgrade/bin/out/export/rbac/conf/acl.json) must be manually merged into apigateway/conf/acl.json on new system.

If the RBAC changes are required to allow users to log into the Admin Node Manager or the API Gateway Manager UI, you must do this before you run the apply step, or sysupgrade will not be able to connect to the new Admin Node Manager and the apply step will fail.

Check for corrupt JARs in ext/lib

If a corrupt JAR file is located in the system-wide or instance-specific ext/lib directories in the old API Gateway installation, the upgrade step logs a warning. The upgrade does not copy the JAR to the new 7.7 installation.

Check for valid API Gateway license

API Gateway requires a valid license. If the license does not exist, or is not valid, the upgrade step logs one of the following errors:

ERROR: A LicenseException ('License not found in /home/axway/Axway-7.5.0/apigateway/conf/licenses') occurred while checking licenses in directory '/home/axway/Axway-7.7/apigateway/conf/licenses'.  
A license may not exist in this directory, or all licenses in this directory may be invalid. Please contact support to acquire a new license. Place the new license in directory '/home/axway/Axway-7.7/apigateway/conf/licenses'. Run sysupgrade again without the --clean option.
ERROR: A LicenseException ('feature "unrestricted" not licensed') occurred while checking for a server license in directory '/home/axway/Axway-7.7/apigateway/conf/licenses'. Please contact support to acquire a new license. Place the new license in directory '/home/axway/Axway-7.7/apigateway/conf/licenses'. Run sysupgrade again without the --clean option.

You must resolve these errors before you can continue to the apply step. If you do not place a valid license in the conf/licenses directory before proceeding to the apply step, the API Gateway will not start.

SSL certificates for management traffic regenerated

When upgrading to API Gateway 7.7 the SSL certificates used for management traffic between Node Managers and API Gateways are always regenerated, so that the old installation and new installation Node Managers cannot communicate with each other.

For example:

SSL certificates used between Node Managers and API Gateways for management traffic will be regenerated in the upgraded system. Please consult documentation if you wish to manually reconfigure the upgraded system to use the SSL certificates from the old version after upgrade.

This message is for information only and does not require any action. If you used custom SSL certificates in your old installation, you can manually add the certificates from your old installation to your new installation after upgrade. For more information, contact Axway Support.

Third-party JDBC JARs

If your old installation of API Gateway uses external databases for OAuth and KPS, the upgrade step logs the following warning:

WARN: OAuth/KPS tables must be upgraded in the following external databases: [u'jdbc:mysql://127.0.0.1:3306/testdb']. You should copy the required JDBC drivers to '/opt/Axway-7.5.1/apigateway/upgrade/lib', so that 'sysupgrade apply' can perform the database upgrades. Alternatively, you can upgrade OAuth/KPS tables manually using the SQL scripts at '/opt/Axway-7.5.1/system/conf/sql/upgrades', then run 'sysupgrade apply' with argument --skip_db_upgrade.

ActiveMQ directory

If the old installation of API Gateway uses embedded ActiveMQ, the upgrade step might log some of the following warnings.

ActiveMQ directory not set

If the directory for ActiveMQ data is not set in the old installation, the following warning appears:

WARN: The ActiveMQ directory is not configured correctly, it is set to '', sysupgrade will NOT export the data.

In this case, sysupgrade proceeds with the upgrade. You should fix the ActiveMQ configuration in the old installation.

ActiveMQ directory does not exist

If the directory for ActiveMQ data set in the old installation does not exist, warnings similar to the following might appear.

WARN: The absolute ActiveMQ directory '/home/user1/does-not-exist' does not exist, sysupgrade will NOT export the data.
WARN: The ActiveMQ directory '/home/user1/AxwayInstalls/GOLD/7.3.1/apigateway/groups/group-2/instance-1/does-not-exist' does not exist, sysupgrade will NOT export the data.

In both of these cases, sysupgrade proceeds with the upgrade. You should fix the ActiveMQ configuration in the old installation.

ActiveMQ directory not under API Gateway installation directory

If the directory set for ActiveMQ data in the old API Gateway installation is not a directory under the API Gateway installation path, the following warning appears:

WARN: The absolute ActiveMQ directory '/home/user1/activemq/data' is NOT a directory under the old version install directory '/home/user1/AxwayInstalls/GOLD/7.4.1/apigateway'. The new version system will use the same directory. When the new version API Gateway uses the data in this directory it may become unusable by the old system. You should take a manual backup of the contents of this directory before proceeding in case you need to run the old version API Gateway again. Sysupgrade will keep a backup of the ActiveMQ data in '/home/user1/dev/sysupgrade1_feature/vordel/install/staging/images/apigateway/upgrade/bin/out/export/activemq/backup' but this may be removed as a result of a clean command, or export --force.

In this case, the upgrade proceeds, and the new API Gateway installation uses the same directory for ActiveMQ data after upgrade.

The new installation might change the data in this directory so that it is no longer usable by the old installation. The sysupgrade process makes a backup of the data in the upgrade/bin/out/export/activemq/backup directory. However, this backup might be removed if you use the clean or export --force commands. We recommend that you create an additional backup to another location in case you need to revert back to using the old installation at any stage.

Check for duplicate API Manager data

If API Manager is running in the old installation, exported KPS data is checked for duplicate users called apiadmin and duplicate organizations called Community. If duplicate data is found, this indicates that KPS is not configured correctly in the old installation.

The upgrade step logs the following error message:

[ERROR] messages from [API Manager] :
ERROR: Found duplicate 'apiadmin' users in KPS table: /opt/Axway-7.7/apigateway/upgrade/bin/out/upgrade/kps/groups/group-2/instance-1/conf/kps/backup/sysexport_api_portal_portaluserstoreldap_json
ERROR: Found duplicate 'Community' organizations in KPS table: /opt/Axway-7.7/apigateway/upgrade/bin/out/upgrade/kps/groups/group-2/instance-1/conf/kps/backup/sysexport_api_portal_portalorganizationstoreldap_json
ERROR: There is a KPS configuration problem on the old system. This must be resolved before upgrade can proceed. Please contact Axway support for assistance.

To resolve this issue, contact Axway Support.

Check for valid FIPS license

If the old API Gateway installation is FIPS-enabled, a FIPS-enabled license is required for the new installation. If the license does not exist, or is not valid, the upgrade step logs the following error:

ERROR: A LicenseException ('feature "FIPS" not licensed') occurred while checking for a FIPS license in directory '/home/user1/dev/apigateway/conf/licenses'. Please contact support to acquire a new license. Place the new license in directory '/home/user1/dev/apigateway/conf/licenses'.

You must resolve this issue before you can continue to the apply step.

Check for Apache Cassandra-backed collections

If you run the sysupgrade upgrade command with the --no_cassandra option, the configuration is checked for Cassandra-backed KPS collections.

If Cassandra-backed collections are found, the upgrade step logs the following error:

ERROR: Group 'Default Group' contains KPS collections with Cassandra data source: ['My Collection']. Either run sysupgrade without '--no_cassandra' option, or remove Cassandra dependency from source system by deleting collections or moving them to a non-Cassandra data source.

You must resolve this error before you can proceed to the apply step.

To resolve this error, delete or update the Cassandra-backed KPS collections in your old installation, and then rerun the export and upgrade steps.

Alternatively, you can rerun the upgrade step without the --no_cassandra option. However, your new installation will require an Apache Cassandra database cluster.

apply command errors and warnings

This section describes example errors and warnings that the sysupgrade apply command generates when applying the upgraded data to the new API Gateway installation.

Errors and warnings are logged to the following file:

Axway-7.7/apigateway/apply/bin/out/logs/apply.log

Incorrect Admin Node Manager host

If the --anm_host specified is not the first Admin Node Manager host and the first Admin Node Manager host is not running, the apply step logs the following error:

ERROR: Node Manager error: Failed to sign CSR for service. None of the available Admin Node Managers have the domain private key, or were able to unlock it with provided passphrase.

The solution is to always use the same --anm_host value for the first Admin Node Manager.

Timeout waiting for deploy to complete

The default transaction timeout of 4 minutes for each Node Manager involved during the sysupgrade apply call might not be enough for the deploy to complete, particularly for large deployment like API Manager, resulting in the following error:

ERROR: com.vordel.api.nm.NodeManagerAPIException: Node Manager error: error occurred on call from Node Manager on 'node.axway.int' to https://node.axway.int:8085' for URL '/api/configuration' and 'PUT'
HTTP status: 500. 
Error code: 102

The instance trace might also have the following error:

ERROR: Transaction to 127.0.0.1 expired after 240000 ms

The solution for this issue is to increase the transaction timeout for node manager and instances. For more information, see How to increase timeouts for apply.