Get help with Connected Azure Gateway

This section provides troubleshooting, error code and known limitations that you may encounter while working with the connected managed environment for Azure Gateway within Amplify Central.

11 minute read

Troubleshooting

This section describe the common error cases you can encounter when using agents for the first time. If your error is not listed, please check the Understand the agent log and Error Codes and Mitigations sections.

Question Answer
Why can’t Discovery Agent connect to Azure Gateway? Make sure that the Azure configuration (AZURE_SUBSCRIPTIONID, AZURE_TENANTID, AZURE_RESOURCEGROUPNAME, AZURE_CLIENTID, AZURE_CLIENTSECRET and AZURE_APIMSERVICENAME) are correctly set.
Why can’t Traceability Agent connect to Azure event hub? Make sure that the Azure configuration (AZURE_SUBSCRIPTIONID, AZURE_TENANTID, AZURE_RESOURCEGROUPNAME, AZURE_CLIENTID, AZURE_CLIENTSECRET, AZURE_APIMSERVICENAME, AZURE_EVENTHUBNAME, AZURE_EVENTHUBNAMESPACE, AZURE_SHAREDACCESSKEYNAME and AZURE_SHAREDACCESSKEYVALUE) are correctly set.
Why can’t my agents connect to Amplify Central? Go to Amplify Central UI > Access > Service Accounts and make sure that the Service Account is correctly named and valid (CENTRAL_AUTH_CLIENTID). Make sure that the organizationID (CENTRAL_ORGANIZATIONID) is correct. Verify the key’s file name and path (CENTRAL_AUTH_PUBLICKEY and CENTRAL_AUTH_PRIVATEKEY).
I have the following error: Amplify Central - FAIL (error creating request header. bad response from AxwayId: 400 Bad Request) Make sure that the Service Account set up is properly spelled (refer to CENTRAL_AUTH_CLIENTID property). Verify that the public/private key pair is the one corresponding to the Service Account (CENTRAL_AUTH_PUBLICKEY and CENTRAL_AUTH_PRIVATEKEY).
Why don’t I see traffic in Amplify Central? Make sure that the TRACEABILITY_HOST and TRACEABILITY_PORT are accessible from the machine where Traceability Agent is installed. If you are using a proxy (TRACEABILITY_PROXYURL), make sure the connections are opened correctly. Refer to Communication ports.
Why isn’t my agent status displayed in the environment details page? It is possible that your agents and their corresponding resources are not linked together. Follow this procedure to ensure the correct usage of agent resources.
Why is my Traceability Agent receiving EOF errors while sending transaction events? It is possible that the number of TRACEABILITY_WORKER is set too high. lower the number of configured workers.

Understand the agent logs

Each log is displayed in a JSON format following this pattern: {“level”:“info|debug|trace|warn”,“msg”:“messageFromTheAgentGoesHere”,“time”:“messageTimestamp”}

The agent logging mechanism is based on logrus (https://github.com/sirupsen/logrus) and can display several levels of information:

  • info (default): informative log about the agent activity (start, stop, discovering API, publishing traffic, etc.).
  • debug: same as info level plus some troubleshooting information (healthcheck, connection to Central / Gateway, etc.).
  • trace: same as debug plus all information related to API call (action / duration / return code / endpoint).

The log level definition is provided using the LOG_LEVEL variable set in the agent configuration file.

The logs can be redirected either to stdout (default) or a file, or both (only applicable for discovery agent), using the LOG_OUTPUT variable set in the agent configuration file.

# redirecting log to stdout (Discovery / Traceability agents)
LOG_OUTPUT=stdout

# redirecting log to file (Discovery / Traceability agents)
LOG_OUTPUT=file

# redirecting log to output and file (Discovery agent only)
LOG_OUTPUT=both

For file output, the agent needs to know where to save the file and what the file name is. By default, the directory is logs where the agent is installed and the file name is based on the binary name you are running: discovery_agent.log or traceability_agent.log. This configuration can be changed using LOG_FILE_PATH and LOG_FILE_NAME variables in the agent configuration file. For the LOG_FILE_PATH, it is recommended to use an absolute path when running the agent in a non container mode.

# Sample configuration to redirect the agent logs to a specific directory and a specific file name
LOG_FILE_PATH=/home/APIC-agents/logs
LOG_FILE=agent-da-resgrp1.log

# Additional configuration for log rotation and backups
# The max size, in megabytes that a log file can grow to - no limit by default.
LOG_FILE_ROTATEEVERYMEGABYTES
# The max number of log file backups to keep - no limit by default.
LOG_FILE_KEEPFILES
# The max age of a backup file, in days - no limit by default.
LOG_FILE_CLEANBACKUPS

Error Codes and Mitigations

In case an error is encountered, the agent will automatically log it based on the logging definition (stdout and/or file).

The error message format is: Component - [Error Code _ErrorCode_] error message.

Below is the list of error codes and possible remediations.

Code # Description Mitigation
Amplify Agent SDK errors
1001 initialization error checking for dependencies to respond, possibly network or settings Typically, a previous error in the log is displayed. See Administer Azure network traffic and Reference - Agent configuration.
1002 timeout error checking for dependencies to respond, possibly network or settings Typically, a previous error in the log is displayed. See Administer Azure network traffic and Reference - Agent configuration.
1003 Periodic health checker or status updater failed. Services are not ready Typically, a previous error in the log is displayed. See Administer Azure network traffic and Reference - Agent configuration.
1004 error starting status update job Internal system error. Contact your system administrator.
1005 error starting agent version checker Build version is missing or of noncompliant semantic versioning. Check the build version of your agent.
1100 general configuration error in CENTRAL CENTRAL_AGENTNAME is set but corresponding resources do not exist. Either create agent resources or remove the CENTRAL_AGENTNAME variable.
1101 error attempting to query for ENVIRONMENT Check that the Environment Name (CENTRAL_ENVIRONMENT) in configuration matches the name on Amplify Central.
1102 could not find specified team in Amplify Central, check CENTRAL_TEAM Check that the Team Name (CENTRAL_TEAM) in configuration matches the name on Amplify Central.
1110 connection to Amplify Central failed Check CENTRAL_URL configuration. See Administer Azure network traffic. If using a proxy to access Amplify, check that CENTRAL_PROXYURL is correctly defined.
1120 request to Amplify Central failed A request is not processed correctly. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1130 request to get authentication token failed Check the CENTRAL_AUTH_* configuration (ClientId, Private and Public keys). Verify that you can connect to platform.axway.com. See Administer Azure network traffic.
1131 token retrieved but was invalid on request to Amplify Central The token to communicate with Platform is not valid. The agent will try to get a new token. Check the CENTRAL_AUTH_* configuration (ClientId, Private and Public keys). See Reference - Agent configuration.
1140 couldn’t find a subscriber email address based on the ID in the subscription event The user requesting this subscription may not have an Email address set. Go to the platform > Organization > Users to verify the user email.
1141 couldn’t contact Amplify Central for subscription, possible network error See Administer Azure network traffic. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1142 couldn’t get subscription data from Amplify Central See Administer Azure network traffic. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1143 couldn’t create or update subscription schema data See Administer Azure network traffic. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1144 unexpected response when managing subscription schema on Amplify Central See Administer Azure network traffic. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1145 unable to create webhook for the subscription definition See Administer Azure network traffic. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1146 unable to create secret for the subscription definition See Administer Azure network traffic. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1147 error parsing filter in configuration. Syntax error There is an issue with AZURE_FILTER variable. See Reference - Agent configuration.
1148 error parsing filter in configuration. Unrecognized expression There is an issue with AZURE_FILTER variable. See Reference - Agent configuration.
1149 error parsing filter in configuration There is an issue with AZURE_FILTER variable. See Reference - Agent configuration.
1150 error parsing filter in configuration. Invalid call argument There is an issue with AZURE_FILTER variable. See Reference - Agent configuration.
1151 error parsing filter in configuration. Invalid selector type There is an issue with AZURE_FILTER variable. See Reference - Agent configuration.
1152 error parsing filter in configuration. Invalid selector expression There is an issue with AZURE_FILTER variable. See Reference - Agent configuration.
1153 error parsing filter in configuration. Invalid operator There is an issue with AZURE_FILTER variable. See Reference - Agent configuration.
1154 error parsing filter in configuration. Unrecognized condition There is an issue with AZURE_FILTER variable. See Reference - Agent configuration.
1155 error getting subscription definition properties in Amplify Central See Administer Azure network traffic. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1156 error updating subscription definition properties in Amplify Central See Administer Azure network traffic. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1157 error getting catalog item API server info properties in Amplify Central See Administer Azure network traffic. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1158 subscription manager is not running likely because the agent can’t communicate with Amplify Central See Administer Azure network traffic. Set LOG_LEVEL=trace in agent configuration to retrieve the failing request and help diagnose the issue.
1160 error getting endpoints for the API specification Verify that the OpenAPI Specification for the discovered API has available endpoints and operations on each endpoint.
1161 error deleting API Service for catalog item in Amplify Central Verify that the catalog item exists in Amplify Central.
1162 error deleting catalog item in Amplify Central Verify that the catalog item exists in Amplify Central.
1163 error retrieving API Service resource instances Verify that the service exists in Amplify Central.
1164 Amplify Central does not contain a team matching the API organization name. The Catalog Item will be assigned to the default team configured in CENTRAL_TEAM variable. Verify that the team exists in Amplify Central.
1300 error communicating with server for subscription notifications (SMTP or webhook), check SUBSCRIPTION config See Manage subscription workflow.
1301 subscription notifications not configured, check SUBSCRIPTION config See Manage subscription workflow.
1302 error creating data for sending subscription notification Check Azure API Manager UI for configuration of API and its subscriptions.
1303 email template not updated because an invalid authType was supplied Check Azure API Manager UI for configuration of API and its subscriptions, specifically enabled authentication on the subscription.
1304 no email template found for action See Manage subscription workflow for the SMTP template settings.
1305 error sending email to SMTP server See Manage subscription workflow for the SMTP settings.
1401 error parsing configuration values, check the config referenced in error See Reference - Agent configuration.
1402 error in overriding configuration using file with environment variables Make sure the env_vars file exists and is properly formatted.
1403 invalid value for statusHealthCheckPeriod. Value must be between 1 and 5 minutes See Reference - Agent configuration variable STATUS_HEALTHCHECKPERIOD.
1404 invalid value for statusHealthCheckInterval. Value must be between 30 seconds and 5 minutes See Reference - Agent configuration variable STATUS_HEALTHCHECKINTERVAL.
1405 a key file could not be read For binary deployment, validate that your user running the agent has access to the key file. For docker deployment on Linux, give the User ID in the container read access to the file setfacl -m u:2500:r private_key.pem.
1410 invalid configuration settings for the logging setup See Reference - Agent configuration for logging configuration options.
1503 http transport is not connected Check the TRACEABILITY_HOST setting. See Reference - Agent configuration.
1504 failed to encode the json content Internal system error. Contact your system administrator.
1505 invalid traceability config See Reference - Agent configuration for the Traceability configuration.
1510 global redactions have not been initialized See Trace redaction.
1511 error while compiling regular expression See Trace redaction.
1520 global sampling has not been initialized See Trace sampling.
1521 invalid sampling configuration See Trace sampling.
1550 error hit while applying redaction See Trace redaction.
1600 error registering the job The agent attempted to register a job, but could not do so successfully.
1601 error executing a job A job defined in the agent failed, see the logs for more information.
1602 error executing a retry job A job defined in the agent failed, see the logs for more information.
1611 periodic healthcheck failed One of the agent healthchecks failed, check the agent logs for more information.
1612 a healthcheck failed the maximum consecutive times One of the agent healthchecks failed, check the agent logs for more information.
1613 terminating agent, another instance of agent already running Another agent of the same type is already running. Check with ps-ef _agent to find the PID of already running agent and kill it if necessary.
1900 unsupported system for service installation The service installation only supports Linux.
1901 systemd is required for service installation The service installation requires an OS with systemd.
1902 service management requires root privileges All service actions require root privileges.
1903 service has already been installed The service has already been installed, you can use “remove” to clean it up and then install it again.
1904 service is running and cannot be removed until stopped The service has to be stopped before removing it.
1905 service is not yet installed The service has to be installed prior to taking any other action.
1906 service is already running The service is already running, nothing to do.
1907 service is already stopped The service is already stopped, nothing to do.
Specific error for Azure Discovery agents
4000 missing or invalid configuration value for Azure APIM service See Administer Azure network traffic and Reference - Agent configuration.
4100 error creating a service in Amplify Central for API on Azure APIM service See Administer Azure network traffic and Reference - Agent configuration.
4101 error getting API specification for API on Azure APIM service See Administer Azure network traffic.
Specific error for Azure Traceability agents
4500 missing or invalid configuration value for Azure APIM service See Administer Azure network traffic and Reference - Agent configuration.

Limitations

  • Azure versioning not taken into account while discovering APIs