403 Errors on API, when there shouldn't be

[unkaputtbar112]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Since https://console.redhat.com/ansible/automation-hub/repo/published/ansible/controller/ refers this repo and issue tracker, i'm creating the issue here, although it feels it might be wrong and it's hard to place it either on the collection or on the API itself, to be honest.

I think, it might be related to this: #14375

I'm using the whole ansible.platform, ansible.controller, infra.aap_configuration, infra.aap_utilities, ... to have configuration as code.
More often than not, i'm getting 403 Errors when executing, randomly, on different tasks, next run the same task is successful and something else fails, i usually need 10-12runs, until i'm "lucky" that everything went fine, without any change to the configuration, just running over and over again.

My guess would be, that the API itself presents 403 falsely already and then the code, since hardcoded to if status_code == 403, then "You don't have permissions to...", falsely prints that statement as well.

The aap_token is a System Administrator token, so there cannot be any real 403 errors and as said, just running again shifts the error to another task.

AWX version

Running Ansible Automation Platform 2.5 with controller version 4.6.16

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

N/A

Modifications

no

Ansible version

2.16.14

Operating system

RHEL 9.2

Web browser

No response

Steps to reproduce

Have a lot of resources to be configured with the infra.aap_configuration.* roles (which use ansible.controller modules ...), best usecase are workflows, they're failing most frequently, but it's not exclusive to workflows! Almost all resources fail occasionally.

Collection              Version
----------------------- ------------
ansible.controller      4.6.16
ansible.eda             2.8.0
ansible.hub             1.0.0
ansible.platform        2.5.20250604
ansible.posix           2.0.0
infra.aap_configuration 3.4.1
infra.aap_utilities     2.7.0
kubernetes.core         6.0.0

Expected results

Not getting 403 Errors.

Actual results

TASK [infra.aap_configuration.controller_workflow_job_templates : Managing Workflows | Wait for finish the workflow management] *******************************************************************************************************
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Inventory sources sync / masked | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Inventory sources sync / masked1 | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Inventory sources sync / masked2 | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Inventory sources sync / masked3 | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Inventory sync / AWS | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Inventory sync / Azure | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Disk Management / Create Disk to create Volume | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Disk Management / Create Disk to extend Volume | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Disk Management / Delete | Wait for finish the workflow creation)
failed: [127.0.0.1] (item=Create/Update Workflow SSA / AWS / WF / Disk Management / Read | Wait for finish the workflow creation) => {"__workflows_job_async_results_item": {"__workflow_loop_item": {"allow_simultaneous": true, "ask_inventory_on_launch": false, "ask_labels_on_launch": false, "ask_limit_on_launch": true, "ask_scm_branch_on_launch": false, "ask_skip_tags_on_launch": false, "ask_tags_on_launch": false, "ask_variables_on_launch": false, "description": "Read disk informations for an AWS EC2 Instance", "destroy_current_nodes": true, "extra_vars": {}, "inventory": "", "job_tags": "", "labels": "", "limit": "", "name": "SSA / AWS / WF / Disk Management / Read", "notification_templates_approvals": [], "notification_templates_error": ["SSA / OpsGenie Notification"], "notification_templates_started": [], "notification_templates_success": [], "organization": "SSA", "scm_branch": "", "simplified_workflow_nodes": [{"identifier": "Reading disk informations from OS", "success_nodes": ["Reading informations from platform and create mapping Platform<>OS"], "unified_job_template": "SSA / General / JT / disk_info"}, {"extra_data": {"cld_playbooks_fetch_gpc_credentials": true, "cld_playbooks_gather_facts": true, "cld_playbooks_include_role": "cld.aws.disk_mgmt", "ext_cld_aws_disk_mgmt_mode": "read"}, "identifier": "Reading informations from platform and create mapping Platform<>OS", "unified_job_template": "SSA / AWS / JT / disk_mgmt"}], "skip_tags": "", "survey": {}, "survey_enabled": false, "survey_spec": {}, "webhook_credential": "", "webhook_service": ""}, "ansible_job_id": "j931849667173.2361760", "ansible_loop_var": "__workflow_loop_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/home/someuser/.ansible_async/j931849667173.2361760", "started": 1}, "ansible_job_id": "j931849667173.2361760", "ansible_loop_var": "__workflows_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "You don't have permission to GET to /api/controller/v2/notification_templates/ (HTTP 403).", "results_file": "/home/someuser/.ansible_async/j931849667173.2361760", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Virtual Machine Management / Create | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Disk Management / Create Disk to create Volume | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Disk Management / Create Disk to extend Volume | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Disk Management / Delete | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Disk Management / Read | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Disk Management / Update | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Virtual Machine Management / Create | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / Avantra Monitoring / Create Maintenance Window | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / Avantra Monitoring / Delete Maintenance Window | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / Commvault Agent / Install | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / Commvault Agent / Uninstall | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / Endpoint connection check | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / GPC / Create OS patchbundle | Wait for finish the workflow creation)

in this rare case it failed on the same Workflow, usually it's failing then on another one, i can run it multiple times and present you the output of runs with different failures, if you want...

TASK [infra.aap_configuration.controller_workflow_job_templates : Managing Workflows | Wait for finish the workflow management] *******************************************************************************************************
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Inventory sources sync / masked | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Inventory sources sync / masked1 | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Inventory sources sync / masked2 | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Inventory sources sync / masked3 | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Inventory sync / AWS | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Inventory sync / Azure | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Disk Management / Create Disk to create Volume | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Disk Management / Create Disk to extend Volume | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Disk Management / Delete | Wait for finish the workflow creation)
failed: [127.0.0.1] (item=Create/Update Workflow SSA / AWS / WF / Disk Management / Read | Wait for finish the workflow creation) => {"__workflows_job_async_results_item": {"__workflow_loop_item": {"allow_simultaneous": true, "ask_inventory_on_launch": false, "ask_labels_on_launch": false, "ask_limit_on_launch": true, "ask_scm_branch_on_launch": false, "ask_skip_tags_on_launch": false, "ask_tags_on_launch": false, "ask_variables_on_launch": false, "description": "Read disk informations for an AWS EC2 Instance", "destroy_current_nodes": true, "extra_vars": {}, "inventory": "", "job_tags": "", "labels": "", "limit": "", "name": "SSA / AWS / WF / Disk Management / Read", "notification_templates_approvals": [], "notification_templates_error": ["SSA / OpsGenie Notification"], "notification_templates_started": [], "notification_templates_success": [], "organization": "SSA", "scm_branch": "", "simplified_workflow_nodes": [{"identifier": "Reading disk informations from OS", "success_nodes": ["Reading informations from platform and create mapping Platform<>OS"], "unified_job_template": "SSA / General / JT / disk_info"}, {"extra_data": {"cld_playbooks_fetch_gpc_credentials": true, "cld_playbooks_gather_facts": true, "cld_playbooks_include_role": "cld.aws.disk_mgmt", "ext_cld_aws_disk_mgmt_mode": "read"}, "identifier": "Reading informations from platform and create mapping Platform<>OS", "unified_job_template": "SSA / AWS / JT / disk_mgmt"}], "skip_tags": "", "survey": {}, "survey_enabled": false, "survey_spec": {}, "webhook_credential": "", "webhook_service": ""}, "ansible_job_id": "j417140448342.2358661", "ansible_loop_var": "__workflow_loop_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/home/someuser/.ansible_async/j417140448342.2358661", "started": 1}, "ansible_job_id": "j417140448342.2358661", "ansible_loop_var": "__workflows_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "You don't have permission to GET to /api/controller/v2/notification_templates/ (HTTP 403).", "results_file": "/home/someuser/.ansible_async/j417140448342.2358661", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Disk Management / Update | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / AWS / WF / Virtual Machine Management / Create | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Disk Management / Create Disk to create Volume | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Disk Management / Create Disk to extend Volume | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Disk Management / Delete | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Disk Management / Read | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Disk Management / Update | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / Azure / WF / Virtual Machine Management / Create | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / Avantra Monitoring / Create Maintenance Window | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / Avantra Monitoring / Delete Maintenance Window | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / Commvault Agent / Install | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / Commvault Agent / Uninstall | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / Endpoint connection check | Wait for finish the workflow creation)
changed: [127.0.0.1] => (item=Create/Update Workflow SSA / General / WF / GPC / Create OS patchbundle | Wait for finish the workflow creation)

Worth to mention, although those errors here refer to the notification_template endpoint sending 403, it's not the only one, it's sometimes directly on the workflow_job_template with id and everything, sometimes on the organization, sometimes on the user...

Additional information

No response

[bogdanmuresan]

Hi, we are encountering same issue in AAP 2.5, after updating to Automation controller 4.6.13 (released on May 28, 2025) .
We are getting random 403s during job templates, inventories, roles, each run on a different object.

We have a support issue opened with RedHat since Jun 20.

I think I see a correlation between the 403s, and a "Failed to connect to DB! TimeoutError" error in /var/log/ansible-automation-platform/gateway/grpc_server.log, on the gateway nodes.

grep -rn -B1 "Failed to connect to DB" /var/log/ansible-automation-platform/gateway/grpc_server.log

There is some code in /usr/lib/python3.11/site-packages/aap_gateway_api/proxy/control_plane.py in the _can_read_from_db method which does some health check on the db, which seems to cause the exception. It seems that on each API call the oauth token is verified against the database.

[unkaputtbar112]

@bogdanmuresan did you get any reply for the issue you've opened?

[bogdanmuresan]

@unkaputtbar112 They asked me to try different settings, none of which worked. They are still investigating. I've sent them sos reports, and also had a screen-sharing session. It might help if you could also open a ticket. Do you also see the db timeout when the 403 occurs?

[unkaputtbar112]

@bogdanmuresan yes, i'm seeing the timeout errors as well:

775-2025-08-17 20:53:56,958 INFO     a7a3b43c-a116-4a4a-b2d2-723074bda2ce ansible_base.oauth2_provider.authentication User configuration_management performed a GET to /api/controller/v2/inventories/?name=SSA+%2F+Azure through the API using OAuth 2 token 1.
776:Failed to connect to DB! TimeoutError
--
1005-2025-08-17 20:54:56,917 INFO     96d9f165-5c7a-45cb-b145-bd10601918e4 ansible_base.oauth2_provider.authentication User configuration_management performed a GET to /api/controller/v2/projects/?name=INI+%2F+AAP+Configuration through the API using OAuth 2 token 1.
1006:Failed to connect to DB! TimeoutError
--
1577-2025-08-17 21:10:50,254 INFO     8a240601-8089-46e6-9adf-e4eaf3c49c84 ansible_base.oauth2_provider.authentication User configuration_management performed a POST to /api/controller/v2/job_templates/235/instance_groups/ through the API using OAuth 2 token 1.
1578:Failed to connect to DB! TimeoutError
--
2409-2025-08-17 21:19:40,508 INFO     229b9719-a8f5-4a1f-b7a9-1fe033d395bf ansible_base.oauth2_provider.authentication User configuration_management performed a POST to /api/controller/v2/workflow_job_template_nodes/ through the API using OAuth 2 token 1.
2410:Failed to connect to DB! TimeoutError
--
2976-2025-08-17 21:26:05,401 INFO     4e6a82f8-e1d9-4355-80fb-426244cc51e7 ansible_base.oauth2_provider.authentication User configuration_management performed a GET to /api/controller/v2/workflow_job_template_nodes/?workflow_job_template=217 through the API using OAuth 2 token 1.
2977:Failed to connect to DB! TimeoutError
--
3441-2025-08-17 21:28:20,364 INFO     886c9f60-1fba-4757-81e5-d3f313f6c08b ansible_base.oauth2_provider.authentication User configuration_management performed a GET to /api/controller/v2/workflow_job_templates/?organization=8&name=SSA+%2F+Azure+%2F+WF+%2F+Disk+Management+%2F+Create+Disk+to+create+Volume through the API using OAuth 2 token 1.
3442:Failed to connect to DB! TimeoutError

Also i noticed, i'm getting connection errors in the UI sometimes, a little red bar appears.
I sadly don't have the access to open the incident / issue, but i'll get in touch with the guys, that can.

[bogdanmuresan]

@unkaputtbar112 Thank you for confirming. I will add a link to your comment to our Red Hat open case.

[unkaputtbar112]

@bogdanmuresan thanks for the pinpointing.
Although it's absolute shit, but until they find a solution, i'm going with locally modifying this:

    def _can_read_from_db(self) -> Optional[external_auth_pb2.CheckResponse]:
        # This is overkill if we have CONN_HEALTH_CHECK=True in the settings
        # But just incase we will also do our own detection
        #timeout = getattr(settings, "PING_PAGE_CHECK_TIMEOUT", 5)
        timeout = getattr(settings, "PING_PAGE_CHECK_TIMEOUT", 20)
        # We need to close out old connections here to prevent accidental usage of obsolete database connections
        close_old_connections()
        try:
            get_db_connection_status('default', timeout)
            return None
        except Exception as e:
            ###
            try:
                get_db_connection_status('default', timeout)
                return None
            except Exception as ee:
                return self._return_no_auth_with_reason(f'Unable to connect to database: {type(ee).__name__}')
            ###
#            return self._return_no_auth_with_reason(f'Unable to connect to database: {type(e).__name__}')

Just that he retries a 2nd time and probably has an increased timeout. Restart of the gateway is neccessary afterwards, as it seems. Maybe coincidental, but the first run after the modification & restart just went through, after having 2 hours of no run succeeding, so i'm sticking with this until they've fixed it.

[bogdanmuresan]

@unkaputtbar112 Did you just increase the timeout? I also tried that, by adding "PING_PAGE_CHECK_TIMEOUT = 10" in
/etc/ansible-automation-platform/gateway/settings.py , and even thought it was green after I restarted the gateway, it then started replicateing again on consecutive runs..

[unkaputtbar112]

@bogdanmuresan i just duplicated the try/expect, so it's just trying again in the expect, i had now multiple consecutive succesful runs. It does not solve the core issue, because it's just a retry, but it works for me for now.

[bogdanmuresan]

@unkaputtbar112 - thank you, I've only noticed the retry after posting my comment. RedHat posted an updated on our ticket that now many accounts are affected with this error so the JIRA is on high priority with engineering.

[bogdanmuresan]

This has been fixed in the AAP 2.5 released on 23rd of September:
https://docs.redhat.com/en/documentation/red_hat_ansible_automation_platform/2.5/html/release_notes/patch_releases#aap-25-20250923

https://issues.redhat.com/browse/AAP-49473