AWX is intermittently slow reporting job end status from container group jobs

[philipsd6]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Job executed in EE are run successfully, but when the pod reports completion, and the job status in the UI remains running for up to 20 minutes.

It seems to be hanging on the receptor work unit release call.

AWX version

24.6.1

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

openshift

Modifications

yes

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Running a simple "hello world" playbook, JT running in the default container group. Execute n times, where n up to 200.

Expected results

The jobs should run to completion and then report final status quickly and be released.

Actual results

Some of them complete, but many of them get stuck on the receptor work unit release call, and that slows down the processing of the job queue.

Additional information

project_sync.yaml has been modified for internal sources, EE are directing logs to external system.

[philipsd6]

We did not identify the root cause for the persistent slowness at scale, however we updated the receptor version in our AWX image to 1.6.2, and are again operating well at scale.