By Lerpong Intaraworrapath | May 17th, 2022
ใน VMware vRealize Automation version 8.5.x เราจะพบว่า ในบางครั้ง เมื่อเรา Login เข้าไปที่ vRealize Automation portal จะไม่สามารถเข้าไปใช้งานได้ จะเจอ ข้อความ “ClientResponse has erroneous status code: 502 Bad Gateway“
Symptoms:
ใน VMware vRealize Automation version 8.5.x หรือหลังจาก ทำการ upgrade to version 8.5.x เมื่อเรา login ในหน้า UI จะเกิด intermittently fails และ services logs contain 502 Bad Gateway exception เนื่องจาก พยายามจะ acquire access tokens from Identity service.
Errors from the identity-service within the identity-service-app-xxxxxxxxx-xxxxx pod contains errors similar to
2021-09-02T13:42:30.824Z ERROR identity-service [host='identity-service-app-9b97cb5f7-n9mdv' thread='reactor-http-epoll-1' user='' org='' trace='4e2d69fb-b884-4ba0-8ca3-60913ce5c830'] reactor.netty.http.server.HttpServer.error:319 - [id:0xa55a54c7,L:/10.244.0.128:8080 - R:/10.244.0.126:45626]
java.io.UncheckedIOException: java.nio.file.FileSystemException: /tmp/synchronoss-file-upload-1080171699777
เมื่อเรา Login vRealize Automation portal จะแสดงผล Error ดังภาพ.
Resolution:
Issues ที่เกิดขึ้น จะแก้โดยใช้ VMware KB86121
https://kb.vmware.com/s/article/86121
Prerequisites:
ทำการ Create snapshot vRealize Automation appliance โดย without memory snapshot
Procedure:
1.ทำการ SSH / PuTTy ไปที่ appliance ตัวใดตัวนึงของ vRA cluster ด้วย user “root”.
2.Run the following command
echo 'IyEvYmluL2Jhc2gKCmlmIFsgIiQoY2F0IC9vcHQvc2NyaXB0cy9zdGF0ZV9lbmZvcmNlbWVudC5zaCB8IGdyZXAgImNsZWFudXBfaWRlbnRpdHlfcG9kc191bnVzZWRfZGlycyIgfCB3YyAtbCkiID09ICIxIiBdCnRoZW4KICAgIGVjaG8gIlRoZSBLQiBmb3IgY2xlYW51cCBvZiB1bnVzZWQgZGlycyBpbiBpZGVudGl0eSBwb2RzIGlzIGFscmVhZHkgYXBwbGllZC4iCiAgICBleGl0IDAKZmkKCnZyYWNsaSBjbHVzdGVyIGV4ZWMgLS0gYmFzaCAtYyAiZWNobyAnSXlFdlltbHVMMkpoYzJnS0NpTWdRMjl3ZVhKcFoyaDBJQ2hqS1NBeU1ESXhJRlpOZDJGeVpTd2dTVzVqTGlBZ1FXeHNJSEpwWjJoMGN5QnlaWE5sY25abFpDNEtJd29qSUZSb2FYTWdZMjlrWlNCcGN5Qm1iM0lnZFhObElHWnliMjBnWW5WcGJIUXRhVzRnWVhWMGIyMWhkR2xqSUhONWMzUmxiWE11SUVSdklHNXZkQ0JqWVd4c0lHbDBJR1p5YjIwS0l5QXpjbVFnY0dGeWRIa2djM2x6ZEdWdGN5d2djbVYxYzJVZ2IzSWdjbVZ3Y205a2RXTmxMZ29qQ2dvaklGUm9hWE1nYzJOeWFYQjBJR2x6SUc5dWJIa2dZWEJ3YkdsallXSnNaU0JtYjNJZ2RsSkJJRGd1TlNCSFFTQjJaWEp6YVc5dUlHRnVaQ0JwZENCemFHOTFiR1FnYm05MElHSmxJSFZ6WldRZ2IyNGdZVzU1SUc5MGFHVnlJSFpTUVNCMlpYSnphVzl1Y3k0S0NtWjFibU4wYVc5dUlHeHZaeWdwSUhzS0lDQnNiMk5oYkNCdGMyYzlJaVF4SWdvZ0lHeHZZMkZzSUd4bGRtVnNQU0lrTWlJS0lDQnNiMk5oYkNCa2REMGtLR1JoZEdVZ0p5c2xXUzBsYlMwbFpDQWxTRG9sVFRvbFV5Y3BDZ29nSUdWamFHOGdJbHNrYkdWMlpXeGRXeVJrZEYwZ0pHMXpaeUlLZlFvS1puVnVZM1JwYjI0Z2JHOW5YMmx1Wm04b0tTQjdDaUFnYkc5bklDSWtNU0lnSWtsT1JrOGlDbjBLQ25ObGRDQXJaUW9LYkc5blgybHVabThnSWtOc1pXRnVhVzVuSUhWd0lHbGtaVzUwYVhSNUlIQnZaSE1nZFc1MWMyVmtJR1JwY25NdUxpNGlDbWxtSUZzZ0xXWWdMM1poY2k5MmJYZGhjbVV2Y0hKbGJIVmtaUzlwWkdWdWRHbDBlUzF6ZG1NdmJHRnpkQzFqYkdWaGJuVndJRjBLZEdobGJnb2dJR2xtSUZzZ0lpUW9abWx1WkNBdmRtRnlMM1p0ZDJGeVpTOXdjbVZzZFdSbEwybGtaVzUwYVhSNUxYTjJZeThnTFc1aGJXVWdiR0Z6ZEMxamJHVmhiblZ3SUMxMGVYQmxJR1lnTFcxdGFXNGdMVGN5TUNCOElIZGpJQzFzS1NJZ1BUMGdJakVpSUYwS0lDQjBhR1Z1Q2lBZ0lDQnNiMmRmYVc1bWJ5QWlTV1JsYm5ScGRIa2djMlZ5ZG1salpTQnplVzVqYUhKdmJtOXpjeTBxSUhOMVltUnBjbVZqZEc5eWFXVnpJR2hoZG1VZ1lXeHlaV0ZrZVNCaVpXVnVJR05zWldGdVpXUWdkWEFnYVc0Z2RHaGxJSEJoYzNRZ01USWdhRzkxY25NdUlnb2dJQ0FnWlhocGRDQXdDaUFnWm1rS1pta0tDbWxrWlc1MGFYUjVYM05sY25acFkyVmZjRzlrYzE5c2FXNWxQU1FvYTNWaVpXTjBiQ0JuWlhRZ2NHOWtjeUF0YmlCd2NtVnNkV1JsSUMxc0lHRndjRDFwWkdWdWRHbDBlUzF6WlhKMmFXTmxMV0Z3Y0NBdExXOTFkSEIxZEQxcWMyOXVjR0YwYUQxN0xtbDBaVzF6TGk1dFpYUmhaR0YwWVM1dVlXMWxmU2tLQ2tsR1V6MG5JQ2NnY21WaFpDQXRjaUF0WVNCd2IyUnpJRHc4UENBaUpHbGtaVzUwYVhSNVgzTmxjblpwWTJWZmNHOWtjMTlzYVc1bElnb0tabTl5SUhCdlpDQnBiaUFpSkh0d2IyUnpXMEJkZlNJS1pHOEtJQ0FnSUdsa1pXNTBhWFI1WDNCdlpGOXpkR0YwZFhNOUpDaHJkV0psWTNSc0lHZGxkQ0J3YjJSeklDMXVJSEJ5Wld4MVpHVWdJaVJ3YjJRaUlDMHRiM1YwY0hWMFBXcHpiMjV3WVhSb1BYc3VjM1JoZEhWekxuQm9ZWE5sZlNrS0NpQWdJQ0JwWmlCYklDSlNkVzV1YVc1bklpQTlQU0FpSkdsa1pXNTBhWFI1WDNCdlpGOXpkR0YwZFhNaUlGMEtJQ0FnSUhSb1pXNEtJQ0FnSUNBZ0lDQWpJRVJsYkdWMFpTQnpkV0lnWkdseWN5QnBiaUF2ZEcxd0lHOXNaR1Z5SUhSb1lXNGdNaUJvYjNWeWN3b2dJQ0FnSUNBZ0lHeHZaMTlwYm1adklDSkVaV3hsZEdsdVp5QjFiblZ6WldRZ1pHbHljeUJtY205dElDUndiMlFpQ2lBZ0lDQWdJQ0FnYTNWaVpXTjBiQ0JsZUdWaklDMXBkQ0F0YmlCd2NtVnNkV1JsSUNJa2NHOWtJaUF0TFNCbWFXNWtJQzkwYlhBZ0xXNWhiV1VnSjNONWJtTm9jbTl1YjNOektpY2dMVzF0YVc0Z0t6RXlNQ0F0WkdWc1pYUmxJREkrTDJSbGRpOXVkV3hzSUh4OElIUnlkV1VLSUNBZ0lHWnBDbVJ2Ym1VS0NuWnlZV05zYVNCamJIVnpkR1Z5SUdWNFpXTWdMUzBnWW1GemFDQXRZeUFpYld0a2FYSWdMWEFnTDNaaGNpOTJiWGRoY21VdmNISmxiSFZrWlM5cFpHVnVkR2wwZVMxemRtTTdJSFJ2ZFdOb0lDOTJZWEl2ZG0xM1lYSmxMM0J5Wld4MVpHVXZhV1JsYm5ScGRIa3RjM1pqTDJ4aGMzUXRZMnhsWVc1MWNDSUtDZz09JyB8IGJhc2U2NCAtZCA+IC9vcHQvc2NyaXB0cy8yODMzMTYxX2NsZWFudXBfaWRlbnRpdHlfcG9kc191bnVzZWRfZGlycy5zaCAmJiBjaG1vZCAreCAvb3B0L3NjcmlwdHMvMjgzMzE2MV9jbGVhbnVwX2lkZW50aXR5X3BvZHNfdW51c2VkX2RpcnMuc2g7IGVjaG8gJy9vcHQvc2NyaXB0cy8yODMzMTYxX2NsZWFudXBfaWRlbnRpdHlfcG9kc191bnVzZWRfZGlycy5zaCcgPj4gL29wdC9zY3JpcHRzL3N0YXRlX2VuZm9yY2VtZW50LnNoIgoK' | base64 -d > /root/kb-identity-pods-cleanup.sh && chmod +x /root/kb-identity-pods-cleanup.sh && /root/kb-identity-pods-cleanup.sh && rm /root/kb-identity-pods-cleanup.sh
Validate the change:
1. Verify the shell script “/opt/scripts/cleanup_identity_pods_unused_dirs.sh” exists on each appliance in the cluster.
2. Verify the shell script “/opt/scripts/state_enforcement.sh” contains the following in the last line
cat /opt/scripts/state_enforcement.sh
3. Verify that the next 2 state-enforcement pods within kube-system namespace complete successfully by running
kubectl get pods -n kube-system
4. ทำการ access เข้าไปที่ vRealize Automation portal
