After upgrading vCenter Server from 7.0U2 to 7.0U3, I encountered an issue with my customer. They attempted to use vCenter Server to deploy an OVA file. They discovered the following error:
“Failed to deploy OVF package.ThrowableProxy.cause A general system error occurred: Transfer failed: Invalid response code: 400, note that HTTP/s proxy is configured for the transfer“
vCenter Server version 7.0.3.01400 build 21477706
Below are some additional workarounds.
Option 1 – Deploy through vSphere ESXi host
1.Login directly through vSphere ESXi host.
2.If you connect with your FQDN and you will still be unable to deploy the OVA file. Change the vSphere ESXi host login to an IP address and try again to deploy the OVA file.
Option 2 – Disable proxy setting in vCenter Server
1.In a Web browser, go to the vCenter Server Management Interface, https://appliance-IP-address-or-FQDN:5480.
Note:If you need to make changes to vCenter Server, we recommend taking a snapshot.
2.Login as root.
3.Go to the Networking section.
4.Click EDIT under the Proxy Setting section.
5.Uncheck both HTTPS and HTTP proxy enabled and click SAVE.
6.The OVA file will be successful.
However, we discovered a new issue in which we were unable to enable the proxy and input the proxy server into the proxy settings.
7.We follow the VMware KB81565 to bring proxy configuration back to vCenter Server.
7.1)Change the HTTPS PROXY settings.
7.2)We make a backup of the proxy file.
7.3)Change the /etc/sysconfig/proxy file. Change the HTTPS_PROXY line to update the value from https to http:
HTTPS_PROXY="https://proxy.domain:80/"
to
HTTPS_PROXY="http://proxy.domain:80/"
Note: The proxy URL is determined by your proxy server.
7.4)If you are using a version previous to 7.0 U1, restart the VCSA.Otherwise, use the command to restart services, then logout and log back in:
In this article, we are planning to upgrade vCenter Server appliance version 7.0.2.00500 build number 18455184 (vCenter Server 7.0 Update 2d) to vCenter Server 7.0 Update 3g (7.0.3.00800) build number 20150588 or higher.
Before upgrading vCenter Server, we must first check its health status by logging onto https://vCenter-Server-FQDN:5480
User: root Password: xxxxx
After login to vCenter Server appliance, go to summary and check on the “Health Status”.
We discovered that “Overall Health” displays a yellow alert warning on “Storage“.
Expand “Storage“, we found alert show “File system /storage/log is low on storage space increase the size of disk /storage/log“
Go to “Monitor“, then “Disks“, and see which hard disks have alerts.
Hard drive 5 (log) has a utilization rate of more than 80%.
Run “df-h” command to display file system disk space statistics in “human-readable” format.
Return to the vCenter Server web client. There was no snapshot on the vCenter Server appliance.
Right-click vCenter Server and choose “Edit Settings“.
Select Hard disks and expand Hard disk 5.
We will increase the size of hard disk 5 from 25 GB to 30 GB and then click OK.
Log in to the vCenter Server Appliance through SSH.
When we use the vSphere Web Client to connect to vCenter Server appliance 7.x. We are unable to access with the message “HTTP Status 500 – Internal Server Error.“
Steps to resolving these issues.
1.SSH into the vCenter Server appliance.
2.To see the status certificate expiration date, use the command below. for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
3.You will see an output similar to:
4.As you can see, the Machine SSL certificate expires on September 1 06:40:37 2022 GMT.
5.The Name, Hostname and VMCA values should match the PNID of the Node where you are replacing the Certificates. PNID should always match the Hostname. In order to obtain the PNID please run these commands: /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
6.Run command below to replace “Machine SSL certificate”. /usr/lib/vmware-vmca/bin/certificate-manager
7.You will have the option to replace or reset the certificate with in output.
Please keep in mind that this command may be used with both vCenter Server appliances 6.x and 7.x.
8.To replace Machine SSL certificate with VMCA Certificate, we choose option 3.
9.Provide credential
10.Enter these values as prompted by the VMCA (See Step 5 to confirm the Name/Hostname/VMCA):
11.To proceed, answer Yes (Y) to the confirmation request.
12.Wait till the status is 100% completed.
13.Re-run command to check Machine SSL certificate for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
14.Machine SSL certificate has been updated to August 31 12:14:11 2024 GMT.
Used vmsa-2021-0028-kb87081.py script from KB 87088 and remove_log4j_class.py from this KB87081.
Used the manual workaround steps in KB87081 and remove_log4j_class.py.
Note: ในบทความนี้ จะใช้ วิธีที่ 2 – Used vmsa-2021-0028-kb87081.py script from KB 87088 and remove_log4j_class.py from this KB87081. โดยจะเป็น vCenter server virtual appliance version 7.0.1.00301
Workaround
Automated Workaround (Recommended)
ทำการ take snapshot “vCenter server virtual appliance”.
ทำการ check disk space available by command “df-h” (capture screen ไว้ด้วย).
ทำการ check vCenter server service by command “service-control –status –all” (capture screen ไว้ด้วย).
13. ทำการ Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client) แล้วเข้าไปที่ path /tmp ใช้ “ls” command ทำการตรวจสอบไฟล์ ที่เราทำการ upload เข้าไป
14. ทำการ run script vmsa-2021-0028-kb87081.py root@xxxx[/tmp]#python vmsa-2021-0028-kb87081.py
15. พิมพ์ Y แล้วกด Enter
16. Script ก็จะทำการ run process เพื่อจะทำ workaround เกี่ยวกับ CVE-2021-44228 and CVE-2021-45046 (ใช้เวลา10 นาที โดยประมาณ). โดย script ก็จะแสดง status ของแต่ละ process และที่สำคัญจะต้องแสดงผล “SUCCESS” -vMON Config files -VMware Update Manager Config files -Analytics service Config files -DBCC Utility Config files
17. เมื่อทำการ run script แรก เสร็จเรียบร้อยแล้วให้ เรา run script ตัวที่ 2 คือ remove_log4j_class.py from this KB87081. root@xxxx[/tmp]#python remove_log4j_class.py
18. พิมพ์ Y แล้วกด Enter
19. Script ก็จะทำการ run process เพื่อจะทำ workaround เกี่ยวกับ CVE-2021-44228 and CVE-2021-45046 (ใช้เวลา10 นาที โดยประมาณ).โดย script ก็จะแสดง status ของแต่ละ process
Verify the changes
เมื่อเราทำการ run script ทั้ง 2 scripts completed แล้วให้เราทำ การ ตรวจสอบให้แน่ใจว่า script ที่เรา run นั้นถูกต้อง
Verify if the stsd, idmd, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter: ps auxww | grep formatMsgNoLookups
โดย จะแสดง ผล -Dlog4j2.formatMsgNoLookups=true
2. Verify the Update Manager changes are shown under “System Properties” in the output of the following two commands: cd /usr/lib/vmware-updatemgr/bin/jetty/ java -jar start.jar --list-config system properties: log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)
3. Verify the Analytics Service changes: grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l โดย จะแสดงค่า “0“
4. Verify the script successfully removed JndiLookup.class from all java files with the following command: python remove_log4j_class.py -r
จะได้ผลลัพธ์ ดังนี้ 2021-12-18T00:04:38 INFO main: Running in dryrun mode 2021-12-18T00:05:04 INFO main: ===== Summary ===== List of vulnerable files: 2021-12-18T00:05:04 INFO main: Done.
Remark
หลังจาก เราทำการ run script เสร็จเมื่อกลับที่ vSphere client browser ของ vCenter server จะพบว่า ที่ Tab -> Monitor ในหน้า Skyline Health จะ show warning “Online health checks execution” (ถ้าไม่เจอแบบนี้ก็ไม่เป็นไร นะครับ) !!!warning ที่แสดง จะหายไป เอง นะครับ!!!
Conclusion
ในบทความนี้จะเป็น workaround สำหรับ vCenter server virtual appliance 7.x เท่านั้น โดย fix version ยังไม่มีออกมาในส่วน version อื่น ให้ทุกท่านติดตาม VMSA-2021-0028 ในการทำ workaround หรือ fix version รวมถึง Product อื่นๆ ของ VMware ที่ได้รับผลกระทบ เช่นกัน.
