Notes from the lab: VMware vCenter 7u2 ADFS changes

When vCenter 7 introduced ADFS integration I jumped on the configuration part in my lab and set it up with the necessary OAUTH integrations:

Now with vCenter 7u2 there are some changes when you have it in place and are upgrading:

The trust store is changed to VECS and you need to change/add that in vCenter:

Well one would think that everything is nice and dandy after this but I completely forgot that at the time I set the whole ADFS integration part on LDAP and of course no signing requirement in place:

These are the extra changes in my environment and need to be changed in vCenter as well:


Keep in mind that you will need to add a certificate for the LDAPS configuration part, after this the whole configuration should work again.

In my setup I would see a long waiting white screen the first time I would log in, that got resolved after a reboot of the vCenter appliance.

Hope it helps!

Notes from the lab: Configuring vCenter 7 with ADFS

With the release of vCenter 7 you can now integrate it with Microsof Active Directory Federation Services (ADFS)

See the following blog article for an overview:
https://blogs.vmware.com/vsphere/2020/03/vsphere-7-identity-federation.html

See the following configuration articles for a setup overview:
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-C5E998B2-1148-46DC-990E-A5DB71F93351.html
https://kb.vmware.com/s/article/78029

With this information I’ve configured my lab environment to a working SAML based login with a few minor issues.

I had my ADFS setup load balanced through a content switching setup for external access. This is working great for my simple office 365 integration point but not so much if you’re trying to do more.

Like stated in the following article don’t terminate the SSL connection:
https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/overview/ad-fs-faq

The issue I came across was that vCenter was failing with the validation of the certificate, at first I thought the missing root/intermediates was the cause of this but this was not the case. Even after uploading my internal root/intermediate and the external certificates root/intermediate the SSL validation check would fail. The chain was valid in every case though.

The resolution was to make my internal DNS entry through a SSL-BRIDGE setup to my ADFS server and afterwards I could finish the configuration part without issues.

Now when presented with vCenter logon page if you put in an account from the federated domain it will redirect you accordingly to the ADFS logon point.

Hope it helps!

Notes from the lab: Migrating Windows vCenter to VCSA 7

In my lab environment I was running Windows vCenter 6.7 and with the release of vCenter 7 a migration is needed because there is no Windows vCenter anymore.

The following articles will give you enough information on how the process works especially the how-to from Vladan Seget:
https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-9A117817-B78D-4BBE-A957-982C734F7C5F.html
https://www.starwindsoftware.com/blog/how-to-migrate-vmware-vcenter-from-windows-to-vcsa-6-7-update-1

Basically the process is the same for vCenter 7 with in my case one issue.

At first try my migration failed at the stage when the migration assistant is shutting down the Windows vCenter and the VCSA 7 is being brought up with the original IP, hence resulting in a conflict and the upgrade is in a broken state. I followed the process and saw that especially Windows Server 2016 has the annoyance to delay the shutdown for minutes, this is a known issue and happens from time to time and to my knowledge has to do when updates were installed (even after multiple reboots).

So the migration assistant is doing its job and completes the migration and is shutting down the VM, except in my case. Resolution for this is when the script is completed and windows says it’s still shutting down etc. do a hard power off and after that the migration is moving further as expected.

In summary:
1. Make sure Windows is up to date with all the updates and there are none outstanding
2. Make sure vCenter is up to date, in my case it was the latest update of 6.7
3. Make a snapshot before you start 😊 saves a lot of hassle later on
4. Force kill the Windows vCenter when it’s shutting down after the migration assistant completes

After this the migration in my case completed flawlessly.

Hope it helps!

Notes from the field: vCenter cannot validate SSO domain

Came across a peculiar issue when adding an second vCenter to the same SSO domain and enable ELM.
The first deployment worked like a charm and the second errored out with the following error:

It turns out there is a known bug when using uppercase FQDN in the configuration wizard, the solution is to put it all in lowercase.
see the following link for reference: https://kb.vmware.com/s/article/56812

Notes from the field: VMware vCenter /dev/mapper/core_vg-core full

Not too long ago I’ve encountered an vCenter instance blowing up the /dev/mapper/core_vg-core with gigabytes of java dump errors.. Just for reference the customers setup is an dual SDDC with respectively an vCenter at each site comprising of vCenter 6.5 U2 and embedded linked mode enabled.

In troubleshooting mode I’ve encountered the following two articles:

https://kb.vmware.com/s/article/2150731

https://kb.vmware.com/s/article/60161

Afterwards decided to open up a support case. This resulted in a session which stated that they had seen this sort of issues arising in 6.7u1 and higher which root caused against hardware level 13 for the appliance and WIA Active Directory integration.

The customer setup had an hardware level 13 deployment on both sites and only one experiencing the problem, and using Active Directory over LDAP integration.

The resolution of the issue was downgrading the VCSA hardware level to version 10.

Supports way is restoring the VCSA with a VAMI back-up restore, my way was re-register the appliance with the VMX file downgraded to the level needed, see https://communities.vmware.com/thread/517825

Hope this helps!

Notes from the field: vCenter VCSA 6.5u2 SEAT cleanup

This is a quick blog to show how an SEAT database failure can be cleared after an sporadic growth and increase to the events part of the SEAT DB in VCSA. I’ll explain the issue origin in an upcoming blog, but in a nutshell the 20gb was reached within six days and crashed the vCenter of a secondary site.

You SSH into the vCenter VCSA and enable shell and afterwards go to the vpostgres directory to complete the tasks. See below entries for reference and testing:

shell.set –enabled true

cd /opt/vmware/vpostgres/current/bin

./psql -d VCDB -U postgres

SELECT nspname || ‘.’ || relname AS “relation”, pg_size_pretty(pg_total_relation_size(C.oid)) AS “total_size”
FROM pg_class C
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE nspname NOT IN (‘pg_catalog’, ‘information_schema’)
AND C.relkind <> ‘i’
AND nspname !~ ‘^pg_toast’
ORDER BY pg_total_relation_size(C.oid) DESC
LIMIT 20;

After these commands you will see the twenty top entries of the database, in my case there were entries from 800mb/1gb+ files and needed to be truncated

for example you filter out the largest and truncate that file:

truncate table vc.vpx_event_1 cascade;

Do this numerous times until the largests sets of events are gone, I did do all of this with a sev1 support case engineer so this is not something to do out of the blue. Hope this helps you out as well.

For reference the following article for vPostgres DB interaction:

https://kb.vmware.com/s/article/2147285