Notes from the field: VMware Horizon instant clone breaks with Kerberos armoring

On my current customer project we’ve encountered a strange issue when some stricter security policies were implemented. Kerberos armoring was enabled which effectively broke the instant clone process for Windows 10 1809/1909 releases but not for 2009 or 21H2.

It all started with a ticket that the image update process in Horizon would error out and fail constantly on the mentioned images. On the newer builds no problem at all. At first we thought it was an Microsoft update of some sorts but after some troubleshooting with colleagues Wesley Kieffer and JP Ruitenbeek it turned out to be new hardening items which got turned on.

When clearing the before setting in the image and then sealing the older images would update just fine. After reproducing the entire setup in my own lab I encountered the same issue as well. Opened up a VMware support case which still is being worked at.

The setup being used is VMware Horizon 7.13.1

The following article also describes how you can troubleshoot the internal VM which is being used for the cloning process: Troubleshooting Instant Clones in the Internal VM Debug Mode

And for some more information regarding Kerberos armoring see What’s New in Kerberos Authentication | Microsoft Docs and Compound Authentication and Active Directory Domain Services claims in Active Directory Federation Services | Microsoft Docs

Hope it helps!

Notes from the lab: Citrix ShareFile and VMware Access SSO

When configuring Citrix ShareFile for an SSO experience with your Microsoft Active Directory setup we have the following guides to use it from Citrix. See How to Configure Single Sign-On (SSO) for ShareFile (citrix.com)

Well I’m having my setup with another Identity Provider in my own lab and still want to achieve an managed SSO setup from my end. To get this to work I checked the setup from an existing integration setup like Microsoft ADFS and reverse engineered it to VMware Access instead.

The following will give you an working SSO setup with VMware Access as your Identity Provider for Citrix ShareFile:

First configure the basic settings of Citrix ShareFile with your URL’s

https://tenant-fqdn-sharefile/saml

https://tenant-fqdn-vmware

Copy paste the certificate information from your VMware Access tenant

Create the application in VMware Access

Use the explicit logon URL of your application in VMware Access in Citrix ShareFile

Use the https://tenant-fqdn-sharefile/saml/acs in VMware Access for consumption

After this make sure that an user from your corporate identity is synchronized in Citrix ShareFile and both of them match so that SSO can be achieved. Afterwards you can nicely click the icon in VMware Access from IDP point of view:

Or you can login via the SP side from the Citrix ShareFile browser landing page:

Hope it helps!

Notes from the lab: VMware vCenter 7u2 ADFS changes

When vCenter 7 introduced ADFS integration I jumped on the configuration part in my lab and set it up with the necessary OAUTH integrations:

Now with vCenter 7u2 there are some changes when you have it in place and are upgrading:

The trust store is changed to VECS and you need to change/add that in vCenter:

Well one would think that everything is nice and dandy after this but I completely forgot that at the time I set the whole ADFS integration part on LDAP and of course no signing requirement in place:

These are the extra changes in my environment and need to be changed in vCenter as well:


Keep in mind that you will need to add a certificate for the LDAPS configuration part, after this the whole configuration should work again.

In my setup I would see a long waiting white screen the first time I would log in, that got resolved after a reboot of the vCenter appliance.

Hope it helps!

Notes from the field: Citrix FAS request not supported

On a recent Citrix FAS deployment I’ve encountered the following error: “Request not supported” when logging in to a published application or desktop.
Article https://support.citrix.com/article/CTX218941 explains that re-enrollment of the domain controller authentication template or another custom template for Kerberos usage should resolve the error.

A little bit of a background on the environment, an already working Microsoft ADCS environment was in play and in use for other services. From a design/security perspective it was designed that two dedicated Microsoft ADCS servers would be used and two Citrix FAS servers connecting these new servers. The setup was working as expected but only above error would keep coming when trying to access an application or desktop.

We tried re-enrolling the domain controller authentication certificate and this didn’t do the trick, then we decided to let the Domain Controllers get the certificate from the new dedicated Microsoft ADCS servers for Citrix FAS and this did do the trick but with a side effect the chain is changed and other services would be negatively impacted so a rollback was needed.

With this information a Microsoft support case was created and ultimately they confirmed that what is mentioned in the Citrix support article should do the trick. Ok we got confirmation and yes it indeed does work when using the new ADCS servers but the issue of the original ADCS environment was still a mystery.

So next up we decided to repoint the Citrix FAS servers to the existing Microsoft ADCS server to root out any chain or other issues that might be in play. The result was exactly the same and a not supported request as the end result.

Digging deeper in the Microsoft ADCS environment it was after checking the “NTAuthCertificates” store that the existing server wasn’t there and the new servers were. This explained the smartcard logon not working when using the existing environment because an requirement for smartcard logon is that the “NTAuthCertificates” store has the issuing certificate authority propagated. After adding the certificate and waiting for replication and a reboot everything was working as expected, also when moving to the new Microsoft ADCS environment for certificate issuing.

See the following screenshot of the Enterprise PKI snap in MMC in which you can check and/or add the missing certificate:

See the following articles for extra information:
https://docs.microsoft.com/en-us/troubleshoot/windows-server/windows-security/import-third-party-ca-to-enterprise-ntauth-store
https://support.microsoft.com/en-za/help/281245/guidelines-for-enabling-smart-card-logon-with-third-party-certificatio

Notes from the lab: Microsoft ADFS and VMware UAG

You don’t see many configuration articles around ADFS and UAG and that’s why I would like to share my setup.

First things first, I’m expecting that there is an working Horizon environment with True SSO enabled for access to the desktop. And a working ADFS environment to add a new application to test with.

My setup:
1 x ADFS for internal usage
1 x WAP for external usage
1 x UAG v3.10 – dedicated for ADFS with its own URL
1 x UAG v3.10 – dedicated for WS1 with its own URL
2 x Horizon Connection Servers
2 x Horizon Enrollment Servers

The following articles helped me in setting this up:
https://docs.vmware.com/en/Unified-Access-Gateway/3.10/com.vmware.uag-310-deploy-config.doc/GUID-E4C8B88F-C771-4829-ABBE-12F7FBF517C3.html
https://communities.vmware.com/thread/625006
https://thevirtualhorizon.com/2019/12/14/integrating-microsoft-azure-mfa-with-vmware-unified-access-gateway-3-8/
https://techzone.vmware.com/enabling-saml-20-authentication-horizon-unified-access-gateway-and-okta-vmware-horizon-operational

Setting this up starts with downloading the “/FederationMetadata/2007-06/FederationMetadata.xml” file of your ADFS setup and saving this on your configuration machine

Now go to the UAG admin appliance on the management port https://xxx:9443 and scroll down to the Identity Bridging Settings, select the gear icon of “Upload Identity Provider Metadata” and in the next screen press on the select link where you can upload the metadata file of ADFS.
Now go to the Horizon settings on the top and select the gearbox and in the next screen select more to show the auth methods to be used. For True SSO select SAML. Next select the uploaded Identity Provider which is now visible and click on the “Download SAML service provider metadata”. Save everything and close the admin portal.

Next in ADFS configure a new “Relying Party Trust” claims aware application and import the downloaded SAML service provider metadata file from UAG
When completed edit the “Claim issuance policy” for the application and create a rule with attribute store active directory selected and provide the input for “User-Principal-Name” with “Name ID” as outgoing claim type.
Edit the application and select encryption and click on remove the encryption certificate. This enabled will not give an valid SAML assertion or logon.

This is all that is needed in ADFS, the application can be assigned anyway you want. The last steps are in Horizon itself.

Go to the admin panel of the connection server and configure an SAML 2.0 Authenticator, create one and name it accordingly (don’t forget to enable True SSO on this connector) and make sure it is an static type. Copy paste the federationmetadata.xml content into the SAML Metadata screen and click OK.

Set the delegation of authentication to VMware Horizon (SAML 2.0 Authenticator) to optional or required. The latter prohibits passthrough authentication straight from the servers itself and will always require a SAML assertion.

This is it, afterwards you should see an ADFS logon possible with SSO in place.

Some extra information for any use case with ADFS/Workspace ONE combined then this procedure will not help. For Workspace ONE integration the only working and supported way is to configure VMware Horizon with VMware Access. I’ve tried it configured with VMware Access and the same UAG and you will get an access denied because the SAML configuration is in place at the Horizon Connection Servers instead of the UAG. ADFS can also be integrated with VMware Access and the SSO can be achieved in that way which is a route you would take when using Workspace ONE.

Hope it helps!

Notes from the field: Citrix FAS SSO not working with invalid CRL

Recently I got contacted by a customer who had problems performing an SSO to a newly build desktop environment.

The setup a greenfield resource domain and forest trust from an existing tenant with a two way trust. Basically everything was correct but the logon from the users would always get terminated at the desktop with invalid credentials.

After a short discussion and remote session and the error messages in the logs with an invalid CRL it was clear that was the issue. Troubleshooted the AIA/CRL locations and basically the defaults where still in play, explained that default push in AD isn’t a recommended approach. If any client can’t access the CRL it will give a deny on further actions (and other clients that don’t understand AD or are joined to AD won’t work as well).

Below screenshots depict a default ADCS installation which in turn pushes out the default legacy templates and also the CRL to LDAP which I see much too often at customers:

Resolution for the CRL error was to revoke all the certificates for usage with FAS, change the CRL/AIA location to a routable and reachable HTTP listener instead of LDAP (preferably an HA setup with a load-balancer in front of it) and push out the new CRL. Afterwards logons where using the SSO capabilities.

Hope it helps!

Notes from the lab: Windows firewall profile not correct after reboot

Just thought of leaving a quick win here. Did you ever had the firewall profile of Windows not correctly mapped after reboots etc.?

This is because after a reboot the Domain Controllers put it in e.g. public profile and this will get passed on to other servers as well. This will effect in not being able to manage machines because of firewall blocks etc.

Solution is to restart the “Network Location Awareness” service and dependent “Network List Service”.
This will reset it to domain profile and after reboots of the other machines which have this it will be updated to domain profile as well. Or restart the service as above that will also do the trick.

Hope it helps!

UPDATE:

Well after migrating to Windows Server 2022 in my homelab the above will not work anymore. Appears the restarting of the NLA service has changed. After some digging around came across the following: Domain Controller thinks its on a Public Network

My working scenario is putting it a delayed start and adding domain services dependency on the NLA service with DNS and NTDS.

Notes from the lab: VMware Horizon and Microsoft MFA NPS Extension

In my own lab environment I have a mixture of EUC components and dual factor configured accordingly, but more and more I see that customers also just use the MFA solution of Microsoft to integrate it for their environments. Why not it’s included with your license right.

So back to the techie part I’ve configured my own NPS setup on a Windows Server 2019 and configured the RADIUS setup. Installed the MFA NPS extension and had a pre-existing configuration for my Citrix ADC appliance.

I’ve configured my Horizon connection server as an RADIUS client and enabled the configuration request and network policies for it as well, configuration type NAS IPv4 Address and the IP-address of the server.

Afterwards put in the configuration part in Horizon itself pointing the RADIUS authentication to the NPS server with all the necessary fields and/or additions that you want.

Well basically all should be working instantly when logging on to the Horizon URL or client.

I did however had some issues when logging in and stuff would time-out, event entries would say that the wrong dual factor request was given. This ultimately came from the fact I didn’t have a primary authentication set in MFA, I’ve checked that I could use my yubikey, SMS or push authentication. The resolution for this was to select primary push in the authenticator app and then it worked instantly.

Reference articles:
https://docs.microsoft.com/en-us/azure/active-directory/authentication/howto-mfa-nps-extension
https://christiaanbrinkhoff.com/2017/02/17/how-to-configure-azure-mfa-for-citrix-netscaler-gateway-radius-by-using-the-new-nps-extension/

(off topic there is an issue with 2019-NPS which I’ve encountered when configuring RADIUS-WIFI authentication, see the resolution here: https://community.ui.com/questions/FYI-Windows-Server-2019-NPS-for-RADIUS-broken-w-fix/364c7c17-b3d3-4973-8dd2-e4e701309300)

Notes from the field: The unexplained Outlook pop-up

Quite recently I’ve had an interesting troubleshoot at a customer. The problem was at first that there was an issue in the newly build Exchange 2019 environment that Outlook clients would open up and ask for credentials in a domain joined environment, so the SSO part of WIA isn’t working and it “seemed” to work after you would put in credentials.

Long story short support case at Microsoft was in play and after some weeks of log troubleshooting no results and a standstill for the customers migration project.

Drilldown to the issue at hand it seemed that the Exchange solution wasn’t working correctly and it was decided to uninstall the software and clean-up the entire setup and do a fresh redeploy. The engineer already repaired some ADDS inconsistencies which perhaps could be an issue but that was not the case.

At this time I got involved because the uninstall wasn’t working correctly on the last server, it kept preventing the uninstall with still some arbitrary mailboxes present and an old hardcoded OAB reference which couldn’t get removed. I’ve also had this kind of behavior with the arbitrary mailboxes on a Exchange 2016 solution and a reboot would resolve the blocking issues, the entities will get populated again and afterwards you could remove them accordingly. That was the solution here as well until the point of the OAB entry we had to delete that one through ADSIEDIT.

Ok.. all is well again and Exchange 2019 is completely removed and a redeploy was the next item. Exchange got redeployed, basic no fuzz and behavior of the client seemed to improve.. well no it came back and the outlook prompts and disconnects were reoccurring after some time again.

Next item was to strip stuff even further, we put on some scenario’s like lets do a complete redeploy of ADDS etc. which would mean a lot of work. So we tried to simulate it in separate lab environments to reproduce the issue and there we found out no issues at all, SSO was working out of the box and things started to point through customization of some sorts or other external factors.

We’ve landed on the point that perhaps the forest trust would be an issue, that was our last item to troubleshoot. There was a full forest trust in place for the migration/transition to the new network which is being used accordingly for resources etc. This was also something we implemented in both lab environments and never had any issues with. Well here comes the kicker… we’ve disabled the trust and the issues were instantly gone!

Ok. We have our little friend and are going to focus on that, it turned out that the name suffix routing enabled of the domain was giving issues, very strange because they are not overlapping of any sorts. It “seemed” all ok from a configuration point of perspective. We focused in on everything that could be an issue and ultimately came to the point that the NETBIOS name and FQDN of ADDS where the same with a dot in it e.g. domain.local for them both.. Right this isn’t something that you can setup anytime soon from Server 2019 through Windows 2000 and is basically not supported or advised to use, this was the case in Windows NT4 but further on no go at all.

Recap of the whole situation we have our workaround for the customer, but the root cause with NETBIOS and a dot in its setup isn’t going away. The migration is focused to move it all as soon as it can and then kill the trust and any connections to the old forest/domain which was giving all this grief.

I’ve had an splendid troubleshoot with the engineer who’s a wizard in ADDS, be sure to connect or follow him on LinkedIn.

Christiaan Ploeger