Notes from the lab: Using VMware Access as IdP for Citrix Gateway

I like to fiddle around with possibilities when it comes to SAML, OAUTH authentications. This all started when a customer engineer triggered me with the possibility of achieving an SSO experience with the Citrix NetScaler and using VMware Access as the source of truth for authentication.

Well guess what this works! And even for the native workspace app users as well. All with the conditional access policies of VMware Access before it.

First things first, I’m expecting that you know how to use the following and already have that in place:

  • Configured Citrix CVAD with Citrix Gateway doesn’t matter if it is Unified or not
  • Advanced authentication profile for SAML usage in Citrix Gateway
  • AAA setup to be used with the SAML / authentication profile
  • Working Citrix session policies set for web and native logins
  • VMware Access (Workspace ONE) on-premises or cloud variant

To start this sort of configuration, first you will need to download the VMware Access IdP SAML metadata or copy the URL of it and paste this in Citrix NetScaler:

The important part is that if you do not allow the NetScaler to have outgoing internet access it cannot validate the SAML metadata discovery, and everything will fail. In this case upload the IdP metadata as a certificate on the NetScaler and deselect the “Import Metadata” option the SAML Server action. Either way make sure that this is working for you.

Regarding the SAML Server action the “${user.userName}” value is the unique object in VMware Access which translates to either sAMAccountName or userPrincipalName according to your setup. With this value it will always work as expected with the logon.

When all the values are correct you can export the SAML metadata to be used in VMware Access as a new SaaS SAML 2.0 application. Paste the XML input in this app and assign the application to your test user/group. Make sure the application id and issuer name align between VMware Access and Citrix NetScaler.

To summarize above and a working setup:

  • Authentication Profile is mandatory otherwise you are not working with advanced authentication on Citrix Gateway
  • Citrix Gateway combines with a AAA for this feature
  • Make sure the URL’s match at both sides otherwise it will fail
  • Don’t forget the Relay State Rule in NetScaler for SAML security

Hope it helps! And if there are any questions give me a ping.

Notes from the field: Citrix Cloud Connector we’re having trouble signing you in

Recently I’ve encountered an issue after installing the Citrix Cloud Connector on new Windows Server 2022 machines. The configuration on my first machine went just fine until the sign-in, here my interest got peeked because I still have IE11 on my box, strange that it uses the sign-in for Citrix Cloud.

Well let’s test the second one and remove IE11 from the box and see what happens:

Oh boy, so we are depending on IE11. Installed it again and voila:

Hope it helps!

UPDATE:

After providing feedback at Citrix Cloud Updates | Cloud Connector installer support for the system default browser and getting the reply that there is an update out I’ve tested the new version and that one works as expected without IE11.

Notes from the lab: Citrix StoreFront 2203 and the cannot complete request

A quick blog regarding my Citrix lab upgrade from Citrix Virtual Apps and Dekstops (CVAD) 1912CU4 to 2203 and the little StoreFront snag I hit.

Summary of my setup:
Two Delivery controllers
Two StoreFront servers cohabitating with Director as well
Two FAS servers
Two WEM servers
One unmanaged VDA worker

And a Citrix ADC HA setup load-balancing the entire solution with keep in mind only TLS1.2 enabled for the services.

All components went flawless through the upgrade only after the maintenance window I could not see any desktops when logging in through StoreFront and got the dreaded “cannot complete your request” when logging in via the Citrix Gateway. At first no error entries that would stand out not even the errors shown in the support article which I’m mentioning in a bit.

The I remembered that there was a hotfix released quite fast after the original 2203 release of StoreFront regarding TLS hardening of the original CVAD setup. The following articles mention it in detail:
Gateway Callback and / or XML Communication fails after upgrade to Storefront 2203
Hotfix – Receiver StoreFront 3.23.1- For Citrix Virtual Apps and Desktops 2203 – English

After doing some checks regarding the XML, StoreFront and all the VIP’s this was definitely the case. The event logs would make clear that the XML could not be reached and when logging in through the gateway the callback URL would not be reachable. Updating the StoreFront servers with the above hotfix resolved everything.

Hope it helps!

Notes from the field: Citrix Files / ShareFile MDX SSO not working

At my latest Citrix Endpoint Management customer there were some issues regarding Citrix Files / ShareFile not achieving an SSO throughout the MDX/MAM enabled applications. Everything outside the MDX/MAM application bubble would work just fine only when tunnelling through the internal only application this would fail. The setup was comprising of a dual IDP setup with Microsoft ADFS and Citrix Endpoint Management itself.

First thing to note was the ACL regarding the customers CEM environment and allowed IP-addresses. Adding those would instantly resolve the messages seen in the debugging logs of IP-address not on the allow list.

Second thing to note when we saw the erroring out of the SSO again, did a sanity check on the ADC configurations and made everything conform the article XenMobile/ShareFile SAML SSO failing

Third thing to note and now comes the kicker after the logs showed a bearer token error when trying to perform the SSO and got the response 401 Unauthorised. The customer also mentioned that “other” SAML applications would also not work and give 401 errors. Ok, we got something here.

Long story short, we opened up a support case, reproduced the issue with an on-premises XenMobile environment as well and found out that when we had the original exploit and mitigation in place of article Secure Hub shows an error and fails to connect after upgrading to a fixed firmware build to address CVE-2020-8299/ CVE-2020-8300 this issue would occur. Ok but we need this in place for Secure Hub to function correctly!

The resolution to add is that for the bearer token there should not be an SSO in place, nothing should touch that one because it performs the SAML and SSO assertions for the applications. So basically this article Post NetScaler upgrade to 11.1, SSO to ShareFile is failing. describes the traffic rule for disabling the SSO on that part and afterwards, voila, presto, eureka! Working SSO for not only the Citrix Files / ShareFile MDX/MAM enabled application but also for all other SAML applications that were failing.

Hope it helps!

Notes from the lab: Citrix ShareFile and VMware Access SSO

When configuring Citrix ShareFile for an SSO experience with your Microsoft Active Directory setup we have the following guides to use it from Citrix. See How to Configure Single Sign-On (SSO) for ShareFile (citrix.com)

Well I’m having my setup with another Identity Provider in my own lab and still want to achieve an managed SSO setup from my end. To get this to work I checked the setup from an existing integration setup like Microsoft ADFS and reverse engineered it to VMware Access instead.

The following will give you an working SSO setup with VMware Access as your Identity Provider for Citrix ShareFile:

First configure the basic settings of Citrix ShareFile with your URL’s

https://tenant-fqdn-sharefile/saml

https://tenant-fqdn-vmware

Copy paste the certificate information from your VMware Access tenant

Create the application in VMware Access

Use the explicit logon URL of your application in VMware Access in Citrix ShareFile

Use the https://tenant-fqdn-sharefile/saml/acs in VMware Access for consumption

After this make sure that an user from your corporate identity is synchronized in Citrix ShareFile and both of them match so that SSO can be achieved. Afterwards you can nicely click the icon in VMware Access from IDP point of view:

Or you can login via the SP side from the Citrix ShareFile browser landing page:

Hope it helps!

Notes from the field: Citrix CEM / XenMobile enabling Certificate Based Authentication (CBA) after enrollment

I think any consultant at some time encountered the scenario of username / password authentication being the only authentication on the Citrix Gateway setup of Citrix CEM / XenMobile.

Afterwards advising the customer to use Certificate Based Authentication (CBA) and then also the sad news okay we need to reenroll all your devices for this to work.

But…. What if I told you there is a middle way for those customers that cannot afford a reenrollment of all their devices and enable the dual-factor situation after enrollment. (little bit of a side note that Citrix Support kind of / sort of well doesn’t support this regarding expected behavior etc. etc.)

You can easily build your test setup for this and stage everything until you will have the correct flow and actions to enable it.

Starting point:

Basic setup working with Citrix Gateway integration and username / password for authentication

Test devices enrolled and fully working

Needed:

Microsoft ADCS server with web-enrollment installed and configured for certificate requests handling

Note: Tiered setup will not work for issuing certificates so a dedicated root or subordinate will be needed with the ADCS Web Enrollment installed on it

Note: Only v2 templates are visible in ADCS Web Enrollment so do not upgrade the template and keep the default Certification Authority / Certificate recipient see the following articles for reference:

Version 3 (CNG) templates won’t appear in certificate web enrollment – Windows Server | Microsoft Docs

Windows Server 2012: Certificate Template Versions and Options – TechNet Articles – United States (English) – TechNet Wiki (microsoft.com)

Note: Validate a working web enrollment request before going any further

Note: And I cannot stress this enough make sure you have a valid working CRL/OCSP as HTTP URL location in ADCS CDP/AIA

Citrix CEM / XenMobile configured with a PKI Entity and used as a Credential Provider see the following configuration articles for reference:

Client certificate or certificate plus domain authentication (citrix.com)

Certificate Based Authentication : Troubleshooting Tips (citrix.com)

Note: Test delivery of a certificate by using an credential policy to your test devices and validate that the certificate is installed correctly

After a valid working test policy you can enable the Deliver user certificate for authentication at the Citrix Gateway integration part:

Now certificates for Citrix Gateway authentication will be generated and delivered after a period of time. You can speed this up a bit by forcing a deploy of the basic delivery group but normally you would need to wait for everyone to get a certificate. Schedule in some time with the customer for this. The delivery can be validated at the device under certificates, there it should give you a NetScaler Gateway Credentials entry:

So at this point Citrix CEM / XenMobile is ready and we need to configure Citrix Gateway. This step will need to be in conjunction with the Citrix Gateway change in Citrix CEM / XenMobile with the following:

Root/Intermediate certificate(s) linked and configured at the VPN Virtual Server with CRL mandatory or OCSP mandatory

LDAP and Cert Policy enabled as cascading primary authentication

Client authentication enabled and Client certificate mandatory

Note: Make sure to attend your certificate policy for UPN or sAMAccountName

Note: The CRL or OCSP mandatory is important because in the way Secure Hub requests certificates and that the certificate itself isn’t revocation aware in Citrix CEM / XenMobile. This way it will trigger a new certificate request and not present the cached older certificate present in Citrix CEM / XenMobile

Note: This change will effectively break access to the Citrix Gateway if you don’t have a valid certificate, so there is also an option to set the client certificate as optional in the migration phase or just do the hard cutover

Now we will change the authentication part in Citrix CEM / XenMobile:

After this change devices will be able to use Certificate based authentication to the VPN virtual server and devices that won’t have the certificate will either be presented with a store error message that will be resolved by either closing the app and reopening or logging off and logging on again in the store.

Note: In some cases there might be devices which do need a reenrollment to work, no point in sugar coating it this is a big change which normally is done at the start of a Citrix CEM / XenMobile deployment

I would say try it out in your lab environment, have done this multiple times and works pretty flawless. This might in turn help you with your customers.

Hope it helps!

Notes from the field: Another cannot complete your request with Citrix FAS

We’ve all seen it time and time again some misconfiguration with Citrix StoreFront and/or Citrix FAS and you’ll be getting the cannot complete your request message in your screen. Digging in the StoreFront logs and you’ll be seeing the most interesting messages of error kind in which you would think am I a rocket professor?

My story for this certain scenario would be a CVAD deployment integrated with FAS and everything working just fine with some minor bumps like adding your resources to the Windows Authorization Access Group and magic occurs things start to work. See Cannot Complete Your Request Error only occurs to certain users connecting from ADC with Azure MFA over to Storefront (citrix.com) for the fun of it and it’s buddy Common Resolutions to “Cannot Complete Your Request” Error when connecting directly to StoreFront Server (citrix.com)

Ok well this works! Happy customer, happy consultant. And after some time of testing the customer started to migrate existing users to this solution… And stuff didn’t work.. The same error as described in the article would occur and well not so happy customer and consultant now. Troubleshooted this and what the hell new users don’t have this problem.. only existing users! Euhm.. okay.. after some more discussion with the customer it was pointed out that this domain has been alive for a while like NT time while and upgraded to the latest and greatest Windows Server 2019.

This triggered me and after some searching came across Apps and APIs require access – Windows Server | Microsoft Docs which explained the truth! We are missing stuff. I did a compare of the groups “Pre-Windows 2000 Compatible Access” and “Windows Authorization Access Group” of the customer with my own and even a brand new test setup and there the following was missing:

Seems like upgrading a domain time and time again stuff won’t get added. After adding these objects all began to work and even the manual added resources don’t need to be in there like the CVAD servers, users object.

Hope it helps!

Notes from the field: VMware UAG and Citrix ADC scenario’s

On a recent project we were testing some scenario’s for the usage of VMware Blast BEAT through Citrix ADC. For some more information regarding Blast see the following article: VMware Blast Extreme Optimization Guide | VMware

Normally you would see that the Citrix ADC setup is an SSL-BRIDGE vserver with accompanying UDP vserver on the same IP for the ports 443 and 8443 which are the default for BLAST and BEAT usage.

There would be situations that port 8443 cannot be used regarding company compliance or restraints that only port 443 is allowed to be used. In this case you would need to change to an SSL-Offloading setup because we can’t intercept any traffic in a SSL-BRIDGE scenario or apply WebSocket usage in a profile.

We need to create an HTTP profile which enables WebSocket connections and we can use on the new SSL-Offloading vservers

Afterwards bind this profile to the SSL-Offloading vservers

Then we need to change the service group which we use in the UDP vserver to 8443, so the frontend is 443 and the backend is 8443

Lastly the changes on the UAG need to be reflected that the firewall connections would also be 443 for TCP/UDP usage and shared services. See the following articles for more information regarding firewall port requirements and alternate setups in the UAG: Firewall Rules for DMZ-Based Unified Access Gateway Appliances (vmware.com) and Blast TCP and UDP External URL Configuration Options (vmware.com)

With this setup you can service everything for the users on port 443. Do keep in mind that port sharing in this way is an excessive CPU load which can seriously impact the setup. See the following article for some clarification: Unified Access Gateway (UAG) high CPU utilization : HTTP 503 error (78419) (vmware.com)

Hope it helps!

Notes from the lab: Citrix ADC and VMware ESX 7u1/7u2

First things first. Citrix ADC at this time isn’t supporting VMware ESX 7.0.1 according to the following article: Support matrix and usage guidelines (citrix.com)

This is something that obviously will get supported in due time. But for the people who are running it just as I am in the lab you would see issues like the ADC instances would lose connectivity or will not load the appropriate network drivers at boot. This is because of the VMXNET3 interface which is causing issues.

Temporary workarounds include just reboot the dam thing until it works… or flip the network interface to E1000 and you have a solid booting/working appliance, well not entirely because the GUI seems to slow down over time to almost not working. I decided to flip it back to VMXNET3 and just do the reboot loop until it works.

Keep in mind that you need to copy/paste your MAC-address before removing the interface and readding them to keep your license etc. nice and working.

Hope it helps!

UPDATE:
ESX 7.0 update 1 build 16850804 is supported, anything above is not at this time. I’ve updated my environment to ESX 7.0 update 2 and the same issues are still present.

Notes from the field: Citrix StoreFront forcing connections through Citrix Gateway

On a recent customer project there was the need to migrate off of VDA TLS encryption and migrate the connections from StoreFront to Citrix Gateway.

The customer previously had StoreFront direct connections and used the VDA TLS encryption setup to provide a TLS encrypted session to the desktop or applications.

The VDA TLS encryption setup was too much engineering labor for the day 2 day operations and therefore they asked for a alternate solution but still provide the client>desktop as an TLS encrypted session.

Here we have two options, the first is to use Citrix Gateway and StoreFront as authentication but this introduces the users with a new logon screen and then delegates the credentials with json to StoreFront.

The second is forcing the connections from StoreFront through the Citrix Gateway by the means of optimal gateway routing, and we don’t have any user experience changes because the logon point is still StoreFront.

Option two was chosen and after a quick and simple deployment a seamless migration with optimal gateway routing is in place.

First all the preparation in place is creating a new Citrix Gateway DNS record and a new StoreFront load-balancer IP into the current setup will be migrated, afterwards configure the CVAD wizard on the NetScaler for a simple Citrix Gateway deployment and unbind any authentication policies because these will not be used. Afterwards configure all the necessary settings for a standard Citrix Gateway deployment and propagate these changes across the cluster. When all this is done edit the web.config file of the store that got configured under the primary StoreFront servers IIS inetpub directory and search for: optimalGatewayForFarmsCollection and make sure there is an entry with optimalGatewayForFarms enabledOnDirectAccess=”true” and save the file. Propagate the changes and after that migrate the old DNS entry to the new StoreFront ip. You will see that after logon the desktop brokering is force through Citrix Gateway.

The following reference articles where used for configuration and testing:

How to Force Connections Through NetScaler Gateway Using Optimal Gateway Feature of StoreFront (citrix.com)

How to Configure Authentication at StoreFront using NetScaler Gateway – NetScaler Configuration (citrix.com)

FAQ: Configuring Authentication at StoreFront using NetScaler Gateway (citrix.com)

SSL configuration on VDA (citrix.com)