Notes from the lab: Citrix StoreFront 2203 and the cannot complete request

A quick blog regarding my Citrix lab upgrade from Citrix Virtual Apps and Dekstops (CVAD) 1912CU4 to 2203 and the little StoreFront snag I hit.

Summary of my setup:
Two Delivery controllers
Two StoreFront servers cohabitating with Director as well
Two FAS servers
Two WEM servers
One unmanaged VDA worker

And a Citrix ADC HA setup load-balancing the entire solution with keep in mind only TLS1.2 enabled for the services.

All components went flawless through the upgrade only after the maintenance window I could not see any desktops when logging in through StoreFront and got the dreaded “cannot complete your request” when logging in via the Citrix Gateway. At first no error entries that would stand out not even the errors shown in the support article which I’m mentioning in a bit.

The I remembered that there was a hotfix released quite fast after the original 2203 release of StoreFront regarding TLS hardening of the original CVAD setup. The following articles mention it in detail:
Gateway Callback and / or XML Communication fails after upgrade to Storefront 2203
Hotfix – Receiver StoreFront 3.23.1- For Citrix Virtual Apps and Desktops 2203 – English

After doing some checks regarding the XML, StoreFront and all the VIP’s this was definitely the case. The event logs would make clear that the XML could not be reached and when logging in through the gateway the callback URL would not be reachable. Updating the StoreFront servers with the above hotfix resolved everything.

Hope it helps!

Notes from the field: Citrix CEM / XenMobile enabling Certificate Based Authentication (CBA) after enrollment

I think any consultant at some time encountered the scenario of username / password authentication being the only authentication on the Citrix Gateway setup of Citrix CEM / XenMobile.

Afterwards advising the customer to use Certificate Based Authentication (CBA) and then also the sad news okay we need to reenroll all your devices for this to work.

But…. What if I told you there is a middle way for those customers that cannot afford a reenrollment of all their devices and enable the dual-factor situation after enrollment. (little bit of a side note that Citrix Support kind of / sort of well doesn’t support this regarding expected behavior etc. etc.)

You can easily build your test setup for this and stage everything until you will have the correct flow and actions to enable it.

Starting point:

Basic setup working with Citrix Gateway integration and username / password for authentication

Test devices enrolled and fully working

Needed:

Microsoft ADCS server with web-enrollment installed and configured for certificate requests handling

Note: Tiered setup will not work for issuing certificates so a dedicated root or subordinate will be needed with the ADCS Web Enrollment installed on it

Note: Only v2 templates are visible in ADCS Web Enrollment so do not upgrade the template and keep the default Certification Authority / Certificate recipient see the following articles for reference:

Version 3 (CNG) templates won’t appear in certificate web enrollment – Windows Server | Microsoft Docs

Windows Server 2012: Certificate Template Versions and Options – TechNet Articles – United States (English) – TechNet Wiki (microsoft.com)

Note: Validate a working web enrollment request before going any further

Note: And I cannot stress this enough make sure you have a valid working CRL/OCSP as HTTP URL location in ADCS CDP/AIA

Citrix CEM / XenMobile configured with a PKI Entity and used as a Credential Provider see the following configuration articles for reference:

Client certificate or certificate plus domain authentication (citrix.com)

Certificate Based Authentication : Troubleshooting Tips (citrix.com)

Note: Test delivery of a certificate by using an credential policy to your test devices and validate that the certificate is installed correctly

After a valid working test policy you can enable the Deliver user certificate for authentication at the Citrix Gateway integration part:

Now certificates for Citrix Gateway authentication will be generated and delivered after a period of time. You can speed this up a bit by forcing a deploy of the basic delivery group but normally you would need to wait for everyone to get a certificate. Schedule in some time with the customer for this. The delivery can be validated at the device under certificates, there it should give you a NetScaler Gateway Credentials entry:

So at this point Citrix CEM / XenMobile is ready and we need to configure Citrix Gateway. This step will need to be in conjunction with the Citrix Gateway change in Citrix CEM / XenMobile with the following:

Root/Intermediate certificate(s) linked and configured at the VPN Virtual Server with CRL mandatory or OCSP mandatory

LDAP and Cert Policy enabled as cascading primary authentication

Client authentication enabled and Client certificate mandatory

Note: Make sure to attend your certificate policy for UPN or sAMAccountName

Note: The CRL or OCSP mandatory is important because in the way Secure Hub requests certificates and that the certificate itself isn’t revocation aware in Citrix CEM / XenMobile. This way it will trigger a new certificate request and not present the cached older certificate present in Citrix CEM / XenMobile

Note: This change will effectively break access to the Citrix Gateway if you don’t have a valid certificate, so there is also an option to set the client certificate as optional in the migration phase or just do the hard cutover

Now we will change the authentication part in Citrix CEM / XenMobile:

After this change devices will be able to use Certificate based authentication to the VPN virtual server and devices that won’t have the certificate will either be presented with a store error message that will be resolved by either closing the app and reopening or logging off and logging on again in the store.

Note: In some cases there might be devices which do need a reenrollment to work, no point in sugar coating it this is a big change which normally is done at the start of a Citrix CEM / XenMobile deployment

I would say try it out in your lab environment, have done this multiple times and works pretty flawless. This might in turn help you with your customers.

Hope it helps!

Notes from the field: Citrix StoreFront forcing connections through Citrix Gateway

On a recent customer project there was the need to migrate off of VDA TLS encryption and migrate the connections from StoreFront to Citrix Gateway.

The customer previously had StoreFront direct connections and used the VDA TLS encryption setup to provide a TLS encrypted session to the desktop or applications.

The VDA TLS encryption setup was too much engineering labor for the day 2 day operations and therefore they asked for a alternate solution but still provide the client>desktop as an TLS encrypted session.

Here we have two options, the first is to use Citrix Gateway and StoreFront as authentication but this introduces the users with a new logon screen and then delegates the credentials with json to StoreFront.

The second is forcing the connections from StoreFront through the Citrix Gateway by the means of optimal gateway routing, and we don’t have any user experience changes because the logon point is still StoreFront.

Option two was chosen and after a quick and simple deployment a seamless migration with optimal gateway routing is in place.

First all the preparation in place is creating a new Citrix Gateway DNS record and a new StoreFront load-balancer IP into the current setup will be migrated, afterwards configure the CVAD wizard on the NetScaler for a simple Citrix Gateway deployment and unbind any authentication policies because these will not be used. Afterwards configure all the necessary settings for a standard Citrix Gateway deployment and propagate these changes across the cluster. When all this is done edit the web.config file of the store that got configured under the primary StoreFront servers IIS inetpub directory and search for: optimalGatewayForFarmsCollection and make sure there is an entry with optimalGatewayForFarms enabledOnDirectAccess=”true” and save the file. Propagate the changes and after that migrate the old DNS entry to the new StoreFront ip. You will see that after logon the desktop brokering is force through Citrix Gateway.

The following reference articles where used for configuration and testing:

How to Force Connections Through NetScaler Gateway Using Optimal Gateway Feature of StoreFront (citrix.com)

How to Configure Authentication at StoreFront using NetScaler Gateway – NetScaler Configuration (citrix.com)

FAQ: Configuring Authentication at StoreFront using NetScaler Gateway (citrix.com)

SSL configuration on VDA (citrix.com)

Notes from the lab: Citrix ADC Native Push OTP not working

I’ve updated my lab environment with Citrix Gateway push OTP support and had some trouble in configuring the Citrix SSO app on my iPhone. For some reason it couldn’t setup the gateway connection and it wasn’t reachable. (Well that was my bad in checking all my devices but I’ll get to that)

Before the push OTP change I’ve worked with the authenticator app behavior and put in the code myself and this worked fine. The change to push OTP isn’t too difficult, and the following articles give you plenty of how-to information:
https://docs.citrix.com/en-us/citrix-gateway/13/push-notification-otp.html
https://docs.citrix.com/en-us/citrix-adc/13/aaa-tm/native-otp-authentication/otp-encryption-tool.html
https://www.carlstalhood.com/native-one-time-passwords-otp-citrix-gateway-13/

Keep in mind that if you have setup a previous OTP setup like I had the encryption part needs to be migrated otherwise it just won’t work. Registration cannot complete if you just flip it on and be done with it. Follow the migration part or just start fresh with OTP encryption enabled by default.

After I corrected the encryption problem I still could not register through the Citrix SSO app for push OTP functionality. It kept giving me the error gateway not reachable, fully checked everything in my environment was up and working. It kept me puzzling that it works fine with the previous authenticator method and if I re-registered it that way it would work (without the push of course because that functionality sits in the Citrix SSO app)

Finally after more troubleshooting I found the problem… because I’ve upgraded and integrated ADFS 2019 in my environment my content switching server and gateway etc. also needed to be SNI aware. Remember that everything worked fine on my Windows devices (even the Citrix SSO VPN functionality which I use quite often) but just not on my iPhone. Turning off SNI was the solution, it seems that the Citrix SSO app on iOS doesn’t support SNI.

Hope it helps!

Notes from the field: Citrix ADC Gateway Native OTP with GSLB

Fun quick fact that I’ve encountered when deploying a ADC Gateway GSLB setup for a customer! You only have to enroll once with the nFactor/Native OTP on one of the ADC’s. (when having a Active Directory Domain across multiple datacenter sites)

The setup of choice:

  • Two ADC appliances in HA set on each site
  • GSLB enabled in active/passive mode for the Gateway across both sites
  • Native OTP enabled and active as the way for authentication
  • Active Directory Domain across two sites

There is no difference in configuration whatsoever because the magic of Native OTP depends on Active Directory.

Configure each ADC identically with the nFactor/Native OTP setup and enable GSLB and you’re done. I must admit at first I thought that I would need to enroll at both gateways independent but happily this is not the case.

For the configuration steps see common examples as below:

https://docs.citrix.com/en-us/netscaler-gateway/12-1/native-otp-support.html

https://www.carlstalhood.com/netscaler-gateway-12-native-one-time-passwords-otp/