Notes from the field: Citrix XenMobile / CEM don’t touch that store name!

Just a quick shout out blog to stress the importance of the store name that XenMobile / CEM uses. This default store name is called “Store”.

If you by any means have changed this store name to anything else, you might run in two issues depending on different scenarios.

Scenario 1:
Citrix XenMobile or CEM and switching from MDX technology to MAM SDK policies causes different behavior regarding the MAM SDK enabled applications and the web sso policy won’t work or connections won’t be active.

Scenario 2:
Citrix CEM and enabling Citrix Gateway integration on-premises which causes the store to lock, and devices won’t be able to properly connect anymore and need a device wipe.

Hope it helps for the still existing environments still out there, and don’t forget and look at the Citrix Product Matrix regarding the valid lifetime of XenMobile and CEM.

A special thanks to Anton van Pelt in helping in with product management contact.

Notes from the lab: Citrix StoreFront 2203 and the cannot complete request

A quick blog regarding my Citrix lab upgrade from Citrix Virtual Apps and Dekstops (CVAD) 1912CU4 to 2203 and the little StoreFront snag I hit.

Summary of my setup:
Two Delivery controllers
Two StoreFront servers cohabitating with Director as well
Two FAS servers
Two WEM servers
One unmanaged VDA worker

And a Citrix ADC HA setup load-balancing the entire solution with keep in mind only TLS1.2 enabled for the services.

All components went flawless through the upgrade only after the maintenance window I could not see any desktops when logging in through StoreFront and got the dreaded “cannot complete your request” when logging in via the Citrix Gateway. At first no error entries that would stand out not even the errors shown in the support article which I’m mentioning in a bit.

The I remembered that there was a hotfix released quite fast after the original 2203 release of StoreFront regarding TLS hardening of the original CVAD setup. The following articles mention it in detail:
Gateway Callback and / or XML Communication fails after upgrade to Storefront 2203
Hotfix – Receiver StoreFront 3.23.1- For Citrix Virtual Apps and Desktops 2203 – English

After doing some checks regarding the XML, StoreFront and all the VIP’s this was definitely the case. The event logs would make clear that the XML could not be reached and when logging in through the gateway the callback URL would not be reachable. Updating the StoreFront servers with the above hotfix resolved everything.

Hope it helps!

Notes from the field: Citrix Files / ShareFile MDX SSO not working

At my latest Citrix Endpoint Management customer there were some issues regarding Citrix Files / ShareFile not achieving an SSO throughout the MDX/MAM enabled applications. Everything outside the MDX/MAM application bubble would work just fine only when tunnelling through the internal only application this would fail. The setup was comprising of a dual IDP setup with Microsoft ADFS and Citrix Endpoint Management itself.

First thing to note was the ACL regarding the customers CEM environment and allowed IP-addresses. Adding those would instantly resolve the messages seen in the debugging logs of IP-address not on the allow list.

Second thing to note when we saw the erroring out of the SSO again, did a sanity check on the ADC configurations and made everything conform the article XenMobile/ShareFile SAML SSO failing

Third thing to note and now comes the kicker after the logs showed a bearer token error when trying to perform the SSO and got the response 401 Unauthorised. The customer also mentioned that “other” SAML applications would also not work and give 401 errors. Ok, we got something here.

Long story short, we opened up a support case, reproduced the issue with an on-premises XenMobile environment as well and found out that when we had the original exploit and mitigation in place of article Secure Hub shows an error and fails to connect after upgrading to a fixed firmware build to address CVE-2020-8299/ CVE-2020-8300 this issue would occur. Ok but we need this in place for Secure Hub to function correctly!

The resolution to add is that for the bearer token there should not be an SSO in place, nothing should touch that one because it performs the SAML and SSO assertions for the applications. So basically this article Post NetScaler upgrade to 11.1, SSO to ShareFile is failing. describes the traffic rule for disabling the SSO on that part and afterwards, voila, presto, eureka! Working SSO for not only the Citrix Files / ShareFile MDX/MAM enabled application but also for all other SAML applications that were failing.

Hope it helps!

Notes from the field: Citrix CEM / XenMobile enabling Certificate Based Authentication (CBA) after enrollment

I think any consultant at some time encountered the scenario of username / password authentication being the only authentication on the Citrix Gateway setup of Citrix CEM / XenMobile.

Afterwards advising the customer to use Certificate Based Authentication (CBA) and then also the sad news okay we need to reenroll all your devices for this to work.

But…. What if I told you there is a middle way for those customers that cannot afford a reenrollment of all their devices and enable the dual-factor situation after enrollment. (little bit of a side note that Citrix Support kind of / sort of well doesn’t support this regarding expected behavior etc. etc.)

You can easily build your test setup for this and stage everything until you will have the correct flow and actions to enable it.

Starting point:

Basic setup working with Citrix Gateway integration and username / password for authentication

Test devices enrolled and fully working

Needed:

Microsoft ADCS server with web-enrollment installed and configured for certificate requests handling

Note: Tiered setup will not work for issuing certificates so a dedicated root or subordinate will be needed with the ADCS Web Enrollment installed on it

Note: Only v2 templates are visible in ADCS Web Enrollment so do not upgrade the template and keep the default Certification Authority / Certificate recipient see the following articles for reference:

Version 3 (CNG) templates won’t appear in certificate web enrollment – Windows Server | Microsoft Docs

Windows Server 2012: Certificate Template Versions and Options – TechNet Articles – United States (English) – TechNet Wiki (microsoft.com)

Note: Validate a working web enrollment request before going any further

Note: And I cannot stress this enough make sure you have a valid working CRL/OCSP as HTTP URL location in ADCS CDP/AIA

Citrix CEM / XenMobile configured with a PKI Entity and used as a Credential Provider see the following configuration articles for reference:

Client certificate or certificate plus domain authentication (citrix.com)

Certificate Based Authentication : Troubleshooting Tips (citrix.com)

Note: Test delivery of a certificate by using an credential policy to your test devices and validate that the certificate is installed correctly

After a valid working test policy you can enable the Deliver user certificate for authentication at the Citrix Gateway integration part:

Now certificates for Citrix Gateway authentication will be generated and delivered after a period of time. You can speed this up a bit by forcing a deploy of the basic delivery group but normally you would need to wait for everyone to get a certificate. Schedule in some time with the customer for this. The delivery can be validated at the device under certificates, there it should give you a NetScaler Gateway Credentials entry:

So at this point Citrix CEM / XenMobile is ready and we need to configure Citrix Gateway. This step will need to be in conjunction with the Citrix Gateway change in Citrix CEM / XenMobile with the following:

Root/Intermediate certificate(s) linked and configured at the VPN Virtual Server with CRL mandatory or OCSP mandatory

LDAP and Cert Policy enabled as cascading primary authentication

Client authentication enabled and Client certificate mandatory

Note: Make sure to attend your certificate policy for UPN or sAMAccountName

Note: The CRL or OCSP mandatory is important because in the way Secure Hub requests certificates and that the certificate itself isn’t revocation aware in Citrix CEM / XenMobile. This way it will trigger a new certificate request and not present the cached older certificate present in Citrix CEM / XenMobile

Note: This change will effectively break access to the Citrix Gateway if you don’t have a valid certificate, so there is also an option to set the client certificate as optional in the migration phase or just do the hard cutover

Now we will change the authentication part in Citrix CEM / XenMobile:

After this change devices will be able to use Certificate based authentication to the VPN virtual server and devices that won’t have the certificate will either be presented with a store error message that will be resolved by either closing the app and reopening or logging off and logging on again in the store.

Note: In some cases there might be devices which do need a reenrollment to work, no point in sugar coating it this is a big change which normally is done at the start of a Citrix CEM / XenMobile deployment

I would say try it out in your lab environment, have done this multiple times and works pretty flawless. This might in turn help you with your customers.

Hope it helps!

Notes from the field: VMware UAG and Citrix ADC scenario’s

On a recent project we were testing some scenario’s for the usage of VMware Blast BEAT through Citrix ADC. For some more information regarding Blast see the following article: VMware Blast Extreme Optimization Guide | VMware

Normally you would see that the Citrix ADC setup is an SSL-BRIDGE vserver with accompanying UDP vserver on the same IP for the ports 443 and 8443 which are the default for BLAST and BEAT usage.

There would be situations that port 8443 cannot be used regarding company compliance or restraints that only port 443 is allowed to be used. In this case you would need to change to an SSL-Offloading setup because we can’t intercept any traffic in a SSL-BRIDGE scenario or apply WebSocket usage in a profile.

We need to create an HTTP profile which enables WebSocket connections and we can use on the new SSL-Offloading vservers

Afterwards bind this profile to the SSL-Offloading vservers

Then we need to change the service group which we use in the UDP vserver to 8443, so the frontend is 443 and the backend is 8443

Lastly the changes on the UAG need to be reflected that the firewall connections would also be 443 for TCP/UDP usage and shared services. See the following articles for more information regarding firewall port requirements and alternate setups in the UAG: Firewall Rules for DMZ-Based Unified Access Gateway Appliances (vmware.com) and Blast TCP and UDP External URL Configuration Options (vmware.com)

With this setup you can service everything for the users on port 443. Do keep in mind that port sharing in this way is an excessive CPU load which can seriously impact the setup. See the following article for some clarification: Unified Access Gateway (UAG) high CPU utilization : HTTP 503 error (78419) (vmware.com)

Hope it helps!

Notes from the lab: Citrix ADC and VMware ESX 7u1/7u2

First things first. Citrix ADC at this time isn’t supporting VMware ESX 7.0.1 according to the following article: Support matrix and usage guidelines (citrix.com)

This is something that obviously will get supported in due time. But for the people who are running it just as I am in the lab you would see issues like the ADC instances would lose connectivity or will not load the appropriate network drivers at boot. This is because of the VMXNET3 interface which is causing issues.

Temporary workarounds include just reboot the dam thing until it works… or flip the network interface to E1000 and you have a solid booting/working appliance, well not entirely because the GUI seems to slow down over time to almost not working. I decided to flip it back to VMXNET3 and just do the reboot loop until it works.

Keep in mind that you need to copy/paste your MAC-address before removing the interface and readding them to keep your license etc. nice and working.

Hope it helps!

UPDATE:
ESX 7.0 update 1 build 16850804 is supported, anything above is not at this time. I’ve updated my environment to ESX 7.0 update 2 and the same issues are still present.

Notes from the field: Citrix StoreFront forcing connections through Citrix Gateway

On a recent customer project there was the need to migrate off of VDA TLS encryption and migrate the connections from StoreFront to Citrix Gateway.

The customer previously had StoreFront direct connections and used the VDA TLS encryption setup to provide a TLS encrypted session to the desktop or applications.

The VDA TLS encryption setup was too much engineering labor for the day 2 day operations and therefore they asked for a alternate solution but still provide the client>desktop as an TLS encrypted session.

Here we have two options, the first is to use Citrix Gateway and StoreFront as authentication but this introduces the users with a new logon screen and then delegates the credentials with json to StoreFront.

The second is forcing the connections from StoreFront through the Citrix Gateway by the means of optimal gateway routing, and we don’t have any user experience changes because the logon point is still StoreFront.

Option two was chosen and after a quick and simple deployment a seamless migration with optimal gateway routing is in place.

First all the preparation in place is creating a new Citrix Gateway DNS record and a new StoreFront load-balancer IP into the current setup will be migrated, afterwards configure the CVAD wizard on the NetScaler for a simple Citrix Gateway deployment and unbind any authentication policies because these will not be used. Afterwards configure all the necessary settings for a standard Citrix Gateway deployment and propagate these changes across the cluster. When all this is done edit the web.config file of the store that got configured under the primary StoreFront servers IIS inetpub directory and search for: optimalGatewayForFarmsCollection and make sure there is an entry with optimalGatewayForFarms enabledOnDirectAccess=”true” and save the file. Propagate the changes and after that migrate the old DNS entry to the new StoreFront ip. You will see that after logon the desktop brokering is force through Citrix Gateway.

The following reference articles where used for configuration and testing:

How to Force Connections Through NetScaler Gateway Using Optimal Gateway Feature of StoreFront (citrix.com)

How to Configure Authentication at StoreFront using NetScaler Gateway – NetScaler Configuration (citrix.com)

FAQ: Configuring Authentication at StoreFront using NetScaler Gateway (citrix.com)

SSL configuration on VDA (citrix.com)

Notes from the lab: Citrix ADC IP Reputation

I’ve been playing around with the Citrix ADC IP Reputation feature – https://docs.citrix.com/en-us/citrix-adc/13/reputation/ip-reputation.html in the lab for some time and to be honest it’s such a small but very effective feature which I almost never see active, why is that?

If you’ve gotten a premium licensed ADC appliance it’s a simple right click>enable and you put in the necessary arguments in a responder policy. See the following article for a quick how to video – https://www.youtube.com/watch?v=WedxwiEVuG4 and basically that is it. The requests are going to be filtered on a Webroot service provider for malicious IP database and you can then drop those from ever getting at you network. (and put in a nifty log action so that you can filter as a syslog entry in Citrix ADM

I put a global responder in place with the expression: CLIENT.IP.SRC.IPREP_IS_MALICIOUS and a reset with accompanied log entry: CLIENT.IP.SRC  + ” connection was dropped by Responder Action for malicious IP when accessing ” + HTTP.REQ.URL the results were pretty much mind blowing, see the following screenshot:Since the exploit CVE-2019-19781 – Vulnerability in Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP appliance – https://support.citrix.com/article/CTX267027 the focus was pretty much around that but with the above rule in place I got more hits on the reputation feature than on the mitigation responder in that time. (And these are probes/attacks we don’t know about!)

So to sum it up this is a very good starting point if something like Application Firewall is a bridge to far but most definitely will improve your security with a simple setup.

Hope it helps.

 

 

Notes from the lab: Citrix ADC Native Push OTP not working

I’ve updated my lab environment with Citrix Gateway push OTP support and had some trouble in configuring the Citrix SSO app on my iPhone. For some reason it couldn’t setup the gateway connection and it wasn’t reachable. (Well that was my bad in checking all my devices but I’ll get to that)

Before the push OTP change I’ve worked with the authenticator app behavior and put in the code myself and this worked fine. The change to push OTP isn’t too difficult, and the following articles give you plenty of how-to information:
https://docs.citrix.com/en-us/citrix-gateway/13/push-notification-otp.html
https://docs.citrix.com/en-us/citrix-adc/13/aaa-tm/native-otp-authentication/otp-encryption-tool.html
https://www.carlstalhood.com/native-one-time-passwords-otp-citrix-gateway-13/

Keep in mind that if you have setup a previous OTP setup like I had the encryption part needs to be migrated otherwise it just won’t work. Registration cannot complete if you just flip it on and be done with it. Follow the migration part or just start fresh with OTP encryption enabled by default.

After I corrected the encryption problem I still could not register through the Citrix SSO app for push OTP functionality. It kept giving me the error gateway not reachable, fully checked everything in my environment was up and working. It kept me puzzling that it works fine with the previous authenticator method and if I re-registered it that way it would work (without the push of course because that functionality sits in the Citrix SSO app)

Finally after more troubleshooting I found the problem… because I’ve upgraded and integrated ADFS 2019 in my environment my content switching server and gateway etc. also needed to be SNI aware. Remember that everything worked fine on my Windows devices (even the Citrix SSO VPN functionality which I use quite often) but just not on my iPhone. Turning off SNI was the solution, it seems that the Citrix SSO app on iOS doesn’t support SNI.

Hope it helps!

Notes from the field: Cannot access Citrix ADC or create HA set

Quite recently I was at a customer where they had an SDX setup with single instances and needed to be upgraded and converted to an HA setup.

Well easy does it I created the instances on the second SDX and started creating HA sets. Numerous went fine and then one started giving errors. Could not propagate from the primary and after checking SSH/SCP access this would fail as well. I logged in through the console of SDX/SVM and saw that the sshd daemon wasn’t starting anymore. (On a side note all of the original SDX instances were upgraded in regard to the exploit of last December)

After some troubleshooting I came across the following discussion article: https://discussions.citrix.com/topic/405628-unable-to-connect-to-adc-nsip-version-121-and-130-using-sshsftp

The discussion referred to an support article regarding false positives and an SSH vulnerability:
https://support.citrix.com/article/CTX209398

After checking the sshd_config file and commenting out the following:
#option UsePrivilegeSeparation
#MACs hmac-sha1,hmac-ripemd160

The sshd daemon started again and the HA propagation and synchronisation started instantly. I’ve had this on several other instances as well and they all needed the above commenting out of the lines.

Hope it helps!