Notes from the field: Citrix XenMobile / CEM don’t touch that store name!

Just a quick shout out blog to stress the importance of the store name that XenMobile / CEM uses. This default store name is called “Store”.

If you by any means have changed this store name to anything else, you might run in two issues depending on different scenarios.

Scenario 1:
Citrix XenMobile or CEM and switching from MDX technology to MAM SDK policies causes different behavior regarding the MAM SDK enabled applications and the web sso policy won’t work or connections won’t be active.

Scenario 2:
Citrix CEM and enabling Citrix Gateway integration on-premises which causes the store to lock, and devices won’t be able to properly connect anymore and need a device wipe.

Hope it helps for the still existing environments still out there, and don’t forget and look at the Citrix Product Matrix regarding the valid lifetime of XenMobile and CEM.

A special thanks to Anton van Pelt in helping in with product management contact.

Notes from the field: Citrix Files / ShareFile MDX SSO not working

At my latest Citrix Endpoint Management customer there were some issues regarding Citrix Files / ShareFile not achieving an SSO throughout the MDX/MAM enabled applications. Everything outside the MDX/MAM application bubble would work just fine only when tunnelling through the internal only application this would fail. The setup was comprising of a dual IDP setup with Microsoft ADFS and Citrix Endpoint Management itself.

First thing to note was the ACL regarding the customers CEM environment and allowed IP-addresses. Adding those would instantly resolve the messages seen in the debugging logs of IP-address not on the allow list.

Second thing to note when we saw the erroring out of the SSO again, did a sanity check on the ADC configurations and made everything conform the article XenMobile/ShareFile SAML SSO failing

Third thing to note and now comes the kicker after the logs showed a bearer token error when trying to perform the SSO and got the response 401 Unauthorised. The customer also mentioned that “other” SAML applications would also not work and give 401 errors. Ok, we got something here.

Long story short, we opened up a support case, reproduced the issue with an on-premises XenMobile environment as well and found out that when we had the original exploit and mitigation in place of article Secure Hub shows an error and fails to connect after upgrading to a fixed firmware build to address CVE-2020-8299/ CVE-2020-8300 this issue would occur. Ok but we need this in place for Secure Hub to function correctly!

The resolution to add is that for the bearer token there should not be an SSO in place, nothing should touch that one because it performs the SAML and SSO assertions for the applications. So basically this article Post NetScaler upgrade to 11.1, SSO to ShareFile is failing. describes the traffic rule for disabling the SSO on that part and afterwards, voila, presto, eureka! Working SSO for not only the Citrix Files / ShareFile MDX/MAM enabled application but also for all other SAML applications that were failing.

Hope it helps!

Notes from the field: Citrix CEM / XenMobile enabling Certificate Based Authentication (CBA) after enrollment

I think any consultant at some time encountered the scenario of username / password authentication being the only authentication on the Citrix Gateway setup of Citrix CEM / XenMobile.

Afterwards advising the customer to use Certificate Based Authentication (CBA) and then also the sad news okay we need to reenroll all your devices for this to work.

But…. What if I told you there is a middle way for those customers that cannot afford a reenrollment of all their devices and enable the dual-factor situation after enrollment. (little bit of a side note that Citrix Support kind of / sort of well doesn’t support this regarding expected behavior etc. etc.)

You can easily build your test setup for this and stage everything until you will have the correct flow and actions to enable it.

Starting point:

Basic setup working with Citrix Gateway integration and username / password for authentication

Test devices enrolled and fully working

Needed:

Microsoft ADCS server with web-enrollment installed and configured for certificate requests handling

Note: Tiered setup will not work for issuing certificates so a dedicated root or subordinate will be needed with the ADCS Web Enrollment installed on it

Note: Only v2 templates are visible in ADCS Web Enrollment so do not upgrade the template and keep the default Certification Authority / Certificate recipient see the following articles for reference:

Version 3 (CNG) templates won’t appear in certificate web enrollment – Windows Server | Microsoft Docs

Windows Server 2012: Certificate Template Versions and Options – TechNet Articles – United States (English) – TechNet Wiki (microsoft.com)

Note: Validate a working web enrollment request before going any further

Note: And I cannot stress this enough make sure you have a valid working CRL/OCSP as HTTP URL location in ADCS CDP/AIA

Citrix CEM / XenMobile configured with a PKI Entity and used as a Credential Provider see the following configuration articles for reference:

Client certificate or certificate plus domain authentication (citrix.com)

Certificate Based Authentication : Troubleshooting Tips (citrix.com)

Note: Test delivery of a certificate by using an credential policy to your test devices and validate that the certificate is installed correctly

After a valid working test policy you can enable the Deliver user certificate for authentication at the Citrix Gateway integration part:

Now certificates for Citrix Gateway authentication will be generated and delivered after a period of time. You can speed this up a bit by forcing a deploy of the basic delivery group but normally you would need to wait for everyone to get a certificate. Schedule in some time with the customer for this. The delivery can be validated at the device under certificates, there it should give you a NetScaler Gateway Credentials entry:

So at this point Citrix CEM / XenMobile is ready and we need to configure Citrix Gateway. This step will need to be in conjunction with the Citrix Gateway change in Citrix CEM / XenMobile with the following:

Root/Intermediate certificate(s) linked and configured at the VPN Virtual Server with CRL mandatory or OCSP mandatory

LDAP and Cert Policy enabled as cascading primary authentication

Client authentication enabled and Client certificate mandatory

Note: Make sure to attend your certificate policy for UPN or sAMAccountName

Note: The CRL or OCSP mandatory is important because in the way Secure Hub requests certificates and that the certificate itself isn’t revocation aware in Citrix CEM / XenMobile. This way it will trigger a new certificate request and not present the cached older certificate present in Citrix CEM / XenMobile

Note: This change will effectively break access to the Citrix Gateway if you don’t have a valid certificate, so there is also an option to set the client certificate as optional in the migration phase or just do the hard cutover

Now we will change the authentication part in Citrix CEM / XenMobile:

After this change devices will be able to use Certificate based authentication to the VPN virtual server and devices that won’t have the certificate will either be presented with a store error message that will be resolved by either closing the app and reopening or logging off and logging on again in the store.

Note: In some cases there might be devices which do need a reenrollment to work, no point in sugar coating it this is a big change which normally is done at the start of a Citrix CEM / XenMobile deployment

I would say try it out in your lab environment, have done this multiple times and works pretty flawless. This might in turn help you with your customers.

Hope it helps!

Notes from the field: Citrix XenMobile / Endpoint Management Per App VPN not working for iOS

This was quite a nice one to troubleshoot, turns out there is a new configuration point for per app VPN and iOS devices, at least it was for me.

If you follow the configuration at https://www.citrix.com/blogs/2016/04/19/per-app-vpn-with-xenmobile-and-citrix-vpn/#:~:text=With%20the%20iOS%20per%20app,applications%20installed%20on%20the%20device. you’ll end up with a config that won’t open up a VPN when accessing the browser. Solution for this is to change the default provider type in the policy from App Proxy to Packet tunnel also mentioned here https://docs.citrix.com/en-us/xenmobile/server/policies/vpn-policy.html and explained it means the following:

Provider Type: A provider type indicates whether the provider is a VPN service or proxy service. For VPN service, choose Packet tunnel. For proxy service, choose App proxy.

Hope it helps!

Notes from the field: XenMobile Certificate Based Authentication lessons learned

Throughout the XenMobile deployments with Certificate Based Authentication(CBA) I came across some items which I thought was worth mentioning.

1. CBA up until Secure Mail 10.6.20 / Secure Hub 10.6.20 was requesting new certificates on SSL exceptions, in effect the exceptions were triggered on every SSL connection error that occurred and thus requesting a new certificate from the PKI, this got resolved in version 10.6.20 by not using Java codes anymore but instead reading the NetScaler Gateway error code which gets presented to the client.

2. The PKI / Credential Provider settings configured with template, validity, CRL and renewal configured on the PKI server won’t work for CBA, this is because CBA is not a payload certificate but only a SIGN method. WiFi certificate which get pushed do honor the validity, renewal and CRL options.

3. With above actions you’ve might get a really large PKI environment which is not necessary and therefore maybe you would need to migrate to a new PKI server, this can be done side by side by creating a new PKI/Credential Provider and configuring those accordingly and migrate in a controlled fashion

4. You might see issued Certificates which aren’t valid anymore or revoked and those devices still get access to the MAM store, this is resolved when you apply CRL mandatory or OCSP mandatory see the following article for some more information regarding CRL: https://docs.citrix.com/en-us/netscaler/12/ssl/manage-cert-revocate-lists.html

Hope these lessons learned help and if there are any comments or questions please feel free to drop them here.

Notes from the field: Be Proactive! Apple ATS is coming

For those who are not aware Apple has an upcoming change regarding App Transport Security (ATS)
https://developer.apple.com/news/?id=12212016b
The date it should be in effect was originally January 2017… but was pushed back for migration purposes, and the new date is yet a mystery.

It will have impact! Be proactive and check your XenMobile / NetScaler environments:

– NetScaler 11.1 will be the preferred build for TLS1.2 and the ECDHE cipher suites
– XenMobile 10.4 RP4 and XenMobile 10.5 have the TLS1.2 and ECDHE cipher suites (plus ATS hotfix)

Once ATS is enforced, Apple will require at least one cipher suite enabled from a specific list of cipher suites. Apple supported ATS cipher suites are:
· TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
· TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
· TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
· TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
· TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
· TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
· TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
· TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
· TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
· TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
· TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

If SSL-Offloading is used in combination with XenMobile, remember that 11.1 is the preferred build.

https://docs.citrix.com/en-us/netscaler/11-1/ssl/supported-ciphers-list-release-11.html
https://docs.citrix.com/en-us/netscaler/11-1/upgrade-downgrade-netscaler-appliance/upgrade-to-release-11-1.html
https://support.citrix.com/article/CTX126793

Notes from the field: XenMobile Location services and SQL deadlocks

Came across a pretty specific issue in a large mobility environment regarding an old value from XenMobile 9 and still present in XenMobile 10, this is called device triangulation, with this the mobile service provider can triangulate the exact location from the device with constant updates regarding there location (this was an old value which was used with SMG and not applicable anymore).
This can cause significant impact on your database server with deadlocks.

The solution is to look for the following value:
Enable Device Triangulation
zdm.device.triangulation.enable
Preffered value: false
Default value: true

This won’t change application actions regarding geofencing because those are application and container specific.

Besides the above there is also an global value optional regarding location services in combination with iOS devices (the annoying popup you’ll get when enrolling the device which security people always ask you about 🙂 the following article will explain the values for XenMobile 9 and XenMobile 10: https://support.citrix.com/article/CTX137614

Thanks to Arnaud Pain for going into more detail regarding location services client side, take a look at his blog: http://arnaudpain.com/index.php/2017/02/06/xenmobile-disable-location-service/

Notes from the field: Quick win: XenMobile remove bulk redeemed enrollments

When you are using enrollment invitations and you don’t clean this up for let’s say an environment with a few thousand of users/devices this could be a time absorbing action to do.
Luckily there is a quick win for this and you’ll want to create a query for “dbo.ENROLLMENT_PASS” on the Database server and remove those entries afterwards the redeemed invitations are gone.

Notes from the field: XenMobile CBA didn’t I revoked that cert?

Just to start it off I’m assuming that the following is in place and fully configured and you are familiar with these concepts:

– XenMobile 10.x cluster (XMS)
– Active Directory (AD)
– Active Directory Certificate Services (ADCS)
– Active Directory Certificate Template(s)
– NetScaler Gateway (NSGW)
– Certificate Based Authentication (CBA)

Which all of them are combined in a XenMobile deployment which is configured to use CBA as an enrollment requirement.

I came across a limitation/by design issue in conjunction with the web enrollment of ADCS that XMS cannot solve, meaning that enrollment and requests for the first time will work just fine but when you revoke or selective wipe a device/user and the latter enrolls again you will get a cached certificate from XMS (you say what…) Revocation in XMS will work just fine but not at this point because according to support the API used in ADCS is not capable of doing a revocation, and basically XMS is using the web-enrollment for this and relying on that.

If you want to check it just enroll a user with the above setup and check for yourself, user gets revoked, you revoke the user certificate in ADCS and enroll again and you will see the cached certificate being issued from XMS (and no new issued certificate from ADCS)2016-10-30-15_51_12-xenmobile-internet-explorer

But there is a workaround/solution for this, query the XMS database for this certificate and select the user certificate to delete..
The following query will give you the certificates which are present on XMS
Select * from dbo.keystore where name like ‘%ag%’

To delete the certificate you execute this query with your ID (in my case 22)
Delete from dbo.keystore where id=22

After this the cached gateway certificate is deleted and with a new enrollment you will also get a new certificate.

UPDATE:

When combining the above with a CRL or OCSP integration on the NetScaler this will give an automatic renewal request for the device, meaning no manual action needed anymore. This seems to be a builtin behaviour client side (Secure Hub) see the following article for more information: https://docs.citrix.com/en-us/netscaler/11-1/ssl/manage-cert-revocate-lists.html

Notes from the field: XenMobile caveats

I’ve done a couple of Xenmobile implementations and found at least two interesting caveats that stood out, when implementing XenMobile and finding resolutions for the problems you’ll get when not adding it in your deployment.

No.1
NTP got introduced again with XenMobile 10.3.x to be configured in the appliance, a little tip enter in an reachable internal server, when you don’t pay attention and let it stay not configured for example on VMware you will get a very nice error message from time to time on the console of your VM: “hrtimer: interrupt took XXXXXX ns” (the xxxxxx is variable) this leaves your node in an failed state and the only resolution then is a reboot of the node.

No.2
ADCS integration and let’s say you will have a tiered set for your ADCS regarding seperation of the roles. The thing that is not documented, is that XenMobile cannot request certificates when there are role seperations, everything needs to be on the same machine.

No.3
Certificate Pinning is something than can be enabled to function against MITM attacks, see Worx Home Certificate Pinning for more information. Usually when you demo or poc/pilot the solution you show al the different flavours that you can choose from. The customer I was started out with e-mailbased enrollment to the environment until the latter we changed to dual factor with certificate based authentication, and for ease of access we changed to upn enrollment with worxpin. Problem is I don’t know why or how, but when changing ADS the certificate pinning part breaks, corrupt certificate messages in worx home log or mismatch errors, you might think what’s going on! Had this kind of fun two times, and conclusion was remove the current certificate pinning / ads part and add the same setup again with the same certificates and all works again. Cloudops confirmed this on both occasions. Bug or not very annoying! I believe an support article is in the worx! (;-p)

Hope these insights help out!