Notes from the field: Citrix CEM / XenMobile enabling Certificate Based Authentication (CBA) after enrollment

I think any consultant at some time encountered the scenario of username / password authentication being the only authentication on the Citrix Gateway setup of Citrix CEM / XenMobile.

Afterwards advising the customer to use Certificate Based Authentication (CBA) and then also the sad news okay we need to reenroll all your devices for this to work.

But…. What if I told you there is a middle way for those customers that cannot afford a reenrollment of all their devices and enable the dual-factor situation after enrollment. (little bit of a side note that Citrix Support kind of / sort of well doesn’t support this regarding expected behavior etc. etc.)

You can easily build your test setup for this and stage everything until you will have the correct flow and actions to enable it.

Starting point:

Basic setup working with Citrix Gateway integration and username / password for authentication

Test devices enrolled and fully working

Needed:

Microsoft ADCS server with web-enrollment installed and configured for certificate requests handling

Note: Tiered setup will not work for issuing certificates so a dedicated root or subordinate will be needed with the ADCS Web Enrollment installed on it

Note: Only v2 templates are visible in ADCS Web Enrollment so do not upgrade the template and keep the default Certification Authority / Certificate recipient see the following articles for reference:

Version 3 (CNG) templates won’t appear in certificate web enrollment – Windows Server | Microsoft Docs

Windows Server 2012: Certificate Template Versions and Options – TechNet Articles – United States (English) – TechNet Wiki (microsoft.com)

Note: Validate a working web enrollment request before going any further

Note: And I cannot stress this enough make sure you have a valid working CRL/OCSP as HTTP URL location in ADCS CDP/AIA

Citrix CEM / XenMobile configured with a PKI Entity and used as a Credential Provider see the following configuration articles for reference:

Client certificate or certificate plus domain authentication (citrix.com)

Certificate Based Authentication : Troubleshooting Tips (citrix.com)

Note: Test delivery of a certificate by using an credential policy to your test devices and validate that the certificate is installed correctly

After a valid working test policy you can enable the Deliver user certificate for authentication at the Citrix Gateway integration part:

Now certificates for Citrix Gateway authentication will be generated and delivered after a period of time. You can speed this up a bit by forcing a deploy of the basic delivery group but normally you would need to wait for everyone to get a certificate. Schedule in some time with the customer for this. The delivery can be validated at the device under certificates, there it should give you a NetScaler Gateway Credentials entry:

So at this point Citrix CEM / XenMobile is ready and we need to configure Citrix Gateway. This step will need to be in conjunction with the Citrix Gateway change in Citrix CEM / XenMobile with the following:

Root/Intermediate certificate(s) linked and configured at the VPN Virtual Server with CRL mandatory or OCSP mandatory

LDAP and Cert Policy enabled as cascading primary authentication

Client authentication enabled and Client certificate mandatory

Note: Make sure to attend your certificate policy for UPN or sAMAccountName

Note: The CRL or OCSP mandatory is important because in the way Secure Hub requests certificates and that the certificate itself isn’t revocation aware in Citrix CEM / XenMobile. This way it will trigger a new certificate request and not present the cached older certificate present in Citrix CEM / XenMobile

Note: This change will effectively break access to the Citrix Gateway if you don’t have a valid certificate, so there is also an option to set the client certificate as optional in the migration phase or just do the hard cutover

Now we will change the authentication part in Citrix CEM / XenMobile:

After this change devices will be able to use Certificate based authentication to the VPN virtual server and devices that won’t have the certificate will either be presented with a store error message that will be resolved by either closing the app and reopening or logging off and logging on again in the store.

Note: In some cases there might be devices which do need a reenrollment to work, no point in sugar coating it this is a big change which normally is done at the start of a Citrix CEM / XenMobile deployment

I would say try it out in your lab environment, have done this multiple times and works pretty flawless. This might in turn help you with your customers.

Hope it helps!

Notes from the field: Another cannot complete your request with Citrix FAS

We’ve all seen it time and time again some misconfiguration with Citrix StoreFront and/or Citrix FAS and you’ll be getting the cannot complete your request message in your screen. Digging in the StoreFront logs and you’ll be seeing the most interesting messages of error kind in which you would think am I a rocket professor?

My story for this certain scenario would be a CVAD deployment integrated with FAS and everything working just fine with some minor bumps like adding your resources to the Windows Authorization Access Group and magic occurs things start to work. See Cannot Complete Your Request Error only occurs to certain users connecting from ADC with Azure MFA over to Storefront (citrix.com) for the fun of it and it’s buddy Common Resolutions to “Cannot Complete Your Request” Error when connecting directly to StoreFront Server (citrix.com)

Ok well this works! Happy customer, happy consultant. And after some time of testing the customer started to migrate existing users to this solution… And stuff didn’t work.. The same error as described in the article would occur and well not so happy customer and consultant now. Troubleshooted this and what the hell new users don’t have this problem.. only existing users! Euhm.. okay.. after some more discussion with the customer it was pointed out that this domain has been alive for a while like NT time while and upgraded to the latest and greatest Windows Server 2019.

This triggered me and after some searching came across Apps and APIs require access – Windows Server | Microsoft Docs which explained the truth! We are missing stuff. I did a compare of the groups “Pre-Windows 2000 Compatible Access” and “Windows Authorization Access Group” of the customer with my own and even a brand new test setup and there the following was missing:

Seems like upgrading a domain time and time again stuff won’t get added. After adding these objects all began to work and even the manual added resources don’t need to be in there like the CVAD servers, users object.

Hope it helps!

Notes from the field: VMware Access Kerberos integration and Office 365

Okay let’s say you have your setup for VMware Access nicely configured with your directory search attribute configured as userPrincipalName because that’s the modern way with all cloud services etc. and configured your inbound Kerberos authentication through the IDP of the Access connector. Everyone is happy and all is working well with external connections, internal connections, mobile connections and what other type of connections we can think of. Then comes the day Office 365 is going to be integrated and still all is working well externally, mobile as well and then you get some calls regarding users who get a prompt unknown user when accessing the portal through Kerberos logon. You get to the trusty old log view and dig in and see message unknown user entries with the UPN value of your internal domain. Well, turns out that when the search attribute is selected as UPN you cannot switch over to your routable domain which is being used in Office 365 and still expect a working Kerberos logon. The only way this little beauty is going to work if is the search attribute is sAMAccountName. After a GSS support case got this one confirmed this is the only way that will work, or you would need to add a global catalog specifically for the domain in question which means double accounts, dedicated domain controller etc. etc. no one wants that!

To summarize sAMAccountName is the value which will work with almost everything, keep in mind that VMware Access is an IDP so we have the values and can transform it to any other solution as we want but specifically in this case the internal Kerberos and VMware Access have a fitty when it’s userPrincipalName. I did test out two different solutions which also worked and that is using internal certificates to be used as an authentication policy, so you add the ADCS setup as a trusted KDC in VMware Access and then will get your SSO that way or integrate ADFS as an IDP and access policy because then you use the Kerberos flow through ADFS.

To give the users still a nice e-mail-based logon experience add group filters to the access policy and that in turn introduces the user sign-in unique identifier experience which you can set to email.

See the following articles for some reference regarding Kerberos:

VMware Workspace ONE Access: Kerberos Authentication Service – Feature Walk-through

Adding Kerberos Authentication Support to Your VMware Identity Manager Connector Deployment

Managing User Attributes in Workspace ONE Access

Hope it helps!

Notes from the field: Citrix FAS request not supported

On a recent Citrix FAS deployment I’ve encountered the following error: “Request not supported” when logging in to a published application or desktop.
Article https://support.citrix.com/article/CTX218941 explains that re-enrollment of the domain controller authentication template or another custom template for Kerberos usage should resolve the error.

A little bit of a background on the environment, an already working Microsoft ADCS environment was in play and in use for other services. From a design/security perspective it was designed that two dedicated Microsoft ADCS servers would be used and two Citrix FAS servers connecting these new servers. The setup was working as expected but only above error would keep coming when trying to access an application or desktop.

We tried re-enrolling the domain controller authentication certificate and this didn’t do the trick, then we decided to let the Domain Controllers get the certificate from the new dedicated Microsoft ADCS servers for Citrix FAS and this did do the trick but with a side effect the chain is changed and other services would be negatively impacted so a rollback was needed.

With this information a Microsoft support case was created and ultimately they confirmed that what is mentioned in the Citrix support article should do the trick. Ok we got confirmation and yes it indeed does work when using the new ADCS servers but the issue of the original ADCS environment was still a mystery.

So next up we decided to repoint the Citrix FAS servers to the existing Microsoft ADCS server to root out any chain or other issues that might be in play. The result was exactly the same and a not supported request as the end result.

Digging deeper in the Microsoft ADCS environment it was after checking the “NTAuthCertificates” store that the existing server wasn’t there and the new servers were. This explained the smartcard logon not working when using the existing environment because an requirement for smartcard logon is that the “NTAuthCertificates” store has the issuing certificate authority propagated. After adding the certificate and waiting for replication and a reboot everything was working as expected, also when moving to the new Microsoft ADCS environment for certificate issuing.

See the following screenshot of the Enterprise PKI snap in MMC in which you can check and/or add the missing certificate:

See the following articles for extra information:
https://docs.microsoft.com/en-us/troubleshoot/windows-server/windows-security/import-third-party-ca-to-enterprise-ntauth-store
https://support.microsoft.com/en-za/help/281245/guidelines-for-enabling-smart-card-logon-with-third-party-certificatio

Notes from the field: Citrix FAS SSO not working with invalid CRL

Recently I got contacted by a customer who had problems performing an SSO to a newly build desktop environment.

The setup a greenfield resource domain and forest trust from an existing tenant with a two way trust. Basically everything was correct but the logon from the users would always get terminated at the desktop with invalid credentials.

After a short discussion and remote session and the error messages in the logs with an invalid CRL it was clear that was the issue. Troubleshooted the AIA/CRL locations and basically the defaults where still in play, explained that default push in AD isn’t a recommended approach. If any client can’t access the CRL it will give a deny on further actions (and other clients that don’t understand AD or are joined to AD won’t work as well).

Below screenshots depict a default ADCS installation which in turn pushes out the default legacy templates and also the CRL to LDAP which I see much too often at customers:

Resolution for the CRL error was to revoke all the certificates for usage with FAS, change the CRL/AIA location to a routable and reachable HTTP listener instead of LDAP (preferably an HA setup with a load-balancer in front of it) and push out the new CRL. Afterwards logons where using the SSO capabilities.

Hope it helps!