Notes from the field: Citrix CEM / XenMobile enabling Certificate Based Authentication (CBA) after enrollment

I think any consultant at some time encountered the scenario of username / password authentication being the only authentication on the Citrix Gateway setup of Citrix CEM / XenMobile.

Afterwards advising the customer to use Certificate Based Authentication (CBA) and then also the sad news okay we need to reenroll all your devices for this to work.

But…. What if I told you there is a middle way for those customers that cannot afford a reenrollment of all their devices and enable the dual-factor situation after enrollment. (little bit of a side note that Citrix Support kind of / sort of well doesn’t support this regarding expected behavior etc. etc.)

You can easily build your test setup for this and stage everything until you will have the correct flow and actions to enable it.

Starting point:

Basic setup working with Citrix Gateway integration and username / password for authentication

Test devices enrolled and fully working

Needed:

Microsoft ADCS server with web-enrollment installed and configured for certificate requests handling

Note: Tiered setup will not work for issuing certificates so a dedicated root or subordinate will be needed with the ADCS Web Enrollment installed on it

Note: Only v2 templates are visible in ADCS Web Enrollment so do not upgrade the template and keep the default Certification Authority / Certificate recipient see the following articles for reference:

Version 3 (CNG) templates won’t appear in certificate web enrollment – Windows Server | Microsoft Docs

Windows Server 2012: Certificate Template Versions and Options – TechNet Articles – United States (English) – TechNet Wiki (microsoft.com)

Note: Validate a working web enrollment request before going any further

Note: And I cannot stress this enough make sure you have a valid working CRL/OCSP as HTTP URL location in ADCS CDP/AIA

Citrix CEM / XenMobile configured with a PKI Entity and used as a Credential Provider see the following configuration articles for reference:

Client certificate or certificate plus domain authentication (citrix.com)

Certificate Based Authentication : Troubleshooting Tips (citrix.com)

Note: Test delivery of a certificate by using an credential policy to your test devices and validate that the certificate is installed correctly

After a valid working test policy you can enable the Deliver user certificate for authentication at the Citrix Gateway integration part:

Now certificates for Citrix Gateway authentication will be generated and delivered after a period of time. You can speed this up a bit by forcing a deploy of the basic delivery group but normally you would need to wait for everyone to get a certificate. Schedule in some time with the customer for this. The delivery can be validated at the device under certificates, there it should give you a NetScaler Gateway Credentials entry:

So at this point Citrix CEM / XenMobile is ready and we need to configure Citrix Gateway. This step will need to be in conjunction with the Citrix Gateway change in Citrix CEM / XenMobile with the following:

Root/Intermediate certificate(s) linked and configured at the VPN Virtual Server with CRL mandatory or OCSP mandatory

LDAP and Cert Policy enabled as cascading primary authentication

Client authentication enabled and Client certificate mandatory

Note: Make sure to attend your certificate policy for UPN or sAMAccountName

Note: The CRL or OCSP mandatory is important because in the way Secure Hub requests certificates and that the certificate itself isn’t revocation aware in Citrix CEM / XenMobile. This way it will trigger a new certificate request and not present the cached older certificate present in Citrix CEM / XenMobile

Note: This change will effectively break access to the Citrix Gateway if you don’t have a valid certificate, so there is also an option to set the client certificate as optional in the migration phase or just do the hard cutover

Now we will change the authentication part in Citrix CEM / XenMobile:

After this change devices will be able to use Certificate based authentication to the VPN virtual server and devices that won’t have the certificate will either be presented with a store error message that will be resolved by either closing the app and reopening or logging off and logging on again in the store.

Note: In some cases there might be devices which do need a reenrollment to work, no point in sugar coating it this is a big change which normally is done at the start of a Citrix CEM / XenMobile deployment

I would say try it out in your lab environment, have done this multiple times and works pretty flawless. This might in turn help you with your customers.

Hope it helps!

Notes from the field: Another cannot complete your request with Citrix FAS

We’ve all seen it time and time again some misconfiguration with Citrix StoreFront and/or Citrix FAS and you’ll be getting the cannot complete your request message in your screen. Digging in the StoreFront logs and you’ll be seeing the most interesting messages of error kind in which you would think am I a rocket professor?

My story for this certain scenario would be a CVAD deployment integrated with FAS and everything working just fine with some minor bumps like adding your resources to the Windows Authorization Access Group and magic occurs things start to work. See Cannot Complete Your Request Error only occurs to certain users connecting from ADC with Azure MFA over to Storefront (citrix.com) for the fun of it and it’s buddy Common Resolutions to “Cannot Complete Your Request” Error when connecting directly to StoreFront Server (citrix.com)

Ok well this works! Happy customer, happy consultant. And after some time of testing the customer started to migrate existing users to this solution… And stuff didn’t work.. The same error as described in the article would occur and well not so happy customer and consultant now. Troubleshooted this and what the hell new users don’t have this problem.. only existing users! Euhm.. okay.. after some more discussion with the customer it was pointed out that this domain has been alive for a while like NT time while and upgraded to the latest and greatest Windows Server 2019.

This triggered me and after some searching came across Apps and APIs require access – Windows Server | Microsoft Docs which explained the truth! We are missing stuff. I did a compare of the groups “Pre-Windows 2000 Compatible Access” and “Windows Authorization Access Group” of the customer with my own and even a brand new test setup and there the following was missing:

Seems like upgrading a domain time and time again stuff won’t get added. After adding these objects all began to work and even the manual added resources don’t need to be in there like the CVAD servers, users object.

Hope it helps!

Notes from the lab: VMware UAG content gateway and an A+ rating

In addition to Jesper Alberts his blog a follow up with another custom UAG edge service which has it quirks called the content gateway. For the SEG article see vJAL.nl – Secure Email Gateway

Now diving in, when you configure the edge service you have the following options to configure Custom Values for Content Gateway and bare in mind that you’ll find this article after your first check on SSL Labs because an disappointing rating is what you get out of the box. See below screenshots for an A+ rating on SSL Labs:

After configuring these options you need to re-save/update the configuration in the UAG as well otherwise the service will not get these changes. And voila an A+

Hope it helps!

Notes from the field: VMware Access Kerberos integration and Office 365

Okay let’s say you have your setup for VMware Access nicely configured with your directory search attribute configured as userPrincipalName because that’s the modern way with all cloud services etc. and configured your inbound Kerberos authentication through the IDP of the Access connector. Everyone is happy and all is working well with external connections, internal connections, mobile connections and what other type of connections we can think of. Then comes the day Office 365 is going to be integrated and still all is working well externally, mobile as well and then you get some calls regarding users who get a prompt unknown user when accessing the portal through Kerberos logon. You get to the trusty old log view and dig in and see message unknown user entries with the UPN value of your internal domain. Well, turns out that when the search attribute is selected as UPN you cannot switch over to your routable domain which is being used in Office 365 and still expect a working Kerberos logon. The only way this little beauty is going to work if is the search attribute is sAMAccountName. After a GSS support case got this one confirmed this is the only way that will work, or you would need to add a global catalog specifically for the domain in question which means double accounts, dedicated domain controller etc. etc. no one wants that!

To summarize sAMAccountName is the value which will work with almost everything, keep in mind that VMware Access is an IDP so we have the values and can transform it to any other solution as we want but specifically in this case the internal Kerberos and VMware Access have a fitty when it’s userPrincipalName. I did test out two different solutions which also worked and that is using internal certificates to be used as an authentication policy, so you add the ADCS setup as a trusted KDC in VMware Access and then will get your SSO that way or integrate ADFS as an IDP and access policy because then you use the Kerberos flow through ADFS.

To give the users still a nice e-mail-based logon experience add group filters to the access policy and that in turn introduces the user sign-in unique identifier experience which you can set to email.

See the following articles for some reference regarding Kerberos:

VMware Workspace ONE Access: Kerberos Authentication Service – Feature Walk-through

Adding Kerberos Authentication Support to Your VMware Identity Manager Connector Deployment

Managing User Attributes in Workspace ONE Access

Hope it helps!

Notes from the field: VMware Workspace ONE UEM and Android Zero Touch

On a recent project we were implementing Android Zero Touch for out of the box enrollment through WS1 UEM. For a detailed explanation what Android Zero Touch is take a look at the following URL: Zero-touch enrollment for IT admins – Android Enterprise Help

When the Zero Touch Portal is enabled through the reseller and you have your access the DPC part of UEM or any other supported EMM vendor can be added and assigned. For WS1 UEM we have the following options for configuration: Enroll Android Device Using Zero Touch Portal

Now comes the part that was the “issue” or better said somewhat misinterpreted throughout the documentation. When you configured the setup through above steps you will always get a prompt after the DPC part through VMware Access that you need to login, but the whole purpose is that there is an auto-enrollment throughout Android Zero Touch and it’s DPC values through WS1 UEM and Access. Well the latter is the blocking part.

If the authentication of WS1 UEM / WS1 Access is configured to use the source of authentication from HUB services as WS1 Access you will break the staging user part from Androids perspective. Apple for example does not have this issue when using DEP/VPP because the staging works different through that program integration with WS1 UEM.

Flipping the authentication over to UEM as primarily source and everything is working nicely. But we don’t want this because Access should be the source of authentication regarding every nicely integration for web applications throughout SAML, other IDP providers etc. etc.

Logged a support case for this but didn’t get any satisfying results regarding the enrollment process. I’ll get back to this because support did help me after we got that one fixed with something else.

Just by luck an internal colleague has a nifty RSS feed active for useful blogs and this one popped up and immediate had my interest: Automated no credential enrollment when using Workspace ONE Access for source of Authentication

OK! That explains a lot after reading and there is another VMware URL regarding extra DPC values for UEM specific items: Additional Supported Enrollment Flags for Android Enrollment and the simple addition of “useUEMAuthentication” did the trick. This effectively disables Access for the enrollment part and allows the use of a staging user again.

Well staging works and… bummer the HUB screen keeps refreshing and we can’t sign in as the owner of the device. And here we come back to VMware support, after some testing I flipped over the staging user account in UEM from “Standard – Users are asked to log in after staging” to “Advanced – Enroll on behalf of another user” and the latter gives the experience that you assign the user after staging and then login and presto no more spinning HUB screen and it works.

After discussing this with support it’s the way that how Zero Touch enrollment works the staging user account needs to be set to advanced, the standard mode isn’t supported in this way of enrolling. And after that case closed!

Hope it helps!

Notes from the field: VMware Access with VMware UAG and JWT validation

It’s been a while since I’ve retested the setup with validating gateway request with JWT entries, because I thought it was depending on an appliance such as F5 for it to work. See Launching Horizon Resources Through Validating Gateways (vmware.com)

I did try and configure it none the less but never got it farther then just enabling JWT in Access with no audience enabled and the UAG also not configured with any WS1 for a working desktop, otherwise it would always error out with something like below:

Well that was then and this is now and here comes a very nice blog post by Nick Burton explaining how easy it is and just works. See Integrating Workspace ONE Access and UAG with JWT – Nick’s IT Blog (nicksitblog.com)

Ok, mind blown and Nick and I got to some trial and error testing and checking the setup of the environments is different whatsoever. Everything seems to be in order.

This is the URL where you can check it btw:

https://<WS1 AccessURL>/SAAS/API/1.0/REST/auth/token?attribute=publicKey&format=pem

So digging in deeper and putting the UAG in DEBUG mode and reproducing the issue gave some interesting feedback in the JWT section of the UAG:

08/03 19:10:01,061[nioEventLoopGroup-10-1]INFO  jwt.JWTArtifactHelper[validateJWT: 215][192.168.30.254][][Horizon][287c-***-dd51-***-7f3c-***-f651]: JWT rejected with error message : Unable to process JOSE object (cause: org.jose4j.lang.InvalidKeyException: An RSA key of size 2048 bits or larger MUST be used with the all JOSE RSA algorithms (given key was only 1024 bits).): JsonWebSignature{“typ”:”JWT”,”alg”:”RS256″,”kid”:”1624905951″}->eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IjE2MjQ5MDU5NTEifQ.eyJqdGkiOiI0M2RkOGY3OS1kNTc5LTRmMTEtYjI5Yi00M2Y4ZTY1NTEwNDAiLCJpc3MiOiJodHRwczovL3RlY2huaWNhbGZlbGxvdy1jb25zdWx0YW5jeS52bXdhcmVpZGVudGl0eS5kZS9TQUFTL2F1dGgiLCJhdWQiOlsiaHR0cHM6Ly91YWcudGVjaG5pY2FsZmVsbG93Lm5sIl0sImFydGlmYWN0IjoiQUFRQUFIVFJLZjJBTVFpMzJNNGxLOVRUMkFNUmRqdmhKVEdMTmErVDhzQVIxeW5DSndSalZFVlk3VlU9IiwidXBuIjoiaGhlcmVzQHRlY2huaWNhbGZlbGxvdy5jb20iLCJleHAiOjE2MjgwMTgwOTgsImlhdCI6MTYyODAxNzc5OH0.ffFHm8zqNyfNJGFl_-at_NL_gEa9PzC88iIBW23jdaOsdXJAOZu6gVD-eiMxWLpX_i9Hje2v6FhqDvetv_M1uutaPgCAZU34-QxmWLN2XK4MT0IaQdLK

It seems that the key size of the tenant is 1024 and 2048 is expected to use this.

So validating tenants again and Nick has a fairly new tenant in which I have an older tenant. I’ve used a separate tenant which is also new and presto it works out of the box there as well. Key size is 2048 and all is fine.

So with this information logged a GSS support case for this and turns out it’s indeed the case that new tenants will get a key size of 2048 and older tenants still have 1024. At this time there is no ETA on when older tenants will get upgraded. If you also want to log a support case and get some more traction reference HW-106923

Hope it helps!

Notes from the field: VMware Access Roles and RBAC bug

On recent projects we where configuring RBAC roles in VMware Access Cloud and stumbled across something annoying which turned out to be a bug. The issue is that when you assign the RBAC roles through super admin, read only admin and directory admin that once added you can’t delete or re-add the same group, it will error out with the following error:


It also isn’t possible to unassing the role anymore, and you might think okay well the role still works! Well no it doesn’t the role is hardcoded and can’t be removed anymore:
Deleting the complete directory and re-adding the directory doesn’t solve it either it will come back no matter what. Logged a GSS support case for this and it turns out this is indeed a bug.

If you have the same log a case and reference HW-123910 so it can get some more traction.

Hope it helps!

Notes from the lab: VMware UAG 2106 and Admin SAML

VMware introduced SAML login capabilities for the admin facing side of UAG with version 2106. See the following article: Release Notes for VMware Unified Access Gateway 2106

This quick home lab blog shows how easy it is and how to integrate this with VMware Workspace ONE Access as your entry point.

First things first, before we start you should have the IDP.xml file of your IDP in place if this is a VMware Access setup or Microsoft ADFS it doesn’t matter, the flow is exactly the same. You upload this at the identity bridging settings part of the UAG.

 

Then you go to the account settings part and select SAML Login Configuration and enable SAML authentication and select your IDP provider.

You click on Download SAML service provider metadata and select the identity provider and external hostname that resolves to the admin port of the UAG. (So yes a FQDN is needed and a valid certificate on the UAG admin facing NIC as well)

The saved XML file is used to import as a custom SAML2.0 web app in VMware Workspace ONE Access and there you can configure a custom access policy if needed.

Afterwards when saving the configuration the admin interface will reset and afterwards only SAML login will work for the admin interface.

Some points to consider:

  • There doesn’t seem to be a fallback login when SAML is configured
  • When configuring the SAML part you might think you are stuck but simply after the SP.xml is downloaded click on cancel
  • This setup should work for any other SAML IDP as well
  • The user or group that is allowed to login via SAML is an ADMIN user

Hope it helps!

Notes from the field: VMware UAG and Citrix ADC scenario’s

On a recent project we were testing some scenario’s for the usage of VMware Blast BEAT through Citrix ADC. For some more information regarding Blast see the following article: VMware Blast Extreme Optimization Guide | VMware

Normally you would see that the Citrix ADC setup is an SSL-BRIDGE vserver with accompanying UDP vserver on the same IP for the ports 443 and 8443 which are the default for BLAST and BEAT usage.

There would be situations that port 8443 cannot be used regarding company compliance or restraints that only port 443 is allowed to be used. In this case you would need to change to an SSL-Offloading setup because we can’t intercept any traffic in a SSL-BRIDGE scenario or apply WebSocket usage in a profile.

We need to create an HTTP profile which enables WebSocket connections and we can use on the new SSL-Offloading vservers

Afterwards bind this profile to the SSL-Offloading vservers

Then we need to change the service group which we use in the UDP vserver to 8443, so the frontend is 443 and the backend is 8443

Lastly the changes on the UAG need to be reflected that the firewall connections would also be 443 for TCP/UDP usage and shared services. See the following articles for more information regarding firewall port requirements and alternate setups in the UAG: Firewall Rules for DMZ-Based Unified Access Gateway Appliances (vmware.com) and Blast TCP and UDP External URL Configuration Options (vmware.com)

With this setup you can service everything for the users on port 443. Do keep in mind that port sharing in this way is an excessive CPU load which can seriously impact the setup. See the following article for some clarification: Unified Access Gateway (UAG) high CPU utilization : HTTP 503 error (78419) (vmware.com)

Hope it helps!

Notes from the lab: VMware vCenter 7u2 ADFS changes

When vCenter 7 introduced ADFS integration I jumped on the configuration part in my lab and set it up with the necessary OAUTH integrations:

Now with vCenter 7u2 there are some changes when you have it in place and are upgrading:

The trust store is changed to VECS and you need to change/add that in vCenter:

Well one would think that everything is nice and dandy after this but I completely forgot that at the time I set the whole ADFS integration part on LDAP and of course no signing requirement in place:

These are the extra changes in my environment and need to be changed in vCenter as well:


Keep in mind that you will need to add a certificate for the LDAPS configuration part, after this the whole configuration should work again.

In my setup I would see a long waiting white screen the first time I would log in, that got resolved after a reboot of the vCenter appliance.

Hope it helps!