Notes from the field: VMware Horizon instant clone breaks with Kerberos armoring

On my current customer project we’ve encountered a strange issue when some stricter security policies were implemented. Kerberos armoring was enabled which effectively broke the instant clone process for Windows 10 1809/1909 releases but not for 2009 or 21H2.

It all started with a ticket that the image update process in Horizon would error out and fail constantly on the mentioned images. On the newer builds no problem at all. At first we thought it was an Microsoft update of some sorts but after some troubleshooting with colleagues Wesley Kieffer and JP Ruitenbeek it turned out to be new hardening items which got turned on.

When clearing the before setting in the image and then sealing the older images would update just fine. After reproducing the entire setup in my own lab I encountered the same issue as well. Opened up a VMware support case which still is being worked at.

The setup being used is VMware Horizon 7.13.1

The following article also describes how you can troubleshoot the internal VM which is being used for the cloning process: Troubleshooting Instant Clones in the Internal VM Debug Mode

And for some more information regarding Kerberos armoring see What’s New in Kerberos Authentication | Microsoft Docs and Compound Authentication and Active Directory Domain Services claims in Active Directory Federation Services | Microsoft Docs

Hope it helps!

UPDATE:

After a long wait from VMware support and pitching in some R&D time this resulted in an it’s going to be a feature update. Currently VMware Horizon code base isn’t aware of Kerberos armoring. It’s not just version 7.x but also 8.x which is affected. The workaround above is the solution until instant clone technology will get an update in the nearby future. This could take up to about six months. I’ll post back the update when it’s final.

Notes from the lab: Citrix ShareFile and VMware Access SSO

When configuring Citrix ShareFile for an SSO experience with your Microsoft Active Directory setup we have the following guides to use it from Citrix. See How to Configure Single Sign-On (SSO) for ShareFile (citrix.com)

Well I’m having my setup with another Identity Provider in my own lab and still want to achieve an managed SSO setup from my end. To get this to work I checked the setup from an existing integration setup like Microsoft ADFS and reverse engineered it to VMware Access instead.

The following will give you an working SSO setup with VMware Access as your Identity Provider for Citrix ShareFile:

First configure the basic settings of Citrix ShareFile with your URL’s

https://tenant-fqdn-sharefile/saml

https://tenant-fqdn-vmware

Copy paste the certificate information from your VMware Access tenant

Create the application in VMware Access

Use the explicit logon URL of your application in VMware Access in Citrix ShareFile

Use the https://tenant-fqdn-sharefile/saml/acs in VMware Access for consumption

After this make sure that an user from your corporate identity is synchronized in Citrix ShareFile and both of them match so that SSO can be achieved. Afterwards you can nicely click the icon in VMware Access from IDP point of view:

Or you can login via the SP side from the Citrix ShareFile browser landing page:

Hope it helps!

Notes from the field: Citrix CEM / XenMobile enabling Certificate Based Authentication (CBA) after enrollment

I think any consultant at some time encountered the scenario of username / password authentication being the only authentication on the Citrix Gateway setup of Citrix CEM / XenMobile.

Afterwards advising the customer to use Certificate Based Authentication (CBA) and then also the sad news okay we need to reenroll all your devices for this to work.

But…. What if I told you there is a middle way for those customers that cannot afford a reenrollment of all their devices and enable the dual-factor situation after enrollment. (little bit of a side note that Citrix Support kind of / sort of well doesn’t support this regarding expected behavior etc. etc.)

You can easily build your test setup for this and stage everything until you will have the correct flow and actions to enable it.

Starting point:

Basic setup working with Citrix Gateway integration and username / password for authentication

Test devices enrolled and fully working

Needed:

Microsoft ADCS server with web-enrollment installed and configured for certificate requests handling

Note: Tiered setup will not work for issuing certificates so a dedicated root or subordinate will be needed with the ADCS Web Enrollment installed on it

Note: Only v2 templates are visible in ADCS Web Enrollment so do not upgrade the template and keep the default Certification Authority / Certificate recipient see the following articles for reference:

Version 3 (CNG) templates won’t appear in certificate web enrollment – Windows Server | Microsoft Docs

Windows Server 2012: Certificate Template Versions and Options – TechNet Articles – United States (English) – TechNet Wiki (microsoft.com)

Note: Validate a working web enrollment request before going any further

Note: And I cannot stress this enough make sure you have a valid working CRL/OCSP as HTTP URL location in ADCS CDP/AIA

Citrix CEM / XenMobile configured with a PKI Entity and used as a Credential Provider see the following configuration articles for reference:

Client certificate or certificate plus domain authentication (citrix.com)

Certificate Based Authentication : Troubleshooting Tips (citrix.com)

Note: Test delivery of a certificate by using an credential policy to your test devices and validate that the certificate is installed correctly

After a valid working test policy you can enable the Deliver user certificate for authentication at the Citrix Gateway integration part:

Now certificates for Citrix Gateway authentication will be generated and delivered after a period of time. You can speed this up a bit by forcing a deploy of the basic delivery group but normally you would need to wait for everyone to get a certificate. Schedule in some time with the customer for this. The delivery can be validated at the device under certificates, there it should give you a NetScaler Gateway Credentials entry:

So at this point Citrix CEM / XenMobile is ready and we need to configure Citrix Gateway. This step will need to be in conjunction with the Citrix Gateway change in Citrix CEM / XenMobile with the following:

Root/Intermediate certificate(s) linked and configured at the VPN Virtual Server with CRL mandatory or OCSP mandatory

LDAP and Cert Policy enabled as cascading primary authentication

Client authentication enabled and Client certificate mandatory

Note: Make sure to attend your certificate policy for UPN or sAMAccountName

Note: The CRL or OCSP mandatory is important because in the way Secure Hub requests certificates and that the certificate itself isn’t revocation aware in Citrix CEM / XenMobile. This way it will trigger a new certificate request and not present the cached older certificate present in Citrix CEM / XenMobile

Note: This change will effectively break access to the Citrix Gateway if you don’t have a valid certificate, so there is also an option to set the client certificate as optional in the migration phase or just do the hard cutover

Now we will change the authentication part in Citrix CEM / XenMobile:

After this change devices will be able to use Certificate based authentication to the VPN virtual server and devices that won’t have the certificate will either be presented with a store error message that will be resolved by either closing the app and reopening or logging off and logging on again in the store.

Note: In some cases there might be devices which do need a reenrollment to work, no point in sugar coating it this is a big change which normally is done at the start of a Citrix CEM / XenMobile deployment

I would say try it out in your lab environment, have done this multiple times and works pretty flawless. This might in turn help you with your customers.

Hope it helps!

Notes from the field: Another cannot complete your request with Citrix FAS

We’ve all seen it time and time again some misconfiguration with Citrix StoreFront and/or Citrix FAS and you’ll be getting the cannot complete your request message in your screen. Digging in the StoreFront logs and you’ll be seeing the most interesting messages of error kind in which you would think am I a rocket professor?

My story for this certain scenario would be a CVAD deployment integrated with FAS and everything working just fine with some minor bumps like adding your resources to the Windows Authorization Access Group and magic occurs things start to work. See Cannot Complete Your Request Error only occurs to certain users connecting from ADC with Azure MFA over to Storefront (citrix.com) for the fun of it and it’s buddy Common Resolutions to “Cannot Complete Your Request” Error when connecting directly to StoreFront Server (citrix.com)

Ok well this works! Happy customer, happy consultant. And after some time of testing the customer started to migrate existing users to this solution… And stuff didn’t work.. The same error as described in the article would occur and well not so happy customer and consultant now. Troubleshooted this and what the hell new users don’t have this problem.. only existing users! Euhm.. okay.. after some more discussion with the customer it was pointed out that this domain has been alive for a while like NT time while and upgraded to the latest and greatest Windows Server 2019.

This triggered me and after some searching came across Apps and APIs require access – Windows Server | Microsoft Docs which explained the truth! We are missing stuff. I did a compare of the groups “Pre-Windows 2000 Compatible Access” and “Windows Authorization Access Group” of the customer with my own and even a brand new test setup and there the following was missing:

Seems like upgrading a domain time and time again stuff won’t get added. After adding these objects all began to work and even the manual added resources don’t need to be in there like the CVAD servers, users object.

Hope it helps!

Notes from the lab: VMware UAG content gateway and an A+ rating

In addition to Jesper Alberts his blog a follow up with another custom UAG edge service which has it quirks called the content gateway. For the SEG article see vJAL.nl – Secure Email Gateway

Now diving in, when you configure the edge service you have the following options to configure Custom Values for Content Gateway and bare in mind that you’ll find this article after your first check on SSL Labs because an disappointing rating is what you get out of the box. See below screenshots for an A+ rating on SSL Labs:

After configuring these options you need to re-save/update the configuration in the UAG as well otherwise the service will not get these changes. And voila an A+

Hope it helps!

Notes from the field: VMware Access Kerberos integration and Office 365

Okay let’s say you have your setup for VMware Access nicely configured with your directory search attribute configured as userPrincipalName because that’s the modern way with all cloud services etc. and configured your inbound Kerberos authentication through the IDP of the Access connector. Everyone is happy and all is working well with external connections, internal connections, mobile connections and what other type of connections we can think of. Then comes the day Office 365 is going to be integrated and still all is working well externally, mobile as well and then you get some calls regarding users who get a prompt unknown user when accessing the portal through Kerberos logon. You get to the trusty old log view and dig in and see message unknown user entries with the UPN value of your internal domain. Well, turns out that when the search attribute is selected as UPN you cannot switch over to your routable domain which is being used in Office 365 and still expect a working Kerberos logon. The only way this little beauty is going to work if is the search attribute is sAMAccountName. After a GSS support case got this one confirmed this is the only way that will work, or you would need to add a global catalog specifically for the domain in question which means double accounts, dedicated domain controller etc. etc. no one wants that!

To summarize sAMAccountName is the value which will work with almost everything, keep in mind that VMware Access is an IDP so we have the values and can transform it to any other solution as we want but specifically in this case the internal Kerberos and VMware Access have a fitty when it’s userPrincipalName. I did test out two different solutions which also worked and that is using internal certificates to be used as an authentication policy, so you add the ADCS setup as a trusted KDC in VMware Access and then will get your SSO that way or integrate ADFS as an IDP and access policy because then you use the Kerberos flow through ADFS.

To give the users still a nice e-mail-based logon experience add group filters to the access policy and that in turn introduces the user sign-in unique identifier experience which you can set to email.

See the following articles for some reference regarding Kerberos:

VMware Workspace ONE Access: Kerberos Authentication Service – Feature Walk-through

Adding Kerberos Authentication Support to Your VMware Identity Manager Connector Deployment

Managing User Attributes in Workspace ONE Access

Hope it helps!

Notes from the field: VMware Workspace ONE UEM and Android Zero Touch

On a recent project we were implementing Android Zero Touch for out of the box enrollment through WS1 UEM. For a detailed explanation what Android Zero Touch is take a look at the following URL: Zero-touch enrollment for IT admins – Android Enterprise Help

When the Zero Touch Portal is enabled through the reseller and you have your access the DPC part of UEM or any other supported EMM vendor can be added and assigned. For WS1 UEM we have the following options for configuration: Enroll Android Device Using Zero Touch Portal

Now comes the part that was the “issue” or better said somewhat misinterpreted throughout the documentation. When you configured the setup through above steps you will always get a prompt after the DPC part through VMware Access that you need to login, but the whole purpose is that there is an auto-enrollment throughout Android Zero Touch and it’s DPC values through WS1 UEM and Access. Well the latter is the blocking part.

If the authentication of WS1 UEM / WS1 Access is configured to use the source of authentication from HUB services as WS1 Access you will break the staging user part from Androids perspective. Apple for example does not have this issue when using DEP/VPP because the staging works different through that program integration with WS1 UEM.

Flipping the authentication over to UEM as primarily source and everything is working nicely. But we don’t want this because Access should be the source of authentication regarding every nicely integration for web applications throughout SAML, other IDP providers etc. etc.

Logged a support case for this but didn’t get any satisfying results regarding the enrollment process. I’ll get back to this because support did help me after we got that one fixed with something else.

Just by luck an internal colleague has a nifty RSS feed active for useful blogs and this one popped up and immediate had my interest: Automated no credential enrollment when using Workspace ONE Access for source of Authentication

OK! That explains a lot after reading and there is another VMware URL regarding extra DPC values for UEM specific items: Additional Supported Enrollment Flags for Android Enrollment and the simple addition of “useUEMAuthentication” did the trick. This effectively disables Access for the enrollment part and allows the use of a staging user again.

Well staging works and… bummer the HUB screen keeps refreshing and we can’t sign in as the owner of the device. And here we come back to VMware support, after some testing I flipped over the staging user account in UEM from “Standard – Users are asked to log in after staging” to “Advanced – Enroll on behalf of another user” and the latter gives the experience that you assign the user after staging and then login and presto no more spinning HUB screen and it works.

After discussing this with support it’s the way that how Zero Touch enrollment works the staging user account needs to be set to advanced, the standard mode isn’t supported in this way of enrolling. And after that case closed!

Hope it helps!

Notes from the field: VMware Access with VMware UAG and JWT validation

It’s been a while since I’ve retested the setup with validating gateway request with JWT entries, because I thought it was depending on an appliance such as F5 for it to work. See Launching Horizon Resources Through Validating Gateways (vmware.com)

I did try and configure it none the less but never got it farther then just enabling JWT in Access with no audience enabled and the UAG also not configured with any WS1 for a working desktop, otherwise it would always error out with something like below:

Well that was then and this is now and here comes a very nice blog post by Nick Burton explaining how easy it is and just works. See Integrating Workspace ONE Access and UAG with JWT – Nick’s IT Blog (nicksitblog.com)

Ok, mind blown and Nick and I got to some trial and error testing and checking the setup of the environments is different whatsoever. Everything seems to be in order.

This is the URL where you can check it btw:

https://<WS1 AccessURL>/SAAS/API/1.0/REST/auth/token?attribute=publicKey&format=pem

So digging in deeper and putting the UAG in DEBUG mode and reproducing the issue gave some interesting feedback in the JWT section of the UAG:

08/03 19:10:01,061[nioEventLoopGroup-10-1]INFO  jwt.JWTArtifactHelper[validateJWT: 215][192.168.30.254][][Horizon][287c-***-dd51-***-7f3c-***-f651]: JWT rejected with error message : Unable to process JOSE object (cause: org.jose4j.lang.InvalidKeyException: An RSA key of size 2048 bits or larger MUST be used with the all JOSE RSA algorithms (given key was only 1024 bits).): JsonWebSignature{“typ”:”JWT”,”alg”:”RS256″,”kid”:”1624905951″}->eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IjE2MjQ5MDU5NTEifQ.eyJqdGkiOiI0M2RkOGY3OS1kNTc5LTRmMTEtYjI5Yi00M2Y4ZTY1NTEwNDAiLCJpc3MiOiJodHRwczovL3RlY2huaWNhbGZlbGxvdy1jb25zdWx0YW5jeS52bXdhcmVpZGVudGl0eS5kZS9TQUFTL2F1dGgiLCJhdWQiOlsiaHR0cHM6Ly91YWcudGVjaG5pY2FsZmVsbG93Lm5sIl0sImFydGlmYWN0IjoiQUFRQUFIVFJLZjJBTVFpMzJNNGxLOVRUMkFNUmRqdmhKVEdMTmErVDhzQVIxeW5DSndSalZFVlk3VlU9IiwidXBuIjoiaGhlcmVzQHRlY2huaWNhbGZlbGxvdy5jb20iLCJleHAiOjE2MjgwMTgwOTgsImlhdCI6MTYyODAxNzc5OH0.ffFHm8zqNyfNJGFl_-at_NL_gEa9PzC88iIBW23jdaOsdXJAOZu6gVD-eiMxWLpX_i9Hje2v6FhqDvetv_M1uutaPgCAZU34-QxmWLN2XK4MT0IaQdLK

It seems that the key size of the tenant is 1024 and 2048 is expected to use this.

So validating tenants again and Nick has a fairly new tenant in which I have an older tenant. I’ve used a separate tenant which is also new and presto it works out of the box there as well. Key size is 2048 and all is fine.

So with this information logged a GSS support case for this and turns out it’s indeed the case that new tenants will get a key size of 2048 and older tenants still have 1024. At this time there is no ETA on when older tenants will get upgraded. If you also want to log a support case and get some more traction reference HW-106923

Hope it helps!

Notes from the field: VMware Access Roles and RBAC bug

On recent projects we where configuring RBAC roles in VMware Access Cloud and stumbled across something annoying which turned out to be a bug. The issue is that when you assign the RBAC roles through super admin, read only admin and directory admin that once added you can’t delete or re-add the same group, it will error out with the following error:


It also isn’t possible to unassing the role anymore, and you might think okay well the role still works! Well no it doesn’t the role is hardcoded and can’t be removed anymore:
Deleting the complete directory and re-adding the directory doesn’t solve it either it will come back no matter what. Logged a GSS support case for this and it turns out this is indeed a bug.

If you have the same log a case and reference HW-123910 so it can get some more traction.

Hope it helps!

Notes from the lab: VMware UAG 2106 and Admin SAML

VMware introduced SAML login capabilities for the admin facing side of UAG with version 2106. See the following article: Release Notes for VMware Unified Access Gateway 2106

This quick home lab blog shows how easy it is and how to integrate this with VMware Workspace ONE Access as your entry point.

First things first, before we start you should have the IDP.xml file of your IDP in place if this is a VMware Access setup or Microsoft ADFS it doesn’t matter, the flow is exactly the same. You upload this at the identity bridging settings part of the UAG.

 

Then you go to the account settings part and select SAML Login Configuration and enable SAML authentication and select your IDP provider.

You click on Download SAML service provider metadata and select the identity provider and external hostname that resolves to the admin port of the UAG. (So yes a FQDN is needed and a valid certificate on the UAG admin facing NIC as well)

The saved XML file is used to import as a custom SAML2.0 web app in VMware Workspace ONE Access and there you can configure a custom access policy if needed.

Afterwards when saving the configuration the admin interface will reset and afterwards only SAML login will work for the admin interface.

Some points to consider:

  • There doesn’t seem to be a fallback login when SAML is configured
  • When configuring the SAML part you might think you are stuck but simply after the SP.xml is downloaded click on cancel
  • This setup should work for any other SAML IDP as well
  • The user or group that is allowed to login via SAML is an ADMIN user

Hope it helps!