Notes from the field: The broken VMware Access, UEM, and HUB portal

On a recent project with a customer, we encountered the issue that the VMware integration of the three products would be sort of “broken”. We first observed the issue after implementing the Intelligent Hub Verify rule set and see that this wouldn’t work. The devices and UEM wouldn’t show in the portal and the access policy would never apply. This in turn was caused by having the wrong OG configured in VMware Access, this was a weird issue because the entire setup was configured that way months earlier and was validated to work.

After resolving that issue, we encountered that the HUB services integration was missing functionality and not completely show the devices being integrated in the HUB portal. After a VMware GSS case with the necessary troubleshooting, we even encountered a P1 scenario that all devices wouldn’t have access anymore to the Intelligent HUB. This was eventually resolved by removing the configuration from Access in UEM and removing the WS1Hubclient OAUTH in Access, the final part was creating a new Built-in IDP for your domain in Access. According to support there are some cases that the IDP can get corrupted. The only solution is to create a new one and migrate the authentication methods over and then delete the corrupted one.

To summarize it, the problem occurred after the migration from 19.x release to 20.x release and higher, the Built-in IDP will get a new name with a migrated from name in it. This then was renamed to a normal name. The troubleshooting part introduced multiple faulty WS1Hubclient OAUTH modules which needed to be removed. The misaligned OG in Access is also to blame that the entire integration wasn’t working. After correcting all this the setup was instantly working again, there is no loss of data because the HUB services co-host within Access.

 

Hope it helps!

Notes from the field: VMware App Volumes LDAP(S) lockout

This is a quick blog to address a lockout issue if you are having troubles with LDAP(S) and or the validation of the certificate. When you want to validate this or for that matter resolve it because you can’t login to the App Volumes Manager anymore do the following on the database:

Select the dbo.ldap_domains entry and click the Select Top 1000 Rows to view the entries

Edit Top 200 Rows to edit the value in question and flip it to your value

Afterwards you can login again with LDAP(S) enabled and/or the verification as well.

Hope it helps!

Notes from the field: VMware Access CRL url too long?

This is just a quick post regarding CRL checking in VMware Access. It seems that when you have the “NEW” UI interface enabled there is a bug when you put in a valid CRL location in the lengths of: http://this.ismycrlfilelocation.crl that it would chop the end off and stay at http://this.ismycrlfilelocation and then a faulty CRL location.

Solution is switching it over to the “OLD” UI interface and put the complete URL in and switch back.

It’s on support their backlog but it is known.

Hope it helps!

Notes from the lab: Using VMware Access as IdP for Citrix Gateway

I like to fiddle around with possibilities when it comes to SAML, OAUTH authentications. This all started when a customer engineer triggered me with the possibility of achieving an SSO experience with the Citrix NetScaler and using VMware Access as the source of truth for authentication.

Well guess what this works! And even for the native workspace app users as well. All with the conditional access policies of VMware Access before it.

First things first, I’m expecting that you know how to use the following and already have that in place:

  • Configured Citrix CVAD with Citrix Gateway doesn’t matter if it is Unified or not
  • Advanced authentication profile for SAML usage in Citrix Gateway
  • AAA setup to be used with the SAML / authentication profile
  • Working Citrix session policies set for web and native logins
  • VMware Access (Workspace ONE) on-premises or cloud variant

To start this sort of configuration, first you will need to download the VMware Access IdP SAML metadata or copy the URL of it and paste this in Citrix NetScaler:

The important part is that if you do not allow the NetScaler to have outgoing internet access it cannot validate the SAML metadata discovery, and everything will fail. In this case upload the IdP metadata as a certificate on the NetScaler and deselect the “Import Metadata” option the SAML Server action. Either way make sure that this is working for you.

Regarding the SAML Server action the “${user.userName}” value is the unique object in VMware Access which translates to either sAMAccountName or userPrincipalName according to your setup. With this value it will always work as expected with the logon.

When all the values are correct you can export the SAML metadata to be used in VMware Access as a new SaaS SAML 2.0 application. Paste the XML input in this app and assign the application to your test user/group. Make sure the application id and issuer name align between VMware Access and Citrix NetScaler.

To summarize above and a working setup:

  • Authentication Profile is mandatory otherwise you are not working with advanced authentication on Citrix Gateway
  • Citrix Gateway combines with a AAA for this feature
  • Make sure the URL’s match at both sides otherwise it will fail
  • Don’t forget the Relay State Rule in NetScaler for SAML security

Hope it helps! And if there are any questions give me a ping.

Notes from the field: Citrix Cloud Connector we’re having trouble signing you in

Recently I’ve encountered an issue after installing the Citrix Cloud Connector on new Windows Server 2022 machines. The configuration on my first machine went just fine until the sign-in, here my interest got peeked because I still have IE11 on my box, strange that it uses the sign-in for Citrix Cloud.

Well let’s test the second one and remove IE11 from the box and see what happens:

Oh boy, so we are depending on IE11. Installed it again and voila:

Hope it helps!

UPDATE:

After providing feedback at Citrix Cloud Updates | Cloud Connector installer support for the system default browser and getting the reply that there is an update out I’ve tested the new version and that one works as expected without IE11.

Notes from the field: The one that Android said no more local

On one of my projects, we’ve encountered a strange issue regarding domain name resolving.

A little background on the canvas painted it’s about a VMware Workspace ONE setup with working web URL’s and UEM enrollments, you name it. We have a nice setup regarding managed devices and these use a per-app VMware tunnel connection to achieve secure application access. All working dandy fine. The devices are based on a Ascom Myco 3 setup with Android 10 and are dedicated used for specific applications.

Then we moved on to the managed devices in a COPE / Work managed format for the normal usage of applications work and personal usage. We also implemented VMware tunnel in a per-app situation and we got some mixed scenario’s as in working, not working or not resolving at all the URL’s with a .local suffix (I know what you are thinking .local isn’t best practice to use etc. etc. well we don’t live in a perfect world and there are a LOT of companies with .local suffixes over here) on Android 11/12, it more of less pointed to Android 12 that changed DNS resolving with mDNS support as in the way how it always should have been, see Android adds mDNS support so .local hostnames finally work (esper.io) for the details.

We had some interaction with VMware support as we first thought that it would be a combination issue with VMware Tunnel because the behavior in a Workspace ONE Web with the built-in tunnel SDK would work just fine. But as soon as we changed the URL’s to a not .local suffix but .net or something else it would instantly work.

On a closing note, I think more and more customers will see this change now that Android 13 is on its way. It’s good to prepare in advance.

Special thanks to my partner in crime on this case, Sidney Laan.

See the following reference articles and blogs for some fun reading and a hint to a new DNS setup:

Local DNS resolution suddenly stopped working… – Google Pixel Community

Why do Android 12, 12L, 13 Preview devices don’t resolve local DNS anymore? – Stack Overflow

Active Directory: Best Practices for Internal Domain and Network Names – TechNet Articles – United States (English) – TechNet Wiki (microsoft.com)

Hope it helps!

Notes from the lab: Migrating Azure AD Connect and then we cannot sync

This is a quick blog post regarding my own Azure AD Connect migration and a nasty error after trying to connect again for an initial connection and synchronisation.

A little insight in my environment, I already had the latest version running of Azure AD Connect namely 2.1.16.0 on my Windows Server 2019. See Azure AD Connect: Version release history – Microsoft Entra | Microsoft Docs

So, I spun up a new Windows Server 2022 and installed the Azure AD Connect role on it, imported my configuration file like described here How to import and export Azure AD Connect configuration settings – Microsoft Entra | Microsoft Docs

And then I got below error when trying to configure my new server:

The error put me onto the blog of Azure AD Connect – Unable to validate credentials due to an unexpected error. – .matrixpost.net but the issue mentioned wasn’t my issue, my GA account doesn’t have any expiry set and the logon was working everywhere else. The point to note is that I have only modern authentication enabled and MFA with number matching enabled in my tenant. So afterwards running the installation in context with /InteractiveAuth did resolve the issue. Afterwards closing, rebooting etc. never gave the error again and al logins are still providing the popup of modern and MFA prompt.

Strange thing is that I’ve had this enabled for a very long time now. Seems that in the latest versions perhaps there are some changes regarding the modern auth popup.

Hope it helps!

Notes from the field: Citrix XenMobile / CEM don’t touch that store name!

Just a quick shout out blog to stress the importance of the store name that XenMobile / CEM uses. This default store name is called “Store”.

If you by any means have changed this store name to anything else, you might run in two issues depending on different scenarios.

Scenario 1:
Citrix XenMobile or CEM and switching from MDX technology to MAM SDK policies causes different behavior regarding the MAM SDK enabled applications and the web sso policy won’t work or connections won’t be active.

Scenario 2:
Citrix CEM and enabling Citrix Gateway integration on-premises which causes the store to lock, and devices won’t be able to properly connect anymore and need a device wipe.

Hope it helps for the still existing environments still out there, and don’t forget and look at the Citrix Product Matrix regarding the valid lifetime of XenMobile and CEM.

A special thanks to Anton van Pelt in helping in with product management contact.

Notes from the lab: Citrix StoreFront 2203 and the cannot complete request

A quick blog regarding my Citrix lab upgrade from Citrix Virtual Apps and Dekstops (CVAD) 1912CU4 to 2203 and the little StoreFront snag I hit.

Summary of my setup:
Two Delivery controllers
Two StoreFront servers cohabitating with Director as well
Two FAS servers
Two WEM servers
One unmanaged VDA worker

And a Citrix ADC HA setup load-balancing the entire solution with keep in mind only TLS1.2 enabled for the services.

All components went flawless through the upgrade only after the maintenance window I could not see any desktops when logging in through StoreFront and got the dreaded “cannot complete your request” when logging in via the Citrix Gateway. At first no error entries that would stand out not even the errors shown in the support article which I’m mentioning in a bit.

The I remembered that there was a hotfix released quite fast after the original 2203 release of StoreFront regarding TLS hardening of the original CVAD setup. The following articles mention it in detail:
Gateway Callback and / or XML Communication fails after upgrade to Storefront 2203
Hotfix – Receiver StoreFront 3.23.1- For Citrix Virtual Apps and Desktops 2203 – English

After doing some checks regarding the XML, StoreFront and all the VIP’s this was definitely the case. The event logs would make clear that the XML could not be reached and when logging in through the gateway the callback URL would not be reachable. Updating the StoreFront servers with the above hotfix resolved everything.

Hope it helps!

Notes from the field: Citrix Files / ShareFile MDX SSO not working

At my latest Citrix Endpoint Management customer there were some issues regarding Citrix Files / ShareFile not achieving an SSO throughout the MDX/MAM enabled applications. Everything outside the MDX/MAM application bubble would work just fine only when tunnelling through the internal only application this would fail. The setup was comprising of a dual IDP setup with Microsoft ADFS and Citrix Endpoint Management itself.

First thing to note was the ACL regarding the customers CEM environment and allowed IP-addresses. Adding those would instantly resolve the messages seen in the debugging logs of IP-address not on the allow list.

Second thing to note when we saw the erroring out of the SSO again, did a sanity check on the ADC configurations and made everything conform the article XenMobile/ShareFile SAML SSO failing

Third thing to note and now comes the kicker after the logs showed a bearer token error when trying to perform the SSO and got the response 401 Unauthorised. The customer also mentioned that “other” SAML applications would also not work and give 401 errors. Ok, we got something here.

Long story short, we opened up a support case, reproduced the issue with an on-premises XenMobile environment as well and found out that when we had the original exploit and mitigation in place of article Secure Hub shows an error and fails to connect after upgrading to a fixed firmware build to address CVE-2020-8299/ CVE-2020-8300 this issue would occur. Ok but we need this in place for Secure Hub to function correctly!

The resolution to add is that for the bearer token there should not be an SSO in place, nothing should touch that one because it performs the SAML and SSO assertions for the applications. So basically this article Post NetScaler upgrade to 11.1, SSO to ShareFile is failing. describes the traffic rule for disabling the SSO on that part and afterwards, voila, presto, eureka! Working SSO for not only the Citrix Files / ShareFile MDX/MAM enabled application but also for all other SAML applications that were failing.

Hope it helps!