Notes from the lab: Citrix StoreFront 2203 and the cannot complete request

A quick blog regarding my Citrix lab upgrade from Citrix Virtual Apps and Dekstops (CVAD) 1912CU4 to 2203 and the little StoreFront snag I hit.

Summary of my setup:
Two Delivery controllers
Two StoreFront servers cohabitating with Director as well
Two FAS servers
Two WEM servers
One unmanaged VDA worker

And a Citrix ADC HA setup load-balancing the entire solution with keep in mind only TLS1.2 enabled for the services.

All components went flawless through the upgrade only after the maintenance window I could not see any desktops when logging in through StoreFront and got the dreaded “cannot complete your request” when logging in via the Citrix Gateway. At first no error entries that would stand out not even the errors shown in the support article which I’m mentioning in a bit.

The I remembered that there was a hotfix released quite fast after the original 2203 release of StoreFront regarding TLS hardening of the original CVAD setup. The following articles mention it in detail:
Gateway Callback and / or XML Communication fails after upgrade to Storefront 2203
Hotfix – Receiver StoreFront 3.23.1- For Citrix Virtual Apps and Desktops 2203 – English

After doing some checks regarding the XML, StoreFront and all the VIP’s this was definitely the case. The event logs would make clear that the XML could not be reached and when logging in through the gateway the callback URL would not be reachable. Updating the StoreFront servers with the above hotfix resolved everything.

Hope it helps!

Notes from the field: Citrix Files / ShareFile MDX SSO not working

At my latest Citrix Endpoint Management customer there were some issues regarding Citrix Files / ShareFile not achieving an SSO throughout the MDX/MAM enabled applications. Everything outside the MDX/MAM application bubble would work just fine only when tunnelling through the internal only application this would fail. The setup was comprising of a dual IDP setup with Microsoft ADFS and Citrix Endpoint Management itself.

First thing to note was the ACL regarding the customers CEM environment and allowed IP-addresses. Adding those would instantly resolve the messages seen in the debugging logs of IP-address not on the allow list.

Second thing to note when we saw the erroring out of the SSO again, did a sanity check on the ADC configurations and made everything conform the article XenMobile/ShareFile SAML SSO failing

Third thing to note and now comes the kicker after the logs showed a bearer token error when trying to perform the SSO and got the response 401 Unauthorised. The customer also mentioned that “other” SAML applications would also not work and give 401 errors. Ok, we got something here.

Long story short, we opened up a support case, reproduced the issue with an on-premises XenMobile environment as well and found out that when we had the original exploit and mitigation in place of article Secure Hub shows an error and fails to connect after upgrading to a fixed firmware build to address CVE-2020-8299/ CVE-2020-8300 this issue would occur. Ok but we need this in place for Secure Hub to function correctly!

The resolution to add is that for the bearer token there should not be an SSO in place, nothing should touch that one because it performs the SAML and SSO assertions for the applications. So basically this article Post NetScaler upgrade to 11.1, SSO to ShareFile is failing. describes the traffic rule for disabling the SSO on that part and afterwards, voila, presto, eureka! Working SSO for not only the Citrix Files / ShareFile MDX/MAM enabled application but also for all other SAML applications that were failing.

Hope it helps!

Notes from the field: VMware Horizon instant clone breaks with Kerberos armoring

On my current customer project we’ve encountered a strange issue when some stricter security policies were implemented. Kerberos armoring was enabled which effectively broke the instant clone process for Windows 10 1809/1909 releases but not for 2009 or 21H2.

It all started with a ticket that the image update process in Horizon would error out and fail constantly on the mentioned images. On the newer builds no problem at all. At first we thought it was an Microsoft update of some sorts but after some troubleshooting with colleagues Wesley Kieffer and JP Ruitenbeek it turned out to be new hardening items which got turned on.

When clearing the before setting in the image and then sealing the older images would update just fine. After reproducing the entire setup in my own lab I encountered the same issue as well. Opened up a VMware support case which still is being worked at.

The setup being used is VMware Horizon 7.13.1

The following article also describes how you can troubleshoot the internal VM which is being used for the cloning process: Troubleshooting Instant Clones in the Internal VM Debug Mode

And for some more information regarding Kerberos armoring see What’s New in Kerberos Authentication | Microsoft Docs and Compound Authentication and Active Directory Domain Services claims in Active Directory Federation Services | Microsoft Docs

Hope it helps!

Notes from the lab: Citrix ShareFile and VMware Access SSO

When configuring Citrix ShareFile for an SSO experience with your Microsoft Active Directory setup we have the following guides to use it from Citrix. See How to Configure Single Sign-On (SSO) for ShareFile (citrix.com)

Well I’m having my setup with another Identity Provider in my own lab and still want to achieve an managed SSO setup from my end. To get this to work I checked the setup from an existing integration setup like Microsoft ADFS and reverse engineered it to VMware Access instead.

The following will give you an working SSO setup with VMware Access as your Identity Provider for Citrix ShareFile:

First configure the basic settings of Citrix ShareFile with your URL’s

https://tenant-fqdn-sharefile/saml

https://tenant-fqdn-vmware

Copy paste the certificate information from your VMware Access tenant

Create the application in VMware Access

Use the explicit logon URL of your application in VMware Access in Citrix ShareFile

Use the https://tenant-fqdn-sharefile/saml/acs in VMware Access for consumption

After this make sure that an user from your corporate identity is synchronized in Citrix ShareFile and both of them match so that SSO can be achieved. Afterwards you can nicely click the icon in VMware Access from IDP point of view:

Or you can login via the SP side from the Citrix ShareFile browser landing page:

Hope it helps!

Notes from the field: Citrix CEM / XenMobile enabling Certificate Based Authentication (CBA) after enrollment

I think any consultant at some time encountered the scenario of username / password authentication being the only authentication on the Citrix Gateway setup of Citrix CEM / XenMobile.

Afterwards advising the customer to use Certificate Based Authentication (CBA) and then also the sad news okay we need to reenroll all your devices for this to work.

But…. What if I told you there is a middle way for those customers that cannot afford a reenrollment of all their devices and enable the dual-factor situation after enrollment. (little bit of a side note that Citrix Support kind of / sort of well doesn’t support this regarding expected behavior etc. etc.)

You can easily build your test setup for this and stage everything until you will have the correct flow and actions to enable it.

Starting point:

Basic setup working with Citrix Gateway integration and username / password for authentication

Test devices enrolled and fully working

Needed:

Microsoft ADCS server with web-enrollment installed and configured for certificate requests handling

Note: Tiered setup will not work for issuing certificates so a dedicated root or subordinate will be needed with the ADCS Web Enrollment installed on it

Note: Only v2 templates are visible in ADCS Web Enrollment so do not upgrade the template and keep the default Certification Authority / Certificate recipient see the following articles for reference:

Version 3 (CNG) templates won’t appear in certificate web enrollment – Windows Server | Microsoft Docs

Windows Server 2012: Certificate Template Versions and Options – TechNet Articles – United States (English) – TechNet Wiki (microsoft.com)

Note: Validate a working web enrollment request before going any further

Note: And I cannot stress this enough make sure you have a valid working CRL/OCSP as HTTP URL location in ADCS CDP/AIA

Citrix CEM / XenMobile configured with a PKI Entity and used as a Credential Provider see the following configuration articles for reference:

Client certificate or certificate plus domain authentication (citrix.com)

Certificate Based Authentication : Troubleshooting Tips (citrix.com)

Note: Test delivery of a certificate by using an credential policy to your test devices and validate that the certificate is installed correctly

After a valid working test policy you can enable the Deliver user certificate for authentication at the Citrix Gateway integration part:

Now certificates for Citrix Gateway authentication will be generated and delivered after a period of time. You can speed this up a bit by forcing a deploy of the basic delivery group but normally you would need to wait for everyone to get a certificate. Schedule in some time with the customer for this. The delivery can be validated at the device under certificates, there it should give you a NetScaler Gateway Credentials entry:

So at this point Citrix CEM / XenMobile is ready and we need to configure Citrix Gateway. This step will need to be in conjunction with the Citrix Gateway change in Citrix CEM / XenMobile with the following:

Root/Intermediate certificate(s) linked and configured at the VPN Virtual Server with CRL mandatory or OCSP mandatory

LDAP and Cert Policy enabled as cascading primary authentication

Client authentication enabled and Client certificate mandatory

Note: Make sure to attend your certificate policy for UPN or sAMAccountName

Note: The CRL or OCSP mandatory is important because in the way Secure Hub requests certificates and that the certificate itself isn’t revocation aware in Citrix CEM / XenMobile. This way it will trigger a new certificate request and not present the cached older certificate present in Citrix CEM / XenMobile

Note: This change will effectively break access to the Citrix Gateway if you don’t have a valid certificate, so there is also an option to set the client certificate as optional in the migration phase or just do the hard cutover

Now we will change the authentication part in Citrix CEM / XenMobile:

After this change devices will be able to use Certificate based authentication to the VPN virtual server and devices that won’t have the certificate will either be presented with a store error message that will be resolved by either closing the app and reopening or logging off and logging on again in the store.

Note: In some cases there might be devices which do need a reenrollment to work, no point in sugar coating it this is a big change which normally is done at the start of a Citrix CEM / XenMobile deployment

I would say try it out in your lab environment, have done this multiple times and works pretty flawless. This might in turn help you with your customers.

Hope it helps!

Notes from the field: Another cannot complete your request with Citrix FAS

We’ve all seen it time and time again some misconfiguration with Citrix StoreFront and/or Citrix FAS and you’ll be getting the cannot complete your request message in your screen. Digging in the StoreFront logs and you’ll be seeing the most interesting messages of error kind in which you would think am I a rocket professor?

My story for this certain scenario would be a CVAD deployment integrated with FAS and everything working just fine with some minor bumps like adding your resources to the Windows Authorization Access Group and magic occurs things start to work. See Cannot Complete Your Request Error only occurs to certain users connecting from ADC with Azure MFA over to Storefront (citrix.com) for the fun of it and it’s buddy Common Resolutions to “Cannot Complete Your Request” Error when connecting directly to StoreFront Server (citrix.com)

Ok well this works! Happy customer, happy consultant. And after some time of testing the customer started to migrate existing users to this solution… And stuff didn’t work.. The same error as described in the article would occur and well not so happy customer and consultant now. Troubleshooted this and what the hell new users don’t have this problem.. only existing users! Euhm.. okay.. after some more discussion with the customer it was pointed out that this domain has been alive for a while like NT time while and upgraded to the latest and greatest Windows Server 2019.

This triggered me and after some searching came across Apps and APIs require access – Windows Server | Microsoft Docs which explained the truth! We are missing stuff. I did a compare of the groups “Pre-Windows 2000 Compatible Access” and “Windows Authorization Access Group” of the customer with my own and even a brand new test setup and there the following was missing:

Seems like upgrading a domain time and time again stuff won’t get added. After adding these objects all began to work and even the manual added resources don’t need to be in there like the CVAD servers, users object.

Hope it helps!

Notes from the lab: VMware UAG content gateway and an A+ rating

In addition to Jesper Alberts his blog a follow up with another custom UAG edge service which has it quirks called the content gateway. For the SEG article see vJAL.nl – Secure Email Gateway

Now diving in, when you configure the edge service you have the following options to configure Custom Values for Content Gateway and bare in mind that you’ll find this article after your first check on SSL Labs because an disappointing rating is what you get out of the box. See below screenshots for an A+ rating on SSL Labs:

After configuring these options you need to re-save/update the configuration in the UAG as well otherwise the service will not get these changes. And voila an A+

Hope it helps!

Notes from the field: VMware Access Kerberos integration and Office 365

Okay let’s say you have your setup for VMware Access nicely configured with your directory search attribute configured as userPrincipalName because that’s the modern way with all cloud services etc. and configured your inbound Kerberos authentication through the IDP of the Access connector. Everyone is happy and all is working well with external connections, internal connections, mobile connections and what other type of connections we can think of. Then comes the day Office 365 is going to be integrated and still all is working well externally, mobile as well and then you get some calls regarding users who get a prompt unknown user when accessing the portal through Kerberos logon. You get to the trusty old log view and dig in and see message unknown user entries with the UPN value of your internal domain. Well, turns out that when the search attribute is selected as UPN you cannot switch over to your routable domain which is being used in Office 365 and still expect a working Kerberos logon. The only way this little beauty is going to work if is the search attribute is sAMAccountName. After a GSS support case got this one confirmed this is the only way that will work, or you would need to add a global catalog specifically for the domain in question which means double accounts, dedicated domain controller etc. etc. no one wants that!

To summarize sAMAccountName is the value which will work with almost everything, keep in mind that VMware Access is an IDP so we have the values and can transform it to any other solution as we want but specifically in this case the internal Kerberos and VMware Access have a fitty when it’s userPrincipalName. I did test out two different solutions which also worked and that is using internal certificates to be used as an authentication policy, so you add the ADCS setup as a trusted KDC in VMware Access and then will get your SSO that way or integrate ADFS as an IDP and access policy because then you use the Kerberos flow through ADFS.

To give the users still a nice e-mail-based logon experience add group filters to the access policy and that in turn introduces the user sign-in unique identifier experience which you can set to email.

See the following articles for some reference regarding Kerberos:

VMware Workspace ONE Access: Kerberos Authentication Service – Feature Walk-through

Adding Kerberos Authentication Support to Your VMware Identity Manager Connector Deployment

Managing User Attributes in Workspace ONE Access

Hope it helps!

Notes from the field: VMware Workspace ONE UEM and Android Zero Touch

On a recent project we were implementing Android Zero Touch for out of the box enrollment through WS1 UEM. For a detailed explanation what Android Zero Touch is take a look at the following URL: Zero-touch enrollment for IT admins – Android Enterprise Help

When the Zero Touch Portal is enabled through the reseller and you have your access the DPC part of UEM or any other supported EMM vendor can be added and assigned. For WS1 UEM we have the following options for configuration: Enroll Android Device Using Zero Touch Portal

Now comes the part that was the “issue” or better said somewhat misinterpreted throughout the documentation. When you configured the setup through above steps you will always get a prompt after the DPC part through VMware Access that you need to login, but the whole purpose is that there is an auto-enrollment throughout Android Zero Touch and it’s DPC values through WS1 UEM and Access. Well the latter is the blocking part.

If the authentication of WS1 UEM / WS1 Access is configured to use the source of authentication from HUB services as WS1 Access you will break the staging user part from Androids perspective. Apple for example does not have this issue when using DEP/VPP because the staging works different through that program integration with WS1 UEM.

Flipping the authentication over to UEM as primarily source and everything is working nicely. But we don’t want this because Access should be the source of authentication regarding every nicely integration for web applications throughout SAML, other IDP providers etc. etc.

Logged a support case for this but didn’t get any satisfying results regarding the enrollment process. I’ll get back to this because support did help me after we got that one fixed with something else.

Just by luck an internal colleague has a nifty RSS feed active for useful blogs and this one popped up and immediate had my interest: Automated no credential enrollment when using Workspace ONE Access for source of Authentication

OK! That explains a lot after reading and there is another VMware URL regarding extra DPC values for UEM specific items: Additional Supported Enrollment Flags for Android Enrollment and the simple addition of “useUEMAuthentication” did the trick. This effectively disables Access for the enrollment part and allows the use of a staging user again.

Well staging works and… bummer the HUB screen keeps refreshing and we can’t sign in as the owner of the device. And here we come back to VMware support, after some testing I flipped over the staging user account in UEM from “Standard – Users are asked to log in after staging” to “Advanced – Enroll on behalf of another user” and the latter gives the experience that you assign the user after staging and then login and presto no more spinning HUB screen and it works.

After discussing this with support it’s the way that how Zero Touch enrollment works the staging user account needs to be set to advanced, the standard mode isn’t supported in this way of enrolling. And after that case closed!

Hope it helps!

Notes from the field: VMware Access with VMware UAG and JWT validation

It’s been a while since I’ve retested the setup with validating gateway request with JWT entries, because I thought it was depending on an appliance such as F5 for it to work. See Launching Horizon Resources Through Validating Gateways (vmware.com)

I did try and configure it none the less but never got it farther then just enabling JWT in Access with no audience enabled and the UAG also not configured with any WS1 for a working desktop, otherwise it would always error out with something like below:

Well that was then and this is now and here comes a very nice blog post by Nick Burton explaining how easy it is and just works. See Integrating Workspace ONE Access and UAG with JWT – Nick’s IT Blog (nicksitblog.com)

Ok, mind blown and Nick and I got to some trial and error testing and checking the setup of the environments is different whatsoever. Everything seems to be in order.

This is the URL where you can check it btw:

https://<WS1 AccessURL>/SAAS/API/1.0/REST/auth/token?attribute=publicKey&format=pem

So digging in deeper and putting the UAG in DEBUG mode and reproducing the issue gave some interesting feedback in the JWT section of the UAG:

08/03 19:10:01,061[nioEventLoopGroup-10-1]INFO  jwt.JWTArtifactHelper[validateJWT: 215][192.168.30.254][][Horizon][287c-***-dd51-***-7f3c-***-f651]: JWT rejected with error message : Unable to process JOSE object (cause: org.jose4j.lang.InvalidKeyException: An RSA key of size 2048 bits or larger MUST be used with the all JOSE RSA algorithms (given key was only 1024 bits).): JsonWebSignature{“typ”:”JWT”,”alg”:”RS256″,”kid”:”1624905951″}->eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IjE2MjQ5MDU5NTEifQ.eyJqdGkiOiI0M2RkOGY3OS1kNTc5LTRmMTEtYjI5Yi00M2Y4ZTY1NTEwNDAiLCJpc3MiOiJodHRwczovL3RlY2huaWNhbGZlbGxvdy1jb25zdWx0YW5jeS52bXdhcmVpZGVudGl0eS5kZS9TQUFTL2F1dGgiLCJhdWQiOlsiaHR0cHM6Ly91YWcudGVjaG5pY2FsZmVsbG93Lm5sIl0sImFydGlmYWN0IjoiQUFRQUFIVFJLZjJBTVFpMzJNNGxLOVRUMkFNUmRqdmhKVEdMTmErVDhzQVIxeW5DSndSalZFVlk3VlU9IiwidXBuIjoiaGhlcmVzQHRlY2huaWNhbGZlbGxvdy5jb20iLCJleHAiOjE2MjgwMTgwOTgsImlhdCI6MTYyODAxNzc5OH0.ffFHm8zqNyfNJGFl_-at_NL_gEa9PzC88iIBW23jdaOsdXJAOZu6gVD-eiMxWLpX_i9Hje2v6FhqDvetv_M1uutaPgCAZU34-QxmWLN2XK4MT0IaQdLK

It seems that the key size of the tenant is 1024 and 2048 is expected to use this.

So validating tenants again and Nick has a fairly new tenant in which I have an older tenant. I’ve used a separate tenant which is also new and presto it works out of the box there as well. Key size is 2048 and all is fine.

So with this information logged a GSS support case for this and turns out it’s indeed the case that new tenants will get a key size of 2048 and older tenants still have 1024. At this time there is no ETA on when older tenants will get upgraded. If you also want to log a support case and get some more traction reference HW-106923

Hope it helps!