Notes from the field: Citrix FAS SSO not working with invalid CRL

Recently I got contacted by a customer who had problems performing an SSO to a newly build desktop environment.

The setup a greenfield resource domain and forest trust from an existing tenant with a two way trust. Basically everything was correct but the logon from the users would always get terminated at the desktop with invalid credentials.

After a short discussion and remote session and the error messages in the logs with an invalid CRL it was clear that was the issue. Troubleshooted the AIA/CRL locations and basically the defaults where still in play, explained that default push in AD isn’t a recommended approach. If any client can’t access the CRL it will give a deny on further actions (and other clients that don’t understand AD or are joined to AD won’t work as well).

Below screenshots depict a default ADCS installation which in turn pushes out the default legacy templates and also the CRL to LDAP which I see much too often at customers:

Resolution for the CRL error was to revoke all the certificates for usage with FAS, change the CRL/AIA location to a routable and reachable HTTP listener instead of LDAP (preferably an HA setup with a load-balancer in front of it) and push out the new CRL. Afterwards logons where using the SSO capabilities.

Hope it helps!

Notes from the field: VMware UAG reverse proxy why doesn’t it work!

When configuring VMware UAG as an reverse proxy I’ve encountered some issues last year that as far as I could see wasn’t all to well documented. My reference article for the configuration was the following: https://techzone.vmware.com/configuring-web-reverse-proxy-identity-bridging-vmware-unified-access-gateway-vmware-workspace-one-operational-tutorial#985671

Basically when you follow it to the letter in your test deployment and with a test site you will not have a working reverse proxy URL. At the time when I encountered this I’ve logged a GSS support case and in the troubleshooting process it was clear that the proxy pattern set wasn’t working whatsoever, the correct one should be (|/(.*)|) instead of (|/intranet(.*)|)

My understanding was that if you would configure the instance id and configured the proxy pattern accordingly it would work but that wasn’t the case. Only when not referencing it and just passing it through it began to work.

When configuring multiple reverse proxy URL’s be sure to create corresponding proxy host patterns on the instance id’s

See the following screenshots for a working reference when using UAG as a reverse proxy for Exchange 2019 and Citrix StoreFront 1912

Hope it helps!

Notes from the field: VMware Horizon Enrollment Server and Core O/S

Recently had an deployment with a customer who has a mandate core o/s deployments are preferred unless the product doesn’t support a core o/s installation.

Well for this deployment we created two core o/s subordinate ADCS servers with the enrollment server software installed and configured. Everything is working fine and dandy, no issues and seems like its golden for production deployment.

Guess again.. all the Horizon products aren’t supported for a core o/s installation, yes it will work most of the time but if you’ll find any errors or in need of GSS support then most likely you would need to install a GUI variant. I’ve had two support cases open and both of them state it’s not supported even though the documentation isn’t explicitly stating that.

Take a look at https://docs.vmware.com/en/VMware-Horizon-7/7.10/horizon-installation/GUID-30AA88CF-8CDF-42E5-97D4-D75B2171434B.html for the ESB release and https://kb.vmware.com/s/article/78652 for the Horizon 8 2006 release.

I’ve asked for a documentation update in the support case and also left feedback that the articles need updating. Please do the same so that we can prevent others from installing a unsupported setup.

Hope it helps!

Notes from the field: VMware Access connector support LDAP Signing and Channel Binding

Quite recently I’ve encountered a random synchronization error that VMware Access connector could not synchronize and would error out with the following error: “Connector communication failed because of invalid data: The specified Bind DN and password could not be used to successfully authenticate against the directory”

At first I stumbled upon the known issues list: https://docs.vmware.com/en/VMware-Workspace-ONE-Access/19.03/rn/VMware-Identity-Manager-1903-Release-Notes.html#knownissues and checked if the computer name was the same as the name in the domain field and that was all correct.

Eventually it came to light that the LDAP Signing and Channel Binding hardening were implemented according to the latest Microsoft update. Well then you can also get this sort of behavior. The solution is present in an hotfix for the connector software.

Knowledge base article can be found here: https://kb.vmware.com/s/article/77158 and the hotfix can be found after logging in at my vmware and the components of Access/Identity Manager

Notes from the field: Citrix XenMobile / Endpoint Management Per App VPN not working for iOS

This was quite a nice one to troubleshoot, turns out there is a new configuration point for per app VPN and iOS devices, at least it was for me.

If you follow the configuration at https://www.citrix.com/blogs/2016/04/19/per-app-vpn-with-xenmobile-and-citrix-vpn/#:~:text=With%20the%20iOS%20per%20app,applications%20installed%20on%20the%20device. you’ll end up with a config that won’t open up a VPN when accessing the browser. Solution for this is to change the default provider type in the policy from App Proxy to Packet tunnel also mentioned here https://docs.citrix.com/en-us/xenmobile/server/policies/vpn-policy.html and explained it means the following:

Provider Type: A provider type indicates whether the provider is a VPN service or proxy service. For VPN service, choose Packet tunnel. For proxy service, choose App proxy.

Hope it helps!

Notes from the lab: Windows firewall profile not correct after reboot

Just thought of leaving a quick win here. Did you ever had the firewall profile of Windows not correctly mapped after reboots etc.?

This is because after a reboot the Domain Controllers put it in e.g. public profile and this will get passed on to other servers as well. This will effect in not being able to manage machines because of firewall blocks etc.

Solution is to restart the “Network Location Awareness” service and dependent “Network List Service”.
This will reset it to domain profile and after reboots of the other machines which have this it will be updated to domain profile as well. Or restart the service as above that will also do the trick.

Hope it helps!

Notes from the lab: Citrix ADC IP Reputation

I’ve been playing around with the Citrix ADC IP Reputation feature – https://docs.citrix.com/en-us/citrix-adc/13/reputation/ip-reputation.html in the lab for some time and to be honest it’s such a small but very effective feature which I almost never see active, why is that?

If you’ve gotten a premium licensed ADC appliance it’s a simple right click>enable and you put in the necessary arguments in a responder policy. See the following article for a quick how to video – https://www.youtube.com/watch?v=WedxwiEVuG4 and basically that is it. The requests are going to be filtered on a Webroot service provider for malicious IP database and you can then drop those from ever getting at you network. (and put in a nifty log action so that you can filter as a syslog entry in Citrix ADM

I put a global responder in place with the expression: CLIENT.IP.SRC.IPREP_IS_MALICIOUS and a reset with accompanied log entry: CLIENT.IP.SRC  + ” connection was dropped by Responder Action for malicious IP when accessing ” + HTTP.REQ.URL the results were pretty much mind blowing, see the following screenshot:Since the exploit CVE-2019-19781 – Vulnerability in Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP appliance – https://support.citrix.com/article/CTX267027 the focus was pretty much around that but with the above rule in place I got more hits on the reputation feature than on the mitigation responder in that time. (And these are probes/attacks we don’t know about!)

So to sum it up this is a very good starting point if something like Application Firewall is a bridge to far but most definitely will improve your security with a simple setup.

Hope it helps.

 

 

Notes from the lab: VMware Horizon and Microsoft MFA NPS Extension

In my own lab environment I have a mixture of EUC components and dual factor configured accordingly, but more and more I see that customers also just use the MFA solution of Microsoft to integrate it for their environments. Why not it’s included with your license right.

So back to the techie part I’ve configured my own NPS setup on a Windows Server 2019 and configured the RADIUS setup. Installed the MFA NPS extension and had a pre-existing configuration for my Citrix ADC appliance.

I’ve configured my Horizon connection server as an RADIUS client and enabled the configuration request and network policies for it as well, configuration type NAS IPv4 Address and the IP-address of the server.

Afterwards put in the configuration part in Horizon itself pointing the RADIUS authentication to the NPS server with all the necessary fields and/or additions that you want.

Well basically all should be working instantly when logging on to the Horizon URL or client.

I did however had some issues when logging in and stuff would time-out, event entries would say that the wrong dual factor request was given. This ultimately came from the fact I didn’t have a primary authentication set in MFA, I’ve checked that I could use my yubikey, SMS or push authentication. The resolution for this was to select primary push in the authenticator app and then it worked instantly.

Reference articles:
https://docs.microsoft.com/en-us/azure/active-directory/authentication/howto-mfa-nps-extension
https://christiaanbrinkhoff.com/2017/02/17/how-to-configure-azure-mfa-for-citrix-netscaler-gateway-radius-by-using-the-new-nps-extension/

(off topic there is an issue with 2019-NPS which I’ve encountered when configuring RADIUS-WIFI authentication, see the resolution here: https://community.ui.com/questions/FYI-Windows-Server-2019-NPS-for-RADIUS-broken-w-fix/364c7c17-b3d3-4973-8dd2-e4e701309300)

Notes from the field: The unexplained Outlook pop-up

Quite recently I’ve had an interesting troubleshoot at a customer. The problem was at first that there was an issue in the newly build Exchange 2019 environment that Outlook clients would open up and ask for credentials in a domain joined environment, so the SSO part of WIA isn’t working and it “seemed” to work after you would put in credentials.

Long story short support case at Microsoft was in play and after some weeks of log troubleshooting no results and a standstill for the customers migration project.

Drilldown to the issue at hand it seemed that the Exchange solution wasn’t working correctly and it was decided to uninstall the software and clean-up the entire setup and do a fresh redeploy. The engineer already repaired some ADDS inconsistencies which perhaps could be an issue but that was not the case.

At this time I got involved because the uninstall wasn’t working correctly on the last server, it kept preventing the uninstall with still some arbitrary mailboxes present and an old hardcoded OAB reference which couldn’t get removed. I’ve also had this kind of behavior with the arbitrary mailboxes on a Exchange 2016 solution and a reboot would resolve the blocking issues, the entities will get populated again and afterwards you could remove them accordingly. That was the solution here as well until the point of the OAB entry we had to delete that one through ADSIEDIT.

Ok.. all is well again and Exchange 2019 is completely removed and a redeploy was the next item. Exchange got redeployed, basic no fuzz and behavior of the client seemed to improve.. well no it came back and the outlook prompts and disconnects were reoccurring after some time again.

Next item was to strip stuff even further, we put on some scenario’s like lets do a complete redeploy of ADDS etc. which would mean a lot of work. So we tried to simulate it in separate lab environments to reproduce the issue and there we found out no issues at all, SSO was working out of the box and things started to point through customization of some sorts or other external factors.

We’ve landed on the point that perhaps the forest trust would be an issue, that was our last item to troubleshoot. There was a full forest trust in place for the migration/transition to the new network which is being used accordingly for resources etc. This was also something we implemented in both lab environments and never had any issues with. Well here comes the kicker… we’ve disabled the trust and the issues were instantly gone!

Ok. We have our little friend and are going to focus on that, it turned out that the name suffix routing enabled of the domain was giving issues, very strange because they are not overlapping of any sorts. It “seemed” all ok from a configuration point of perspective. We focused in on everything that could be an issue and ultimately came to the point that the NETBIOS name and FQDN of ADDS where the same with a dot in it e.g. domain.local for them both.. Right this isn’t something that you can setup anytime soon from Server 2019 through Windows 2000 and is basically not supported or advised to use, this was the case in Windows NT4 but further on no go at all.

Recap of the whole situation we have our workaround for the customer, but the root cause with NETBIOS and a dot in its setup isn’t going away. The migration is focused to move it all as soon as it can and then kill the trust and any connections to the old forest/domain which was giving all this grief.

I’ve had an splendid troubleshoot with the engineer who’s a wizard in ADDS, be sure to connect or follow him on LinkedIn.

Christiaan Ploeger

Notes from the field: Configuring AFAS Online with Azure

I have a quick win for those who are also in the process of migrating an ADFS configured AFAS Online setup to Azure Active Directory. I’ve already had an support call with them and besides the point they don’t support any troubleshooting IDP setups they did their best which in turn got me to sharing this.

So down to the point, the following article describes the SSO needed part for AFAS Online: https://help.afas.nl/help/EN/SE/plv2_Config_SSO.htm

The parts that need to be adjusted are at the endpoint part, they refer to the federation metadata document which is not the one you need. This needs to be the OpenID Connect metadata document listed at the endpoints. Microsoft now defaults to the /v2.0/ part. (On a side note there might be some situations you will want to use the v1 document which is not listed anymore as an endpoint to copy, to use this just delete the /v2.0/ part and the old version will be used)

The final part is the configuration adjustment in AFAS Online, there when you fill in all the values the documentation states that “Scopes” is an optional field which in turn isn’t. I’ve only got it to work with this filled with email and the same at the claim part.

If you don’t fill out the scopes section it will error out with missing claim “upn” if that is the one you chose or “email”

Hope it helps!