Notes from the lab: Citrix ADC IP Reputation

I’ve been playing around with the Citrix ADC IP Reputation feature – https://docs.citrix.com/en-us/citrix-adc/13/reputation/ip-reputation.html in the lab for some time and to be honest it’s such a small but very effective feature which I almost never see active, why is that?

If you’ve gotten a premium licensed ADC appliance it’s a simple right click>enable and you put in the necessary arguments in a responder policy. See the following article for a quick how to video – https://www.youtube.com/watch?v=WedxwiEVuG4 and basically that is it. The requests are going to be filtered on a Webroot service provider for malicious IP database and you can then drop those from ever getting at you network. (and put in a nifty log action so that you can filter as a syslog entry in Citrix ADM

I put a global responder in place with the expression: CLIENT.IP.SRC.IPREP_IS_MALICIOUS and a reset with accompanied log entry: CLIENT.IP.SRC  + ” connection was dropped by Responder Action for malicious IP when accessing ” + HTTP.REQ.URL the results were pretty much mind blowing, see the following screenshot:Since the exploit CVE-2019-19781 – Vulnerability in Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP appliance – https://support.citrix.com/article/CTX267027 the focus was pretty much around that but with the above rule in place I got more hits on the reputation feature than on the mitigation responder in that time. (And these are probes/attacks we don’t know about!)

So to sum it up this is a very good starting point if something like Application Firewall is a bridge to far but most definitely will improve your security with a simple setup.

Hope it helps.

 

 

Notes from the lab: VMware Horizon and Microsoft MFA NPS Extension

In my own lab environment I have a mixture of EUC components and dual factor configured accordingly, but more and more I see that customers also just use the MFA solution of Microsoft to integrate it for their environments. Why not it’s included with your license right.

So back to the techie part I’ve configured my own NPS setup on a Windows Server 2019 and configured the RADIUS setup. Installed the MFA NPS extension and had a pre-existing configuration for my Citrix ADC appliance.

I’ve configured my Horizon connection server as an RADIUS client and enabled the configuration request and network policies for it as well, configuration type NAS IPv4 Address and the IP-address of the server.

Afterwards put in the configuration part in Horizon itself pointing the RADIUS authentication to the NPS server with all the necessary fields and/or additions that you want.

Well basically all should be working instantly when logging on to the Horizon URL or client.

I did however had some issues when logging in and stuff would time-out, event entries would say that the wrong dual factor request was given. This ultimately came from the fact I didn’t have a primary authentication set in MFA, I’ve checked that I could use my yubikey, SMS or push authentication. The resolution for this was to select primary push in the authenticator app and then it worked instantly.

Reference articles:
https://docs.microsoft.com/en-us/azure/active-directory/authentication/howto-mfa-nps-extension
https://christiaanbrinkhoff.com/2017/02/17/how-to-configure-azure-mfa-for-citrix-netscaler-gateway-radius-by-using-the-new-nps-extension/

(off topic there is an issue with 2019-NPS which I’ve encountered when configuring RADIUS-WIFI authentication, see the resolution here: https://community.ui.com/questions/FYI-Windows-Server-2019-NPS-for-RADIUS-broken-w-fix/364c7c17-b3d3-4973-8dd2-e4e701309300)

Notes from the field: The unexplained Outlook pop-up

Quite recently I’ve had an interesting troubleshoot at a customer. The problem was at first that there was an issue in the newly build Exchange 2019 environment that Outlook clients would open up and ask for credentials in a domain joined environment, so the SSO part of WIA isn’t working and it “seemed” to work after you would put in credentials.

Long story short support case at Microsoft was in play and after some weeks of log troubleshooting no results and a standstill for the customers migration project.

Drilldown to the issue at hand it seemed that the Exchange solution wasn’t working correctly and it was decided to uninstall the software and clean-up the entire setup and do a fresh redeploy. The engineer already repaired some ADDS inconsistencies which perhaps could be an issue but that was not the case.

At this time I got involved because the uninstall wasn’t working correctly on the last server, it kept preventing the uninstall with still some arbitrary mailboxes present and an old hardcoded OAB reference which couldn’t get removed. I’ve also had this kind of behavior with the arbitrary mailboxes on a Exchange 2016 solution and a reboot would resolve the blocking issues, the entities will get populated again and afterwards you could remove them accordingly. That was the solution here as well until the point of the OAB entry we had to delete that one through ADSIEDIT.

Ok.. all is well again and Exchange 2019 is completely removed and a redeploy was the next item. Exchange got redeployed, basic no fuzz and behavior of the client seemed to improve.. well no it came back and the outlook prompts and disconnects were reoccurring after some time again.

Next item was to strip stuff even further, we put on some scenario’s like lets do a complete redeploy of ADDS etc. which would mean a lot of work. So we tried to simulate it in separate lab environments to reproduce the issue and there we found out no issues at all, SSO was working out of the box and things started to point through customization of some sorts or other external factors.

We’ve landed on the point that perhaps the forest trust would be an issue, that was our last item to troubleshoot. There was a full forest trust in place for the migration/transition to the new network which is being used accordingly for resources etc. This was also something we implemented in both lab environments and never had any issues with. Well here comes the kicker… we’ve disabled the trust and the issues were instantly gone!

Ok. We have our little friend and are going to focus on that, it turned out that the name suffix routing enabled of the domain was giving issues, very strange because they are not overlapping of any sorts. It “seemed” all ok from a configuration point of perspective. We focused in on everything that could be an issue and ultimately came to the point that the NETBIOS name and FQDN of ADDS where the same with a dot in it e.g. domain.local for them both.. Right this isn’t something that you can setup anytime soon from Server 2019 through Windows 2000 and is basically not supported or advised to use, this was the case in Windows NT4 but further on no go at all.

Recap of the whole situation we have our workaround for the customer, but the root cause with NETBIOS and a dot in its setup isn’t going away. The migration is focused to move it all as soon as it can and then kill the trust and any connections to the old forest/domain which was giving all this grief.

I’ve had an splendid troubleshoot with the engineer who’s a wizard in ADDS, be sure to connect or follow him on LinkedIn.

Christiaan Ploeger

Notes from the field: Configuring AFAS Online with Azure

I have a quick win for those who are also in the process of migrating an ADFS configured AFAS Online setup to Azure Active Directory. I’ve already had an support call with them and besides the point they don’t support any troubleshooting IDP setups they did their best which in turn got me to sharing this.

So down to the point, the following article describes the SSO needed part for AFAS Online: https://help.afas.nl/help/EN/SE/plv2_Config_SSO.htm

The parts that need to be adjusted are at the endpoint part, they refer to the federation metadata document which is not the one you need. This needs to be the OpenID Connect metadata document listed at the endpoints. Microsoft now defaults to the /v2.0/ part. (On a side note there might be some situations you will want to use the v1 document which is not listed anymore as an endpoint to copy, to use this just delete the /v2.0/ part and the old version will be used)

The final part is the configuration adjustment in AFAS Online, there when you fill in all the values the documentation states that “Scopes” is an optional field which in turn isn’t. I’ve only got it to work with this filled with email and the same at the claim part.

If you don’t fill out the scopes section it will error out with missing claim “upn” if that is the one you chose or “email”

Hope it helps!

Notes from the lab: Configuring vCenter 7 with ADFS

With the release of vCenter 7 you can now integrate it with Microsof Active Directory Federation Services (ADFS)

See the following blog article for an overview:
https://blogs.vmware.com/vsphere/2020/03/vsphere-7-identity-federation.html

See the following configuration articles for a setup overview:
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-C5E998B2-1148-46DC-990E-A5DB71F93351.html
https://kb.vmware.com/s/article/78029

With this information I’ve configured my lab environment to a working SAML based login with a few minor issues.

I had my ADFS setup load balanced through a content switching setup for external access. This is working great for my simple office 365 integration point but not so much if you’re trying to do more.

Like stated in the following article don’t terminate the SSL connection:
https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/overview/ad-fs-faq

The issue I came across was that vCenter was failing with the validation of the certificate, at first I thought the missing root/intermediates was the cause of this but this was not the case. Even after uploading my internal root/intermediate and the external certificates root/intermediate the SSL validation check would fail. The chain was valid in every case though.

The resolution was to make my internal DNS entry through a SSL-BRIDGE setup to my ADFS server and afterwards I could finish the configuration part without issues.

Now when presented with vCenter logon page if you put in an account from the federated domain it will redirect you accordingly to the ADFS logon point.

Hope it helps!

Notes from the lab: Migrating Windows vCenter to VCSA 7

In my lab environment I was running Windows vCenter 6.7 and with the release of vCenter 7 a migration is needed because there is no Windows vCenter anymore.

The following articles will give you enough information on how the process works especially the how-to from Vladan Seget:
https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-9A117817-B78D-4BBE-A957-982C734F7C5F.html
https://www.starwindsoftware.com/blog/how-to-migrate-vmware-vcenter-from-windows-to-vcsa-6-7-update-1

Basically the process is the same for vCenter 7 with in my case one issue.

At first try my migration failed at the stage when the migration assistant is shutting down the Windows vCenter and the VCSA 7 is being brought up with the original IP, hence resulting in a conflict and the upgrade is in a broken state. I followed the process and saw that especially Windows Server 2016 has the annoyance to delay the shutdown for minutes, this is a known issue and happens from time to time and to my knowledge has to do when updates were installed (even after multiple reboots).

So the migration assistant is doing its job and completes the migration and is shutting down the VM, except in my case. Resolution for this is when the script is completed and windows says it’s still shutting down etc. do a hard power off and after that the migration is moving further as expected.

In summary:
1. Make sure Windows is up to date with all the updates and there are none outstanding
2. Make sure vCenter is up to date, in my case it was the latest update of 6.7
3. Make a snapshot before you start 😊 saves a lot of hassle later on
4. Force kill the Windows vCenter when it’s shutting down after the migration assistant completes

After this the migration in my case completed flawlessly.

Hope it helps!

Notes from the lab: Citrix ADC Native Push OTP not working

I’ve updated my lab environment with Citrix Gateway push OTP support and had some trouble in configuring the Citrix SSO app on my iPhone. For some reason it couldn’t setup the gateway connection and it wasn’t reachable. (Well that was my bad in checking all my devices but I’ll get to that)

Before the push OTP change I’ve worked with the authenticator app behavior and put in the code myself and this worked fine. The change to push OTP isn’t too difficult, and the following articles give you plenty of how-to information:
https://docs.citrix.com/en-us/citrix-gateway/13/push-notification-otp.html
https://docs.citrix.com/en-us/citrix-adc/13/aaa-tm/native-otp-authentication/otp-encryption-tool.html
https://www.carlstalhood.com/native-one-time-passwords-otp-citrix-gateway-13/

Keep in mind that if you have setup a previous OTP setup like I had the encryption part needs to be migrated otherwise it just won’t work. Registration cannot complete if you just flip it on and be done with it. Follow the migration part or just start fresh with OTP encryption enabled by default.

After I corrected the encryption problem I still could not register through the Citrix SSO app for push OTP functionality. It kept giving me the error gateway not reachable, fully checked everything in my environment was up and working. It kept me puzzling that it works fine with the previous authenticator method and if I re-registered it that way it would work (without the push of course because that functionality sits in the Citrix SSO app)

Finally after more troubleshooting I found the problem… because I’ve upgraded and integrated ADFS 2019 in my environment my content switching server and gateway etc. also needed to be SNI aware. Remember that everything worked fine on my Windows devices (even the Citrix SSO VPN functionality which I use quite often) but just not on my iPhone. Turning off SNI was the solution, it seems that the Citrix SSO app on iOS doesn’t support SNI.

Hope it helps!

Notes from the field: Cannot access Citrix ADC or create HA set

Quite recently I was at a customer where they had an SDX setup with single instances and needed to be upgraded and converted to an HA setup.

Well easy does it I created the instances on the second SDX and started creating HA sets. Numerous went fine and then one started giving errors. Could not propagate from the primary and after checking SSH/SCP access this would fail as well. I logged in through the console of SDX/SVM and saw that the sshd daemon wasn’t starting anymore. (On a side note all of the original SDX instances were upgraded in regard to the exploit of last December)

After some troubleshooting I came across the following discussion article: https://discussions.citrix.com/topic/405628-unable-to-connect-to-adc-nsip-version-121-and-130-using-sshsftp

The discussion referred to an support article regarding false positives and an SSH vulnerability:
https://support.citrix.com/article/CTX209398

After checking the sshd_config file and commenting out the following:
#option UsePrivilegeSeparation
#MACs hmac-sha1,hmac-ripemd160

The sshd daemon started again and the HA propagation and synchronisation started instantly. I’ve had this on several other instances as well and they all needed the above commenting out of the lines.

Hope it helps!

Notes from the field: Configuring SentinelOne SSO with VMware Workspace ONE Access

SentinelOne’s configuration can be achieved after you have a valid account and support login. Afterwards its pretty easy to configure the SSO part.

In the cloud console of SentinelOne go to Settings>>Integrations>>SSO

Configure the following items for SSO usage:

IDP Redirect URL:

https://workspaceoneaccessurl:443/SAAS/API/1.0/GET/apps/launch/app/uniqueapplicationid

IssuerID:

https://workspaceoneaccessurl/SAAS/API/1.0/GET/metadata/idp.xml

Configure the rest of the items at your own requirements but don’t forget to upload the IDP public certificate of Workspace ONE Access.

Make copies of the Assertion Consumer Service URL and SP Entity ID to use in Workspace ONE Access.

For the configuration part of Workspace ONE Access just add a new manual SAML 2.0 application and provide the following information:

Single Sign On URL: This is the Assertion Consumer Service URL of SentinelOne

Recipient URL: This is the Assertion Consumer Service URL of SentinelOne

Application ID: this is the SP Entity ID URL of SentinelOne

Username Format: Unspecified

Username Value: ${user.email}

Don’t forget you only get an application id in Workspace ONE Access if you’ve created an application. So first up create the application with bogus input to get your id and update it accordingly.

Notes from the field: Configuring Autotask PSA with VMware Workspace ONE Access

Autotask PSA SSO configuration can be found at the following url: https://ww13.autotask.net/help/Content/AdminSetup/1FeaturesSettings/ResourcesUsers/Security/SSSO_OIDC.htm

For the configuration part of Workspace ONE Access SSO you can see the available API at this url: https://code.vmware.com/apis/57/idm#/

The problem is that Autotask PSA SSO doesn’t work/supports the setup of VMware Workspace ONE Access. I worked around this issue by having a federated setup to our Office 365 tenant and adding the Autotask application there and ultimately publishing the application as a custom application link and still provide the requested SSO.

Add a Web Application Link in Workspace ONE Access and provide the following as your target url:

https://myapps.microsoft.com/o365tenant/signin/applicationname/uniqueguidoftheapplication