Notes from the field: Cannot access Citrix ADC or create HA set

Quite recently I was at a customer where they had an SDX setup with single instances and needed to be upgraded and converted to an HA setup.

Well easy does it I created the instances on the second SDX and started creating HA sets. Numerous went fine and then one started giving errors. Could not propagate from the primary and after checking SSH/SCP access this would fail as well. I logged in through the console of SDX/SVM and saw that the sshd daemon wasn’t starting anymore. (On a side note all of the original SDX instances were upgraded in regard to the exploit of last December)

After some troubleshooting I came across the following discussion article: https://discussions.citrix.com/topic/405628-unable-to-connect-to-adc-nsip-version-121-and-130-using-sshsftp

The discussion referred to an support article regarding false positives and an SSH vulnerability:
https://support.citrix.com/article/CTX209398

After checking the sshd_config file and commenting out the following:
#option UsePrivilegeSeparation
#MACs hmac-sha1,hmac-ripemd160

The sshd daemon started again and the HA propagation and synchronisation started instantly. I’ve had this on several other instances as well and they all needed the above commenting out of the lines.

Hope it helps!

Notes from the field: Configuring SentinelOne SSO with VMware Workspace ONE Access

SentinelOne’s configuration can be achieved after you have a valid account and support login. Afterwards its pretty easy to configure the SSO part.

In the cloud console of SentinelOne go to Settings>>Integrations>>SSO

Configure the following items for SSO usage:

IDP Redirect URL:

https://workspaceoneaccessurl:443/SAAS/API/1.0/GET/apps/launch/app/uniqueapplicationid

IssuerID:

https://workspaceoneaccessurl/SAAS/API/1.0/GET/metadata/idp.xml

Configure the rest of the items at your own requirements but don’t forget to upload the IDP public certificate of Workspace ONE Access.

Make copies of the Assertion Consumer Service URL and SP Entity ID to use in Workspace ONE Access.

For the configuration part of Workspace ONE Access just add a new manual SAML 2.0 application and provide the following information:

Single Sign On URL: This is the Assertion Consumer Service URL of SentinelOne

Recipient URL: This is the Assertion Consumer Service URL of SentinelOne

Application ID: this is the SP Entity ID URL of SentinelOne

Username Format: Unspecified

Username Value: ${user.email}

Don’t forget you only get an application id in Workspace ONE Access if you’ve created an application. So first up create the application with bogus input to get your id and update it accordingly.

Notes from the field: Configuring Autotask PSA with VMware Workspace ONE Access

Autotask PSA SSO configuration can be found at the following url: https://ww13.autotask.net/help/Content/AdminSetup/1FeaturesSettings/ResourcesUsers/Security/SSSO_OIDC.htm

For the configuration part of Workspace ONE Access SSO you can see the available API at this url: https://code.vmware.com/apis/57/idm#/

The problem is that Autotask PSA SSO doesn’t work/supports the setup of VMware Workspace ONE Access. I worked around this issue by having a federated setup to our Office 365 tenant and adding the Autotask application there and ultimately publishing the application as a custom application link and still provide the requested SSO.

Add a Web Application Link in Workspace ONE Access and provide the following as your target url:

https://myapps.microsoft.com/o365tenant/signin/applicationname/uniqueguidoftheapplication

Notes from the field: Configuring OpsGenie (without Atlassian Access) with VMware Workspace ONE Access

OpsGenie can use SAML SSO without the use of Atlassian Access, see the following url: https://docs.opsgenie.com/docs/single-sign-on-with-opsgenie

For the configuration part of Workspace ONE Access just add a new manual SAML 2.0 application and provide the following information according to above article:

  • Single Sign On URL https://app.opsgenie.com/auth/saml?id=”uniquesamlidprovided
  • Recipient URL https://app.opsgenie.com/auth/saml?id=”uniquesamlidprovided
  • Application ID https://app.opsgenie.com/auth/saml?id=”uniqesamlidprovided
  • Username Format = Unspecified

Username Value = ${user.email}

Notes from the field: Configuring Atlassian Access with Workspace ONE Access

Atlassian Access is the SSO portal being used for SSO access across Jira, Confluence etc. for the configuration part see the following url: https://confluence.atlassian.com/cloud/saml-single-sign-on-943953302.html

For the configuration part of Workspace ONE Access just add a new manual SAML 2.0 application and provide the following information according to above article:

  • Single Sign On URL https://auth.atlassian.com/login/callback?connection=saml”uniquesamlidprovided
  • Recipient URL https://auth.atlassian.com/login/callback?connection=saml”uniquesamlidprovided
  • Application ID https://auth.atlassian.com/saml/”uniqesamlidprovided
  • Username Format = Unspecified
  • Username Value = ${user.email}
  • Relay State URL = https://id.atlassian.com/login

Add the custom attribute mappings for firstname, lastname and userprincipalname.

Notes from the field: vCloud usage meter doesn’t meter NSX

A while back I had an support case with VMware support regarding NSX integration and that it wasn’t getting metered by vCloud Usage Meter in a customer deployment. Turns out that Usage meter looks for a Global Transport Zone before the discovery of a Universal Transport Zone and metering can occur. So if you are in a setup that only has Universal Global Transport Zones it is expected behavior to see no NSX monitoring hits being satisfied in Usage meter. This can be resolved by adding a Global Transport Zone as a fictive addition so that it will meter your setup.

Notes from the field: Windows 2019 Storage Replica lock-up on VMware

On one of my latest projects consisting of a new Windows Server 2019 setup on VMware and making use of Storage Replica in a server to server setup for replicating home drives and profiles I came across a random lock-up of the VM and by that inaccessible shares.

The setup was all working until the failover part. It seems there is an delay of some sort and the failover isn’t instant or takes a while to be active with the server being unresponsive and disconnecting any form of management to the VM in question(VM tools are not responding as well and console login will not work in this failover time). I’ve tried the actions again of doing a storage replica failover and I got an BSOD on the VM stating: HAL INITIALIZATION FAILED I’ve tried all of this in a separate test setup and had this working without any problems on Server 2016, and Server 2019. Only this time it gave me this strange behavior. The difference in my own setup is HW level 14 and this new one had HW level 15 and the hosts are 6.7 13981272 build and my own setup is 6.7 14320388 build (older builds have also worked fine for me)

After some troubleshooting and providing the BSOD dump findings to VMware GSS support it became clear that version 10341 of VMware tools was the troublemaker. The solution was to upgrade to the latest 10346 VMware tools. The vmm2core tool provided me with the means of creating a dump file with the VM in question.

Notes from the field: Hyper-V to VMware migrated VM’s cannot install VMware Tools

One of my last projects I needed to convert Hyper-V VM’s to VMware, this all went fine with the offline capability of vcenter converter and the migration succeeded. Only after trying to install the VMware tools this would hang on starting the VGauth services and several other dependencies. For reference the VM’s in question are a mixture of 2008R2 / 2012R2. After some troubleshooting and searching the knowledgebase I stumbled across this article: https://kb.vmware.com/s/article/55798

So for the project I didn’t had any ok to patch the servers that was out of scope for this one, the mitigation was to install older VMware tools (10.2.5 to be exact) afterwards the tools installed fine.

On a side note when finalizing the converted VM don’t forget to delete the hidden older hyper-v network adapter, this can still provide conflicts if not removed.

Notes from the lab: Citrix ADC Native OTP and AdminSDHolder

While doing some lab work I came across an issue that the Domain Admin accounts could not register on the manageotp site while Domain Users could. This got me figuring it out.

For the use of Native OTP on the ADC we need to use an bind account for Active Directory which has the appropriate write permissions on the userParameters value of the users.

When we delegate control of the exact write permission of the userParameters everything is fine for normal users but administrator accounts won’t work. When we use a service account with full blown domain administrator permissions as the bind account then it works.

After some researching I came across this old article which explained the behavior:
https://support.microsoft.com/nl-nl/help/817433/delegated-permissions-are-not-available-and-inheritance-is-automatical

Long story short, if any user is also a member of a high privileged group the AdminSDHolder protection will prevent this. There is a way that inheritance can be enabled but this is mostly not recommended as you will open up a whole lot of extra security risks.

If it isn’t needed then just delegate control of the needed permissions otherwise use an bind account with domain admin permissions.

For some in depth knowledge of AdminSDHolder and it’s workings see the following article:
https://www.petri.com/active-directory-security-understanding-adminsdholder-object

Notes from the field: vCenter cannot validate SSO domain

Came across a peculiar issue when adding an second vCenter to the same SSO domain and enable ELM.
The first deployment worked like a charm and the second errored out with the following error:

It turns out there is a known bug when using uppercase FQDN in the configuration wizard, the solution is to put it all in lowercase.
see the following link for reference: https://kb.vmware.com/s/article/56812