Notes from the lab: Windows Server 2016 and MDT gen2 secure boot

For some upcoming projects and also lab use cases I’ve decided to brush up on some MDT/Automation tasks.

For this I’ve deployed a new Windows Server 2016 Hyper-V VirtualMachine Gen2 with secure boot enabled and installed MDT server and configured it to an up and running environment(yeah.. right).

At first I thought there were some inconsistencies with the installer because I kept getting an error on the windows overlay filter driver and it’s signature, didn’t pay much attention and kept going configured the MDT Deployment share and everything with it. Clicked on the update share item and…. boom kept getting an error on unable to mount the wim file of winpe and well a broken MDT setup..

Did some searching and came across the following articles: https://blogs.technet.microsoft.com/configurationmgr/2017/04/14/known-issue-with-the-windows-adk-for-windows-10-version-1703/ and https://blogs.technet.microsoft.com/mniehaus/2017/05/16/quick-workaround-for-adk-1703-issue/

My solution for now was to disable secure boot (because it’s an lab environment) but hope it gets resolved by Microsoft in an upcoming update, imho these workarounds shouldn’t be necessary.

Notes from the field: XenMobile Certificate Based Authentication lessons learned

Throughout the XenMobile deployments with Certificate Based Authentication(CBA) I came across some items which I thought was worth mentioning.

1. CBA up until Secure Mail 10.6.20 / Secure Hub 10.6.20 was requesting new certificates on SSL exceptions, in effect the exceptions were triggered on every SSL connection error that occurred and thus requesting a new certificate from the PKI, this got resolved in version 10.6.20 by not using Java codes anymore but instead reading the NetScaler Gateway error code which gets presented to the client.

2. The PKI / Credential Provider settings configured with template, validity, CRL and renewal configured on the PKI server won’t work for CBA, this is because CBA is not a payload certificate but only a SIGN method. WiFi certificate which get pushed do honor the validity, renewal and CRL options.

3. With above actions you’ve might get a really large PKI environment which is not necessary and therefore maybe you would need to migrate to a new PKI server, this can be done side by side by creating a new PKI/Credential Provider and configuring those accordingly and migrate in a controlled fashion

4. You might see issued Certificates which aren’t valid anymore or revoked and those devices still get access to the MAM store, this is resolved when you apply CRL mandatory or OCSP mandatory see the following article for some more information regarding CRL: https://docs.citrix.com/en-us/netscaler/12/ssl/manage-cert-revocate-lists.html

Hope these lessons learned help and if there are any comments or questions please feel free to drop them here.

Notes from the field: Be Proactive! Apple ATS is coming

For those who are not aware Apple has an upcoming change regarding App Transport Security (ATS)
https://developer.apple.com/news/?id=12212016b
The date it should be in effect was originally January 2017… but was pushed back for migration purposes, and the new date is yet a mystery.

It will have impact! Be proactive and check your XenMobile / NetScaler environments:

– NetScaler 11.1 will be the preferred build for TLS1.2 and the ECDHE cipher suites
– XenMobile 10.4 RP4 and XenMobile 10.5 have the TLS1.2 and ECDHE cipher suites (plus ATS hotfix)

Once ATS is enforced, Apple will require at least one cipher suite enabled from a specific list of cipher suites. Apple supported ATS cipher suites are:
· TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
· TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
· TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
· TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
· TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
· TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
· TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
· TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
· TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
· TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
· TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

If SSL-Offloading is used in combination with XenMobile, remember that 11.1 is the preferred build.

https://docs.citrix.com/en-us/netscaler/11-1/ssl/supported-ciphers-list-release-11.html
https://docs.citrix.com/en-us/netscaler/11-1/upgrade-downgrade-netscaler-appliance/upgrade-to-release-11-1.html
https://support.citrix.com/article/CTX126793

Notes from the field: NetScaler SDX LACP Flapping issue

I came across a peculiar issue regarding a new NetScaler SDX 14020 setup in combination with a Cisco Nexus C9372-PX-E and C9336PQ infrastructure, a new buildup of the SDX/VPX with multiple HA instances spinning and a working environment. LA sets configured for HA probes and everything nice and easy separated through vlan access. Long story short, at first it looked like a bug regarding the combination of NetScaler and Cisco: https://support.citrix.com/article/CTX215720 and created an support case with the follow ups with it, afterwards it seemed that the untagged management vlan setup was overlapping from data channels and the root cause for this was at the Cisco ACI side of things, the EPG(EndpointGroup) and BridgeDomain were overlapping in that case. The solution was to create a new and dedicated EPG/BridgeDomain for the data channels of the NetScaler.

So lessons learned:

  • Double check the setup of the ACI even if you get the “yes it’s correct” statement from your customer

Notes from the field: XenMobile Location services and SQL deadlocks

Came across a pretty specific issue in a large mobility environment regarding an old value from XenMobile 9 and still present in XenMobile 10, this is called device triangulation, with this the mobile service provider can triangulate the exact location from the device with constant updates regarding there location (this was an old value which was used with SMG and not applicable anymore).
This can cause significant impact on your database server with deadlocks.

The solution is to look for the following value:
Enable Device Triangulation
zdm.device.triangulation.enable
Preffered value: false
Default value: true

This won’t change application actions regarding geofencing because those are application and container specific.

Besides the above there is also an global value optional regarding location services in combination with iOS devices (the annoying popup you’ll get when enrolling the device which security people always ask you about 🙂 the following article will explain the values for XenMobile 9 and XenMobile 10: https://support.citrix.com/article/CTX137614

Thanks to Arnaud Pain for going into more detail regarding location services client side, take a look at his blog: http://arnaudpain.com/index.php/2017/02/06/xenmobile-disable-location-service/

Notes from the field: Provisioning Services 7.11/7.12 TLS 1.2 issue

Citrix Provisioning Servers can be showing a offline status because the SQL native client version (11.0.2100.60) installed with it will not support TLS 1.2 and due to this it will give an error in event viewer with event ID 11 – Undefined database error

Installing the latest version of SQL native client on the PVS servers should resolve the issue.

Notes from the field: Quick win: XenMobile remove bulk redeemed enrollments

When you are using enrollment invitations and you don’t clean this up for let’s say an environment with a few thousand of users/devices this could be a time absorbing action to do.
Luckily there is a quick win for this and you’ll want to create a query for “dbo.ENROLLMENT_PASS” on the Database server and remove those entries afterwards the redeemed invitations are gone.

Notes from the field: XenServer 7 mouse alignment MCS/PVS machines and XenServer 7 MCS XenTools

Came across two bugs on a XenServer 7 deployment in combination with XenDesktop/XenApp 7.12 worth sharing:

The first is a mouse alignment issue which results in VNC mouse pointer slowness or disalignment of the pointer on a console session in XenServer, the following can check the status of the usb and usb_tablet parameters on the vm’s:
## xe vm-list uuid=[of the provisioned machine] params=platform
which will give the output of the VM and the following command will set the value’s:
## xe vm-param-set uuid=[of the provisioned machine] platform:usb=true
## xe vm-param-set uuid=[of the provisioned machine] platform:usb_tablet=true
confirm the settings with the xe vm-list command and afterwards reboot the machine and the issue is resolved.

The second bug is in a newly provisioned MCS catalog the XenTools of all provisioned machines won’t get installed, there is a private fix for this with Citrix Support under LC6769, the definitive solution shall be updated in 7.13 according to support.

Notes from the field: XenMobile CBA didn’t I revoked that cert?

Just to start it off I’m assuming that the following is in place and fully configured and you are familiar with these concepts:

– XenMobile 10.x cluster (XMS)
– Active Directory (AD)
– Active Directory Certificate Services (ADCS)
– Active Directory Certificate Template(s)
– NetScaler Gateway (NSGW)
– Certificate Based Authentication (CBA)

Which all of them are combined in a XenMobile deployment which is configured to use CBA as an enrollment requirement.

I came across a limitation/by design issue in conjunction with the web enrollment of ADCS that XMS cannot solve, meaning that enrollment and requests for the first time will work just fine but when you revoke or selective wipe a device/user and the latter enrolls again you will get a cached certificate from XMS (you say what…) Revocation in XMS will work just fine but not at this point because according to support the API used in ADCS is not capable of doing a revocation, and basically XMS is using the web-enrollment for this and relying on that.

If you want to check it just enroll a user with the above setup and check for yourself, user gets revoked, you revoke the user certificate in ADCS and enroll again and you will see the cached certificate being issued from XMS (and no new issued certificate from ADCS)2016-10-30-15_51_12-xenmobile-internet-explorer

But there is a workaround/solution for this, query the XMS database for this certificate and select the user certificate to delete..
The following query will give you the certificates which are present on XMS
Select * from dbo.keystore where name like ‘%ag%’

To delete the certificate you execute this query with your ID (in my case 22)
Delete from dbo.keystore where id=22

After this the cached gateway certificate is deleted and with a new enrollment you will also get a new certificate.

UPDATE:

When combining the above with a CRL or OCSP integration on the NetScaler this will give an automatic renewal request for the device, meaning no manual action needed anymore. This seems to be a builtin behaviour client side (Secure Hub) see the following article for more information: https://docs.citrix.com/en-us/netscaler/11-1/ssl/manage-cert-revocate-lists.html