Notes from the lab: NetScaler VPX nsnet_connect prevents logon

When I started to rebuild my lab I came across the most strangest thing when configuring my NetScaler’s again. First a little background regarding my setup:

VMware ESXi 6.5u1 Hypervisors

NetScaler VPX 1000 Platinum Appliances

Distributed vSwitches with vlan trunks enabled

Dedicated NSVLAN for management (tagged)

Data transport vlan tagged

 

Whilst configuring and setting op the first and secondary nodes I’ve let the default appliance imports intact, that is 2vcpu and 2gb of ram and changed the E1000 nic’s to VMXNET3 and upgraded the VM compatibility format to the latest level. Nothing wrong here and started configuring both appliances with their NSIP’s respectively. Created the HA set and all was well.

 

Then it was time to put in the second nic which I’m going to use for my data transport with all vlan tagged interfaces and ip’s. Gave both appliances a shutdown and configured the nic’s accordingly (so it seemed at the time it was late 😊)

 

First node came back flawlessly but the second node wasn’t reachable anymore.. So put open the hypervisor console and I saw error messages regarding the nic and that the instance had crashed. When I would log in with the nsroot account I would get nsnet_connect prevents logon… Well ok.. that one was familiar to me with in mind the switch of E1000 and VMXNET3 devices (had this when upgrading a customer’s setup and that was the VM compatibility level, because you will need the latest build to be able to use VMXNET3, the default appliance level isn’t enough) but I’ve got both appliances up to date… I thought what the !%!@% and logged in with the nsrecover username to be able to login to the shell and dig in deeper. Thank god that worked and I was able to run the command ns_hw_err.bash which will check for any hardware error. And yes I instantly got the nic not present and reachable message. Looked at the configuration of the nic’s and a nice homer simpson moment the nic in question was still a E1000.. right… so turned it off and removed the nic, re-added it with the same MAC and presto all is well again.

 

Moral of the story double check your network settings when using VMXNET3!!!!

Notes from the field: XenDesktop RemotePC and Multi Licensing

Recently I got involved at a customer location which was going to use Remote PC catalogs in combination with their XenDesktop / XenApp 7.15 environment. This was no problem whatsoever to configure but on closer testing I encountered a bug that when you create for example a delivery group called “Windows 10 Remote PC” and adding more than one desktop the second, third and so on would get the published name of the local computer name e.g. WSDELL34951 which doesn’t comply with a standard name. The following can be observed for the delivery group name:Normally you would see at “PublishedName” an empty value, to correct this take a note of the “Uid” number and put in the following command:In this case my id was 4, and voila this will correct the name in StoreFront like in the following screenshot:

For the Multi Licensing part this needs to be done at the same level in powershell, see the following article:Multi-type licensing

In the previous screenshot you will see:

“LicenseModel” & “ProductCode” these need to be compliant with their respective edition of XenApp or XenDesktop license model, management is then per delivery group and not applicable for the entire site anymore. This would be a default for every new delivery group that will be created unless like in above screenshot you will add the “LicenseModel” & “ProductCode”

 

Hope this helps!

Notes from the lab: Exchange Server 2016 CU6 broken by default??

I came across the most peculiar issue I’ve seen so far with Exchange 2016.
Installed a greenfield setup and the ECP/OWA page was broken by default with the following entry in event viewer:
——————————————————————————————————————————————————–
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 9-9-2017 22:26:57
Event time (UTC): 9-9-2017 20:26:57
Event ID: 53b3f1166cb147408cb97bc79483c3f5
Event sequence: 2
Event occurrence: 1
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/2/ROOT/owa-4-131494624100042355
Trust level: Full
Application Virtual Path: /owa
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\owa\
Machine name: EX01

Process information:
Process ID: 7756
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM

Exception information:
Exception type: TargetInvocationException
Exception message: Exception has been thrown by the target of an invocation.
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Owin.Loader.DefaultLoader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder)
at Owin.Loader.DefaultLoader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder)
at Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint()
at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

Encryption certificate is absent
at Microsoft.Exchange.Security.Authentication.Utility.GetCertificates()
at Microsoft.Exchange.Clients.Owa2.Server.Core.notifications.SignalR.SignalRStartup.Configuration(IAppBuilder app)

Request information:
Request URL: https://localhost:444/owa/exhealth.check
Request path: /owa/exhealth.check
User host address: 127.0.0.1
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\SYSTEM

Thread information:
Thread ID: 25
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace: at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Owin.Loader.DefaultLoader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder)
at Owin.Loader.DefaultLoader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder)
at Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint()
at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

Custom event details:
———————————————————————————————————————————————————
After some digging I came across this blog: https://justaucguy.wordpress.com/2014/12/01/exchange-2013-cu6-owa-something-went-wrong/ and https://blogs.technet.microsoft.com/rmilne/2017/06/27/exchange-2016-cu6-released/
the first one mentions of replacing the sharedwebconfig, which wasn’t my error but tried it anyway without any change, and the other triggered me with certificates… okay I checked them via the Exchange Management Shell and also there no issues..

Finally I got the bugger in IIS, it appears that a wrong certificate got bound at installation (yeah two clean servers and even some re-runs in other lab setups give me the same) but the solution was to unbound the certificate it had and bind the Microsoft Exchange Server Auth Certificate and do a IISreset.

Problem was instantly solved in my case. (the second blog above mentions that in an upgrade scenario the Microsoft Exchange Auth Certificate could get deleted so beware!!)

See the following reference regarding the binding in IIS:

Hope this helps!

 

Notes from the lab: Windows Server 2016 black screen when launching any application

I came across an issue in my lab environment where the screen will go black while launching a session on Windows Server 2016. This is with XenApp/XenDesktop 7.15 LTSR
The following registry entry: DisableLogonUISuppression (D WORD Value 0) did not resolve the issue as stated in the following articles:
https://support.microsoft.com/en-us/help/4034661/windows-10-update-kb4034661 and https://support.citrix.com/article/CTX225819

Ultimatly after some trial and error the deletion of all subkeys from below registry entries resolved it:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Connectivity
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Configuration

Notes from the lab: Windows Server 2016 and MDT gen2 secure boot

For some upcoming projects and also lab use cases I’ve decided to brush up on some MDT/Automation tasks.

For this I’ve deployed a new Windows Server 2016 Hyper-V VirtualMachine Gen2 with secure boot enabled and installed MDT server and configured it to an up and running environment(yeah.. right).

At first I thought there were some inconsistencies with the installer because I kept getting an error on the windows overlay filter driver and it’s signature, didn’t pay much attention and kept going configured the MDT Deployment share and everything with it. Clicked on the update share item and…. boom kept getting an error on unable to mount the wim file of winpe and well a broken MDT setup..

Did some searching and came across the following articles: https://blogs.technet.microsoft.com/configurationmgr/2017/04/14/known-issue-with-the-windows-adk-for-windows-10-version-1703/ and https://blogs.technet.microsoft.com/mniehaus/2017/05/16/quick-workaround-for-adk-1703-issue/

My solution for now was to disable secure boot (because it’s an lab environment) but hope it gets resolved by Microsoft in an upcoming update, imho these workarounds shouldn’t be necessary.

Notes from the field: XenMobile Certificate Based Authentication lessons learned

Throughout the XenMobile deployments with Certificate Based Authentication(CBA) I came across some items which I thought was worth mentioning.

1. CBA up until Secure Mail 10.6.20 / Secure Hub 10.6.20 was requesting new certificates on SSL exceptions, in effect the exceptions were triggered on every SSL connection error that occurred and thus requesting a new certificate from the PKI, this got resolved in version 10.6.20 by not using Java codes anymore but instead reading the NetScaler Gateway error code which gets presented to the client.

2. The PKI / Credential Provider settings configured with template, validity, CRL and renewal configured on the PKI server won’t work for CBA, this is because CBA is not a payload certificate but only a SIGN method. WiFi certificate which get pushed do honor the validity, renewal and CRL options.

3. With above actions you’ve might get a really large PKI environment which is not necessary and therefore maybe you would need to migrate to a new PKI server, this can be done side by side by creating a new PKI/Credential Provider and configuring those accordingly and migrate in a controlled fashion

4. You might see issued Certificates which aren’t valid anymore or revoked and those devices still get access to the MAM store, this is resolved when you apply CRL mandatory or OCSP mandatory see the following article for some more information regarding CRL: https://docs.citrix.com/en-us/netscaler/12/ssl/manage-cert-revocate-lists.html

Hope these lessons learned help and if there are any comments or questions please feel free to drop them here.

Notes from the field: Be Proactive! Apple ATS is coming

For those who are not aware Apple has an upcoming change regarding App Transport Security (ATS)
https://developer.apple.com/news/?id=12212016b
The date it should be in effect was originally January 2017… but was pushed back for migration purposes, and the new date is yet a mystery.

It will have impact! Be proactive and check your XenMobile / NetScaler environments:

– NetScaler 11.1 will be the preferred build for TLS1.2 and the ECDHE cipher suites
– XenMobile 10.4 RP4 and XenMobile 10.5 have the TLS1.2 and ECDHE cipher suites (plus ATS hotfix)

Once ATS is enforced, Apple will require at least one cipher suite enabled from a specific list of cipher suites. Apple supported ATS cipher suites are:
· TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
· TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
· TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
· TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
· TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
· TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
· TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
· TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
· TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
· TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
· TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

If SSL-Offloading is used in combination with XenMobile, remember that 11.1 is the preferred build.

https://docs.citrix.com/en-us/netscaler/11-1/ssl/supported-ciphers-list-release-11.html
https://docs.citrix.com/en-us/netscaler/11-1/upgrade-downgrade-netscaler-appliance/upgrade-to-release-11-1.html
https://support.citrix.com/article/CTX126793

Notes from the field: NetScaler SDX LACP Flapping issue

I came across a peculiar issue regarding a new NetScaler SDX 14020 setup in combination with a Cisco Nexus C9372-PX-E and C9336PQ infrastructure, a new buildup of the SDX/VPX with multiple HA instances spinning and a working environment. LA sets configured for HA probes and everything nice and easy separated through vlan access. Long story short, at first it looked like a bug regarding the combination of NetScaler and Cisco: https://support.citrix.com/article/CTX215720 and created an support case with the follow ups with it, afterwards it seemed that the untagged management vlan setup was overlapping from data channels and the root cause for this was at the Cisco ACI side of things, the EPG(EndpointGroup) and BridgeDomain were overlapping in that case. The solution was to create a new and dedicated EPG/BridgeDomain for the data channels of the NetScaler.

So lessons learned:

  • Double check the setup of the ACI even if you get the “yes it’s correct” statement from your customer

Notes from the field: XenMobile Location services and SQL deadlocks

Came across a pretty specific issue in a large mobility environment regarding an old value from XenMobile 9 and still present in XenMobile 10, this is called device triangulation, with this the mobile service provider can triangulate the exact location from the device with constant updates regarding there location (this was an old value which was used with SMG and not applicable anymore).
This can cause significant impact on your database server with deadlocks.

The solution is to look for the following value:
Enable Device Triangulation
zdm.device.triangulation.enable
Preffered value: false
Default value: true

This won’t change application actions regarding geofencing because those are application and container specific.

Besides the above there is also an global value optional regarding location services in combination with iOS devices (the annoying popup you’ll get when enrolling the device which security people always ask you about 🙂 the following article will explain the values for XenMobile 9 and XenMobile 10: https://support.citrix.com/article/CTX137614

Thanks to Arnaud Pain for going into more detail regarding location services client side, take a look at his blog: http://arnaudpain.com/index.php/2017/02/06/xenmobile-disable-location-service/