Notes from the field: Windows 2019 Storage Replica lock-up on VMware

On one of my latest projects consisting of a new Windows Server 2019 setup on VMware and making use of Storage Replica in a server to server setup for replicating home drives and profiles I came across a random lock-up of the VM and by that inaccessible shares.

The setup was all working until the failover part. It seems there is an delay of some sort and the failover isn’t instant or takes a while to be active with the server being unresponsive and disconnecting any form of management to the VM in question(VM tools are not responding as well and console login will not work in this failover time). I’ve tried the actions again of doing a storage replica failover and I got an BSOD on the VM stating: HAL INITIALIZATION FAILED I’ve tried all of this in a separate test setup and had this working without any problems on Server 2016, and Server 2019. Only this time it gave me this strange behavior. The difference in my own setup is HW level 14 and this new one had HW level 15 and the hosts are 6.7 13981272 build and my own setup is 6.7 14320388 build (older builds have also worked fine for me)

After some troubleshooting and providing the BSOD dump findings to VMware GSS support it became clear that version 10341 of VMware tools was the troublemaker. The solution was to upgrade to the latest 10346 VMware tools. The vmm2core tool provided me with the means of creating a dump file with the VM in question.

Notes from the field: Hyper-V to VMware migrated VM’s cannot install VMware Tools

One of my last projects I needed to convert Hyper-V VM’s to VMware, this all went fine with the offline capability of vcenter converter and the migration succeeded. Only after trying to install the VMware tools this would hang on starting the VGauth services and several other dependencies. For reference the VM’s in question are a mixture of 2008R2 / 2012R2. After some troubleshooting and searching the knowledgebase I stumbled across this article: https://kb.vmware.com/s/article/55798

So for the project I didn’t had any ok to patch the servers that was out of scope for this one, the mitigation was to install older VMware tools (10.2.5 to be exact) afterwards the tools installed fine.

On a side note when finalizing the converted VM don’t forget to delete the hidden older hyper-v network adapter, this can still provide conflicts if not removed.

Notes from the lab: Citrix ADC Native OTP and AdminSDHolder

While doing some lab work I came across an issue that the Domain Admin accounts could not register on the manageotp site while Domain Users could. This got me figuring it out.

For the use of Native OTP on the ADC we need to use an bind account for Active Directory which has the appropriate write permissions on the userParameters value of the users.

When we delegate control of the exact write permission of the userParameters everything is fine for normal users but administrator accounts won’t work. When we use a service account with full blown domain administrator permissions as the bind account then it works.

After some researching I came across this old article which explained the behavior:
https://support.microsoft.com/nl-nl/help/817433/delegated-permissions-are-not-available-and-inheritance-is-automatical

Long story short, if any user is also a member of a high privileged group the AdminSDHolder protection will prevent this. There is a way that inheritance can be enabled but this is mostly not recommended as you will open up a whole lot of extra security risks.

If it isn’t needed then just delegate control of the needed permissions otherwise use an bind account with domain admin permissions.

For some in depth knowledge of AdminSDHolder and it’s workings see the following article:
https://www.petri.com/active-directory-security-understanding-adminsdholder-object

Notes from the field: uberAgent to the rescue!!

We all know it, the once in a while “it’s slow logging  on..” and then it gets dropped at the escalation desk for a resolution. So I got the call for troubleshooting this issue. Since I knew from previous experiences that uberAgent is the troubleshooting tool you will want for this I contacted them and requested the consulting license at https://uberagent.com/ (thanks to Helge Klein) did the installation of Splunk / Uberagent and got myself a monitoring baseline to work with. A little background on the setup:

  • vSphere 6.0
  • XenDesktop 7.15 / MCS – Windows 8.1 & Windows Server 2012 R2
  • RES WorkspaceManager 10.1.300.1

The problem was at times users would have a profile initialization of 90 seconds! and at times the user shell would hang..

After a period of two weeks I would have my baseline with uberAgent and filtered out that this would be random very early start of the day or just after break time. No funny business whatsoever in the environment and no lack of resources e.g. iops or cpu/memory exhaustion, drilling down in some user trending with uberAgent I came to a somewhat recurring user base that experienced the issue. Ok! That helps and after that I could reproduce it with the useraccounts in question displaying the following screen:Dropped this in the resrockstars.slack.com group and got a quick reply from Dennis van Dam in regards to traceviewer and came to the following:This in turn pointed me out to the following support article:Problem resolved and a happy customer! Hope this helps you out as well.

Reference article:

HOWTO: Create a trace file

Notes from the lab: Exchange Server 2016 CU6 broken by default??

I came across the most peculiar issue I’ve seen so far with Exchange 2016.
Installed a greenfield setup and the ECP/OWA page was broken by default with the following entry in event viewer:
——————————————————————————————————————————————————–
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 9-9-2017 22:26:57
Event time (UTC): 9-9-2017 20:26:57
Event ID: 53b3f1166cb147408cb97bc79483c3f5
Event sequence: 2
Event occurrence: 1
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/2/ROOT/owa-4-131494624100042355
Trust level: Full
Application Virtual Path: /owa
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\owa\
Machine name: EX01

Process information:
Process ID: 7756
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM

Exception information:
Exception type: TargetInvocationException
Exception message: Exception has been thrown by the target of an invocation.
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Owin.Loader.DefaultLoader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder)
at Owin.Loader.DefaultLoader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder)
at Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint()
at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

Encryption certificate is absent
at Microsoft.Exchange.Security.Authentication.Utility.GetCertificates()
at Microsoft.Exchange.Clients.Owa2.Server.Core.notifications.SignalR.SignalRStartup.Configuration(IAppBuilder app)

Request information:
Request URL: https://localhost:444/owa/exhealth.check
Request path: /owa/exhealth.check
User host address: 127.0.0.1
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\SYSTEM

Thread information:
Thread ID: 25
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace: at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Owin.Loader.DefaultLoader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder)
at Owin.Loader.DefaultLoader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder)
at Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint()
at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

Custom event details:
———————————————————————————————————————————————————
After some digging I came across this blog: https://justaucguy.wordpress.com/2014/12/01/exchange-2013-cu6-owa-something-went-wrong/ and https://blogs.technet.microsoft.com/rmilne/2017/06/27/exchange-2016-cu6-released/
the first one mentions of replacing the sharedwebconfig, which wasn’t my error but tried it anyway without any change, and the other triggered me with certificates… okay I checked them via the Exchange Management Shell and also there no issues..

Finally I got the bugger in IIS, it appears that a wrong certificate got bound at installation (yeah two clean servers and even some re-runs in other lab setups give me the same) but the solution was to unbound the certificate it had and bind the Microsoft Exchange Server Auth Certificate and do a IISreset.

Problem was instantly solved in my case. (the second blog above mentions that in an upgrade scenario the Microsoft Exchange Auth Certificate could get deleted so beware!!)

See the following reference regarding the binding in IIS:

Hope this helps!

 

Notes from the lab: Windows Server 2016 and MDT gen2 secure boot

For some upcoming projects and also lab use cases I’ve decided to brush up on some MDT/Automation tasks.

For this I’ve deployed a new Windows Server 2016 Hyper-V VirtualMachine Gen2 with secure boot enabled and installed MDT server and configured it to an up and running environment(yeah.. right).

At first I thought there were some inconsistencies with the installer because I kept getting an error on the windows overlay filter driver and it’s signature, didn’t pay much attention and kept going configured the MDT Deployment share and everything with it. Clicked on the update share item and…. boom kept getting an error on unable to mount the wim file of winpe and well a broken MDT setup..

Did some searching and came across the following articles: https://blogs.technet.microsoft.com/configurationmgr/2017/04/14/known-issue-with-the-windows-adk-for-windows-10-version-1703/ and https://blogs.technet.microsoft.com/mniehaus/2017/05/16/quick-workaround-for-adk-1703-issue/

My solution for now was to disable secure boot (because it’s an lab environment) but hope it gets resolved by Microsoft in an upcoming update, imho these workarounds shouldn’t be necessary.