Notes from the field: VMware UAG reverse proxy why doesn’t it work!

When configuring VMware UAG as an reverse proxy I’ve encountered some issues last year that as far as I could see wasn’t all to well documented. My reference article for the configuration was the following: https://techzone.vmware.com/configuring-web-reverse-proxy-identity-bridging-vmware-unified-access-gateway-vmware-workspace-one-operational-tutorial#985671

Basically when you follow it to the letter in your test deployment and with a test site you will not have a working reverse proxy URL. At the time when I encountered this I’ve logged a GSS support case and in the troubleshooting process it was clear that the proxy pattern set wasn’t working whatsoever, the correct one should be (|/(.*)|) instead of (|/intranet(.*)|)

My understanding was that if you would configure the instance id and configured the proxy pattern accordingly it would work but that wasn’t the case. Only when not referencing it and just passing it through it began to work.

When configuring multiple reverse proxy URL’s be sure to create corresponding proxy host patterns on the instance id’s

See the following screenshots for a working reference when using UAG as a reverse proxy for Exchange 2019 and Citrix StoreFront 1912

Hope it helps!

Notes from the field: The unexplained Outlook pop-up

Quite recently I’ve had an interesting troubleshoot at a customer. The problem was at first that there was an issue in the newly build Exchange 2019 environment that Outlook clients would open up and ask for credentials in a domain joined environment, so the SSO part of WIA isn’t working and it “seemed” to work after you would put in credentials.

Long story short support case at Microsoft was in play and after some weeks of log troubleshooting no results and a standstill for the customers migration project.

Drilldown to the issue at hand it seemed that the Exchange solution wasn’t working correctly and it was decided to uninstall the software and clean-up the entire setup and do a fresh redeploy. The engineer already repaired some ADDS inconsistencies which perhaps could be an issue but that was not the case.

At this time I got involved because the uninstall wasn’t working correctly on the last server, it kept preventing the uninstall with still some arbitrary mailboxes present and an old hardcoded OAB reference which couldn’t get removed. I’ve also had this kind of behavior with the arbitrary mailboxes on a Exchange 2016 solution and a reboot would resolve the blocking issues, the entities will get populated again and afterwards you could remove them accordingly. That was the solution here as well until the point of the OAB entry we had to delete that one through ADSIEDIT.

Ok.. all is well again and Exchange 2019 is completely removed and a redeploy was the next item. Exchange got redeployed, basic no fuzz and behavior of the client seemed to improve.. well no it came back and the outlook prompts and disconnects were reoccurring after some time again.

Next item was to strip stuff even further, we put on some scenario’s like lets do a complete redeploy of ADDS etc. which would mean a lot of work. So we tried to simulate it in separate lab environments to reproduce the issue and there we found out no issues at all, SSO was working out of the box and things started to point through customization of some sorts or other external factors.

We’ve landed on the point that perhaps the forest trust would be an issue, that was our last item to troubleshoot. There was a full forest trust in place for the migration/transition to the new network which is being used accordingly for resources etc. This was also something we implemented in both lab environments and never had any issues with. Well here comes the kicker… we’ve disabled the trust and the issues were instantly gone!

Ok. We have our little friend and are going to focus on that, it turned out that the name suffix routing enabled of the domain was giving issues, very strange because they are not overlapping of any sorts. It “seemed” all ok from a configuration point of perspective. We focused in on everything that could be an issue and ultimately came to the point that the NETBIOS name and FQDN of ADDS where the same with a dot in it e.g. domain.local for them both.. Right this isn’t something that you can setup anytime soon from Server 2019 through Windows 2000 and is basically not supported or advised to use, this was the case in Windows NT4 but further on no go at all.

Recap of the whole situation we have our workaround for the customer, but the root cause with NETBIOS and a dot in its setup isn’t going away. The migration is focused to move it all as soon as it can and then kill the trust and any connections to the old forest/domain which was giving all this grief.

I’ve had an splendid troubleshoot with the engineer who’s a wizard in ADDS, be sure to connect or follow him on LinkedIn.

Christiaan Ploeger

Notes from the lab: Exchange Server 2016 CU6 broken by default??

I came across the most peculiar issue I’ve seen so far with Exchange 2016.
Installed a greenfield setup and the ECP/OWA page was broken by default with the following entry in event viewer:
——————————————————————————————————————————————————–
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 9-9-2017 22:26:57
Event time (UTC): 9-9-2017 20:26:57
Event ID: 53b3f1166cb147408cb97bc79483c3f5
Event sequence: 2
Event occurrence: 1
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/2/ROOT/owa-4-131494624100042355
Trust level: Full
Application Virtual Path: /owa
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\owa\
Machine name: EX01

Process information:
Process ID: 7756
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM

Exception information:
Exception type: TargetInvocationException
Exception message: Exception has been thrown by the target of an invocation.
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Owin.Loader.DefaultLoader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder)
at Owin.Loader.DefaultLoader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder)
at Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint()
at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

Encryption certificate is absent
at Microsoft.Exchange.Security.Authentication.Utility.GetCertificates()
at Microsoft.Exchange.Clients.Owa2.Server.Core.notifications.SignalR.SignalRStartup.Configuration(IAppBuilder app)

Request information:
Request URL: https://localhost:444/owa/exhealth.check
Request path: /owa/exhealth.check
User host address: 127.0.0.1
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\SYSTEM

Thread information:
Thread ID: 25
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace: at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Owin.Loader.DefaultLoader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder)
at Owin.Loader.DefaultLoader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder)
at Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint()
at System.Threading.LazyInitializer.EnsureInitializedCore[T](T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory)
at Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

Custom event details:
———————————————————————————————————————————————————
After some digging I came across this blog: https://justaucguy.wordpress.com/2014/12/01/exchange-2013-cu6-owa-something-went-wrong/ and https://blogs.technet.microsoft.com/rmilne/2017/06/27/exchange-2016-cu6-released/
the first one mentions of replacing the sharedwebconfig, which wasn’t my error but tried it anyway without any change, and the other triggered me with certificates… okay I checked them via the Exchange Management Shell and also there no issues..

Finally I got the bugger in IIS, it appears that a wrong certificate got bound at installation (yeah two clean servers and even some re-runs in other lab setups give me the same) but the solution was to unbound the certificate it had and bind the Microsoft Exchange Server Auth Certificate and do a IISreset.

Problem was instantly solved in my case. (the second blog above mentions that in an upgrade scenario the Microsoft Exchange Auth Certificate could get deleted so beware!!)

See the following reference regarding the binding in IIS:

Hope this helps!