Sep 19

Written by:
9/19/2009 2:11 PM 

I've worked this week on a critical production issue involving Provisioning Services 5.1.

The symptoms were the following :

• explorer.exe freezing randomly for some seconds (8 to 30)
• Logons stalling randomly for 8 to 30 seconds
• Microsoft Office Applications (Word, Excel, Powerpoint ...) Freezing randomly when doing File / Open or File Save for 8 to 30 second
• Start Menu freezing for 8 to 30 seconds
• ...

Actually, anything that can use the SMB Network stack was randomly freezing on Provisioned devices.

During our tests, we've confirmed that the Master Target (the device used to create the vDisk) was not impacted at all, and that we were able to reproduce only on provisioned target devices.

We've been able to reproduce the issue and found that even if the symptom was an application freezing, the issue was impacting all XenApp / Terminal Server sessions on the Target Device at the same time.

We've also found that the usual SMB tuning tips for XenApp Servers were raising the timeout from 8 to 30 seconds.

The case is now escalated and hopefully we should get an hotfix for it (and I'll update this post as soon as we've validated it).

Fortunately we have able been to find a workaround and as this case is critical for users I'm sharing it (to be honest we just found it )

To explain the problem and the workaround, I'll describe the Architecture we're using to understand the context and see if the workaround could be applied in your implementation.

My customer is using Citrix Provisioning Services 5.1 to deploy Provisioned XenApp Servers.

The screenshot below is describing the Global XenApp / PVS Architecture of the project ( Web Interfaces, DataStore, UPM, EdgeSight components are not present because not concerned).

 

 

To optimize network performances, we've used a multihomed network configuration to separate streaming and production workloads.

As we were relying on two interfaces, we've also ensured that vDisk access and Streaming we separated on the Provisioning Services Servers side.

The screenshot below is describing the conceptual network configuration.

This configuration is very important, because the root cause of the freezes is as far as we've found the Multihoming.

We did not find many documentation about PVS and multihoming, except "Use a dedicated interface when feasible" or using the Configuration Wizard to dedicate an interface for streaming.

During our investigations, we found that the "browstat status" was issuing unexpected results on  Provisioned Devices.

Actually, we could see that the results on the Production vLan were not correct and were binding to the streaming VLan.

Then we decided to search deeper on this issue and tried to disable Microsoft services on the streaming VLAN Interface, as shown on the screenshot below.

 

After testing these settings, we've found that disabling both on the Streaming VLAN Interface was solving our issue and decided to apply this workaround until the appropriate fix will be provided.

Important Note : As this workaround could impact the "Enable Automatic Password Support" option, we've decided to disable it on our Provisioning Servers (in the Provisioning Services Console / server Properties of Each PVS Server) while waiting for the hotfix to be released.