Blog: Citrix

We had an issue come up with a group of Citrix Presentation Servers that were recently introduced as a pilot for one of our customers. All the Citrix servers are newly built using mostly application defaults. Our first application silo contained routine applications like Office, IE, and file share resources. Setup and testing went very smooth. However, the second silo we built published a core application that was launched using a batch file. Testing of the application by our network engineers and by the customer’s head of IT raised no concerns. Everything worked as expected. The application was published to a small set of regular users chosen as a pilot group and we were quickly alerted that none of the users could launch the application. Here is the error that was displayed: [more]

“The desktop you are trying to open is currently available only to administrators”

We were aware of some ongoing Terminal Server licensing issues that this customer was experiencing so we automatically assumed that was the problem. We scrambled to put another Windows 2003 server on the network to run TS licensing thinking it was only a matter of time before the other Citrix servers started having issues. By the end of the day, we had the new TS licensing server online and all the settings deployed via group policy to the problematic servers. A reboot of the servers would fix it…not so much. After the reboot, the issue persisted. Back to square one. After digging through all the settings a few times, I finally found the following setting:

On the surface this setting doesn’t look like it would cause any issues. After all, we were using a published application and it looks like this setting would only prevent users from opening a published desktop on the server. However, after doing some testing, we found that the ICA protocol doesn’t consider launching a batch file to be a published application. Only files with .exe extensions count. Our testing failed to uncover this issue because we had unintentionally done all of our testing with administrator accounts and all the applications published on the first silo were standard .exe apps. Unchecking this box fixed the problem.

As part of a Citrix environment overhaul, another network engineer and I discovered a very frustrating limitation of using group policy with Citrix published applications. The problem centers around the inability to apply IE group policy settings using loopback mode processing. I'll warn you ahead of time, there are lots of details so hang with me....and remember this is all going to converge at the application of group policy. Here is what we found...

When a user with an empty roaming profile (new user) has their profile created as the result of running a published application, the user portion of the registry hive (ntuser.dat) is not created in its entirety. The users' hive can be loaded and a number of noticeable differences exist between it and the default user registry hive. If the user profile is created by logging on locally (console), via RDP to the same machine, or via Citrix published desktop on the same machine, the profile that is created is complete. I was unable to find any noticeable differences between the default user registry hive and that of the newly created roaming user profile when the profile was created in this way. Additionally, once an incomplete profile had been created via published application session, the profile could NOT be "fixed" by logging on via RDP or published desktop. Once the registry hive was created in an incomplete fashion, it seemed to be affected from then on. So why are we talking profiles...I thought this was about group policy? Well, it is...I'm getting there. [more]

We found that users running published applications did not have group policy correctly applied. We were trying to set policies on Internet Explorer using Internet Control Panel settings in the user portion of the GPO. Specifically, IE security zone settings such as trusted and intranet sites would not apply. We also noticed that each security zone seemed to be locked. In the Security tab of the Internet Options dialog box, all the icons were the same....blue IE symbol with a lock next to it. The "Sites" button and the "Custom Level" button were also grayed out. So, here is the where the profile problem merges with the group policy problem. I found that by manually exporting certain keys from the default user profile registry hive under \Software\Microsoft\Windows\CurrentVersion\Internet Settings\ and importing them into in a incomplete user registry hive, I could fix the problem. That is, once the keys existed in the user registry hive that pertained to the settings I was trying to set via group policy, the policy was applied correctly...no issues. Makes sense right....if the group policy is setting registry keys in order to apply certain policies, it’s not going to work if the keys don't exist in the first place.

So things have come full circle. Group policy isn't working because the user profile is messed up. So why is the user profile not getting created correctly? Well, this is actually a Microsoft problem --> http://support.microsoft.com/kb/899270. And the script they provide doesn’t work…we tried it. Actually, there is more to the problem than that, but here is a summary of the information that we gathered. By design, Citrix published applications, remote applications in Windows 2008, and the "start this application on connection" functionality of RDP (mstsc.exe) running against Windows 2003 servers implement limited logon functionality so that the session footprint is smaller than a normal desktop session. Part of the "limited functionality" is that the user session does not start explorer.exe. So, any application that depends wholly or in part on explorer.exe could have issues. Some of the important pieces of functionality that explorer.exe implements are the following:

1. The run registry entry
2. The RunOne registry entry
3. Startup applications 

If you have ever noticed the small gray box that is displayed the first time you log on as a new user, you have seen the effects of explorer.exe running at session logon. It goes by fast, but it says something like "applying internet explorer customizations", "setting up windows media player..."...stuff like that. That little box is normally initiated by explorer.exe. It is called runonce.exe. What we found was that if we initiated runonce.exe in a logon script, the user was created correctly when running published application; thus, group policy was applied correctly as well. Testing also showed that this process could also fix a previously created broken user registry hive (ntuser.dat). All we had to do is add the following to our logon.bat file

start /MIN %windir%\system32\runonce.exe /AlternateShellStartup

Citrix has documented this problem in a support article (http://support.citrix.com/article/CTX104374) and they refer back to the previous MS KB listed above. Numerous forums threads exist on this issue and we were unable to find a resolution elsewhere that did not include scripting registry imports to the user profile at logon. This workaround seems to be a more flexible and reliable.

One of our new customers using VMware has not been happy with the performance of some of their virtual machines that had been setup before they were our client. Specifically, a couple of their Citrix VMs and a SQL Server 2005 VM have been “sluggish since they were built” according to the IT staff. I did some basic diagnostics on the SQL Server VM and it did seem to have some performance problems. However, since they had already bought a new physical server and started moving the databases to the new install we didn’t spend much time trying to make the VM run well. We decided to upgrade to VMware vSphere v4.1 upgrade before attempting to address any of the performance issues, since less than stellar performance on virtual terminal servers was normal on VMware v3.5.

During the upgrade, I needed to vMotion some of the VMs around to take down one of the ESX hosts. I kept getting a very generic error on several of the VMs and the migration would fail. I lmust have ooked at every setting a dozen times until I finally just shut the VM down and opened up the VMX file to see what might be causing the issue. The problem was that the person who had built the VMs originally had included processor affinity settings in the VMX file. This binds a VM to a specific subset of the physical processors/cores on the ESX hosts. For example, on the SQL Server VM, it was bound to cores 0 & 1. With this setting, ESX was forced to schedule core 0 & core 1 for all operations even though the server had 8 cores. Additionally, on ESX, the service console has processor affinity on cores 0 & 1, but it holds the highest priority. So basically the SQL Server VM (and the other VM I found that was co-scheduled on cores 0 & 1) were fighting with the service console for processor cycles. After removing the processor affinity, the CPU wait time counter in vCenter for that VM dropped 6x. I ended up finding 10-12 VMs with processor affinity set, so that explained why the performance was terrible. 

The moral of this story is to not manually schedule the processors when you configure ESX or Virtual Machines.  Chances are the ESX schedule will be much better than any manual configuration you could put together.