I recenty ran across this little gem:

[error] SSL Library Error 47 on pm.domain.com:443 with peer 10.20.30.40: An unclassified SSL network error occurred. (error code: error:1408A0C1:lib(20):func(138):reason(193))
[warn] SSL handshake from client failed

Well… now isn’t that just so helpful.  After verifying and re-verifying my certificate setup I wasn’t getting anywhere.  I ended up opening the httpd.conf file that the Citrix XTE service depends on and noticed a line near the bottom:

 #Allowed CipherSuite (ALL,COM,GOV)
SSLCipherSuite GOV

I didn’t go to the trouble to lookup what ‘exactly’ this means but I theorized that I don’t really want to limit what kind of SSL cipher I can use so why not change it to read:

 SSLCipherSuite ALL

After making that simple, three letter change, I was now able to hit the Citrix Password Manager service URL from a  web browser and I saw the SSL Error 47 messages in my error.log dissappear.

Hopefully this will save someone else the hours of troubleshooting that I spent on this one.

I should point out that this application had been working well from within the LAN when users were connecting with ICA from a WYSE ThinOS based terminal.  However when connecting from a Windows based workstation using a XenApp or XenDesktop plugin the performance suffered.  My first thought was something was different about the way in which the terminal handled multimedia, SpeedScreen, HDX, etc.

Thinking that I could reproduce this I started down the path of trying older clients until I found one that worked.  Then I could read the release notes and try to figure out exactly what was or wasn’t needed to make this application perform well.  Aside from taking a trip down memory lane by installing some pretty old clients I was never able to make the application perform well from the Windows based workstation.

Now I know that disabling HDX Multimedia Acceleration is just a check box away in XenApp, the same cannot be said for XenDesktop.  The console for XenDesktop 3 does not have anywhere near the number of configuration items as its close relative XenApp 5 does.

Only in the release notes for an updated Virtual Desktop Agent (CTX122963) did I finally find my answer.

On the virtual desktop:


Set the following DWORD to false

HKLM\SOFTWARE\Citrix\ICA Client\Engine\Lockdown Profiles\All Regions\Lockdown\Virtual Channels\Multimedia\SpeedScreenMMA

On the Desktop Delivery Controller:

1. At a command prompt, cd to C:\Program Files\Citrix\Desktop Delivery Controller.
2. Execute PorticaSetDefaults.exe /o <filename.xml> to create an editable settings XML file.
3. Open the xml file created in the previous step and locate the SpeedScreen Multimedia section.
4. Change the entry for <Enabled>true</Enabled> (default) of the SpeedScreen Multimedia section to <Enabled>false</Enabled>.
5. Save your changes.
6. Reimport the settings using Porticasetdefaults /i. The settings will take effect the next time a Virtual Desktop Controller registers against this Desktop Delivery Controller.
7. Repeat the previous step on every Desktop Delivery Controller that conducts VDA registrations.

Now from a single DVD you can install XenApp 5.0 with HFRU5, Password Manager, EdgeSight, SmartAuditor, Web Interface, etc.  It made my upgrade process so easy all I had to do was run through the wizard select the components I wanted to upgrade and after a reboot all was done.

Even on a new installation I was able to save quite a bit of time as all of the components for Password Manager and EdgeSight were installed as a part of the overall XenApp installation.

Great job Citrix on getting all of this stuff together in one place!

There are two multipathing drivers that have been available since XenServer 5.0 SP2.  They are the DMP (dynamic multipath) and MPP (multipath proxy).  DMP is the one that is used by default.  In fact when you go into XenCenter and enable the check box for ‘multipathing’ you are essentially turning on DMP.  Normally this works well but the IBM DS4800 and other storage controllers that use RDAC (redundant disk array controller) need a little more.

If you just go through and enable the DMP driver you’ll be able to see, format, and use the storage from the DS4800… but what you’ll eventually notice is that the LUN will continually failover to another path using a different storage controller.  This is called thrashing and its not a good thing.

To stop the thrashing you’ll need to do one of two things:  1) tell the DS4800 to disallow AVT (automatic volume transfer) or 2) tell the XenServer to pick a path and stick with it until a failover is necessary (i.e. STOP THE ROUND ROBIN)

The first option of disabling AVT is actually the easier of the two and is done on the storage controller itself.  Don’t worry you can set this by LUN or even by host so you won’t be in danger of tampering with your other SAN attached systems.  The way to do this in the IBM DS4800 is to change the host type to something that doesn’t perform AVT.  The host type of Linux VMware Cluster or LNXCLUVMWARE should do the trick.

The second method of chaning the driver over from DMP to MPP is not as easy.  If you go out and do some searches on the matter you’ll find that Citrix states in CTX121364 ”This is the real RDAC driver supported by IBM and other vendors” about the MPP driver.  What you won’t find in that article are steps to enable and actually use the MPP driver.  Your searching might even land you at CTX118791 where you might find specific reference to the IBM DS4xxx.  But once again you’ll be hard pressed to actually find out how to get this thing working.

I recently spoke with the XenServer support team and got some instructios on how to enable the MPP driver in place of the DMP driver.  Its not an official CTX article yet but my understanding is that they are working on it.  Even when it does get released it won’t be for the faint of heart.  I should also point out that the cute little “4 of 4 paths active’ notification that you see in XenCenter ONLY works with DMP so once you switch over to MPP you’ll lose that information.

Changing the host type on the storage controller is definately the easiest and it will allow you to continue to see the multipathing information in XenCenter.  But it definately seems to struggle with performance.  I think this is largely due to the fact that XenServer is still trying to use round robin multipathing but the IBM DS4800 is not allowing access to the LUN through one of the controllers… meaning up to 50% of your I/O requests from XenServer will not be fullfilled.

I have to admit, this really isn’t a new trend in the world of certifications. And to be fair you can’t really expect an exam and corresponding certification to roll out of the door the same day products go RTM.

My best example of this is the exact certification program that I mentioned earlier…  the CCEA/CCEE program.  The current CCEA is based on the Citrix Access Suite 4.0 which includes Access Gateway 4.2, Password Manager 4.1, Presentation Server 4.0, and Web Interface 4.2.  Seriously?  I don’t know about the rest of the components of this bunch but I can tell you that Presentation Server 4.0 will be end of life and end of support by the end of the calendar year 2009!  And it is STILL the most current engineer level certification that Citrix offers!

In defense of Citrix, they have announced and released almost all of the bits and pieces for the successor of the CCEA.. the CCEE program.  At the time of writing all but one exam are available for this certification with the final exam not scheduled to be released until the fourth quarter of 2009.  No big deal right?  Well…  except that nearly every single product that will be in the ‘final’ exam will already be at least one revision behind!

The final CCEE exam has been rumored to be similar to the final exam in the CCEA program where candidates will be expected to perform a series of tasks in addition to answering multiple choice questions.  It is rumored that the exam will cover the following products:  Provisioning Server 5.0, XenApp 5.0 for Windows Server 2008, XenDesktop 2.0, and XenServer 5.0.

The problem?  Today the current releases are:  Provisioning Server 5.1, XenApp 5.0 for Windows Server 2008, XenDesktop 3.0, and XenServer 5.5.  The test hasn’t even been relased and it is already rumored to be based on older technology.

So I pose the question again… Can Citrix certifications (or any vendor for that matter) keep up with product releases?

I completely agree that licensing can be a real pain sometimes especially when there are frequent changes as there seem to have been lately with  Citrix XenApp licensing.  I would love to post a simple listing of what products are included with the XenApp Platinum license… but unfortunately it just isn’t that simple.  It turns out that for some products like EdgeSight portions of the product are included with Advanced and Enterprise with the all of the individual product features being exposed only in the Platinum license.

OK so I admit that I may not have cleared up any licensing mysteries here, but that wasn’t my goal.  On with the show.  Here are what I feel are the most often overlooked features or products in the XenApp Platinum suite.

Citrix EdgeSight for Load Testing- If you can honestly tell me that you have never had to answer the question “how many users can I expect to get on a single XenApp server” then I would say you haven’t built too many XenApp environments.  Citrix EdgeSight for Load Testing is an amazing product that allows you to record and replay interactions with XenApp published applications.  Once you have your set of instructions compiled you can then setup a test by replaying those instructions over and over again until you find the breaking points or performance thresholds you are looking for.

Citrix Password Manager- Quite possibly my favorite Citrix product   I use Password Manager EVERYDAY for single sign-on to what feels like about 1000 different applications.  Password Manager as you may have guessed is a single sign-on product that also has self service password functionality.  I find that right out of the box (without complex scripting) I have a very high success rate with configuring applications for single sign-on.  Furthermore, Citrix has expanded the licensing terms of Password Manger to extend its reach to both desktop and offline users!  If you are constantly setting, forgetting, and resetting password on websites you NEED to try Password Manager.

Citrix SmartAuditor- If you have ever worked in any type of support role you know how difficult it can be to diagnose or troubleshoot a problem without some vision into what the end user is seeing.  SmartAuditor can address this problem area by recording sessions and allowing for storage and playback at a later date.  SmartAuditor may also help to address some compliance issues you may have because you can now have a complete visual representation for auditing instead of combing through logs from multiple systems.  I do have to say that I really liked the older name of this product:  Project IRIS (ICA Record ICA Surveillance).

I should point out that I purposely left out the XenApp Platinum features that require additional appliances.  I should also point out that EdgeSight for Load Testing is actually available to XenApp Enterprise and XenApp Platinum customers.

Ok so there are more than just a few problems…  Basically Hot Desktop doesn’t work with Password Manager Agents 4.6.265 and 4.6.266.

When using either one of these two agents you will find that Hot Desktop installs without incident but the process that executes the session.xml functions only appears to be triggered on the first logon after a reboot.  That’s a BIG problem because the whole idea behind Hot Desktop is the fast switching from one user account to another!  So if you have to reboot each time a user logs off just to prepare the system for the next user then you probably aren’t shaving very much time off of the logon process.

The good news is that Password Manager Agent 4.6.264 DOES work properly with Hot Desktop.  The bad news… it doesn’t support the new XenApp Platinum licensing model for desktop and offline usage.  As it turns out this isn’t that big of a problem because it’s not very likely that you would need the Hot Desktop functionality in a disconnected or offline mode.

I briefly spoke with Citrix support and they verbally acknowledged that in fact they were aware the functionality was missing or broken in the two latest agents.  I didn’t get a very clear sense of when or if a fix would be available but I did hear that a new release of Password Manager might be on the horizon.

Some of my biggest complaints about XenServer when compared to VMware ESX was that XenServer was lacking some of the features that have become critical to the successful adoption and implementation of a hypervisor.  Specifically, XenServer has been missing dynamic load balancing  and snapshot capabilities.  Also included in XenServer 5.5 is support for Active Directory authentication through XenCenter.  You can read the full list of new features in the release notes found here:  http://support.citrix.com/product/xens/v5.5/

Now it would be easy to argue that all of these features have been available from VMware since the inception of VI3 through ESX 3.0 and VirtualCenter 2.0 which was originally released in June of 2006!

So is this just Citrix playing catch up?  Are they really three years behind VMware?

You can easily jump over to the Citrix http://support.citrix.com/product/provsvr/psv5.1 to review the entire listing of new features but I thought I would point out a few of the features that I am most excited about.

• NIC Teaming – this is huge for environments that are taking advantage of Provisioning to physical hardware for things like XenApp servers as you can now add some redundancy at the network level.
• Offline Database Support – similar to the local host cache functionality in a XenApp environment, Provisioning Server can now survive even if connectivity to the SQL database is lost.
• Windows XP Embedded Support – streaming an operating system to a thin client… what could be better?  I can’t wait to make my own Ardence video with a room full of thin clients booting into Windows XP, Vista, 7, and Red Hat!

There are still lots of other features that have come out with this new release like the offline support with Hyper-V, user assigned vDisks, and the ability to store multiple partitions within a single vDisk.

One good bit of news is that current CCEA and CCIA holders will be able to take fewer tests in order to upgrade their certifications.  To my knowledge this is the first time that Citrix has offered an upgrade path for their higher level certifications.  Current CCEA’s will only have to take one exam!  Current CCIA’s will have to take two exams but note that one of these two will also earn the CCEE!

Also of note is that Citrix has announced two tracks the more traditional track is now known as the Virtualization path while the newer networking products will be grouped into the Application Networking path.  Only the requirements for the Virtualization path have been released but look for the Application Networking path details to come out later in the year.

New CCEE candidates will have to take a minimum of five exams that encompass the XenApp, XenDesktop, and XenServer product lines.  Gone are the Access Gateway and Password Manager exams that were requirements for the previous version of the CCEA.  Not gone, however, is the practical simulation exam that will be the stumbling block for those wishing to breeze through the program.

Rumor has it that the simulation exam will contain somewhere around 20-30 simulation items and another 10-20 multiple choice questions.  Rumors are also circulating that the content will be based on three major technologies but will be much more balanced with a bulk of the exam focusing on XenApp.