*Words of Wisdom From The Elder

There is an extremely nasty bug in WCF (Windows Communication Foundation) that I have discovered as of late.  I know it is a bug because after several weeks with an open support call into Microsoft it was confirmed as a bug.  It was so perplexing because it was hard to replicate.  It even took the .Net Framework developers a week or more to track it down.  Here’s a full blown error message:

An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/2/ROOT

Process ID: 4428

Exception: System.ServiceModel.EndpointNotFoundException

Message: The service ‘~/unittestqueue’ does not exist.

StackTrace:    at System.ServiceModel.ServiceHostingEnvironment.NormalizeVirtualPath(String virtualPath)
   at System.ServiceModel.Channels.MsmqHostedTransportManager.HostedBindingFilter.MatchFound(String host, String name, Boolean isPrivate)
   at System.ServiceModel.Channels.MsmqBindingMonitor.MatchQueue(MatchState state)
   at System.ServiceModel.Channels.MsmqBindingMonitor.ProcessFoundQueues(MessageQueue[] queues, Dictionary`2 knownQueues, Boolean isPrivate)
   at System.ServiceModel.Channels.MsmqBindingMonitor.OnTimer(Object state)
   at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
   at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object o)
   at System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
   at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
   at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
   at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
   at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
   at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Ok, you see the error message above where it says the service ‘~unittestqueue` does not exist?  Well that queue DOES exist on the server but there isn’t a service mapped to that queue. 

The kicker to this is that I’m not even using the unittestqueue at all within my code.  This begs the question, why is a queue that I’m not even using showing up in my error list?  The netMsmqBinding I had configured used another queue called logEntries.  Here is the WCF configuration so you can see how this is configured.

   1: <system.serviceModel>

   2:         <serviceHostingEnvironment>

   3:             <baseAddressPrefixFilters>

   4:                 <add prefix="http://keithelder.net/"/>

   5:             </baseAddressPrefixFilters>

   6:         </serviceHostingEnvironment>

   7:         <client/>

   8:         <bindings>

   9:             <netMsmqBinding>

  10:                 <binding name="msmqBinding">

  11:                     <security mode="None">

  12:                         <transport msmqAuthenticationMode="None" msmqProtectionLevel="None"/>

  13:                         <message clientCredentialType="None"/>

  14:                     </security>

  15:                 </binding>

  16:             </netMsmqBinding>

  17:         </bindings>

  18:         <services>

  19:    <service behaviorConfiguration="LogEntriesBehavior"

  20:     name="KeithElder.MsmqService">

  21:     <endpoint address="net.msmq://keithelder.net/private/LogEntries"

  22:      binding="netMsmqBinding" bindingConfiguration="msmqBinding" name="msmq"

  23:      contract="KeithElder.ILogMsmqService" />

  24:     <endpoint binding="mexHttpBinding" bindingConfiguration="" name="wsdl"

  25:      contract="IMetadataExchange" />

  26:    </service>

  27:   </services>

  28:         <behaviors>

  29:             <serviceBehaviors>

  30:                 <behavior name="LogEntriesBehavior">

  31:                     <serviceMetadata httpGetEnabled="true"/>

  32:                     <serviceDebug includeExceptionDetailInFaults="false"/>

  33:                     <serviceThrottling/>

  34:                 </behavior>

  35:             </serviceBehaviors>

  36:         </behaviors>

  37:     </system.serviceModel>

The endpint is set to net.msmq://keithelder.net/private/LogEntries.  This means that messages received on this endpoint will be stored in the private queue logEntries.  Once stored, the messages call the method written in WCF to take the message from the queue and process it.  Really, this is all slick when it works and opens the door for a lot of scenarios. 

Obviously we aren’t supposed to be getting this error, especially about a queue WE AREN”T EVEN USING!

By the way, when this error occurs, it kicks off the jitdebugger and will slow your machine to a crawl.  It isn’t something fun to debug, trust me.

The Fix

If you have run into this error, and find this entry on my blog somehow, here is the workaround for this.  Again, this is a workaround, not optimal by any means.  There are two things you have to do (at least as far as I can tell with my testing).

Fix #1

Do not have any additional private queues other than the queue required for your service.  Obviously this has problems because other applications may be writing to private queues as well. 

Fix #2

Deploy the application to a virtual directory in IIS.  This is where I first started noticing the problem because I couldn’t get our test server to work when I deployed the code even though it worked on my machine. 

Fix #3 (added 12/3/2008)

Support from Microsoft said you can do one or the other, but I am still getting errors in the event log even though the application is deployed in a virtual directory with two private queues.  If I delete all queues and have it deployed in a virtual directory things are happy.  Changing the endpoint to the name of the path of the service does fix the problem and allows multiple queues to be on the machine.  I’ve tested this extensively.  Here’s an example:

So instead of having the queue named “LogEntries”, changing it to “myvirtualapp/service.svc” fixed the problem and things work normally.

Official Fix

I was told by Microsoft Support this bug would be fixed in the next service pack and also the .Net Framework 4.0. 

I just spent two hours trying to figure out why I couldn’t deploy a Windows Communications Service to my local IIS setup.  I kept getting the famous error:

503 – Service Unavailable

This error is about as helpful as a toothpick in a snow storm.

It turned out my problem was simple to solve, I just wasn’t paying attention. 

I had an application pool setup with my domain credentials so the service could run under my username and access the resources it needed to.  The problem was it had been so long since I  had worked on the application, my password for my domain account had expired.  Thus the application pool wouldn’t start.  I changed the password and things are working now. 

Hopefully this will help someone out there not waste a few hours of work.  I’m off to kick myself for not catching this earlier.