*Words of Wisdom From The Elder

Recently I had to decide which messaging technology I was going to use in an application.  After doing some research on the topic and talking to some fellow developers, it seems others are having the same question I did, which one do you use where.  The answer is of course “it depends”.  Each technology is capable of receiving messages and queing them.  However, they each serve different purposes and one might be overkill over another.

Message Queuing Overview
If you are not familar with messaging in applications that’s OK.  As a developer maybe you’ve never needed to use message queuing in your applications.  More than likely though you have used an application that implemented some type of message queue. The best real world example I can think of where message queues are used would be somewhere like Amazon.Com during Christmas season.  If you have ever placed an order on Amazon.Com you know you do not get an automatic email as soon as you hit the “Place Order” button.  Sometimes an email arrives in 15 minutes, and at other times it may take an 1 hour.   The reason is your order is actually queued up.  When you hit the “Place Order” button a message is sent (probably in XML) to a specific new order queue.  This doesn’t mean that all orders on all web sites are queued up in this similar fashion.  I’ve even written shopping carts that process everything synchronously from start to finish such as:  process credit card, update inventory,  send order to ware house for shipping, email customer, etc.  But for large scale sites message queuing is a must.

There are defineable steps that happen when an order is placed.  If you are a developer and writing a shopping cart, what do you do if you cannot authorize the credit card because the payment processor is down?  Do you tell the customer to come back and try again?  Probably not!  If you stored the order in a message queue, it could simply keep trying to process the order until the payment processor comes back online.  This is a perfect example where messaging in an application can be used.   Let’s not forget congestion on servers either.  In other words if all of sudden your traffic spikes during Christmas rush or because of  a Super Bowl commercial, the more people placing orders, the slower the order process will be if you write things synchronously without message queues.

By sending messages to a queue you do not overload your front-end applications and your backend applications can process the items in a queue as they get to it.  Again, imagine a large ecommerce site that just ran a Super Bowl ad.  People are now placing orders as fast as they can and all of the orders are simply being queued on the backend.  The front-end web site is taking orders as fast as people can enter them.  However, the backend is just taking the next order in the queue and processing it and will continue to do so until the queue empties.  Message queues can be applied to other scenarios as well. 

There are three technologies we can leverage in our .Net applications that support messaging.  Each one serves a different purpose. 

Here is a quick overview of each one:

MSMQ

• Built into Windows and is free
• Primarily recommended for windows to windows applications
• Messages cannot be larger than 4MB in size.  Depending on what type of messages you are queing, this may be a show stopper right there.

Biztalk

• Uses orchestrations to define how a message should be routed
• Great at transforming data from one message to another.
• Built in rules engine
• Hefty learning curve
• Allows system to integrate with other systems

Service Broker

• Provides exactly once, In-order semantics
• Transactional
• Great for data (makes sense since it is apart of the database)
• Built into SQL Server 2005
• Database to database

Recent Service Broker Example
The purpose of this blog post was to get back to my original question, which one do you use where.  Recently I had to tackle this question and ultimately decided the best solution for my application was Service Broker.  It isn’t to say the other two are bad.  Just in this case, I have a really good fit since I am dealing with data.

I have an application that is a Smart Client which uses web services extensively to communicate with the database.  I ran into a situation where I had a very simple message that i needed to send to the web service.  Something like this:

<TransferContacts>
<OriginalOwner>Bob</OriginalOwner>
<NewOwner>Sam</NewOwner>
<TransferType Type=”All” />
</TransferContacts>

This is a very simple XML message.  However, this simple message could mean thousands upon thousands of records in the database have to get updated.  There are audit trails to update, history tracking, etc.  This could obviously take some time to process and I needed to make sure it completed all of the tasks. 

This is where Service Broker comes in.  From the user perspective submitting a transfer request in the application is simple and over at a blink of an eye.  They press a submit transfer request button and the message is sent to the web service and the web service places the message in a transfer request queue within Service Broker.  The user is notified their request has been received and that it will be processed shortly in the user interface.

Service Broker is perfect for this type of message.  As soon as Service Broker sees a message in the queue a stored procedure is called to handle all of the updates.  In my case I used Service Broker because everything is “data driven”.  In other words all the updates are being done in the database.  I am also guaranteed that everything is processed transactionally and in-order. 

Why not MSMQ?
The problem with MSMQ in this case above is that I still have to ultimately call a stored procedure in a database, and make sure everything survives the transaction.  MSMQ could have stored the message perfectly well, but I would be writing more code to process things out of the queue, rather than simply invoking an already written stored procedure.

Why not Biztalk?
In the case against Biztalk, it is just a huge overkill.  I am not transforming data, nor running business rules, nor runnning a long winded orchestration.   If I needed to alert other systems about the transfer, then Biztalk would have been used.  There is a Service Broker Adapter for Biztalk 2006 planned (Jesus Rodriquez mentioned on his blog) so Biztalk and Service Broker can communicate together.

Conclusion
The next time you need to implement message queues in your application be sure you weigh all of the ins and outs of what you are doing before you pick a technology.  You can easily pick by doing this:

Application to Application – MSMQ
System to System – Biztalk
Database to Database – Service Broker

Now of course this may not apply for all of your needs, but it is a good guiding principle to at least start with.  Of course Biztalk can communicate to a database, so it isn’t black and white.  If you have any experience or thoughts on the subject, drop me a line or a comment.