The other day I stumbled upon what I thought was an interesting problem, with an accompanying illustrative solution. The problem at hand is related to concurrency, which seems to be an inevitable problem when working with message bases solutions. I will describe the problem, and a solution that I found appealing, utilizing lock and ConcurrentDictionary . The examples are invented for this post in specific, and does not reflect a real world system.

The system at hand is message based, there are message handlers (singletons) that handle messages using multiple threads. One handler could e.g. create an order. The system will handle n (= number of threads) messages at a time, creating orders based on a given id.

A handler could look something like this:

 public class Handler
{
    public DateTime StartedAt;
    public int TimeStamp => (DateTime.Now - StartedAt).Seconds;

    internal void Handle(int key)
    {
        WriteLine($"[{TimeStamp}] {ThreadId()} creating order {key}");
        CreateOrder(key);
    }
}

Because of concurrency and a missing unique constraint in the database, the same order could be created multiple times. The CreateOrder-method would typically first check if an order with the same key exists and if so reply with a different message or simply return successfully. The problem is when executing in parallel, two threads might get a negative result on the same id, thus both creating the order successfully. This can be illustrated by executing the handler using the following program (numbers in brackets stating the relative time of execution):

 
class Program
{
    static void Main(string[] args)
    {
        var workflow = new Handler { StartedAt = DateTime.Now };
        Parallel.Invoke(
            () => workflow.Handle(1),
            () => workflow.Handle(1),
            () => workflow.Handle(2),
            () => workflow.Handle(1));
        ReadLine();
    }
}

[0] 3 creating order 1
[0] 6 creating order 1
[0] 1 creating order 1
[0] 4 creating order 2

Here we see that 3 threads are creating the same order at the same time. One possible solution to this would be some kind of constraint in the database, which would make the duplicate messages fail and retry. On the second attempt the first thread would have created the order, and the "exists"-check would return positive. But let's say that you are not able to enforce such changes in the database, or that the infrastructure is different and does not provide automatic retries.

The solution then will have to be locking on application level. This could be achieved using C# lock-keyword as follows:

 
public class Handler
{
    public DateTime StartedAt;
    public int TimeStamp => (DateTime.Now - StartedAt).Seconds;
    private static object _lockObject = new Object();

    internal void Handle(int key)
    {
        lock (_lockObject)
        {
            WriteLine($"[{TimeStamp}] {ThreadId()} creating order {key}");
            CreateOrder(key);
        }
    }
}

This would produce the following output from the previous program:

[0] 3 creating order 1
[5] 1 creating order 1
[10] 4 creating order 2
[15] 5 creating order 1

Now we see that only one thread at a time is allowed to do work. Problem now is that we have limited the program to one thread, which would hit performance hard. What we need is a lock that only locks a specific id from being worked on at any time, while utilizing all threads.

 
public class Handler
{
    public DateTime StartedAt;
    public int TimeStamp => (DateTime.Now - StartedAt).Seconds;
    static ConcurrentDictionary<int, object> LockObjects = new ConcurrentDictionary<int, object>();

    internal void Handle(int key)
    {
        var lockObject = LockObjects.GetOrAdd(key, new object());
        lock (lockObject)
        {
            try
            {
                WriteLine($"[{TimeStamp}] {ThreadId()} creating {key}");
                CreateOrder(key);
            }
            finally
            {
                LockObjects.TryRemove(key, out _);
            }
        }
    }
}

Now the first thread will add an object in the dictionary and lock it. The next thread will get the same object and try to lock it, as it is already locked the thread will block until the first thread releases its lock. The third thread are handling on a different key, and will thus not be affected by the first two threads. We now have the desired behavior with different orders being created in parallel, but never more than one thread working on the same order at the same time.

[0] 5 creating 2
[0] 1 creating 1
[5] 3 creating 1
[10] 4 creating 1 

We see that two threads start work at different orders from the beginning, while the duplicates creating order 1, waits for the previous messages to be done. It is important to note that for this to work, ConcurrentDictionary must be used, allowing each thread to utilize GetOrAdd, when locking, since we do not know if there already is a object for the given id present. Same when removing the lock object in the end, it might already be removed be another concurrent thread.

NB: If the message handlers are run in a transaction, this solution might not be sufficient, since the database update will not be committed until the handler returns, leaving a short window after releasing the lock where another thread might create the same order. This would be solved by running the "update"-code in a separate transaction scope committing before releasing the lock.