In the previous post I touched on an example of enabling scaling out using .NET caching (System.Web.HttpRuntime.Cache).  Well, a while ago I had the opportunity to use caching again – the result was that with roughly 10 lines of code, the scalability and performance characteristics was greatly enhanced.  The component in question has a long history:

The component is a complex rate lookup system which searches for a scenario that satisfies a certain set of criteria.  When originally designed (7 years ago), the assumption was that the engine would work on a matrix of roughly 30 variables with a possible scenario list of about 500 to 1000 scenarios.  Because of the size of the matrix and some variables which are bound to live data, this was implemented as a SQL stored procedure.  Initial performance was OK, and no need existed for scaling out. (We could handle about 10 requests per second, more than enough at that point)

2 Years on, one of the clients loaded a rate table with 1.6 million scenarios, and this just about killed the SQL machine.  I managed to enhance the performance of the stored procedure by building some intelligence into the selection criteria used for the scenarios – instead of evaluating all 1.2 million scenarios, we were back to about 500 scenarios again. (Performance degraded to about 1 request in 2 minutes, after the change we could handle about 20 requests per second)

Another 2 years later, the volumes picked up – to the point where requests started to kill the SQL server.  Scaling up was done, but architecturally we needed to enable scaling out as well.  At this point we enabled scaling out by adding a layer between the requesting component and the database.  In this layer we could now implement the ability to cache request/response pairs.  When a new request comes in, a bit of logic is used to compare this with the request/response pairs in memory, trying to find a similar request and response in memory before firing a request to the database.  This lowered the load on the SQL machine, and the combination of scaling up and scaling out gave the required performance characteristics. (We could handle about 200 requests per second)

Another 3 years later volumes dictated that we look at the performance again.  The caching implemented in the previous step was local to each requesting process, and it relieved the pressure on the SQL box, but we still saw a lot of the same requests coming from different threads.  With 500 processes requesting values constantly, we needed a central caching ability rather than one which is local to the process.  At this point I implemented the .NET caching on a per server basis.  Due to the structure of the application, the code changes in total resulted in less than 10 lines of code.  The processing is slightly slower than the previous solution (there is an overhead as the cache is not local to the threads/processes doing the request).  It however enhanced our ability to scale out dramatically – the central SQL machine is still the bottleneck, but the ability to share the cache, as well as the ability to have a long lived cache moves a hug eamount of work off the SQL machine, and this more than compensates for the slight overhead of communicating over process boundaries. (After running the solution for a while (and thus populating the cache), we have seen more than 2000 requests per second, with the central machine still doing fine)

The result of all this was a 100 –fold increase in the ability to handle requests, and all it required was a 10 line code change.  How great is that!

What will the future hold for this way of using caching?  Currently there are a couple of products on the market providing caching which can be scaled across server boundaries. My prediction is that we will get a version of this built into the .NET framework pretty soon.  Hopefully before I have my next big bottleneck in above mentioned application.

As a side note: For technical reasons related to the original application, I did not have the option of scaling out the SQL machines: Even thought the rate table data is relatively static, I did not have this level of freedom