So this is one of my pet topics: I have been in the IT industry for near to two decades.  Started out working with some assembler, wrote some low level transport layers on networks, worked with a variety of operating systems and at some point did some open source stuff on Linux , then worked with database systems etc.

At some point I entered the Microsoft world: I became and MCSD and MCSE around 1996, and at that point I actually knew all applications/ software offered by Microsoft at an expert level.   The combination off all of this knowledge and experience enabled me to architect enterprise level, high performance, high scalability applications.

Today (13 years later) it is not possible to be an expert on all Microsoft software anymore (and my guess is that goes for other vendors as well – in fact, in my opinion it is not even possible to be an expert on a single tool such as SQL Server anymore.  You need to choose a field within SQL Server, and become an expert on that.  SQL Server is just an example, this is true for most areas of the total IT industry.  An exponential increase in technology, complexity, options etc. is making it near impossible to keep up. 

A concern I have is: How do new guys/graduates catch up?  Is it still possible for a graduate to start as a junior developer and work his way up to become a good systems architect?  It is pretty much impossible to keep up as well as catch up detailed knowledge about all the layers of abstractions that senior resources have aggregated over years and years.  The only workable solution seems to be:

• You need to have a base knowledge about the abstraction layers you are working with
• You need to trust these abstractions
• This will enable you to focus on keeping up

This sounds workable, but again my critical mind is not happy.  Sure, I will entrust my “Hello World” application to someone who has this level of knowledge (And I will probably get a much better user experience than anything I can ever develop or even image ;-) ).  Will I trust this guy when it comes to mission critical, high performance, highly scalable applications?  I don’t think so!

The interesting thing about this is: I attended a session presented by Rockford Lhotka at Tech-Ed, Barcelona recently, where Rocky was speaking about “How to Manage Technology And Not have it Manage You”.  The interesting thing about the audience was that we were all mostly “seasoned IT staff”  ;-).  The above concern seemed to be a shared by the whole group. I take this as a sign that I’m not paranoid, and that the above really is an issue.

Are their any solutions/remedies to this situation?  No silver bullet available I’m afraid.  But I suggest the following:

• Junior resources should be encouraged to pick up a broad level of experience before starting to specialise in a certain technology/ architecture/ type of application.
• Junior guys that want to move on to become system architects must make it their mission to understand the mind of the guys who designed the systems they work with.  In all applications there are numerous trade-offs and design decisions – Make an effort to understand the thought processes and motivations behind these decisions. Work with stuff which turned out great, as well as things which turned out to be not so great
• Senior guys – take care when creating your API set/ abstraction/reusable module  - note that someone will at some point trust that whatever you created is doing the best job possible.

In the previous post I touched on an example of enabling scaling out using .NET caching (System.Web.HttpRuntime.Cache).  Well, a while ago I had the opportunity to use caching again – the result was that with roughly 10 lines of code, the scalability and performance characteristics was greatly enhanced.  The component in question has a long history:

The component is a complex rate lookup system which searches for a scenario that satisfies a certain set of criteria.  When originally designed (7 years ago), the assumption was that the engine would work on a matrix of roughly 30 variables with a possible scenario list of about 500 to 1000 scenarios.  Because of the size of the matrix and some variables which are bound to live data, this was implemented as a SQL stored procedure.  Initial performance was OK, and no need existed for scaling out. (We could handle about 10 requests per second, more than enough at that point)

2 Years on, one of the clients loaded a rate table with 1.6 million scenarios, and this just about killed the SQL machine.  I managed to enhance the performance of the stored procedure by building some intelligence into the selection criteria used for the scenarios – instead of evaluating all 1.2 million scenarios, we were back to about 500 scenarios again. (Performance degraded to about 1 request in 2 minutes, after the change we could handle about 20 requests per second)

Another 2 years later, the volumes picked up – to the point where requests started to kill the SQL server.  Scaling up was done, but architecturally we needed to enable scaling out as well.  At this point we enabled scaling out by adding a layer between the requesting component and the database.  In this layer we could now implement the ability to cache request/response pairs.  When a new request comes in, a bit of logic is used to compare this with the request/response pairs in memory, trying to find a similar request and response in memory before firing a request to the database.  This lowered the load on the SQL machine, and the combination of scaling up and scaling out gave the required performance characteristics. (We could handle about 200 requests per second)

Another 3 years later volumes dictated that we look at the performance again.  The caching implemented in the previous step was local to each requesting process, and it relieved the pressure on the SQL box, but we still saw a lot of the same requests coming from different threads.  With 500 processes requesting values constantly, we needed a central caching ability rather than one which is local to the process.  At this point I implemented the .NET caching on a per server basis.  Due to the structure of the application, the code changes in total resulted in less than 10 lines of code.  The processing is slightly slower than the previous solution (there is an overhead as the cache is not local to the threads/processes doing the request).  It however enhanced our ability to scale out dramatically – the central SQL machine is still the bottleneck, but the ability to share the cache, as well as the ability to have a long lived cache moves a hug eamount of work off the SQL machine, and this more than compensates for the slight overhead of communicating over process boundaries. (After running the solution for a while (and thus populating the cache), we have seen more than 2000 requests per second, with the central machine still doing fine)

The result of all this was a 100 –fold increase in the ability to handle requests, and all it required was a 10 line code change.  How great is that!

What will the future hold for this way of using caching?  Currently there are a couple of products on the market providing caching which can be scaled across server boundaries. My prediction is that we will get a version of this built into the .NET framework pretty soon.  Hopefully before I have my next big bottleneck in above mentioned application.

As a side note: For technical reasons related to the original application, I did not have the option of scaling out the SQL machines: Even thought the rate table data is relatively static, I did not have this level of freedom