Stephen Kennedy


Software engineer in the Microsoft technology stack, focusing on native and high-scale systems development. Works for Readify.


Company Backed Open Source

What are realistic expectations to have around open source software backed by well known technology companies?

This blog post was originally going to be part of a much larger post discussing the failover gap with the 'full Microsoft Redis'. The details of that can be found in a previous post here. As such while the content focuses largely on Redis it is probably applicable in other areas.

Before I begin

This is not a criticism of Microsoft Open Technologies, StackExchange, Marc Gravell, or any open source contributor. I am immensely grateful for the lengths that these individuals and organizations go to in order to make our lives as developers and technology enthusiasts easier / better.

Also, I've not used it, but NCache but it is a commercial option with open source licensing (and no support). I've conveniently left it out of this post, but it is an option to consider if you are looking at caching technologies on the .Net framework. Likewise, there is a ServiceStack client that looks to have Sentinel support if you are willing to pay.

Branded technology

Most Microsoft developers would appreciate the tooling support that Microsoft offers in its products. Visual Studio is highly regarded and many organisations choose to leverage Microsoft technologies because they allow developers to focus on business solutions. Microsoft has earned a certain level of trust based on their product history.

While Microsoft's Redis is an open source project it is still perceived (subsidary or not) as being a Microsoft produced technology.

As such I think there is a bit of an issue with Microsoft's Redis story and it wasn't immediately obvious.

Out of the box

Most organizations prefer a solution that just work 'out of the box'. (After all in software there is always some level of integration involved)

Microsoft did have a caching solution that largely worked out of the box and it was called AppFabric. But the writing has been on the wall for a while now in regards to AppFabric's future when Microsoft effectively replaced it with Redis on Azure.

Microsoft has recently announced that Windows Server AppFabric is going to end of support

Microsoft has been talking up Redis (and offering it as part of the Azure infrastructure) for a while now on the Azure platform. So the move to Redis on self-hosted platforms for many organizations that leverage Microsoft technologies must seem like an easy choice.

Redis works fine on Azure during a failover as Microsoft's infrastructure is able to handle the node failure. However, if you want to use Microsoft's session state provider (and thus inherit the StackExchange.Redis client) then you need to handle the failover yourself. See my previous blog post for more detail.

The integration breakdown

This issue would not have occured if one of the following occurred.

  • Microsoft's Redis implementation supported Redis 3.0.0. (StackExchange.Redis supports Redis Cluster)
  • The StackExchange.Redis client supported handling Redis Sentinel's pub/sub channel for switch-master messages.
    • The client supports listening to Redis's pub/sub channel but you need to write your own logic to explicitly listen to and handle these messages.
    • In addition Microsoft's Redis session state provider would need to support this functionality as well (assuming it needed to be configuration driven).
  • Microsoft had their own Redis client that supported the Redis Sentinel functionality they themselves have written for Redis.

Okay there are a number of organizations (contributors) involved here.

Marc Gravell (the StackExchange Redis client author) admits it doesn't have good support for Sentinel. His reason if you look at that link and dig for other posts is that StackExchange doesn't use Sentinel so he'd have to spend his own personal time (which he also devotes to other open source projects) and he doesn't feel comfortable enough about Sentinel to commit to it. This is both fair and reasonable.

The version of Redis Microsoft uses on Azure is 2.8.* (Sorry, I'm too lazy to spin up an Azure Redis cluster just to verison check it). So I suspect (no proof) that they are using some variant of the Microsoft Open Technologies version of Redis and 2.8 is enough for Azure. That is, at least, until customers start wanting sharding on Redis clusters.

For some, open source is journey

Certain organizations shy away from open source software. They don't completely trust it, and this is despite having full access to the source code. They don't want to be ultimately responsible / accountable if there any bugs or issues that arise from using that software. This is why even when products (such as NCache) are available and free for commercial use some organizations will pay often at times expensive licensing fees just for that support.

Large organizations such as Microsoft, GitHub, and Facebook are immensely important when it comes to getting the wider software community behind open source software.

That being said, many organizations assume that they are going to get the same level of full 'out of the box' solution that they might get for a licensed product from these vendors. And when that doesn't happen it can:

  • Damage the vendors brand.
  • Be used as an excuse to avoid open source software.

Now, I'm not saying that this is fair or right, but this something that does happen.

But you're missing the point of open source

No, I get it. The proper resolution to this problem is that either myself or the client that I am working for write a solution and submit a pull request. If I can find the time to write a more generic solution (in C#) that Marc Gravell (StackExchange.Redis author) is comfortable with accepting I just might do that.

TLDR

  • Each of the different organizations involved in this developed a solution to meet their needs, but there is a failover gap on Windows Redis with the Microsoft Redis session state provider (with a dependency on the StackExchange.Redis client).
  • The large technology companies are immensely important when it comes to getting organizations over the line with open source. I know I've seen a massive sea change in software practices since Microsoft became more open source friendly. (Note: Obviously they are not the only vendor doing this)
  • Expectations around open source software can and should be different from commercial products. But, rightly or wrongly, when a big vendor is behind an open source technology and it doesn't meet the expections of an organization it can be used as an excuse against open source.
  • When you effectively deprecate a technology in favour of another, you should ensure that all major features of the old technology retain parity in some way with the new, and that includes failover.