Scaling to 30K: Two-level caches

Like everyone else, we use memcached to speed access to frequently used data. Getting started with memcached in Django is as easy as:

from django.core.cache import cache

cache.set('key', 'value')

Add a line to to point to your cache server, and you’re off. As ever, the Django docs are superb.

Memcached is very fast and handles lots of connections with ease. However, as we drove the scaling past 10K requests/sec, we noticed that the cache server holding the active buzzer data was becoming network-bandwidth limited. Since network bandwidth is one of the harder things to scale on EC2, we needed a way to keep the Django web tier from hitting the cache quite so hard.

The solution: implement a two-level cache, just like the one in your CPU. By putting a second, read-only cache on each of the Django boxes, a huge number of reads are serviced locally without hitting the network. Result: dramatically lower loads on the main cache servers, and the load on them only growing with write activity.

The trade-off is that the data sat in the caches on the web boxes can be out of date – and invalidating a cache key across all the caches is non-trivial. But even a very short cache expiry time (we use 1 second) saves a huge amount of reads, while keeping the stale data problem under control.

The code for the two-level cache is available on GitHub.

Using the TwoLayerCache

Using the two layer cache is simple:

  1. Download the code
  2. Put the file in your application
  3. Run memcached on each of your Django boxes
  4. Add the following to your
    LOCAL_CACHE_ADDR = ('',)
  5. Change your code from:
    from django.core.cache import cache


    from dualcache import cache
  6. That’s it.

Other than changing the code to point to the dual-level cache rather than Django’s standard cache, everything should keep working just as-is.

How it works

The basic idea is that any read operation is first tried against the local cache, and if it’s found the value is returned – without contacting the global cache, which is whatever you’ve set up as Django’s cache. Otherwise the global cache is queried and the value(s) or None returned as normal.
Writes and anything which might alter data are intercepted and the local copy deleted, so that future reads will re-fetch from the global cache.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s