Redis

Redis is an open-source, networked, in-memory, key-value data store with optional durability, written in ANSI C.

MediaWiki & Wikimedia use cases for Redis

Session storage

The Wikimedia Foundation has been using Redis as a memcached replacement for session storage since the eqiad switchover in January, because it has a replication feature which can be used to synchronise data between the two data centres. It allowed us to switch from Tampa to Ashburn without logging everyone out.

Job queue

We previously stored the MW job queue in MySQL. This gave us lots of useful features, like replication and indexing for duplicate removal, but it has often been hard to manage the performance implications of the high insert rate. Among its many features, Redis embeds a Lua interpreter on the server side. The new Redis job queue class provides a rich feature set superior to the MySQL job queue, mainly through several server-side Lua scripts which provide high-level job queue functions. Redis is also used to keep a hash table that tracks which job queues actually have jobs, so runners know where to look. Updates to this table are push-based, so it is always up-to-date.

Features

Extension:GettingStarted's early implementation of a category-based recommender system has used Redis to store a list of tasks served via a Special page.

Further reading

General

• Official site (see esp. Introduction to Redis)
• The Redis article on the English Wikipedia.
• Redis/INCR
• Getting to Know Redis (PDF)
• Redis, from the Ground Up
• Redis and Relational Data
• Redis Cookbook (book; not great, but see ch. "Analytics and Time-Based Data")
• Interview with Salvatore Sanfilippo (code-oriented but still useful)
• Redis DB (Google Group)

Analytics

• Redis at Disqus (their entire analytics platform runs on Redis)
• Effective Web App Analytics with Redis
• How YouPorn uses Redis (video)
• Realtime metrics using Redis bitmaps

Tooling

• redis-py is the library of choice for Python
• Getting Started: Redis and Python
• Redis and Python (presentation slides)
• Resque for jobs
• Redisco, a Python ORM for Redis
• py-analytics (I haven't used this)

Informed Opinions

• Antirez: You Need To Think In Terms Of Organizing Your Data For Fetching

Miscellaneous

• Storing hundreds of millions of simple key-value pairs (how Instagram uses Redis)