As part of the pre-Christmas push we rolled out a series of infrastructure improvements including both new additional servers and new servers to replace old ones. One set of servers to be upgraded were our cache servers. These have two roles day-to-day, firstly they’re our Akamai (CDN) origin servers and secondly they store our primary memcached instances.
Each server has multiple instances of memcached running on different ports. There are multiple sets for both the “data” and “view” segments. Each of the new servers has 64GB RAM.
The main challenges when planning to migrate the data is mitigating the impact on the platform. memcached is a core element. It’s used to store HTML fragment caches and various parts of both site and member data.
There is a tight line between getting the migration completed versus keeping the application as quick as possible during the whole procedure.
There are a few approaches which are briefly outlined below:
- Big Bang. Update all the configuration files and simply push that code live. This has a huge hit on performance as all the new instances are cold, it is however the quickest option.
- Migration instance by instance. Each instance would be moved and relevant configuration files pushed live. This way only 1 or X instances is “cold”. This limits impact, but, also takes time as only 1 instance can be done at any one time.
- Write to both. The application code could be altered to make all writes to both the existing set of instances and the new. This isn’t great as we’d need a full QA cycle to validate the code works. It’s also something we’d have to implement then pull back out down the line.
- Mirror traffic. Similar to the above, but, this time lower down the stack. Essentially duplicate all the TCP level traffic so it warms in parallel and keeps both instances in sync meaning new writes, deletes occur on both existing and new sets.
In the end the best solution which minimised the impact to the platform
was to mirror all the traffic to the existing instances across to the
tcpcopy helped us to get this done. As well as migrating
tcpcopy could be used to load test, stress test and benchmark.
The plan involved running the
tcpcopy for 24 hours, see how full the new
instances reached then decided if another 24 hours should elapse before
the code changes to point at new instances are deployed.
There would be some data “missing” however all freqently used data keys, core and shared information will all exist and as such mitigate the performance concerns.
tcpcopy is pretty straight forward. Assuming all
requirements are in place:
wget --no-check-certificate \ https://github.com/downloads/wangbin579/tcpcopy/tcpcopy-0.6.5.tar.gz && \ tar xvfz tcpcopy-0.6.5.tar.gz && cd tcpcopy-0.6.5 && \ ./configure && make && make install
Once that’s installed on your target server (new cache servers) you’ll need to setup the iptables rules. Note there are a couple of extra’s here to ensure that connections from the localhost and from the office IP were not queued to allowing monitoring.
modprobe ip_queue; iptables -A OUTPUT -o lo -j ACCEPT; iptables -A OUTPUT -d 220.127.116.11/32 -j ACCEPT; iptables -A OUTPUT -p tcp --match multiport --sports 11211:11212 -j QUEUE;
You can then fire up the intercepter, the
-d option backgrounds the
process as a daemon.
Next you’ll need to head over to the source servers. In our case this is the existing cache servers.
tcpcopy -d -x 11211-192.168.2.184:11211 tcpcopy -d -x 11212-192.168.2.184:11212
The format of the copy is
localport-targetip:targetport. You can
tweak with more options as well, see the manual for
full details. You can also include logging if needed by using
In order to see how the copies were going 3 monitoring methods were used.
Direct monitoring via
[root@server ~] telnet 0 11211 Trying 0.0.0.0... Connected to 0. Escape character is '^]'. stats STAT pid 1234 STAT uptime 1234 STAT time 111111111 ...
Ruby key dumper
This small Ruby key dumper allows you to dump out a selection of the keys. It’s not great and memcached doesn’t allow a full dump of keys in anyway. It allows you to see what’s there and validate it’s content though.
Overall instance stats from
This is a perl program available on google code which allows you monitor a given set of instances, much like regular top.