benjamingolub.com

Google App Engine + Memcached + FF To Go = Bliss

Google App Engine gained support for memcached today, a very high performance caching system, and I’m already using it in FF To Go.

I’ll share some code with you so you know how it works. This is the code that renders the public feed:

def public(request):
    f = friendfeed.FriendFeed()
    try:
        start = int(request.GET.get('start', 0))
    except:
        start = 0
    service = request.GET.get('service', None)
    num = int(request.session.get('num', NUM))
    key = 'public_%i_%i_%s' % (num, start, service)
    data = memcache.get(key)
    if not data:
        try:
            data = f.fetch_public_feed(num=num, start=start, service=service)
        except Exception, e:
            return HttpResponseRedirect('/%s/' % e)
        memcache.set(key, data, CACHE_TIME)
    extra_context = {
        'entries': data['entries'],
        'next': start + num,
    }
    if start > 0:
        extra_context['has_previous'] = True
        extra_context['previous'] = start - num
    return render_to_response('public.html', extra_context, context_instance=RequestContext(request))

And that’s it! The FriendFeed team should be happy; I’m using less of their resources because I don’t need to query them as much as I used to. Now this is really only useful on the FriendFeed public feed because all other feeds are likely to be accessed using your FriendFeed credentials and have user specific differences (ie hiding settings) which we don’t want to cache but the public feed does not respond to your hide settings so it is a good candidate. It’s nice to see that using memcached on Google App Engine is no different from using it on your own server.

Am I the first to use memcached on Google App Engine?

Viewing 4 Comments

ericflo 3 months ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

I think you might be better off caching the entire response, as then subsequent requests don't even have to go through the template system, context processors, etc.

For example:

...
key = 'public_%i_%i_%s' % (num, start, service)
response = memcache.get(key)
if not response:
try:
data = f.fetch_public_feed(num=num, start=start, service=service)
except Exception, e:
return HttpResponseRedirect('/%s/' % e)
extra_context = {
'entries': data['entries'],
‘next’: start + num,
}
if start > 0:
extra_context['has_previous'] = True
extra_context['previous'] = start - num
response = render_to_response(’public.html’, extra_context, context_instance=RequestContext(request))
memcache.set(key, response, CACHE_TIME)
return response

reply edit reblog flag record video comment

2 /people/ericflo/ /people/ericflo/following/

ericflo 3 months ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

Sorry, Disqus is not so great for code snippets :(

reply edit reblog flag record video comment

2 /people/ericflo/ /people/ericflo/following/

Benjamin Golub 3 months ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

It's a little trickier than that; because the user can also set the font size and the header contains a link to the user's friendfeed page. But yes; this is exactly what I do on RSSmeme (and as I'm sure you are aware it's how the Django cache middleware works). I cache the entire response (I also cache the results from the DB so hitting the html and then the API for the same data doesn't result in 2 DB requests). Since this runs on Google App Engine and the CPU is not a bottleneck at all I'm not worried about the time spent on the template system too much.

But thanks!

reply edit reblog flag record video comment

25 /people/bgolub/ /people/bgolub/following/ http://benjamingolub.com/ 15500414 bgolub bgolub benjamingolub bgolub

ericflo 3 months ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

Ahh I see--foot, meet mouth :) Still, it's cool stuff.

reply edit reblog flag record video comment

2 /people/ericflo/ /people/ericflo/following/