Fast polling using C, memcached, nginx and libevent

Why limiting to 4 (overflow?)
MServer memcache_servers[4];

Pretty cool, the more I find out about the tech at Plurk the more I start to admire you guys, especially compared to the crap that twitter was running.

Here's a quick code review of your C code for security/safety reasons:

add_mserver -- Should have asserts protecting against NULL on the host param and that NUM_OF_SERVERS isn't out of range.

init_memcache_servers -- You don't validate the return of memcached_create.

parse_path -- You don't validate the return of malloc. Using strlen is a buffer overflow. Pass in the length of the string or use a lib like bstring. For loop uses \0 instead of a predetermined length, which you already have from strlen. This function is externally accessible so these are real problems.

memcache_handler -- Arbitrary fixed size for key. Using strcpy rather than a string with a predefined size. Ironically, your strlen checking for no buffer overflows has a buffer overflow. Use strnlen, actually just avoid all str functions that also don't have a length param. You don't check the return value of parse_path. You pass in key to
memcached_get but the str functions that make key are notorious for not honoring the \0. You should know the size of key after building and add your own \0 to double check it.

main -- You don't check return values of any function in here.

Finally, run this under valgrind and assume valgrind is always right. It sometimes isn't, but it's usually more right than a programmer so it's your job to prove why valgrind is wrong. Usually if you tricked valgrind then your code is too tricky. Quit doing that.

Cool, hope that helps clean it up. Feel free to shoot me edits for feedback.

In your path parsing bits, you could make use of standard functions like strchr() etc. to avoid over allocating the string.

Something like:

char* parse_path(const char *uri) {
  char *ret;
  int len;
  char *qmark = strchr(uri, '?');
  if (qmark) {
    len = qmark - uri;
  } else {
    len = strlen(uri);
  }
  ret = malloc(len + 1);
  memcopy(ret, uri, len);
  ret[len] = '\0';
  return ret;
}

Not a huge difference overall as your strings are short and short lived, but it's a bit more readable aside from the little bit of pointer math.

Is there a reason why you haven't used memcached support from nginx?

http://weichhold.com/2008/09/1...

Comet isn't especially hard, especially once you know how to use libevent.

Can you share any numbers? 2% cpu on a single server? How often a single web page polls for updates? (ie: what is the worse-scenario latency)

@Marek Majkowski:

I didn't use nginx'es memcached module since it only supports one memcached server and only GET requests.

You need http://www.reversehttp.net/

ret = malloc(strlen(uri) + 1);

=>

ret = strdup(uri);


... or you might consider modifying it in-place since you're not using req->uri elsewhere...

This could just be me, but why would you do this over a proxy setup with Varnish, Squid og Apache with mpm_event and mod_cache? Seems a little like you've reinvented the wheel.

Also, as a previous commenter has suggested, why did you not just use the memcached support in nginx, and if it was because of the lack of multiple memcache servers, why not patch it and send this patch back to the community?

Or I should perhaps read what the code is before suggesting something like that :-) Disregard (or delete if you're lenient) my comment

Zed Shaw:
Thanks for the great review. I'll update the code and re-publish it.

Brynjar:
It's arbitrary. I'll probably create a config reader in the next iteration and remove this limit.

Charlie:
Thanks for the suggestion.

Marek:
Every live user polls after updates every couple of seconds, so it's thousands of polling requests pr. second.

Mads Sülau Jørgensen:
There are two reasons why I didn't patch nginx with this support:

• Separation of concerns: Running a separate polling server architecture I can monitor it and scale it independently
• Patching nginx would not have been that easy

I prefer using memcached than Varnish or other caching options and memcached gives me a lot of options (deletion of multiple keys at once, consistent hashing and easy and proven scaling to terabytes of cache - - just to name a few).

Ah, but you could, easily, monitor a running nginx instance. The same goes for a running varnish instance. I'm not necessarily talking about running the nginx memcached module in the same process or the same server as your main proxy is running. So you'd archive the seperation by running another process or running the process on another box. Easy.

And the memcached module of nginx does not seem /that/ difficult to work with. Sure you'd have to do alot of handling of dead servers, and timeouts that could block the whole server. The current one probably handles this already. I don't use it, so I'm not one to say.

With your keys structured correctly, you can flush (purge) multiple keys easily in varnish. You can provide your own hashing function as well. You could also use nginx's hash balancer to send the request to the "correct" backend server.

The thing about hacking some server together with libevent in a language, that might not be onces main language is, that you'll end up missing a lot of things, and it may end up exploding in your face. From the top of my head, what happens if a memcached server dies, memory leaks, what happens if the process segfaults (see memory leak).

It might be super-duper fast, but is it stable :)

Mads Sülau Jørgensen:

plurk@web04:~$ ps aux | grep poll_server
plurk     29835  2.3  0.0   2376  1048 ?        
S    Mar24  27:23 /home/plurk/poll_server/poll_server 0.0.0.0 16000

It's pretty stable (has not crashed since I started it), uses very little CPU and very little memory. If I stop the server, the nginx server will just route everything to Python servers. I doubt Varnish or patching Nginx would have been faster or more stable - - and they would have added a big conceptual complexity.

Personally, I would much rather fix errors in a 100 line program than a thousands line program. It's separation of concerns and separation of complexity. So sure, I could have extended nginx+memcached or hacked Varnish - - but I chose to keep it simple, learn some new things and solve the problem in a very specialized way.

Amix, you should listen to these guys, they're right. They're not criticizing your work, but giving you high level advice. Your system might seem stable now but unfortunately that's not necessarily an indicator of its lack of defects or security.

By using an off the shelf battle tested solution like a varnish fronted HTTP cache through nginx, you're standing on the shoulders of giants. For this particular problem, it seems that this would work just fine.

One of the things I've learned over the years is that you should actually enjoy being a position when you can throw out code. One less thing to worry about, one less thing to go wrong. Another thing is that you can never be sure that there no bugs lurking in your polling server -- Zed outlined a few things and of course there are probably more. Eliminating this risk and shifting the burden onto more trustworthy components is not just the smart thing to do, but your obligation as an engineer it will result in better stability and security.

There is no hacking in the varnish solution what so ever. Varnish is a http proxy cache, it would be doing what is was designed to do. And my guess is, that it would be doing it well. My guess is also, that you'd end up using it alot, and to do more that using it with this one goal.

With a specialized solution, you have to handle a new process, you have to handle the errors that process can give, and you have to keep it running.

Using the tools already available makes more sense to me, and that's what I'm doing when I run into a problem. Others, more proficient programmers usually have had the same problem before me.

Hi Amix,

Regarding memcached and Facebook: they're not actually using memcached as is. As far as I know, they've modified it to run under UDP.

Ed, facebook contributed a lot of code back to memcached. They are responsible for the threading in memcached, and possibly the udp sockets as well. So they are in deed using a stock memcached.

Hey,

Very nice. I really like libevent. What about funky path like /../../etc/passwd and ~/.ssh/authorized_keys. You may already be sure you get clean uri's but you can never be too safe.

Martin

Mads Sülau Jørgensen:

You're wrong ;-) Check out their work at Github:

http://github.com/fbmarc/faceb...

They are certainly not using stock, they have a heavily modified version. They even removed a few files compared to stock.

I know memcached is fast, but this:

cached = memcached_get(tcp_client, key, strlen(key),
&string_length, &flags, &rc);

still seems like a costly operation to do inside a nonblocking
event loop.

Chris, So I am :)

I actually thought they'd gotten most of their changes comitted back into the main memcached. Seems I was a wee bit off ;)

Mads Sülau Jørgensen:
Varnish is a good HTTP proxy and cache, no doubt about that, but it lacks a lot of things to be used in my setup. For example, how can I invalidate 1000 keys at once and do it in a split of a second? Their purging capabilities seem to be pretty primitive...

Plus, who said that I won't iterate over this and go into a new direction - for example, benchmark this against a long-polling "comet" solution. Going into this direction would mean dropping Varnish - while I only need to change and add some lines of code to create a long polling solution of this.

Great stuff!

When talking about speed and async architectures I always find the Talkinator (Java, from the Mailinator guy) amazing.

"In my benchmarks now, on my quad-core desktop the talkinator server can push about 39000 messages per second."

Stephan
http://twitter.com/codemonkeyi...

Well, with a properly designed URL schema, you'd do something like this:

PURGE /poll/user/(?:mads|john|abc*)+/status

And then you'd have the VCL to handle that purge via purge_url(). The above will then purge the urls matching the regular expression, (so mads or john or starting with abc).

It actually works both fast, and well. It does not purge the cache when you tell it, but when you request an object. That makes invalidation _fast_.

See http://varnish.projects.linpro... for more information.

You can more or less configure varnish to do what you want it to do via it's configuration file.

As for the comet solution, your post more or less said that. Right about where it said something about push being hard to scale, which strikes me as a very odd comment. With push, you just have to be able to handle a horde of mostly idle connections, for which http://code.google.com/p/erlyc... or something like that, would probably be the way to go.

While Zed Shaw provided some very valuable feedback on your C code, I'll just nitpick here a bit:

If you know beforehand that request URIs aren't going to be longer than a certain amount, use a fixed size char array rather than mallocing/freeing on every request. Malloc/Free is *relatively* expensive and if you're handling that many requests per second and doing a malloc/free on each one, it will add up.

So, instead, define a max length constant and use that with your request URI buffer. No freeing necessary.

Mads Sülau Jørgensen:
I am pretty tired of keep repeating myself. So this will be the last comment I answer you.

Varnish is a good cache, but it's not really suited where I am headed. An example, how would you change the Varnish solution to support long polling? I can explore this option pretty easily with a basic understanding of how to build a custom server that can perform really good and that can be customized for my needs. I personally want to learn how to build things myself (also things that are complex and that require some thinking and engineering). I know I'll need this skill longer down the road and practice makes a master - so the more practice I have, the better I'll be at doing this.

Mitchell:
Thanks, great tip.

Stephan Schmidt:
I am unsure if Talkinator is using non-blocking IO, because of this:

• Kill the myth please. NIO is *not* faster than IO

he also did a great post on the architecture of Talkinator:

• Benchmarking Talkinator

Both of these are very interesting and recommended reads.

I am unsure if this applies to C thought, especially since libevent is such an amazing library to work with.

Looks very nice. I am going to try this out on my Django site and see how it scales up.

"The performance of the server is over 2400 request pr. second on a not optimized Mac Book." - That is really brilliant!

I have a question, your example looks like you have say 1000 people looking for an update that is common to all. What if I have 1000 users looking for individual updates? This would fill up the cache with 1000 users data right? Will it scale as well?

I'd like to see your ab results with the -k flag. (Keep-alive)

1) Run ab test with more requests. Your test was only 2 sec long. It is not representative.
2) In cache-benchmark-test you should stuck in your network, which is not your case with your only 347.03 [Kbytes/sec], so keep optimising.
3) Out apache-memcache cache is about 12000 req\sec on one machine and with bigger file than yours (approx. 100 KiB).

Very interesting article for those of us working with nginx at scale. Thanks for taking the time to write it.

Please excuse me for a very beginner question:

How does the C code get invoked by Nginx?
Is it by FastCGI? or Is the code itself a Nginx module?

Thanks in advance.

btw, any good Nginx reference site u guys recommend?

@Frank

http://wiki.nginx.org/Main

it is not invoked by nginx. code creates little "http server" that listen to defined port. nginx just passes request to it and returns it's response.

Regarding nginx and multiple memcached's, there's a patch I've used in production for more than a year now:

http://openhack.ru/nginx-patch...

I've been told that's it's still meant to be rolled into memcached proper at some point, but not clear on the status of this.

Lars:
Great link, thanks!