The main issue with non-blocking servers

There is a problem with the general understanding of how non-blocking servers work. And some have even built whole frameworks around non-blocking technology without solving the basic issues of non-blocking servers (like Python's Tornado web-framework).

The gist is that if you use non-blocking servers then all of your scalability issues fade away and you have amazing performance. The problem is that if you want to use non-blocking servers then all of your IO and network calls have to be non-blocking as well. And this is where people fail, because they use a non-blocking server that uses blocking libraries.

In this post we'll dig deeper into how threaded servers work and how non-blocking servers work and why you should not use a non-blocking server unless your libraries are non-blocking as well.

Non-blocking servers perform better

First off, I am not arguing that blocking servers have better performance than non-blocking. In most cases non-blocking servers are much better at handling many thousands concurrent users. This can be best seen in a benchmark that WebFaction did where they switched from a threaded to a non-blocking server:

• A little holiday present: 10,000 reqs/sec with Nginx!

In their switch from Apache to nginx they found that:

nginx could handle a lot more req. pr. sec:

nginx used a lot less memory than Apache:

Amazing stuff, why not use this non-blocking technology in your Python/Java/Ruby/PHP framework?

How blocking servers work

Blocking servers are usually threaded and a request is handled by a thread, this can be visualized like so:

This is usually true about threaded servers:

• Handling many thousands concurrent requests is usually costly because the server needs to spawn threads to handle these and threads aren't that cheap
• The libraries you use have to be thread safe, meaning that multiple threads should be able to use them concurrently

How non-blocking servers work

Non-blocking servers aren't threaded and they use an IO loop and events to handle requests, this can be visualized like this:

This is usually true about non-blocking servers:

• Handling many thousands of concurrent requests isn't a problem and non-blocking servers excel at this and the reason why they are used for Comet
• Everything performed in the IO loop has to be non-blocking, else the IO loop gets blocked and your server will stall until this blocking operation is complete
• Thread safety isn't an issue

Where Tornado (and others) go wrong

I'll take Tornado as an example, but a lot of other non-blocking solutions have similar issues.

The problem with Tornado is that they use a non-blocking server that uses blocking libraries. What this means is following:

• Tornado's MySQL usage is blocking, meaning that a 1sec query will block your IO loop for 1 sec and you won't be able to handle any requests during this 1sec
• Doing an expensive system call with Tornado? This system call will block your IO loop
• Doing an expensive template rendering with Torando? This will block your IO loop

And like I stated earlier blocking the IO loop is fatal for performance as you won't be able to process any events while the IO loop is blocked!

Conclusion

The bottom line is that non-blocking technology is smart and the performance is usually great, but in order to take full advantage of non-blocking technology then all of your IO and network calls have to be non-blocking as well. Using a non-blocking server with blocking libraries is usually a recipe for disaster if some of your blocking calls begin to be costly.

Do also note that MOST things in languages like Python, Ruby, Java or PHP are blocking by default - - so you should be very wary when you use a non-blocking server with these languages.