The main issue with non-blocking servers
There is a problem with the general understanding of how non-blocking servers work. And some have even built whole frameworks around non-blocking technology without solving the basic issues of non-blocking servers (like Python's Tornado web-framework).
The gist is that if you use non-blocking servers then all of your scalability issues fade away and you have amazing performance. The problem is that if you want to use non-blocking servers then all of your IO and network calls have to be non-blocking as well. And this is where people fail, because they use a non-blocking server that uses blocking libraries.
In this post we'll dig deeper into how threaded servers work and how non-blocking servers work and why you should not use a non-blocking server unless your libraries are non-blocking as well.
Non-blocking servers perform better
First off, I am not arguing that blocking servers have better performance than non-blocking. In most cases non-blocking servers are much better at handling many thousands concurrent users. This can be best seen in a benchmark that WebFaction did where they switched from a threaded to a non-blocking server:
nginx could handle a lot more req. pr. sec:
nginx used a lot less memory than Apache:
Amazing stuff, why not use this non-blocking technology in your Python/Java/Ruby/PHP framework?
How blocking servers work
Blocking servers are usually threaded and a request is handled by a thread, this can be visualized like so:
This is usually true about threaded servers:
How non-blocking servers work
Non-blocking servers aren't threaded and they use an IO loop and events to handle requests, this can be visualized like this:
This is usually true about non-blocking servers:
Where Tornado (and others) go wrong
I'll take Tornado as an example, but a lot of other non-blocking solutions have similar issues.
The problem with Tornado is that they use a non-blocking server that uses blocking libraries. What this means is following:
And like I stated earlier blocking the IO loop is fatal for performance as you won't be able to process any events while the IO loop is blocked!
The bottom line is that non-blocking technology is smart and the performance is usually great, but in order to take full advantage of non-blocking technology then all of your IO and network calls have to be non-blocking as well. Using a non-blocking server with blocking libraries is usually a recipe for disaster if some of your blocking calls begin to be costly.
Do also note that MOST things in languages like Python, Ruby, Java or PHP are blocking by default - - so you should be very wary when you use a non-blocking server with these languages.