Proactive vs. Reactive scaling
Posted this comment
on Hacker News which I think you might enjoy:
Having scaled a site to millions of users and billions of data items I think the strategy of extinguishing fires as they arise is the wrong strategy. It will lead to lots of stress, lots of sleepless nights and lots of unneeded bugs. I know, because we used this strategy and I would not recommend it. There's a better way that's less stressful and better for your health.
It's the idea of being proactive and reactive - - and as a good developer you should master both and be able to switch between these states.
For example, backups and anticipations of future load should be proactive. If your load is close to maxing out today then don't wait till your web site is down before you begin optimizations. Know everything about your systems so you can take an educated guess of how many more users you can handle. Think about which strategies you can apply if you want to scale 10x, 100x, 1000x of your current load.
Reactive programming is great when you are exploring new territory and are unsure how many would use something. This is the state where you don't think about scaling to millions of users. This said, as a good developer always keep in mind how something can be scaled or how fast something is.
In other words: it's about finding balance between the proactive state and the reactive state. Don't optimize prematurely, but don't optimize when everything is burning down either. You have lots of data so use this to your advantage and make educated guess about your way of scaling.
Similar blog post from 2008: Reactive vs. proactive development