Displaying timezones better in Python

I have released timezones for Python, it makes timezones more user-friendly for the users by formatting timezones better and auto-guessing timezone based on the user's IP address.

You can:

The library provides:

  • User friendly rendering of common timezones. pytz.common_timezones includes 430 common timezones, without any smart sorting or display of useful information such as timezone offsets. This provides an awful experience if you just present this to the users - - like it's done in Django
  • Auto-guessing a user's timezone based on the user's IP. This is done via pygeoip
  • Supporting fixed offsets timezones, such as "GMT +1:00"

Example usage of the library:

from timezones.tz_rendering import html_render_timezones

html_timezones = html_render_timezones(select_name=timezone',
                                       current_selected=current_timezone,
                                       user_ip=get_current_ip(),
                                       first_entry=_('Select your timezone'))

A screenshot of the library in use [in Wedoist]:

python-timezones screenshot

Updating caching headers for Amazon S3 and CloudFront

I made a major blunder when setting caching headers for Amazon S3 and CloudFront. Making such a blunder makes my sites slower and costs more in bandwidth. In this little blog post I will detail how to fix this and make sure you use correct caching headers.

Use the correct syntax

The first rule, make sure that the syntax is correct. Correct syntax looks like this:

  • Cache-Control: max-age=155520000, public
  • Expires: Sat, 29 Apr 2017 13:31:45-0000 GMT

For me, I used following syntax (it's wrong and wont be understood by browsers!):

  • Cache-Control: max-age 155520000

Read more in RFC 2616 for all the details sounding headers.

Be greedy and use file versioning

Use file versioning (for example make md5 hash a part of the name). You are forced to do this anyway since CloudFront does not support invalidations that well.

Already using file versioning? Great, then set your expires a lot of years in the future, since the filename will change when the files changes (i.e. you don't have to worry about invalidating old files).

Made a blunder? Use my script to update all S3 files in a bucket

Before you update headers to every S3 object make sure that the code works by testing it on dummy objects. I had a lot of issues getting it to work, since it will replace older metadata and not just update it. You can use my script (but it's not bulletproof, so be sure that any missing headers that you use are copied over to the updated metadata).

You will need to do following:

  • Using the script below test it out on dummy S3 objects
  • Update headers for every S3 object
  • Create new Amazon CloudFront distributions after the S3 objects are updated. Can be done via aws.amazon.com
  • Update DNS records to use the new distributions
#!/usr/bin/env python
"""
    fix_s3_cache_headers
    ~~~~~~~~

    Updates S3 objects with new cache-control headers.

    Usage::
        python fix_cloudfront.py <bucket_name> <keys>*

    Examples::
        Updates all keys of avatars.wedoist.com bucket::
            python fix_cloudfront.py avatars.wedoist.com

        Updates only one key::
            python fix_cloudfront.py avatars.w.com d39c2.gif

    Read more here::
        http://amix.dk/blog/post/19687

    :copyright: by Amir Salihefendic ( http://amix.dk/ )
    :license: MIT
"""
import sys
import mimetypes
import email
import time
import types
from datetime import datetime, timedelta

from boto.s3.connection import S3Connection
from boto.cloudfront import CloudFrontConnection


#--- AWS credentials ----------------------------------------------
AWS_KEY = '...'
AWS_SECRET = '...'


#--- Main function ----------------------------------------------
def main(s3_bucket_name, keys=None):
    s3_conn = S3Connection(AWS_KEY, AWS_SECRET)

    bucket = s3_conn.get_bucket(s3_bucket_name)

    if not keys:
        keys = bucket.list()

    for key in keys:
        if type(key) == types.StringType:
            key_name = key
            key = bucket.get_key(key)
            if not key:
                print 'Key not found %s' % key_name
                continue

        # Force a fetch to get metadata
        # see this why: http://goo.gl/nLWt9
        key = bucket.get_key(key.name)

        aggressive_headers = _get_aggressive_cache_headers(key)
        key.copy(s3_bucket_name, key, metadata=aggressive_headers, preserve_acl=True)

        print 'Updated headers for %s' % key.name


#--- Helpers ----------------------------------------------
def _get_aggressive_cache_headers(key):
    metadata = key.metadata

    metadata['Content-Type'] = key.content_type

    # HTTP/1.0 (5 years)
    metadata['Expires'] = '%s GMT' %\
        (email.Utils.formatdate(
            time.mktime((datetime.now() +
            timedelta(days=365*5)).timetuple())))

    # HTTP/1.1 (5 years)
    metadata['Cache-Control'] = 'max-age=%d, public' % (3600 * 24 * 360 * 5)

    return metadata


if __name__ == '__main__':
    main( sys.argv[1],
          sys.argv[2:] )
Code · Python · Stuff Permanent link 2. May

Product features vs. UI elements on the screen

Like noted in The essence of minimal product design successful products hide complexity from the users. Balsamiq has a great graph illustrating this as well.

 features vs. buttons

from The Balsamiq Mockups Manifesto

Design · Stuff · Todoist · Wedoist Permanent link 28. Apr

Hiring talented iOS and Android developers

We are expanding the team at Todoist and Wedoist with iOS and Android programmers.

Some of our stats:

  • Over 300.000 users
  • Rapid growth
  • Our business is profitable
  • We started fulltime with the company just 8 months ago. Imagine the future :-)

Join us either freelance or full-time and work on something that makes the world more productive.

Send your resume to amix@amix.dk, be sure to include some code you are proud of (or a link to your GitHub/BitBucket profile).

Announcements · Code · Todoist · Wedoist Permanent link 15. Mar

Open sourced coffee-watcher, less-watcher and watcher_lib

I have updated/published following libraries today:
  • coffee-watcher: a script that can watch a directory and recompile your .coffee scripts if they change
  • less-watcher: a script that can watch a directory and recompile your .less scripts if they change
  • watcher_lib: A library that can watch a directory and recompile files if they change. Can be used to build watcher scripts

Basically these scripts are useful for development as you don't need to think about recompiling your files. You can also use watcher_lib to implement custom watchers.

coffee-watcher

Documentation

sudo npm install coffee-watcher
coffee-watcher -p [prefix] -d [directory]

Options:
  -d  Specify which directory to scan.
  -p  Which prefix should the compiled files have?
      Default is style.coffee will be compiled to .coffee.style.css
  -h  Prints help

less-watcher

Documentation

sudo npm install less-watcher
less-watcher -p [prefix] -d [directory]

Options:
  -d  Specify which directory to scan.
  -p  Which prefix should the compiled files have?
      Default is style.less will be compiled to .less.style.css
  -h  Prints help

watcher_lib

Documentation

sudo npm install watcher_lib

How to build a generic watcher (here is less-watcher's implementation):

# Use `watcher-lib`, a library that abstracts away most of the implementation details.
watcher_lib = require 'watcher_lib'

# Searches through a directory structure for *.less files using `find`.
# For each .less file it runs `compileIfNeeded` to compile the file if it's modified.
findLessFiles = (dir) ->
    watcher_lib.findFiles('*.less', dir, compileIfNeeded)

# Keeps a track of modified times for .less files in a in-memory object,
# if a .less file is modified it recompiles it using compileLessScript.
#
# When starting the script all files will be recompiled.
WATCHED_FILES = {}
compileIfNeeded = (file) ->
    watcher_lib.compileIfNeeded(WATCHED_FILES, file, compileLessScript)

# Compiles a file using `lessc`. Compilation errors are printed out to stdout.
compileLessScript = (file) ->
    fnGetOutputFile = (file) -> file.replace(/([^\/\\]+)\.less/, "#{argv.p}$1.css")
    watcher_lib.compileFile("lessc #{ file }", file, fnGetOutputFile)

# Starts a poller that polls each second in a directory that's
# either by default the current working directory 
# or a directory that's passed through process arguments.
watcher_lib.startDirectoryPoll(argv.d, findLessFiles)

css_image_concat: Improve performance by concating your images

I have updated my CSS image contact script from 2007. This script can concat images into one image and create a CSS file with classes. This is a super useful optimization when you want decrease issuing a lot of HTTP requests due to a lot of small images (like icons).

What is the idea behind this?

The idea is to take all separated images and concat them to one image file:

Concat images

This means that only one HTTP request is made to fetch all the images.

CSS is then used to display an image (by using background offsets):

.cmp_email_icon {
    background: transparent url(all_images.gif) 0 -48px no-repeat;
    width: 21px;
    height: 16px;
}

Installing it

sudo pip install css_image_concat

This script also requires ImageMagick.

Using it

$ css_image_concat static/icons static/all_icons.png static/all_icons.css
Parsed 18 in static/icons
Written CSS file: static/all_icons.css
Written image file: static/all_icons.png
----

GitHub and PyPi

You can checkout the code from here:

Focusing is about saying no - Steve Jobs (WWDC'97)

When Steve Jobs returned back to Apple in 1997 he fired thousands of people and discontinued lots of projects. In WWDC'97 he explained why:

Apple suffered for several years from lousy engineering management. There were people that were going off in 18 different directions... What happened was that you looked at the farm that's been created with all these different animals going in all different directions, and it doesn't add up - the total is less than the sum of the parts. We had to decide: What are the fundamental directions we are going in? What makes sense and what doesn't? And there were a bunch of things that didn't.

Focusing is saying yes, right? No. Focusing is about saying no. You've got to say, no, no, no. The result of that focus is going to be some really great products where the total is much greater than the sum of the parts.

Design · Interesting · Stuff Permanent link 20. Feb

Niall Ferguson: The 6 killer apps of prosperity

I would recommend watching this TED talk: Niall Ferguson: The 6 killer apps of prosperity

Over the past few centuries, Western cultures have been very good at creating general prosperity for themselves. Historian Niall Ferguson asks: Why the West, and less so the rest? He suggests half a dozen big ideas from Western culture -- call them the 6 killer apps -- that promote wealth, stability and innovation. And in this new century, he says, these apps are all shareable.

The Course of the Empire 1

The Course of the Empire 2

The Course of the Empire 3

The Course of the Empire 4

The Course of the Empire 5

The course of the Empire

Life · Political · Tips Permanent link 14. Feb

Interview with DoesWhat.com

I gave a short interview to DoesWhat.com. You can read it here:

What are you most excited about at the moment?
I want to make the world more productive. I wake up each morning and I am excited to work towards this goal.

Life · Tips · Todoist · Wedoist Permanent link 10. Feb

Vim: Annotate strings with gettext (the macro way)

If your application does not use gettext strings then you must do a lot of rewrites of following form:
  • "My String" needs to be rewritten to _("My String")
  • And in templates you need to rewrite My String to ${ _("My String") }

Doing this over huge codebase is a pain in the ass and automating it can be error prone. Luckily we can write a couple of Vim macros that can help with this job!

The basic ideas is to select text and press si to annotate it with gettext function.

To implement this simply pass this to your vimrc:

vmap si s(i_<esc>f)
au FileType mako vmap si s"i${ _(<esc>2f"a) }<esc>

You use it in a following manner:

  • You select some text (in visual mode)
  • You press si
  • Your text gets transformed from "... Text selection..." to _( "... Text selection..." )
  • There's extra support for annotation inside mako templates

The above snippet requires surround.vim plugin!

The above reason is why I love to use Vim. I can spend 10 minutes on creating a macro that saves me hours of manual labor!

Tips · VIM Editor Permanent link 8. Feb

The new way to do Cross Domain Ajax/Comet: Cross-Origin Resource Sharing

Wedoist Comet

Wedoist features a realtime system that's powered by comet where the state is synced across all open sessions. Today we have rewritten our client side implementation to use Cross-Origin Resource Sharing, which is a relatively new technology to do cross domain communication. This technology is supported by following browsers:

  • Firefox 3.5+
  • Safari 4+
  • Google Chrome 3+
  • Internet Explore 8+

Implementing comet is a challenge, both on the backend and the client side. In this blog post I will only feature the client side implementation.

How is our comet setup?

comet0X.wedoist.com points to a nginx server that proxies to JBoss Netty servers. This makes it possible to support SSL and load balance.

This solution scales really well, but we need to do cross-domain requests to comet0X.wedoist.com. For this we used ScriptCommunicator, which is one of my more popular open-source projects. It basically uses script tags for communication and it resolves to hacks in order to detect errors.

Using Cross-Origin Resource Sharing

The main points of Cross-Origin Resource Sharing are following:

  • Your backend server specifies which origins it accepts requests from via a special Access-Control-Allow-Origin header
  • On Chrome, Safari of Firefox you use XMLHttpRequest object and check for withCredentials property
  • On Internet Explorer 8+ you use XDomainRequest object
  • For everything else you resolve back to script tags

Our implementation of this strategy is shared below.

Specifying Access-Control-Allow-Origin

You need to specify what origins you are accepting requests from and you do this by adding a special Access-Control-Allow-Origin header.

In nginx you can do following to only accept from https://wedoist.com;

location / {
    add_header Access-Control-Allow-Origin https://wedoist.com;
    proxy_pass http://0.0.0.0:8001/;
}

To accept from everywhere you would do this (it isn't recommended to due CSRF exploits!):

location / {
    add_header Access-Control-Allow-Origin *;
    proxy_pass http://0.0.0.0:8001/;
}

Implementation of the new script communicator

You can download and fork the code here:
https://github.com/amix/Script...

Using the new script communicator

To use the new script communicator simply do following:

var url = 'http://some_domain.com/give_me_js_back.php?i=42&callback=foocall'
ScriptCommunicator.sourceJavascript(url, on_success, on_error)

on_success and on_error are functions. on_error is only supported by browser that support Cross-Origin Resource Sharing, i.e.:

  • Firefox 3.5+
  • Safari 4+
  • Google Chrome 3+
  • Internet Explore 8+

Like the old ScriptCommunicator implementation your server should return JavaScript that can be eval'ed.

References

Why we picked Amazon AWS over Rackspace and dedicated hosting

Amazon AWS

Last saturday we switched Wedoist from Rackspace's Cloud to Amazon AWS. In this post I will present why we have done this and why we picked Amazon AWS instead of Rackspace's Cloud or dedicated hosting.

Why not Rackspace's Cloud?

We initially used Rackspace because we thought they had lots of experience and expertise in hosting. Now we are not so sure and we don't have a huge trust in their systems or their network.

Our problem was that a lot of users had reported performance issues. Wedoist is used in about 50 countries and each time we debugged these issues they were from users that came from "remote countries" such as Brazil, Hong Kong or Taiwan. Having developed web-applications for a long time we were sure it was a network issue and even I could reproduce this issue in Chile. This was our main reason for moving to Amazon AWS: Rackspace's network does not seem to work that well in "remote areas" of the world.

The second reason is that Amazon AWS seems to be a superior product that offers more solutions such as:

Amazon AWS also seems to be in rapid development, for example, Amazon DynamoDB was released last week (and it seems to be an awesome database solution!)

Why not dedicated hosting?

You will get a lot more hardware by going for dedicated hosting (either buying or leasing servers). For example, if you read pinboard's article on dedicated hosting, A Short Rant About Hosting, you will find prices that are much cheaper than Amazon's.

Here are our reasons why we didn't go the dedicated hosting route:

Cost isn't everything:
For us cost isn't the only thing we care about. Our focus is on building software and serving our users in best ways possible. Setting things up ourselfs would save money, but would require a lot more work, be more limited and possible produce more errors.

We don't have resources to use multiple data centers:
Some things would not be possible at all, because setting it up would be very expensive - - an example is minute snapshots of our databases that are stored in 3 different data centers and can be restored with a few clicks. Implementing backup strategies is complex, especially if you have a lot of data, and Amazon AWS saves us a lot of headaches in this area.

We can easily scale:
With Amazon AWS scaling up or down is easy: we can add servers, we can upgrade to bigger machines, we can downgrade, we can script everything so extra machines are added in the morning and shutdown in the afternoon. All this with a few clicks or a few scripts and in matter of minutes.

Our focus isn't on managing servers:
The bottom line is that we can focus on building great software. Using Amazon AWS is more costly than dedicated hosting, but our products aren't free, so paying for premium hosting isn't a huge deal.

Amazon AWS seems to be a marketleader
Amazon AWS has an impressive customer base that includes Dropbox, Netflix or Yelp (just to name a few). We feel in good company by being there as well.

Todoist · Wedoist Permanent link 30. Jan

Model View Controller Rewrite (JavaScript)

I think it's worthwhile rewriting large amounts of code if it adds significant advantage such as cleaner and more maintainable codebase. I have taken some screenshots along the way to document how I do a rewrite.

The context

In 2006, when I started Todoist, MochiKit was one of the better JavaScript libraries. Today MochiKit is largely forgotten and the state of the art JavaScript libraries are backbone.js, spine.js or ember.js. All of these newer libraries use the Model-View-Controller (MVC) pattern, a pattern I have written about in the past:

Model View Controller rewrite in JavaScript/CoffeeScript

I will now show a series of screenshots of what the problem is and how I find a solution that makes the code simpler and more maintainable.

Each project in Todoist has a counter that shows how many tasks the project holds. In the old codebase there's one creator and 7 (yes 7!) update functions:

Todoist MVC rewrite 1

I could rewrite this to use fewer update functions or even just one, but I could also go an extra mile to ensure that such blunders do not happen in the future. The better way to do this is to use the MVC pattern: the basic idea is that we create a counter that autoupdates itself each time the model changes.

I looked at how modern libraries handled this (I was largely inspired by backbone.js ) and I began scribbling down how an API could look like:

Todoist MVC rewrite 2

I did not want to use backbone.js directly as I don't like the way routes work or that it's dependent on underscore (adding backbone.js and underscore would result in 2000 lines of extra code). I also think I can learn more by implementing stuff myself and I can do a customized solution that works well with my existing code.

After this step I began to implement a test to see if my API made sense:

Todoist MVC rewrite 3

I tried to use the new code for the counter (I had two solutions, either binding it to a project object or using ProjectsModel). I tried to use them both throughout some code to see how they would behave in different contexts:

Todoist MVC rewrite 4

Todoist MVC rewrite 5

Then I set these solutions against each other - - reflecting on pros and cons of each solution:

Todoist MVC rewrite 8

The solution I ended up with was following:

Todoist MVC rewrite 9

Would love to hear about how you attack a rewrite. Happy hacking!

Hiring CoffeeScript and Python programmers

We are expanding the team at Todoist and Wedoist with CoffeeScript and Python programmers.

Some of our stats:

  • Over 300.000 users
  • Rapid growth
  • Our business is profitable
  • We started fulltime with the company just 6 months ago. Imagine the future :-)

Join us either freelance or full-time and work on something that makes the world more productive.

Send your resume to amix@amix.dk, be sure to include some code you are proud of (or a link to your GitHub/BitBucket profile).

Announcements · Code · Todoist · Wedoist Permanent link 18. Jan
© Amir Salihefendic. Powered by Skeletonz.