Circular imports in Python

I have to say that this restriction has bitten me so many times now and it's one lame restriction.

Basically, Python has an awful support for circular imports, a simple example: 

# a.py
import b
x = 1

# b.py
import a
x = a.x + 1 #circular definition, x isn't defined

Now I understand why it's like this (Python is very dynamic and import is a executable statement yada yada), but a lot of times it's just useful for two modules to reuse each others code. Here are some examples of this:

  • Cache + Model code: Cache uses model and model has to invalidate cache on updates
  • Worker + Model: Model adds tasks to a worker (i.e. uses worker) and worker reuses code from model
  • Users + Items: Items uses Users code (such as getUserById) and Users reuses Items code (such as deleteItems(uid))

Now a solution for this could be to create more modules or do some hacks, but generally, I really hope they solve this issue since it's really annoying and prevents some designs.

Circular imports are solved in languages such as Perl or Java.

Code · Python • 8. Sep 2008
12 comments so far

PHP's solution to this problem is to have _once version of include and require (include_once and require_once). PHP keeps an internal cache of what files have already been included and the _once versions only include if the file hasn't already been loaded by PHP.

Daniel, I suspect your argument for a _once version is misinformed. That _is_ what import does. A module is only loaded once, and further imports just add a reference to the cached module.

I haven't seen code that tries to do this, where solving it doesn't improve the code quality anyway.

You know, whenever I see circular dependencies like this, I'm thinking "this is just bad design".

If you want your coupling this strong, you put your code in the same module.

We have encountered the problem multiple times in Inyoka too, and it's still not really solved there. I think in Python it's important to think about some layers for your application, and if you need additional dependencies, do lazy imports.

I haven't much experiences in Perl, but circular imports aren't that easy in Java. You just have to face the problems on compile time and you might have to Clean/Build your project multiple times and/or increase the resolve-limit of the compiler to get it working.

Thomas Mailund:
The problem is that you'll eventually end up with modules with thousands lines of code. A good example of this is http://www.feedparser.org/ - it has just one file with 4000 lines of code.

I agree thought that circular dependencies aren't good, but in some cases they are needed (for example, for cache and model).

Very seldom do you require circular imports. Escalate or demote the parts that are necessarily circular to a higher or lower level, respectively, or you may find yourself with a Big Ball of Mud (TM). Read Lakos' "Large Scale C++ Software Design" for more on this - much of what he says is very relevant for all languages.

Tomas:
How would you solve the issue with separating models and caches in two modules without using circular imports? The only "clean" solution (at least for Python) is to use one module, which gives bloat - the other solution is to hack circular imports.

Hi amix:

Generally speaking, you have two choices:

a) Escalate the circular functionality. Cut out those parts of the functionality that are circular and put them in a "higher", more knowledgeable module that controls the model and the cache models. In essence, make the model and cache modules simpler and dumber, and have one extra smarter "control" module.

b) Demote the circular functionality. If applicable, cut out those parts of the functionality that are circular and put them it a simpler, dumber module. Then the model and cache modules can import and use that.

Both approaches can require a bit of thinking and hooks in the right places to make it work. It tends to increase the amount of code a little. But the overall structure of any larger code base is very much enhanced without circular dependencies.

Again, I recommend Lakos for this.

Tomas:
I really dislike the idea of creating control modules just to solve a problem that's largely a language's fault. Your solutions would make the code base harder to understand since you'll get more code and more complexity and you'll get bloat in number of modules. It's just a hack, like it's an hack to cram all code in one module or to do late imports to solve this issue :) Thanks for your pointers thought.

Anyway, I have now coded in Python for a few years and this problem of circular references keeps biting back and I know that a lot of people have had troubles with them. I think Python should solve this problem so we as programmers don't have to find "creative" ways of solving this issue.

Amix:

You're right that it certainly wouldn't hurt if Python supported circular imports in a nicer fashion, and sure, I would like it fixed. However, I want to stress that my points are language agnostic - in no language should you use circular dependencies. You shouldn't try to eliminate them only for the languages where they happen to be inconvenient.

Some specific comments:

> Your solutions would make the code base harder to understand since you'll get more code and more complexity and you'll get bloat in number of modules.

Have you tried these methods, or are you speculating?

In my experience isolating modules such that they are less dependent on one another makes the code base easier to understand. You do get more modules, but you get _less_ complexity. Allow me to illustrate:

One decent metric of complexity is connections of knowledge between modules. In a hierarchical system of N modules, the complexity is bounded by some O(lg(N)*N) function. In a circular system it is bounded by O(N^2). Try drawing the knowledge chart yourself and see which one makes sense.

Imagine you are a person trying to understand a code base you have never seen before. Since you are not super-human, you try to lift out a module at a time and study it in isolation. In my experience, code bases that have circular dependencies tend to resist such a method, since no module can be understood on its own.

This effect only really comes into its own for largish (100000 loc) code bases with multiple people. I can personally attest to its devastating effect at those scales. For small code bases (maybe 20000 loc or so), especially ones you write yourself, you can probably comprehend it in one bite anyway and so these things are less important. If you are a particularly smart cookie you can get away with bigger code bases but eventually your neurons will overflow...

> It's just a hack, like it's an hack to cram all code in one module or to do late imports to solve this issue :)

I assume that you mean that Python's way of doing circular imports are hackish, in which case I agree.

You know, whenever I see circular dependencies like this, I'm thinking "this is just bad design".

If you want your coupling this strong, you put your code in the same module.

I know I'm a bit late, but I've just been hurt by this issue. I have been using python for small projects for six years know, but I've spent hours to hunt down an initialization problem just to find that the cilcular imports work like this.

I would argue that circular import always means bad design. Actually in a sufficiently complex system it will happen. I would even say that the larger the system the harder it is to avoid it.

In my case I'm writing a multithreaded application that has a web interface (provided by Django). The the two has to communicate and share some data. Though they are quite loosely coupled, I thought it would be elegant to have them in one process instead of having them communicate over XML-RPC or some other interface. (In java I would never thought separating them.)

The problem manifested in a module-global variable being reinitialized to None, but of course it took me a while to figure out that it was reinitialized on the second import, because I could never imagine that :). There is a 'main.py' that starts the multiprocess daemon, creates some objects and then calls 'WebUI.py' to start Django. It also passes in one of the objects they need to share, and that is a global in WebUI. Then WebUI.py starts Django and then when a web request is made over the management UI, django will load the views which will try to get the object from WebUI, but as there is a cycle know (WebUI -> Django view -> WebUI), python will reimport WebUI and will run the initialization code again (x = None), and it will clear the previous contents. Actually the main problem here, and this is why it's not a problem with java, is the hyped feature of python, that needn't and can't separately declare a variable. You can't say 'var x' (or 'def x') like in javascript.

Post a comment
Commenting on this post has expired.
© 2000-2009 amix. Powered by Skeletonz.