import from anywhere

Update (20140102): This solution is not perfect either. As discussed here, each time imp.load_source is run on a module, the module-level code in that module will be re-executed, which could cause problems in more complicated cases. Thanks to alb1 for the info.

---

Problem
You have a project structure with several submodules and in a source you want to import something from another submodule by using relative import. Example:

project/
    main.py
    sub1/
        __init__.py
        helper.py
    sub2/
        __init__.py
        utils.py

As long as you launch main.py only, there are no big problems. However, I like adding simple tests to the end of modules after the ‘if __name__ == "__main__":‘ line. Say utils.py wants to import something from helper.py. How to do it properly so that I can still execute utils.py as a stand-alone script? Unfortunately, this is a real pain in Python :(

Solution
I think I managed to find a simple and elegant solution for this problem. Let’s see utils.py:

def load_src(name, fpath):
    import os, imp
    return imp.load_source(name, os.path.join(os.path.dirname(__file__), fpath))

load_src("helper", "../sub1/helper.py")
import helper
from helper import hi    # if you need just this, "import helper" is not required

print helper.PI
print hi()

Provide two parameters to load_src. The first one is a name. You will be able to import the module using this name. The second parameter is the relative path of the source file you want to import. Note that we can only load files this way, not directories!

I prefer this way because you need to import the desired module explicitly (using the import keyword). From the Zen of Python we know that “explicit is better than implicit”.

Another way would be this:

helper = load_src("helper", "../sub1/helper.py")
print helper.PI

Here helper is imported implicitly, so after loading it you can refer to its content (e.g. helper.PI). I prefer the first version; that makes the source code more readable.

Update: Version 2
Here is an updated version that supports file names given either in absolute or relative path.

def load_src(name, fpath):
    import os, imp
    p = fpath if os.path.isabs(fpath) \
        else os.path.join(os.path.dirname(__file__), fpath)
    return imp.load_source(name, p)

Alternatives
You could also modify the sys.path list to include the parent of “sub1/“. Then you could import helper like this: “from sub1 import helper“. However, modifying sys.path is considered by many as a bad solution.

Import a file from another directory of the project

Imagine the project structure on the right side. The project is called “Sortings”, thus everything is under this directory. It’s a good idea to put your project in a package, thus all the source files are in the “sortings” package. Here we have a “config.py” file that is used by almost all the source files in the project. We also have a subpackage called “sorts”, and in this subpackage there is a script called “shell.py”. Say “shell.py” wants to include “config.py” from the parent directory. It can be done this way:

# shell.py
from sortings import config

Now you see why we put everything under the named package “sortings”.

It works well from the IDE, great. However, if you go to the command-line and you want to launch “shell.py”, you’ll get a surprise:

$ ./shell.py
Traceback (most recent call last):
  File "./shell.py", line 3, in
    from sortings import config
ImportError: No module named sortings

Hmmm… Let’s see the PYTHONPATH:

$ echo $PYTHONPATH
:/home/jabba/python/lib/jabbapylib:/home/jabba/python/lib

He is right, the “Sortings” directory is not in the PYTHONPATH, thus it cannot find the module “sortings”.

Let’s add the required directory then. Open “shell.py” and add these lines to the beginning. It is important that these lines should be executed first.

import sys
sys.path.append('../..')

from sortings import config
...

Now it works… as long as you launch “shell.py” from the current directory. Step up to the parent directory and try to launch it again:

$ ./shell.py    # works
$ cd ..
$ sorts/shell.py
Traceback (most recent call last):
  File "sorts/shell.py", line 6, in
    from sortings import config
ImportError: No module named sortings

Damn, what’s wrong again? Well, we appended a relative path, not an absolute path. Let’s change the beginning of “shell.py” again:

import os
import sys
sys.path.append(os.path.abspath(os.path.dirname(__file__) + '/' + '../..'))

from sortings import config
...

Super! Now you can call “shell.py” from everywhere, even from crontab.

Another way
There is another way too. Add the absolute path of the directory “Sortings” to PYTHONPATH. If you want it permanent, add the following to your “.bashrc” file:

export PYTHONPATH=$PYTHONPATH:/abs/path/to/Sortings

Note that this is not the recommended way though.