Archive

Archive for January, 2014

pythonium: a Python to JavaScript translator

January 17, 2014 Leave a comment

Pythonium is a Python 3 to Javascript translator written in Python that produce fast portable JavaScript code.

Example:

$ echo "for i in range(10): print(i)" >> loop.py
$ pythonium -V loop.py
var iterator_i = range(10);
for (var i_iterator_index=0; i_iterator_index < iterator_i.length; i_iterator_index++) {
    var i = iterator_i[i_iterator_index];
    console.log(i);
}

I haven’t tried it yet, so this post is a reminder for me to check it out.

Categories: python Tags:

What is a BDFL?

January 14, 2014 Leave a comment

A BDFL, a term originally used by Python creator Guido van Rossum, is basically a leader of an open-source project who resolves disputes and has final say on big decisions.” (source)

Categories: python Tags: ,

Python news in French

January 13, 2014 Leave a comment

I just came across the site http://news.humancoders.com which is a news collector in French. Users can submit and discuss news here. It has a subpage dedicated to Python.



Human Coders News
est un service permettant de partager les meilleures ressources trouvées sur la toile à propos d’un thème précis. Vous pouvez consulter l’ensemble des news sur la page d’accueil, ou bien, cliquer sur un sujet pour filtrer.

Categories: python Tags: ,

a command line progress bar for your loops

January 10, 2014 Leave a comment

Problem
When I was working on Project Euler, there were several problems that I solved with a brute force approach and thus the runtime was several hours. To see some changes and to know approximately when it finishes, I added a simple counter to the main loop that showed the progress in percentage.

Is there a simpler way to add a progress bar to a loop?

Solution
The project tqdm addresses this problem. Reddit discussion is here.

Usage example:

import time
from tqdm import tqdm

def main():
    for i in tqdm(range(10000)):
        time.sleep(.005)

Notes:

  • you can use xrange too: tqdm(xrange(10000))
  • you can write trange: trange(10000)

On the github page of the project you will find an animated gif too.

Categories: python Tags: , , ,

monkeypatching the string type

January 8, 2014 Leave a comment

Problem
A monkey patch is a way to extend or modify the run-time code of dynamic languages without altering the original source code.” (via wikipedia) That is, we have the standard library, and we want to add new features to it. For instance, in the stdlib a string cannot tell whether it is a palindrome or not, but we would like to extend the string type to support this feature:

>>> s = "racecar"
>>> print(s.is_palindrome())    # Warning! It won't work.
True

Is it possible in Python?

Solution
As pointed out in this thread, built-in types are implemented in C and you cannot modify them in runtime. As I heard Ruby allows this, but it doesn’t work in Python.

However, there is a workaround if you really want to do something like this. You can make a subclass of the built-in type and then you can extend it as you want. Example:

from __future__ import (absolute_import, division,
                        print_function, unicode_literals)

class MyStr(unicode):
    """
    "monkeypatching" the unicode class

    It's not real monkeypatching, just a workaround.
    """ 
    def is_palindrome(self):
        return self == self[::-1]

def main():
    s = MyStr("radar")
    print(s.is_palindrome())

####################

if __name__ == "__main__":
    main()
Categories: python Tags: , ,

Capture the exit code, the stdout, and the stderr of an external command

January 8, 2014 Leave a comment

Update (20140110): The get_exitcode_stdout_stderr function was improved. Thanks to @Rhomboid for the tip. You will find the old version at the end of the post.


Problem
You have an arbitrary program that you want to execute as an external command, i.e. you have a wrapper around it. In the wrapper script you want to get the exit code, the stdout, and the stderr of the executed program.

Solution
Let the external command be this simple C program saved as test.c:

#include <stdio.h>

int main()
{
    printf("go 2 stdout\n");
    fprintf(stderr, "go 2 stderr\n");
    return 3;
}

Compile and run it:

$ gcc test.c
$ ./a.out
go 2 stdout
go 2 stderr

Note that this external command can be anything, written in any language.

After executing it, we pose the following questions:

  1. What was its exit code?
  2. What did it send to the stdout?
  3. What did it send to the stderr?

Here is a wrapper that can capture all these three things. Thanks to @Rhomboid for his tip on improving the get_exitcode_stdout_stderr function.

#!/usr/bin/env python
# encoding: utf-8

from __future__ import (absolute_import, division,
                        print_function, unicode_literals)

import sys
import shlex
from subprocess import Popen, PIPE


def frame(text):
    """
    Put the text in a pretty frame.
    """
    result = """
+{h}+
|{t}|
+{h}+
""".format(h='-' * len(text), t=text).strip()
    return result


def get_exitcode_stdout_stderr(cmd):
    """
    Execute the external command and get its exitcode, stdout and stderr.
    """
    args = shlex.split(cmd)

    proc = Popen(args, stdout=PIPE, stderr=PIPE)
    out, err = proc.communicate()
    exitcode = proc.returncode
    #
    return exitcode, out, err


def main(params):
    cmd = ' '.join(params)
    exitcode, out, err = get_exitcode_stdout_stderr(cmd)

    print(frame("EXIT CODE"))
    print(exitcode)

    print(frame("STDOUT"))
    print(out)

    print(frame("STDERR"))
    print(err)

####################

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print('Usage: {} <external_command>'.format(sys.argv[0]))
        sys.exit(1)
    # else
    main(sys.argv[1:])

Running example

$ ./wrapper.py ./a.out
+---------+
|EXIT CODE|
+---------+
3
+------+
|STDOUT|
+------+
go 2 stdout

+------+
|STDERR|
+------+
go 2 stderr

The script wrapper.py can be called in various ways:

$ ./wrapper.py python print.py    # pass the command in several pieces
...
$ ./wrapper.py "python print.py"    # you can pass the command as one argument
...

Fine, but what is it good for?
I’m working on an online judge whose job is to execute a program with different parameters and decide whether it’s correct or not. I want to execute the programs in a sandboxed (secure) environment and I need to get the output (and the errors) of the programs to analyze how they ran. The wrapper script above is a first step in this direction.


Update (20140110)
First I solved this problem by writing the stdout and stderr to temporary files. As it turned out, temp. files are not necessary, so the post above was updated accordingly. Here I leave my old solution:

import shlex
import tempfile
from subprocess import call

# Warning! The code above is better! This is the old version!

def get_exitcode_stdout_stderr(cmd):
    """
    Execute the external command and get its exitcode, stdout and stderr.
    """
    args = shlex.split(cmd)

    try:
        fout = tempfile.TemporaryFile()
        ferr = tempfile.TemporaryFile()
        exitcode = call(args, stdout=fout, stderr=ferr)
        fout.seek(0)
        ferr.seek(0)
        out, err = fout.read(), ferr.read()
    finally:
        fout.close()
        ferr.close()
    #
    return exitcode, out, err

<old>
You might be tempted to redirect the stdout and stderr in the function call(...) to a string (to a StringIO) instead of a file. Unfortunately it doesn’t work. Although a StringIO behaves like a file-like object, it’s not a file, thus it doesn’t have a fileno() method and you would get an error because of this (see this thread for instance).

So, we must redirect the outputs to files. After reading the content of these files, they can be removed, thus we use tempfiles from the standard library. When a tempfile.TemporaryFile is closed, it is removed, so we don’t need to unlink them (more info @pymotw and @docs).
</old>

Python equivalent of Java .jar files

January 5, 2014 Leave a comment

Problem
In Java, you can distribute your project in JAR format. It is essentially a ZIP file with some metadata. The project can be launched easily:

$ java -jar project.jar

What is its Python equivalent? How to distribute a Python project (with several modules and packages) in a single file?

Solution
The following is based on this post, written by bheklilr. Thanks for the tip.

Let’s see the following project structure:

MyApp/
    MyApp.py          <--- Main script
    alibrary/
        __init__.py
        alibrary.py
        errors.py
    anotherlib/
        __init__.py
        another.py
        errors.py
    configs/
        config.json
        logging.json

Rename the main script to __main__.py and compress the project to a zip file. The extension can be .egg:

myapp.egg/             <--- technically, it's just a zip file
    __main__.py        <--- Renamed from MyApp.py
    alibrary/
        __init__.py
        alibrary.py
        errors.py
    anotherlib/
        __init__.py
        another.py
        errors.py
    configs/
        config.json
        logging.json

How to zip it? Enter the project directory (MyApp/) and use this command:

zip -r ../myapp.egg .

Now you can launch the .egg file just like you launch a Java .jar file:

$ python myapp.egg

You can also use command-line arguments that are passed to __main__.py.

Categories: python Tags: , , , ,

2014: moving towards Python 3

January 3, 2014 Leave a comment

My new year’s resolution is to move (slowly) towards Python 3. I don’t want to switch yet but I will use things in my code that will facilitate the transition in the future. Thus, from now on I will use this skeleton for my new scripts:

#!/usr/bin/env python
# encoding: utf-8

from __future__ import (absolute_import, division,
                        print_function, unicode_literals)

def main():
    pass

####################

if __name__ == "__main__":
    main()

A future statement must appear near the top of the module. The only lines that can appear before a future statement are:

  • the module docstring (if any),
  • comments,
  • blank lines, and
  • other future statements.

More info on the __future__ imports

Compatibility layers

import from anywhere

January 1, 2014 Leave a comment

Update (20140102): This solution is not perfect either. As discussed here, each time imp.load_source is run on a module, the module-level code in that module will be re-executed, which could cause problems in more complicated cases. Thanks to alb1 for the info.


Problem
You have a project structure with several submodules and in a source you want to import something from another submodule by using relative import. Example:

project/
    main.py
    sub1/
        __init__.py
        helper.py
    sub2/
        __init__.py
        utils.py

As long as you launch main.py only, there are no big problems. However, I like adding simple tests to the end of modules after the ‘if __name__ == "__main__":‘ line. Say utils.py wants to import something from helper.py. How to do it properly so that I can still execute utils.py as a stand-alone script? Unfortunately, this is a real pain in Python :(

Solution
I think I managed to find a simple and elegant solution for this problem. Let’s see utils.py:

def load_src(name, fpath):
    import os, imp
    return imp.load_source(name, os.path.join(os.path.dirname(__file__), fpath))

load_src("helper", "../sub1/helper.py")
import helper
from helper import hi    # if you need just this, "import helper" is not required

print helper.PI
print hi()

Provide two parameters to load_src. The first one is a name. You will be able to import the module using this name. The second parameter is the relative path of the source file you want to import. Note that we can only load files this way, not directories!

I prefer this way because you need to import the desired module explicitly (using the import keyword). From the Zen of Python we know that “explicit is better than implicit”.

Another way would be this:

helper = load_src("helper", "../sub1/helper.py")
print helper.PI

Here helper is imported implicitly, so after loading it you can refer to its content (e.g. helper.PI). I prefer the first version; that makes the source code more readable.

Update: Version 2
Here is an updated version that supports file names given either in absolute or relative path.

def load_src(name, fpath):
    import os, imp
    p = fpath if os.path.isabs(fpath) \
        else os.path.join(os.path.dirname(__file__), fpath)
    return imp.load_source(name, p)

Alternatives
You could also modify the sys.path list to include the parent of “sub1/“. Then you could import helper like this: “from sub1 import helper“. However, modifying sys.path is considered by many as a bad solution.