Posts on Python and Big Data

See Neal Caren’s tutorials and posts on Python and Big Data here:
http://nealcaren.web.unc.edu/big-data/
.

How to make a python, command-line program autocomplete arbitrary things

This entry is based on this SO post.

Problem
You have an interactive command-line Python script and you want to add autocompletion to it when hitting the TAB key.

Solution
Here is a working example (taken from here):

import readline

addrs = ['angela@domain.com', 'michael@domain.com', 'david@test.com']

def completer(text, state):
    options = [x for x in addrs if x.startswith(text)]
    try:
        return options[state]
    except IndexError:
        return None

readline.set_completer(completer)
readline.parse_and_bind("tab: complete")

while True:
    inp = raw_input("> ")
    print "You entered", inp

bash-like functionalities in command-line Python script

Problem
You have an interactive Python script that reads input from the command line. You want to add bash-like functionalities to it like moving the cursor with the arrows, jump to the front with Home, jump to the end with End, browse previous commands with the up arrow, etc.

Solution
You won’t believe what is needed for this:

import readline

Yes, that’s it. Just import it and you are good to go.

Minimal example:

import readline

while True:
    inp = raw_input("> ")
    print "You entered", inp

Alternative
Before discovering the readline module, I used to start my scripts with “rlwrap“:

rlwrap my_script.py

It does the trick too. I put the line above in a script called “my_script.sh” and I launched this latter one. However, “import readline” is simpler.

update all pip packages

Problem
You want to update all the pip-installed Python packages on your system.

Solution
At
http://stackoverflow.com/questions/2720014/
there is a very nice little script for this task:

import pip
from subprocess import call

for dist in pip.get_installed_distributions():
    call("pip install --upgrade " + dist.project_name, shell=True)

I modified it a bit to update packages in ascending order by name in an ignore-case way:

#!/usr/bin/env python

import os
import pip

dists = []
for dist in pip.get_installed_distributions():
    dists.append(dist.project_name)

for dist_name in sorted(dists, key=lambda s: s.lower()):
    cmd = "sudo pip install {0} -U".format(dist_name)
    print '#', cmd
    #os.system(cmd)

I want to update the packages globally, that’s why the “sudo“. If you run it, it will just print the commands to the stdout (dry run). If everything is OK, uncomment the last line.

Stack Overflow has a Python info page

Stack Overflow has a Python info page at
http://stackoverflow.com/tags/python/info
.

There are other info pages too, just replace the substring “python” in the link above with a different keyword.

Get yourself analyzed on GitHub

If you have a GitHub account, you can get yourself analyzed with The Open Source Report Card.

Here is my report for instance. Scroll down to the bottom and try it with your username.

It will also list users whose activity is similar to yours.

Problems with PIL? Use Pillow instead!

Problem
PIL started to report the following error: “IOError: decoder zip not available“. I tried to solve this issue following these tips but I had no luck:

• 
  http://obroll.com/install-python-pil-python-image-library-on-ubuntu-11-10-oneiric/
  
• 
  http://www.lancejian.com/2011/04/04/pil-ioerror-decoder-jpeg-not-available.html
  

The funny thing is that PIL shows the following statistics:

--------------------------------------------------------------------
    PIL 1.1.7 SETUP SUMMARY
    --------------------------------------------------------------------
    version       1.1.7
    platform      linux2 2.7.4 (default, Apr 19 2013, 18:28:01)
                  [GCC 4.7.3]
    --------------------------------------------------------------------
    *** TKINTER support not available (Tcl/Tk 8.5 libraries needed)
    --- JPEG support available
    --- ZLIB (PNG/ZIP) support available
    --- FREETYPE2 support available
    *** LITTLECMS support not available
    --------------------------------------------------------------------

Notice the line “ZLIB (PNG/ZIP) support available“. That’s a lie! :)

Solution
I got fed up with PIL and switched to Pillow, which is a fork of PIL. It solved the problem for me. Remove PIL and install Pillow:

sudo pip uninstall PIL
sudo pip install pillow -U

Pillow produces the following statistics:

--------------------------------------------------------------------
    SETUP SUMMARY (Pillow 2.0.0 fork, originally based on PIL 1.1.7)
    --------------------------------------------------------------------
    version      2.0.0 (Pillow)
    platform     linux2 2.7.4 (default, Apr 19 2013, 18:28:01)
                 [GCC 4.7.3]
    --------------------------------------------------------------------
    *** TKINTER support not available
    (Tcl/Tk 8.5 libraries needed)
    --- JPEG support available
    --- ZLIB (PNG/ZIP) support available
    *** TIFF G3/G4 (experimental) support not available
    --- FREETYPE2 support available
    *** LITTLECMS support not available
    *** WEBP support not available
    --------------------------------------------------------------------

Now test if everything is OK:

$ python
Python 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Image
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named Image
>>> from PIL import Image
>>>

The first attempt is used with PIL. If it fails: good, PIL is removed. The second one is for Pillow. If you can import successfully: good!

If you want to port an existing PIL project to Pillow, just replace “import Image” with “from PIL import Image“.