update all pip packages

Problem
You want to update all the pip-installed Python packages on your system.

Solution
At http://stackoverflow.com/questions/2720014/ there is a very nice little script for this task:

import pip
from subprocess import call

for dist in pip.get_installed_distributions():
    call("pip install --upgrade " + dist.project_name, shell=True)

I modified it a bit to update packages in ascending order by name in an ignore-case way:

#!/usr/bin/env python

import os
import pip

dists = []
for dist in pip.get_installed_distributions():
    dists.append(dist.project_name)

for dist_name in sorted(dists, key=lambda s: s.lower()):
    cmd = "sudo pip install {0} -U".format(dist_name)
    print '#', cmd
    #os.system(cmd)

I want to update the packages globally, that’s why the “sudo“. If you run it, it will just print the commands to the stdout (dry run). If everything is OK, uncomment the last line.

Update (20140121)
I made a github project of it: https://github.com/jabbalaci/update-pip-packages.

Stack Overflow has a Python info page

Stack Overflow has a Python info page at http://stackoverflow.com/tags/python/info .

There are other info pages too, just replace the substring “python” in the link above with a different keyword.

Get yourself analyzed on GitHub

If you have a GitHub account, you can get yourself analyzed with The Open Source Report Card.

Here is my report for instance. Scroll down to the bottom and try it with your username.

It will also list users whose activity is similar to yours.

Problems with PIL? Use Pillow instead!

Problem
PIL started to report the following error: “IOError: decoder zip not available“. I tried to solve this issue following these tips but I had no luck:

• http://obroll.com/install-python-pil-python-image-library-on-ubuntu-11-10-oneiric/
• http://www.lancejian.com/2011/04/04/pil-ioerror-decoder-jpeg-not-available.html

The funny thing is that PIL shows the following statistics:

--------------------------------------------------------------------
    PIL 1.1.7 SETUP SUMMARY
    --------------------------------------------------------------------
    version       1.1.7
    platform      linux2 2.7.4 (default, Apr 19 2013, 18:28:01)
                  [GCC 4.7.3]
    --------------------------------------------------------------------
    *** TKINTER support not available (Tcl/Tk 8.5 libraries needed)
    --- JPEG support available
    --- ZLIB (PNG/ZIP) support available
    --- FREETYPE2 support available
    *** LITTLECMS support not available
    --------------------------------------------------------------------

Notice the line “ZLIB (PNG/ZIP) support available“. That’s a lie! :)

Solution
I got fed up with PIL and switched to Pillow, which is a fork of PIL. It solved the problem for me. Remove PIL and install Pillow:

sudo pip uninstall PIL
sudo pip install pillow -U

Pillow produces the following statistics:

--------------------------------------------------------------------
    SETUP SUMMARY (Pillow 2.0.0 fork, originally based on PIL 1.1.7)
    --------------------------------------------------------------------
    version      2.0.0 (Pillow)
    platform     linux2 2.7.4 (default, Apr 19 2013, 18:28:01)
                 [GCC 4.7.3]
    --------------------------------------------------------------------
    *** TKINTER support not available
    (Tcl/Tk 8.5 libraries needed)
    --- JPEG support available
    --- ZLIB (PNG/ZIP) support available
    *** TIFF G3/G4 (experimental) support not available
    --- FREETYPE2 support available
    *** LITTLECMS support not available
    *** WEBP support not available
    --------------------------------------------------------------------

Now test if everything is OK:

$ python
Python 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import Image
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named Image
>>> from PIL import Image
>>>

The first attempt is used with PIL. If it fails: good, PIL is removed. The second one is for Pillow. If you can import successfully: good!

If you want to port an existing PIL project to Pillow, just replace “import Image” with “from PIL import Image“.

Splinter: open Firefox in fullscreen mode

Problem
With Splinter you can automate a browser window (click on a button, type in some text, etc). You can also use a Firefox instance beside Chrome and some other browsers. But how to open the Firefox instance in fullscreen (as if you had clicked on the “maximize” button)? Strangely, there is no command-line option for this :(

Solution
Well, under Linux there are some tools that allows you to interact with windows:

• xwininfo
• xdotool
• wmctrl

When the Firefox instance is opened, it becomes the active window and I ask its window ID with “xdotool getactivewindow”. Then, with “wmctrl” I can toggle this window to fullscreen.

Demonstration:

jabba@jabba-uplink:~$ xdotool getactivewindow
109051940
jabba@jabba-uplink:~$ python
Python 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> hex(109051940)
'0x6800024'
jabba@jabba-uplink:~$ wmctrl -i -r 0x6800024 -b toggle,maximized_vert,maximized_horz

The same in Python is available in my jabbapylib library here.

Splinter patch: open the Chrome browser window in a maximized way

Problem
With Splinter, I would like to open the Chrome browser window in a maximized way, i.e. it should fill the whole screen.

Solution
As a temporary solution, I patched my /usr/local/lib/python2.7/dist-packages/splinter/driver/webdriver/chrome.py file by adding the following line:

options.add_argument("--start-maximized")

_{I also reported this idea to the authors of Splinter here.}

Scrape an HTML table

Problem
You want to extract an HTML table and get it in .csv, .json, etc. format.

Solution
I found a nice solution in this SO thread. The script is here: tablescrape.py.