Author Archive

Get the IMDb Top 250 list

August 19, 2016 Leave a comment

From IMDb you want to get the list of the Top 100 movies.

There is a Top 250 list here: To access IMDb info, I use the excellent imdbpy package. It has a get_top250_movies() function but it returns an empty list :)

During my research I found this post on SO. It suggests that one should download the official IMDb dump from here. The Top 250 list is in the file ratings.list.gz. However, this file doesn’t contain the IMDb IDs of the movies, so it’s good for nothing :(

There was only one solution left: let’s do some scraping. Here is the Python code that did the job for me. I didn’t use BeautifulSoup just plain ol’ regular expressions:

import requests
import re

top250_url = ""

def get_top250():
    r = requests.get(top250_url)
    html = r.text.split("\n")
    result = []
    for line in html:
        line = line.rstrip("\n")
        m ='data-titleid="tt(\d+?)">', line)
        if m:
            _id =
    return result

It returns the IMDb IDs of the Top 250 movies. Then, using the imdbpy package you can ask all the information about a movie, since you have the movie ID.


Categories: python Tags: , , , ,

string distances

August 17, 2016 Leave a comment

See the Jellyfish project: “Jellyfish is a python library for doing approximate and phonetic matching of strings“.

Jellyfish implements the following algorithms: Levenshtein Distance, Damerau-Levenshtein Distance, Jaro Distance, Jaro-Winkler Distance, Match Rating Approach Comparison, Hamming Distance.

See the project page for more info.

Categories: python Tags: ,

compile lxml on Ubuntu 16.04

August 4, 2016 Leave a comment

lxml doesn’t want to compile on Ubuntu 16.04.


$ sudo apt install libxml2-dev libxslt1-dev python-dev zlib1g-dev

I was getting the error “/usr/bin/ld: cannot find -lz“. It turned out that the package zlib1g-dev was the cure…

Note that this is for Python 2. For Python 3 you might need to install the package python3-dev.

Categories: python, ubuntu Tags: ,

installing a Flask webapp on a Digital Ocean Ubuntu 16.04 box using Systemd

August 4, 2016 Leave a comment

I’ve updated my Digital Ocean Flask notes on GitHub. Now it includes information about installing a Flask webapp on a Digital Ocean Ubuntu 16.04 box using Systemd.

Categories: flask, python, ubuntu Tags: ,


July 17, 2016 Leave a comment

Using Flask-RESTful, I needed an API endpoint that accepts JSON data.

I found the solution here: You can copy / paste that code. Note that the JSON data is POSTed to your API endpoint, thus you need to implement the post() method.

However, how to test it?

1) using cURL:

$ curl -i -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d "{\"Hello\":\"Karl\"}" http://domain/your_api_endpoint

Damn, that’s compicated, right? Is there an easier way?

2) using httpie:
You can install httpie with your favorite package manager. Then:

$ http POST http://domain/your_api_endpoint Hello=Karl

get the tweets of a user and save them in CSV

July 16, 2016 Leave a comment
Categories: python Tags: , , ,

remove tags from HTML

July 13, 2016 Leave a comment

You have an HTML string and you want to remove all the tags from it.

Install the package “bleach” via pip. Then:

>>> import bleach
>>> html = "Her <h1>name</h1> was <i>Jane</i>."
>>> cleaned = bleach.clean(html, tags=[], attributes={}, styles=[], strip=True)
>>> html
'Her <h1>name</h1> was <i>Jane</i>.'
>>> cleaned
'Her name was Jane.'

Tip from here.

Categories: python Tags: , ,

Get every new post delivered to your Inbox.

Join 120 other followers