Posts Tagged ‘scraping’

scraping Steam, get through the age check

You want to scrape Steam but sometimes Steam brings up an age check. How to get through it from a script? Examples: Fallout: New Vegas (age check), PST: EE (additional maturity check?).


import requests

    'birthtime': '283993201',
    'mature_content': '1',
URL = ""

r = requests.get(URL, cookies=COOKIES)

Tip from here.

Categories: python Tags: , ,

Get the IMDb Top 250 list

August 19, 2016 Leave a comment

From IMDb you want to get the list of the Top 100 movies.

There is a Top 250 list here: To access IMDb info, I use the excellent imdbpy package. It has a get_top250_movies() function but it returns an empty list :)

During my research I found this post on SO. It suggests that one should download the official IMDb dump from here. The Top 250 list is in the file ratings.list.gz. However, this file doesn’t contain the IMDb IDs of the movies, so it’s good for nothing :(

There was only one solution left: let’s do some scraping. Here is the Python code that did the job for me. I didn’t use BeautifulSoup just plain ol’ regular expressions:

import requests
import re

top250_url = ""

def get_top250():
    r = requests.get(top250_url)
    html = r.text.split("\n")
    result = []
    for line in html:
        line = line.rstrip("\n")
        m ='data-titleid="tt(\d+?)">', line)
        if m:
            _id =
    return result

It returns the IMDb IDs of the Top 250 movies. Then, using the imdbpy package you can ask all the information about a movie, since you have the movie ID.


Categories: python Tags: , , , ,