Archive

Posts Tagged ‘static’

mypy — optional static typing

July 25, 2018 Leave a comment

Problem
Python is awesome, it’s my favourite programming language (what a surprise :)). Dynamic typing allows one to code very quickly. However, if you have a large codebase (and I ran into this problem with my latest project JiVE), things start to get complicated. Let’s take an example:

def extract_images(html):
    lst = []
    ...
    return lst

If I look at this code, then what is “html”? Is it raw HTML (string), or is it a BeautifulSoup object? What is the return value? What is in the returned list? Does it return a list of URLs (a list of strings)? Or is it a list of custom objects that wrap the URLs?

If we used a statically-typed language (e.g. Java), these questions wouldn’t even arise since we could see the types of the parameters and the return value.

Solution
I’ve heard about mypy but I never cared to try it. Until now… “Mypy is an experimental optional static type checker for Python that aims to combine the benefits of dynamic (or “duck”) typing and static typing. Mypy combines the expressive power and convenience of Python with a powerful type system and compile-time type checking.” (source)

Thus, the example above could be written like this:

from typing import List
from bs4 import BeautifulSoup
from myimage import Image

def extract_images(html: BeautifulSoup) -> List[Image]:
    lst = []
    ...
    return lst

And now everything is clear and there is no need to carefully analyse the code to figure out what goes in and what comes out.

This new syntax was introduced in Python 3.6. Actually, if you run such an annotated program, the Python interpreter ignores all these hints. So Python won’t become a statically-typed language. If you want to make these hints work, you need to use “mypy”, which is a linter-like static analyzer. Example:

$ pip install mypy    # you can also install it globally with sudo
$ mypy program.py     # verify a file
$ mypy src/           # verify every file in a folder

With mypy you can analyse individual files and you can also analyse every file in a folder.

If you get warnings that mypy doesn’t find certain modules, then run mypy with these options:

$ mypy program.py --ignore-missing-imports --follow-imports=skip

IDE support
I highly recommend using PyCharm for large(r) projects. PyCharm has its own implementation of a static type checker. You can use type hints out of the box and PyCharm will tell you if there’s a problem. It’s a good idea to combine PyCharm with Mypy, i.e. when you edit the code in the IDE, run mypy from time to time in the terminal too.

Getting started
To get started, I suggest watching/reading these resources:

It’s really easy to get started with mypy. It took me only 2 days to fully annotate my project JiVE. Now the code is much easier to understand IMO.

Tips
If you have a large un-annotated codebase, proceed from bottom up. Start to annotate files that are leaf nodes in the dependency tree/graph. Start with modules that are most used by others (e.g. helper.py, utils.py, common.py). Then proceed upwards until you reach the main file (that I usually call main.py, which is the entry point of the whole project).

You don’t need to annotate everything. Type hints are optional. The more you add, the better, but if there’s a function that you find difficult to annotate, just skip it and come back to it later.

Annotate the function signatures (type of arguments, type of the return value). Inside a function I don’t annotate every variable. If mypy drops a warning and says a variable should be annotated, then I do it.

Sometimes mypy drops an error on a line but you don’t want to annotate it. In this case you can add a special comment to tell mypy to ignore this line:

    ...    # type: ignore

If a function’s signature has no type hints at all, mypy will skip it. If you want mypy to check that function, then add at least one type hint to it. You can add for instance the return type. If the function is a procedure, i.e. it has no return value, then indicate None as the returned type:

def hello() -> None:
    print("hello")

You can add type hints later. That is, you can write your project first, test it, and when it works fine, you can add type hints at the end.

When to use mypy?
For a small script it may not be necessary but it could add a lot to a large(r) project.

Notes
If you read older blog posts, you may find that they mention the package “mypy-lang”. It’s old. Install the package “mypy” and forget “mypy-lang”. More info here.

Categories: python Tags: , , , ,

[django] init_app.py: complete the initialization of an app.

August 4, 2014 Leave a comment

I wrote a simple script called init_app.py that completes the initialization of a Django application.

You create a new app. like this:

./manage.py startapp APP_NAME

However, the generated app. is far from complete. Put init_app.py next to manage.py (in the same folder) and execute the following command:

./init_app.py APP_NAME

Currently, it performs the following actions:

  • inside the app., create the folder static/APP_NAME
  • inside the app., create the folder templates/APP_NAME
  • inside the app., create a sample urls.py file

Links

Static HTML filelist generator

March 26, 2011 1 comment

Problem

On our webserver I had some files in a directory that I wanted to browse online. However, the webserver didn’t generate a list of links on these files when I pointed the browser to this directory.

Solution

Idea: write a script that traverses the current directory recursively and prints all files with a link to them. I didn’t need any fancy features so the script can produce a very simple output.

Download link: here. Source code:

#!/usr/bin/env python

# index_gen.py

import os
import os.path
import sys

class SimpleHtmlFilelistGenerator:
    # start from this directory
    base_dir = None

    def __init__(self, dir):
        self.base_dir = dir

    def print_html_header(self):
        print """<html>
<body>
<code>
""",

    def print_html_footer(self):
        print """</code>
</body>
</html>
""",

    def processDirectory ( self, args, dirname, filenames ):
        print '<strong>', dirname + '/', '</strong>', '<br>'
        for filename in sorted(filenames):
            rel_path = os.path.join(dirname, filename)
            if rel_path in [sys.argv[0], './index.html']:
                continue   # exclude this generator script and the generated index.html
            if os.path.isfile(rel_path):
                href = "<a href=\"%s\">%s</a>" % (rel_path, filename)
                print '&nbsp;' * 4, href, '<br>'

    def start(self):
        self.print_html_header()
        os.path.walk( self.base_dir, self.processDirectory, None )
        self.print_html_footer()

# class SimpleHtmlFilelistGenerator

if __name__ == "__main__":
    base_dir = '.'
    if len(sys.argv) > 1:
        base_dir = sys.argv[1]
    gen = SimpleHtmlFilelistGenerator(base_dir)
    gen.start()

Usage:

Simply launch it in the directory where you need the filelist. Redirect the output to index.html:

./index_gen.py >index.html

Don’t forget to set the rights of index.html (chmod 644 index.html).

Demo:

Update (20141202)
This version here works but it’s quite primitive. We made a much better version; check it out here: https://pythonadventures.wordpress.com/2014/12/02/static-html-file-browser-for-dropbox/.