Speed up Python with Cython
This post is based on a conversation with our local Python guru, Yves :)
You have a script that you would like to speed up. For instance, there is a function that is called lots of times and you suspect it causes a bottleneck.
With Cython, it is possible to compile a module to C source that you can then compile with GCC. The resulting binary can be imported in your Python script just as if it were a normal module. Since it’s a compiled module, you can expect some speed gains.
Example #01 (pure Python)
Let’s see the following simple script. It enumerates numbers up to a given threshold and tests if the given number is prime. At the end it prints the number of primes found.
Pure Python solution:
#!/usr/bin/env python from prime import is_prime UPTO = 10**7 / 4 def main(): i = 1 cnt = 0 while i <= UPTO: if is_prime(i): cnt += 1 i += 1 print cnt if __name__=="__main__": main()
def is_prime(n): if n == 2: return True if n % 2 == 0: return False i = 3 maxi = n**0.5 + 1 while i <= maxi: if n % i == 0: return False i += 2 return True
According to the Unix
time command, the execution time is
29.69 sec. on my laptop.
Example #02 (with Cython, first try)
Now let’s compile
This will produce prime.c. Now compile it:
gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -I/usr/include/python2.7 -o prime.so prime.c
The output is the binary prime.so.
There is nothing to change in the main file, “
from prime import is_prime” will import
Execution time: 21.11 sec.
Example #03 (with Cython, second try)
This paragraph is an update (20111027), incorporating the remarks of James.
By adding static type declarations, Cython can perform much better.
#!/usr/bin/env python import pyximport # add it here, before importing cython code pyximport.install() from prime import is_prime # cython code is imported here UPTO = 10**7 / 4 def main(): i = 1 cnt = 0 while i <= UPTO: if is_prime(i): cnt += 1 i += 1 print cnt if __name__=="__main__": main()
prime.pyx (notice the .pyx extension):
def is_prime(int n): if n == 2: return True if n % 2 == 0: return False cdef int i = 3 cdef double maxi = n**0.5 + 1 while i <= maxi: if n % i == 0: return False i += 2 return True
The line in the first code “import pyximport; pyximport.install()” will ensure that the cython module is automatically built when imported, thus there is no need to run cython or gcc.
Execution time: 2.15 sec. Lesson learned: use static type declarations in your Cython code whenever possible.
Example (with PyPy)
Just out of curiosity, I tried to launch the script with PyPy too. PyPy is a fast, compliant alternative implementation of the Python language, written in Python itself. Since it uses a JIT compiler, PyPy is often faster than the standard Python interpreter (see a presentation here).
Execution time (hang on!): 2.35 sec.
Well, the difference is quite spectacular in the case of this example but it doesn’t mean that PyPy is always faster. In a completely different problem setting the end result can be just the opposite. So always make some tests and then choose the solution which is best for you.
If your program seems to run slowly, first try to polish the code and use some better algorithms / data structures. If it’s still slow, you can try to compile some parts of it with Cython. However, bare in mind that you hurt portability. But before transforming your program to a half Python / half C monster, try PyPy too. Maybe you don’t need Cython at all.