Archive

Archive for July, 2013

Generate random hash

Problem
MongoDB generates 96 bit hash values that are used as primary keys. In a project of mine I also needed randomly generated primary keys so I decided to go the MongoDB way. So the question is: how to generate 96 bit hash values with Python?

Solution

#!/usr/bin/env python

import random


def my_hash(bits=96):
    assert bits % 8 == 0
    required_length = bits / 8 * 2
    s = hex(random.getrandbits(bits)).lstrip('0x').rstrip('L')
    if len(s) < required_length:
        return my_hash(bits)
    else:
        return s


def main():
    for _ in range(3):
        print my_hash()

#########################################################

if __name__ == "__main__":
    main()

Sample output:

f4bf4a4c949d7beee38d84a3
457ef2f29f462a4f1e54b61e
dc921ad1e6c32bc8ce8503c8

Another (but about 3.5 times slower) solution:

def my_hash(bits=96):
    assert bits % 8 == 0
    return os.urandom(bits/8).encode('hex')

urandom needs the number of bytes as its parameter.

Tips from here.

Update (20130813)
I found a related work called SimpleFlake. SimpleFlake generates 64 bit IDs, where the ID is prefixed with a millisecond timestamp and the remaining bits are completely random. It has the advantage that IDs show the chronological order of ID creation.

A basic socket client server example

July 6, 2013 2 comments

Here I present a basic socket client server example. It can be used as a starting point for a more serious project.

Problem
You want to have a server that is listening on a port. The server can receive data and these pieces of data must be processed one after the other. That is, the server must manage a queue and the received data are put in this queue. The elements in the queue must be processed one by one.

We also need a client whose job is to send data to the server.

(1) Common config part

# config.py
PORT=3030

The port number won’t be repeated in the server and in the client. Instead, it is read from a config file.

(2) The server

#!/usr/bin/env python
# server.py

import socket
import select
import config as cfg
import Queue
from threading import Thread
from time import sleep
from random import randint
import sys

class ProcessThread(Thread):
    def __init__(self):
        super(ProcessThread, self).__init__()
        self.running = True
        self.q = Queue.Queue()

    def add(self, data):
        self.q.put(data)

    def stop(self):
        self.running = False

    def run(self):
        q = self.q
        while self.running:
            try:
                # block for 1 second only:
                value = q.get(block=True, timeout=1)
                process(value)
            except Queue.Empty:
                sys.stdout.write('.')
                sys.stdout.flush()
        #
        if not q.empty():
            print "Elements left in the queue:"
            while not q.empty():
                print q.get()

t = ProcessThread()
t.start()

def process(value):
    """
    Implement this. Do something useful with the received data.
    """
    print value
    sleep(randint(1,9))    # emulating processing time

def main():
    s = socket.socket()         # Create a socket object
    host = socket.gethostname() # Get local machine name
    port = cfg.PORT                # Reserve a port for your service.
    s.bind((host, port))        # Bind to the port
    print "Listening on port {p}...".format(p=port)

    s.listen(5)                 # Now wait for client connection.
    while True:
        try:
            client, addr = s.accept()
            ready = select.select([client,],[], [],2)
            if ready[0]:
                data = client.recv(4096)
                #print data
                t.add(data)
        except KeyboardInterrupt:
            print
            print "Stop."
            break
        except socket.error, msg:
            print "Socket error! %s" % msg
            break
    #
    cleanup()

def cleanup():
    t.stop()
    t.join()

#########################################################

if __name__ == "__main__":
    main()

We create a thread that is running in the background and manages the queue. In a loop it checks if there is an element in the queue. If yes, then it takes out the first element and processes it.

In the main function we start listening on a given port. Incoming data are passed to the thread, where the thread puts it in the queue.

You can stop the server with CTRL+C. The cleanup method stops the thread nicely: it changes a variable to False, thus the thread quits from its infinite loop. Notice the parameters of q.get: we block for 1 second only. This way we have a chance to stop the thread even if the queue is empty. With t.join() we wait until it completely stops.

(3) The client

#!/usr/bin/env python
# client.py

import config as cfg
import sys
import socket

def main(elems):
    try:
        for e in elems:
            client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            host = socket.gethostname()
            client.connect((host, cfg.PORT))
            client.send(e)
            client.shutdown(socket.SHUT_RDWR)
            client.close()
    except Exception as msg:
        print msg

#########################################################

if __name__ == "__main__":
    main(sys.argv[1:])

Command line parameters are passed one by one to the server to be processed.

Usage
Start the server in a terminal:

$ ./server.py 
Listening on port 3030...
........

Start the client in another terminal and pass some values to the server:

$ ./client.py 1 5 8

Switch back to the server:

$ ./server.py 
Listening on port 3030...
............1
5
8
....

Related links

[ discussion @reddit ]

Categories: python Tags: , ,

Add History and Tab Completion to the Default Python Shell

Problem
You want history and TAB completion in your default Python shell. You are aware of bpython and ipython, but you miss this functionality in the default shell.

Solution
Edit ~/.bashrc:

# ~/.bashrc
export PYTHONSTARTUP=$HOME/.pythonstartup.py

Download pythonstartup.py from here and rename it to ~/.pythonstartup.py.

Call Python from C

July 1, 2013 1 comment

Problem
You want to embed Python code inside a C program.

Solution
See this: http://docs.python.org/2/extending/embedding.html.

Example:

#include "Python.h"

int main(int argc, char *argv[])
{
  Py_SetProgramName(argv[0]);  /* optional but recommended */
  Py_Initialize();
  PyRun_SimpleString("from time import time,ctime\n"
                     "print 'Today is',ctime(time())\n");
  Py_Finalize();
  return 0;
}

Compilation:

gcc -I/usr/include/python2.7 cool.c -lpython2.7
# or:
clang -I/usr/include/python2.7 -lpython2.7 cool.c

Sample output:

Today is Mon Jul  1 00:53:10 2013

For more details, please refer to this page.

Update (20140216)
I tried this example today and clang dropped me this error:

/usr/include/limits.h:124:16: fatal error: 'limits.h' file not found

I have Clang 3.2 on my machine and it’s a known bug. There is a workaround:

$ cd /usr/lib/clang/3.2/
$ sudo ln -s /usr/lib/llvm-3.2/lib/clang/3.2/include