Archive for the ‘bash’ Category

convert a file to an UTF-8-encoded text

December 16, 2017 Leave a comment

I wrote a simple script that takes an input file, changes its character encoding to UTF-8, and prints the result to the screen.

It’s actually a wrapper around the Unix commands “file” and “iconv“. The goal was to make its usage as simple as possible. The script is here:


$ input.txt

The program tries to detect the encoding of the input file.


Categories: bash, python Tags: ,

create a virtual environment easily

August 30, 2015 Leave a comment

I prefer to put my larger projects in a virtual environment (there are people who put every project in a virt. env. …). I keep my projects in Dropbox (thus they are available on all my machines), but the virt. env.’s are kept outside Dropbox since they can grow quite big (and they are easily reproducible).

For creating virt. env.’s, I use virtualenvwrapper, which (by default) puts virt. env.’s in the folder ~/.virtualenvs. Say I have a project in Dropbox, and I want to create a virt. env. for it. How to do it easily?

First, you need to know if your project is written in Python 2 or Python 3. Then, you need to use the mkvirtualenv command but I always need to look up its syntax. Solution: in the root folder of my project I want a script that will create a virt. env. for the project. Here is the script:

#!/usr/bin/env bash


# which Python version to use in the created virt. env. (2 or 3)

source `which` && mkvirtualenv -p `which python${PYTHON_VER}` $f

Just set PYTHON_VER and launch the script. It will figure out the name of the current folder and create a virt. env. with this name. For instance, if you have your project in ~/projects/stuff, then the virt. env. will be created in the folder ~/.virtualenvs/stuff.

Related links

Categories: bash, python Tags: ,

working with zip files

August 21, 2015 Leave a comment


In a project of mine I had to deal with folders, where a folder can contain several thousands of small text files. I kept this project on Dropbox, so I could use it on all my machines. However, Dropbox is quite slow when trying to synchronize several thousand files. So I decided to put files in a folder into a zip file.

So the question is: how to deal with zip files? How to do basic operations with them: create zip, delete from zip, list zip, add to zip, move to zip, extract from zip, etc.


In this project of mine I used the external zip command as well as the zipfile package from the stdlib. Let’s see both of them.

Manipulating zip files from the command-line
Let’s see some examples. Compress every .json file in the current directory except the desc.json file:

zip -9 *.json -x desc.json

The switch “-9” gives the best compression, is the output, and “-x” is short for “--exclude“. From Python you can call it as an external command with os.system() for instance.

The previous example creates a zip file and leaves the original files. Now let’s move files into a zip file (and delete the original files when they were added successfully to the archive):

zip -9 -m *.json -x desc.json

Delete a file from an archive:

zip -d desc.json

It will delete desc.json from the zip file.

List the content of a zip file:


Add a file to the archive:

zip -g new.json

Where “-g” means: grow.

Extract just one file from a zip file:

# basic:
unzip this.json

# extract to a specific folder:
unzip this.json -d /extract/here/

It will extract this.json from the archive.

Read the content of a zip file in Python
OK, say we have a zip file that contains some files. How to get the filenames? How to read them? I found some nice examples here.

List the file names in a zip file:

import zipfile

zfile = zipfile.ZipFile("", "r")

for name in zfile.namelist():

Read files in a zip file:

import zipfile

zfile = zipfile.ZipFile("", "r")

for name in zfile.namelist():
    data =


Categories: bash, python Tags: ,

fancy text tables

December 28, 2014 Leave a comment

Instead of simply printing some data on the screen, I wanted to put them in a nicely formatted ASCII table.

After some research I found a nice package for this purpose: python-tabulate. It supports both Python 2 and Python 3 (yes, from now on it’s also important for me).

Its usage is very simple. Here is a snippet that creates random usernames and passwords:

from tabulate import tabulate

table = []
headers = ["Username #1", "Username #2", "Password #1", "Password #2"]
for _ in range(10):
    name1 = get_username_1()
    name2 = get_username_2()
    pass1 = get_password_1(8)
    pass2 = get_password_2(12)
    table.append([name1, name2, pass1, pass2])
#        print("{:15}{:15}{:15}{:15}".format(name1, name2, pass1, pass2))    # this is the past :)
print(tabulate(table, headers=headers, tablefmt="psql"))


| Username #1   | Username #2   | Password #1   | Password #2   |
| Adarah        | hasana        | ygyQsF6u      | uTzPqZMDNJ6x  |
| Alary         | begahi        | YqW4aY7q      | ipZuX0sX2RFg  |
| Solita        | otomot        | Xwliu9yi      | IjeFibVFaoZq  |
| Casony        | rikari        | fw6dk5gt      | zbAXO8gd33Lh  |
| Anne          | asakou        | MXsXpz43      | aYNiJTwojULG  |
| Joby          | mgomam        | vZjiCuyT      | qc3Q9caAenJw  |
| Kallita       | aremon        | j1ZD1QU9      | AIEsykmYodfy  |
| Cara          | iumina        | 75UzkKgK      | lK92GdAxn441  |
| Fuscie        | goomio        | uof2C7ct      | HFgVlAZ9PSmv  |
| Dean          | utinon        | gycncz9f      | 61oJzUGdDVKf  |

The module supports various formatting styles. For more examples, check out the official page.

Update (20191029)

The project has moved to .

Categories: bash, python Tags: , ,

deploy a git project to a remote host

October 23, 2013 Leave a comment

You have a project that you develop on your local machine. You want to deploy it (upload) to a production server.

For instance, you have a homepage and you want to upload it to your university server.

I assume your project on the local machine is stored in a dedicated folder. Turn this folder into a git repository. Example:

$ cd /home/jabba/projects/homepage
$ cat index.html
Awesome homepage v0
$ cat .gitignore
$ git init
$ git add .
$ git commit -m "initial commit"

We also need an uploader script called :

git ls-files -z | rsync --files-from - --copy-links -av0 . --rsh="ssh -p$TO_PORT" $TO_HOST:$TO_DIR

Just launch the script and it will copy all the checked in files to the server.

However! If you remove some files on your local machine, this script won’t remove them from the remote host. That is, you will have to delete files which got removed from the project.

This tip is from here.

Related work

  • Using Git to manage a web site (Shortly: you have a local git repo with your project. You create another git repo on the remote host and you push changes to this remote git repo. A customized post-receive hook will copy everything to the specified destination folder.)
Categories: bash Tags: , ,