Archive

Archive for the ‘bash’ Category

convert a file to an UTF-8-encoded text

December 16, 2017 Leave a comment

I wrote a simple script that takes an input file, changes its character encoding to UTF-8, and prints the result to the screen.

It’s actually a wrapper around the Unix commands “file” and “iconv“. The goal was to make its usage as simple as possible. The script is here: to_utf8.py.

Usage:

$ to_utf8.py input.txt

The program tries to detect the encoding of the input file.

Links

Categories: bash, python Tags: ,

create a virtual environment easily

August 30, 2015 Leave a comment

Problem
I prefer to put my larger projects in a virtual environment (there are people who put every project in a virt. env. …). I keep my projects in Dropbox (thus they are available on all my machines), but the virt. env.’s are kept outside Dropbox since they can grow quite big (and they are easily reproducible).

For creating virt. env.’s, I use virtualenvwrapper, which (by default) puts virt. env.’s in the folder ~/.virtualenvs. Say I have a project in Dropbox, and I want to create a virt. env. for it. How to do it easily?

Solution
First, you need to know if your project is written in Python 2 or Python 3. Then, you need to use the mkvirtualenv command but I always need to look up its syntax. Solution: in the root folder of my project I want a script that will create a virt. env. for the project. Here is the script:

#!/usr/bin/env bash

# mk_venv.sh

# which Python version to use in the created virt. env. (2 or 3)
PYTHON_VER=2

a=`pwd`
f=${a##*/}
source `which virtualenvwrapper.sh` && mkvirtualenv -p `which python${PYTHON_VER}` $f

Just set PYTHON_VER and launch the script. It will figure out the name of the current folder and create a virt. env. with this name. For instance, if you have your project in ~/projects/stuff, then the virt. env. will be created in the folder ~/.virtualenvs/stuff.

Related links

Categories: bash, python Tags: ,

working with zip files

August 21, 2015 Leave a comment

Problem

In a project of mine I had to deal with folders, where a folder can contain several thousands of small text files. I kept this project on Dropbox, so I could use it on all my machines. However, Dropbox is quite slow when trying to synchronize several thousand files. So I decided to put files in a folder into a zip file.

So the question is: how to deal with zip files? How to do basic operations with them: create zip, delete from zip, list zip, add to zip, move to zip, extract from zip, etc.

Solution

In this project of mine I used the external zip command as well as the zipfile package from the stdlib. Let’s see both of them.

Manipulating zip files from the command-line
Let’s see some examples. Compress every .json file in the current directory except the desc.json file:

zip -9 files.zip *.json -x desc.json

The switch “-9” gives the best compression, files.zip is the output, and “-x” is short for “--exclude“. From Python you can call it as an external command with os.system() for instance.

The previous example creates a zip file and leaves the original files. Now let’s move files into a zip file (and delete the original files when they were added successfully to the archive):

zip -9 -m files.zip *.json -x desc.json

Delete a file from an archive:

zip -d files.zip desc.json

It will delete desc.json from the zip file.

List the content of a zip file:

zipinfo files.zip

Add a file to the archive:

zip -g files.zip new.json

Where “-g” means: grow.

Extract just one file from a zip file:

# basic:
unzip files.zip this.json

# extract to a specific folder:
unzip files.zip this.json -d /extract/here/

It will extract this.json from the archive.

Read the content of a zip file in Python
OK, say we have a zip file that contains some files. How to get the filenames? How to read them? I found some nice examples here.

List the file names in a zip file:

import zipfile

zfile = zipfile.ZipFile("files.zip", "r")

for name in zfile.namelist():
    print(name)

Read files in a zip file:

import zipfile

zfile = zipfile.ZipFile("files.zip", "r")

for name in zfile.namelist():
    data = zfile.read(name)
    print(data)

Links

Categories: bash, python Tags: ,

fancy text tables

December 28, 2014 Leave a comment

Problem
Instead of simply printing some data on the screen, I wanted to put them in a nicely formatted ASCII table.

Solution
After some research I found a nice package for this purpose: python-tabulate. It supports both Python 2 and Python 3 (yes, from now on it’s also important for me).

Its usage is very simple. Here is a snippet that creates random usernames and passwords:

from tabulate import tabulate

table = []
headers = ["Username #1", "Username #2", "Password #1", "Password #2"]
for _ in range(10):
    name1 = get_username_1()
    name2 = get_username_2()
    pass1 = get_password_1(8)
    pass2 = get_password_2(12)
    table.append([name1, name2, pass1, pass2])
#        print("{:15}{:15}{:15}{:15}".format(name1, name2, pass1, pass2))    # this is the past :)
print(tabulate(table, headers=headers, tablefmt="psql"))

Output:

+---------------+---------------+---------------+---------------+
| Username #1   | Username #2   | Password #1   | Password #2   |
|---------------+---------------+---------------+---------------|
| Adarah        | hasana        | ygyQsF6u      | uTzPqZMDNJ6x  |
| Alary         | begahi        | YqW4aY7q      | ipZuX0sX2RFg  |
| Solita        | otomot        | Xwliu9yi      | IjeFibVFaoZq  |
| Casony        | rikari        | fw6dk5gt      | zbAXO8gd33Lh  |
| Anne          | asakou        | MXsXpz43      | aYNiJTwojULG  |
| Joby          | mgomam        | vZjiCuyT      | qc3Q9caAenJw  |
| Kallita       | aremon        | j1ZD1QU9      | AIEsykmYodfy  |
| Cara          | iumina        | 75UzkKgK      | lK92GdAxn441  |
| Fuscie        | goomio        | uof2C7ct      | HFgVlAZ9PSmv  |
| Dean          | utinon        | gycncz9f      | 61oJzUGdDVKf  |
+---------------+---------------+---------------+---------------+

The module supports various formatting styles. For more examples, check out the official page.

Update (20191029)

The project has moved to https://github.com/astanin/python-tabulate .

Categories: bash, python Tags: , ,

deploy a git project to a remote host

October 23, 2013 Leave a comment

Problem
You have a project that you develop on your local machine. You want to deploy it (upload) to a production server.

For instance, you have a homepage and you want to upload it to your university server.

Solution
I assume your project on the local machine is stored in a dedicated folder. Turn this folder into a git repository. Example:

$ cd /home/jabba/projects/homepage
$ cat index.html
Awesome homepage v0
$ cat .gitignore
.gitignore
deploy.sh
$ git init
$ git add .
$ git commit -m "initial commit"

We also need an uploader script called deploy.sh :

TO_HOST=<username>@<remote_host>
TO_PORT=<port>
TO_DIR=<remote_dir_on_server_with_absolute_path>
git ls-files -z | rsync --files-from - --copy-links -av0 . --rsh="ssh -p$TO_PORT" $TO_HOST:$TO_DIR

Just launch the script and it will copy all the checked in files to the server.

However! If you remove some files on your local machine, this script won’t remove them from the remote host. That is, you will have to delete files which got removed from the project.

Credits
This tip is from here.

Related work

  • Using Git to manage a web site (Shortly: you have a local git repo with your project. You create another git repo on the remote host and you push changes to this remote git repo. A customized post-receive hook will copy everything to the specified destination folder.)
Categories: bash Tags: , ,