pandas: add new columns; reorder columns

Problem
I had a CSV file and (1) I wanted to add some new columns, and (2) I wanted to reorder the columns.

Solution
Instead of doing it manually, I used the pandas library for this job.

Input file:

number,season,episode,airdate,title,TVmaze link
1,1,1,24 Jun 15,"eps1.0_hellofriend.mov","http://www.tvmaze.com/episodes/157154/mr-robot-1x01-eps10hellofriendmov"
2,1,2,01 Jul 15,"eps1.1_ones-and-zer0es.mpeg","http://www.tvmaze.com/episodes/167379/mr-robot-1x02-eps11ones-and-zer0esmpeg"
3,1,3,08 Jul 15,"eps1.2_d3bug.mkv","http://www.tvmaze.com/episodes/167380/mr-robot-1x03-eps12d3bugmkv"
4,1,4,15 Jul 15,"eps1.3_da3m0ns.mp4","http://www.tvmaze.com/episodes/167381/mr-robot-1x04-eps13da3m0nsmp4"
5,1,5,22 Jul 15,"eps1.4_3xpl0its.wmv","http://www.tvmaze.com/episodes/167382/mr-robot-1x05-eps143xpl0itswmv"
6,1,6,29 Jul 15,"eps1.5_br4ve-trave1er.asf","http://www.tvmaze.com/episodes/167383/mr-robot-1x06-eps15br4ve-trave1erasf"
7,1,7,05 Aug 15,"eps1.6_v1ew-s0urce.flv","http://www.tvmaze.com/episodes/167384/mr-robot-1x07-eps16v1ew-s0urceflv"
8,1,8,12 Aug 15,"eps1.7_wh1ter0se.m4v","http://www.tvmaze.com/episodes/167385/mr-robot-1x08-eps17wh1ter0sem4v"
9,1,9,19 Aug 15,"eps1.8_m1rr0r1ng.qt","http://www.tvmaze.com/episodes/167386/mr-robot-1x09-eps18m1rr0r1ngqt"
10,1,10,02 Sep 15,"eps1.9_zer0-day.avi","http://www.tvmaze.com/episodes/167387/mr-robot-1x10-eps19zer0-dayavi"

Desired output:

number,season,episode,prod_code,airdate,title,special,TVmaze link
1,1,1,,24 Jun 15,eps1.0_hellofriend.mov,,http://www.tvmaze.com/episodes/157154/mr-robot-1x01-eps10hellofriendmov
2,1,2,,01 Jul 15,eps1.1_ones-and-zer0es.mpeg,,http://www.tvmaze.com/episodes/167379/mr-robot-1x02-eps11ones-and-zer0esmpeg
3,1,3,,08 Jul 15,eps1.2_d3bug.mkv,,http://www.tvmaze.com/episodes/167380/mr-robot-1x03-eps12d3bugmkv
4,1,4,,15 Jul 15,eps1.3_da3m0ns.mp4,,http://www.tvmaze.com/episodes/167381/mr-robot-1x04-eps13da3m0nsmp4
5,1,5,,22 Jul 15,eps1.4_3xpl0its.wmv,,http://www.tvmaze.com/episodes/167382/mr-robot-1x05-eps143xpl0itswmv
6,1,6,,29 Jul 15,eps1.5_br4ve-trave1er.asf,,http://www.tvmaze.com/episodes/167383/mr-robot-1x06-eps15br4ve-trave1erasf
7,1,7,,05 Aug 15,eps1.6_v1ew-s0urce.flv,,http://www.tvmaze.com/episodes/167384/mr-robot-1x07-eps16v1ew-s0urceflv
8,1,8,,12 Aug 15,eps1.7_wh1ter0se.m4v,,http://www.tvmaze.com/episodes/167385/mr-robot-1x08-eps17wh1ter0sem4v
9,1,9,,19 Aug 15,eps1.8_m1rr0r1ng.qt,,http://www.tvmaze.com/episodes/167386/mr-robot-1x09-eps18m1rr0r1ngqt
10,1,10,,02 Sep 15,eps1.9_zer0-day.avi,,http://www.tvmaze.com/episodes/167387/mr-robot-1x10-eps19zer0-dayavi

Python code:

import pandas as pd

def main():
    df = pd.read_csv('bad.csv')

    # add these two extra columns to the end
    df["prod_code"] = ""
    df["special"] = ""
    cols = df.columns.tolist()
    # reorder columns
    cols = cols[:3] + [cols[-2]] + cols[3:5] + [cols[-1]] + [cols[-3]]
    # "commit" the reordering
    df = df[cols]
    # write the output without Pandas' first index column
    df.to_csv('out.csv', index=False)

[nodejs] Node.js one-liner

Problem
You want to execute a Node.js snippet in the command-line non-interactively. Why? Maybe you want to include it in a Bash script.

Solution

$ node -p '"jabba".split("").reverse().join("")'
abbaj

The option “-p” means “print result of --eval“.

Same thing in Python

$ python2 -c "print 'abc'.upper()"
ABC

# or, it can be multiple lines too
$ python2 -c "
for x in range(5):
    print x
print 'Finished'
"

The option “-c” means “command”.

[nodejs] create a TCP time server

Problem
I found this exercise at https://github.com/workshopper/learnyounode and here is my solution. Write a TCP server that listens on the port provided by the first argument to your program. For each connection you must write the current date & 24 hour time in this format: “YYYY-MM-DD HH:mm” (example: “2015-11-23 16:11”).

Solution
We will use moment.js to suck less with dates and times.

#!/usr/bin/env node

"use strict";

var net = require('net');
var moment = require('moment');    // install with npm

const port = process.argv[2];

function main() {
    console.log("The server is listening on port " + port + "...");
    var server = net.createServer(function (socket) {
        socket.end(moment().format('YYYY-MM-DD HH:mm') + "\n");
    }).listen(port);
}

main();

Start it and the server will listen to incoming connections. When a request is processed, it doesn’t stop. It continues listening for the next connection.

Test it
Start the server and run the following command in another terminal:

$ nc 127.0.0.1 8000
2015-11-23 16:21

The command “nc” is the netcat command. Under Manjaro it’s in the gnu-netcat package.

[nodejs] working with big integers

Python
Under Python it’s very easy to work with big integers (called ‘long’ type in Python 2). The programmer doesn’t have to worry about the overflow error, the interpreter takes care of it. For instance, let’s calculate 2¹²⁸:

>>> 2 ** 128
340282366920938463463374607431768211456

NodeJS
In JavaScript there is just a ‘number’ type, which is floating point. Actually, there is no integer type in JS :) And big integers are not supported, so you need a library if you want to work with huge numbers.

There is a very nice package called big-integer. It has an excellent documentation with lots of examples. Its usage is quite simple, similar to Java:

#!/usr/bin/env node

"use strict";

var bigInt = require("big-integer");

/*
 * 2 ** 128 == 340282366920938463463374607431768211456
 */

function main() {
    console.log(bigInt(2).pow(128).toString());
}

main();

[nodejs] raw_input in Node.js

Problem
How to read in Node.js from the console? For instance, how to rewrite the following Python script?

n1 = int(raw_input("1st number: "))
n2 = int(raw_input("2nd number: "))

print "The sum is:", n1+n2

Solution

#!/usr/bin/env node

"use strict";

var readline = require('readline');

var rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout
});

function processNums(n1, n2) {
    n1 = Number(n1);
    n2 = Number(n2);
    console.log("Their sum: " + (n1 + n2));
}

function start() {
    rl.question('1st number: ', function (x) {
        rl.question('2nd number: ', function (y) {
            rl.close();

            processNums(x, y);
        });
    });
}

function main() {
    start();
}

main();

Tip from here.

Dive into JavaScript

I’ve decided to dive into JavaScript. I had some superficial knowledge of it, but nothing serious. I’ve used Flask for my pet projects, but they should also look nice on the client side. So expect some JavaScript and Node.js related posts in the future. But fear not, my favourite programming language is still Python :)

endswith also accepts a tuple

Have you ever written something like this?

fname = 'movie.avi'
if fname.endswith('avi') or fname.endswith('mp4'):
    print("It's a movie.")

The function endswith also accepts a tuple. Just saying.

fname = 'movie.avi'
if fname.endswith(('avi', 'mp4')):
    print("It's a movie.")

Meaning: if it ends as ‘avi‘ or ‘mp4‘, then…

This also works with startswith, of course.

_{Thanks to one of my students, Marton Sz. who solved one of his exercises using this trick.}

automatic text summarization

See https://github.com/miso-belica/sumy . In the README there is a list of alternative projects.