Archive

Posts Tagged ‘json’

pretty JSON output with Flask-RESTful

March 15, 2016 Leave a comment

Problem
Flask-RESTful is an awesome way to write REST APIs. In debug mode, its output is nicely indented, easy to read. However, in production mode the JSON is compressed and hard to read.

If debug is False, how to have a nicely formatted output?

Solution

from flask import Flask
from flask_restful import Api

app = Flask(__name__)
api = Api(app)

if production:
    print("# running in production mode")
    HOST = '0.0.0.0'
    DEBUG = False
    # START: temporary help for the UI developers, remove later
    settings = app.config.get('RESTFUL_JSON', {})
    settings.setdefault('indent', 2)
    settings.setdefault('sort_keys', True)
    app.config['RESTFUL_JSON'] = settings
    # END
else:
    print("# running in development mode")
    HOST='127.0.0.1'
    DEBUG = True

# ...

if __name__ == '__main__':
    app.run(debug=DEBUG, host=HOST, port=1234)

Note that here I use the development server shipped with Flask, which is not suitable for real production. So this solution is between development and production.

Categories: flask, python Tags: , , ,

detect duplicate keys in a JSON file

March 6, 2016 1 comment

Problem
I want to edit a JSON file by hand but I’m afraid that somewhere I introduce a duplicate key by accident. If it happens, then the second key silently overwrites the first one. Example:

$ cat input.json 
{
    "content": {
        "a": 1,
        "a": 2
    }
}

Naive approach:

import json

with open("input.json") as f:
    d = json.load(f)

print(d)

# {'content': {'a': 2}}

If there is a duplicate key, it should fail! But it remains silent and you have no idea that you just lost some data.

Solution
I found the solution here.

import json

def dict_raise_on_duplicates(ordered_pairs):
    """Reject duplicate keys."""
    d = {}
    for k, v in ordered_pairs:
        if k in d:
           raise ValueError("duplicate key: %r" % (k,))
        else:
           d[k] = v
    return d

def main():
    with open("input.json") as f:
        d = json.load(f, object_pairs_hook=dict_raise_on_duplicates)

    print(d)

Now you get a nice error message:

Traceback (most recent call last):
  File "./check_duplicates.py", line 28, in <module>
    main()
  File "./check_duplicates.py", line 21, in main
    d = json.load(f, object_pairs_hook=dict_raise_on_duplicates)
  File "/usr/lib64/python3.5/json/__init__.py", line 268, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib64/python3.5/json/__init__.py", line 332, in loads
    return cls(**kw).decode(s)
  File "/usr/lib64/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.5/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
  File "./check_duplicates.py", line 13, in dict_raise_on_duplicates
    raise ValueError("duplicate key: %r" % (k,))
ValueError: duplicate key: 'a'

If your json file has no duplicates, then the code aboce nicely prints its content.

Categories: python Tags: ,

opening gzipped JSON files

January 7, 2015 Leave a comment

Problem
I have a project where the input JSON file is almost 7 MB. I keep this project in my Dropbox folder, so that 7 MB text file seems to be a waste. Any way to reduce its size?

Solution
I zipped it up with gzip: “gzip -9 input.json“. This command produced a 1.3 MB “input.json.gz” file and deleted the original. Good. But how to open it in Python?

Normal way (without gzip):

import json

with open("input.json") as f:
    d = json.load(f)

Compressed way (with gzip):

import json
import gzip

with gzip.open("input.json.gz", "rb") as f:
    d = json.loads(f.read().decode("ascii"))

I didn’t notice any performance penalty. The application that first reads this json file starts as fast as before.

Categories: python Tags: ,

XML to dict / XML to JSON

December 29, 2014 Leave a comment

Problem
You have an XML file and you want to convert it to dict or JSON.

Well, if you have a dict, you can convert it to JSON with “json.dump()“, so the real question is: how to convert an XML file to a dictionary?

Solution
There is an excellent library for this purpose called xmltodict. Its usage is very simple:

import xmltodict

# It doesn't work with Python 3! Read on for the solution!
def convert(xml_file, xml_attribs=True):
    with open(xml_file) as f:
        d = xmltodict.parse(f, xml_attribs=xml_attribs)
        return d

This worked well under Python 2.7 but I got an error under Python 3. I checked the project’s documentation and it claimed to be Python 3 compatible. What the hell?

The error message was this:

Traceback (most recent call last):
  File "/home/jabba/Dropbox/python/lib/jabbapylib2/apps/xmltodict.py", line 247, in parse
    parser.ParseFile(xml_input)
TypeError: read() did not return a bytes object (type=str)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./xml2json.py", line 27, in <module>
    print(convert(sys.argv[1]))
  File "./xml2json.py", line 17, in convert
    d = xmltodict.parse(f, xml_attribs=xml_attribs)
  File "/home/jabba/Dropbox/python/lib/jabbapylib2/apps/xmltodict.py", line 249, in parse
    parser.Parse(xml_input, True)
TypeError: '_io.TextIOWrapper' does not support the buffer interface

I even filed an issue ticket :)

After some debugging I found a hint here: you need to open the XML file in binary mode!

XML to dict (Python 2 & 3)
So the correct version that works with Python 3 too is this:

import xmltodict

def convert(xml_file, xml_attribs=True):
    with open(xml_file, "rb") as f:    # notice the "rb" mode
        d = xmltodict.parse(f, xml_attribs=xml_attribs)
        return d

XML to JSON (Python 2 & 3)
If you want JSON output:

import json
import xmltodict

def convert(xml_file, xml_attribs=True):
    with open(xml_file, "rb") as f:    # notice the "rb" mode
        d = xmltodict.parse(f, xml_attribs=xml_attribs)
        return json.dumps(d, indent=4)
Categories: python Tags: , , , , , ,

Simple JSON time service (with timezone support)

Categories: python Tags: , ,

Serializations: data <-> XML, data <-> JSON, XML <->JSON

January 20, 2012 Leave a comment

Problems

  1. Python data to XML and back
  2. Python data to JSON and back
  3. XML to JSON and back

Solution

import json
import xmlrpclib
from xml2json import Xml2Json

def data_to_xmlrpc(data):
    # http://www.reddit.com/r/Python/comments/ole01/dictionary_to_xml_in_20_lines/
    """Return value: XML RPC string."""
    return xmlrpclib.dumps((data,)) # arg. is tuple

def xmlrpc_to_data(xml):
    # http://www.reddit.com/r/Python/comments/ole01/dictionary_to_xml_in_20_lines/
    """Return value: python data."""
    return xmlrpclib.loads(xml)[0][0]

def data_to_json(data):
    """Return value: JSON string."""
    data_string = json.dumps(data)
    return data_string

def json_to_data(data_string):
    """Return value: python data."""
    data = json.loads(data_string)
    return data

def xml_to_json(xml):
    """Return value: JSON string."""
    res = Xml2Json(xml).result
    return json.dumps(res)

def json_to_xmlrpc(data_string):
    """Return value: XML RPC string."""
    data = json.loads(data_string)
    return data_to_xmlrpc(data)

def xmlrpc_to_json(xmlrpc):
    """Return value: JSON string."""
    data = xmlrpc_to_data(xmlrpc)
    return data_to_json(data)

The full source code (serialize.py) together with the imported xml2json.py are available here. This work is part of my jabbapylib library.

Examples
Unit tests are here, they show you how to use these functions. Examples with comments are here.

Categories: python Tags: , , ,

Prettify a JSON file

August 10, 2011 1 comment

Problem
You want to make a JSON file (string) human-readable.

Solution

curl -s http://www.reddit.com/r/nsfw/.json | python -mjson.tool

For some more alternatives please refer to this post.

Update (20121021)
Another way is to use jq, a powerful command-line tool for querying JSON files.

Categories: python Tags: , , ,