Archive
An SQLite example
Here you will find a very basic program that stores data in SQLite. It is a simple example. If I need to write a program that needs sqlite, it can serve as a basis and I won’t need to write it from scratch.
Problem
Visit the page /r/earthporn, extract the image URLs and save the URLs of images stored on imgur in an sqlite table.
Solution
earthporn.py:
#!/usr/bin/env python
# encoding: utf-8
import requests
import database as db # see below
subreddit = "earthporn"
def main():
db.init() # Important! It will connect to the DB.
r = requests.get('http://www.reddit.com/r/{sr}/.json'.format(sr=subreddit))
d = r.json()
children = d["data"]["children"]
for e in children:
url = e["data"]["url"]
if "imgur.com" in url:
# print url
db.add_image(url, subreddit)
####################
if __name__ == "__main__":
main()
database.py:
#!/usr/bin/env python
"""
Sqlite database handler.
"""
import os
import sqlite3
import atexit
import random
import termcolor
PATH = os.path.dirname(os.path.abspath(__file__))
SCHEMA = """
CREATE TABLE "images" (
"url" TEXT PRIMARY KEY NOT NULL,
"subreddit" TEXT,
"insert_date" DEFAULT CURRENT_TIMESTAMP
)
"""
SQLITE_DB = PATH + '/images.sqlite'
conn = None
def get_random_image():
query = "SELECT url FROM images"
cursor = conn.cursor()
cursor.execute(query)
result = cursor.fetchall()
return random.choice(result)[0]
def add_image(url, subreddit=None):
try:
if subreddit:
query = "INSERT INTO images (url, subreddit) VALUES (?, ?)"
conn.execute(query, (url, subreddit))
else:
query = "INSERT INTO images (url, subreddit) VALUES (?, NULL)"
conn.execute(query, (url,))
#
print termcolor.colored("# {url} added to DB".format(url=url), "cyan")
except sqlite3.IntegrityError:
print termcolor.colored("# the image {0} is already in the DB...".format(url), "red")
def get_all_images():
query = "SELECT url FROM images"
cursor = conn.cursor()
cursor.execute(query)
result = cursor.fetchall()
return result
def create_db():
"""
Create the DB if not exists.
"""
global conn
conn = sqlite3.connect(SQLITE_DB)
conn.executescript(SCHEMA)
def init(commit=True):
"""
Initialize the DB.
"""
global conn
if commit:
atexit.register(commit_and_close)
else:
atexit.register(close)
if not os.path.exists(SQLITE_DB):
create_db()
if not conn:
conn = sqlite3.connect(SQLITE_DB)
def commit():
"""
Commit.
"""
if conn:
conn.commit()
def close():
"""
Close.
"""
if conn:
conn.close()
def commit_and_close():
"""
Commit and close DB connection.
As I noticed, commit() must be called, otherwise changes
are not committed automatically when the program terminates.
"""
if conn:
conn.commit()
conn.close()
####################
if __name__ == "__main__":
init()
The module database.py is responsible for handling the database. When you need to work with the database, just include this file and call the appropriate function(s).
imgur album downloader
This little script can download all the images in an imgur gallery.
Improving the sidebar of /r/python
In the sidebar of /r/python, there was a very unpythonic infinite loop for a long time:
while 1:
# do something
Today I sent a message to the moderators and they changed it:
Update: I got a message from them.
A course on Udemy about Python GUI programming with Qt
http://www.reddit.com/r/Python/comments/14453f/hey_rpython_ive_started_teaching_a_course_on/
If you are lucky, you can put your hand on a free coupon :)
Official page at Udemy: http://www.udemy.com/python-gui-programming/.
submitting a link to reddit and adding a comment
Problem
You want to send a large number of links to reddit and you want to get it done with a script. How to send links and how to add comments to them?
Solution
First, install the reddit api:
sudo pip install reddit -U
Then you can use my RedBot to perform the task. It is part of my jabbapylib library.
#!/usr/bin/env python
"""
Simple API for:
* posting links to reddit
* commenting on a post
This requires that you have a reddit account. Put your username and
password in the following files:
* ~/reddit_username.txt
* ~/reddit_password.txt
To use RedBot, just make a subclass of it and give it a cool name:
class HyperBot(RedBot):
def __init__(self):
super(HyperBot, self).__init__()
self.name = 'HyperBot'
Now you are ready to flood reddit :)
# from jabbapylib.reddit import red
"""
import reddit
from jabbapylib.filesystem import fs
from jabbapylib.platform import platform
USERNAME_TXT = '{home}/reddit_username.txt'.format(home=platform.get_home_dir())
PASSWORD_TXT = '{home}/reddit_password.txt'.format(home=platform.get_home_dir())
#
USERNAME = fs.read_first_line(USERNAME_TXT)
PASSWORD = fs.read_first_line(PASSWORD_TXT)
class RedBot(object):
def __init__(self):
self.name = 'RedBot'
self.username = USERNAME
self.password = PASSWORD
#
self.r = reddit.Reddit(user_agent=self.name)
self.r.login(username=self.username, password=self.password)
self.last_post = None # Submission object
self.permalink = None # URL of the last post
def submit_link(self, url, subreddit, title):
"""
The return value (res) is a Submission object or None.
URL of the newly created post: res.permalink
"""
try:
self.last_post = self.r.submit(subreddit, title, url=url)
self.permalink = self.last_post.permalink
print '# url to send: {url}'.format(url=url)
print '# submitted to: {pl}'.format(pl=self.permalink)
return self.last_post
except:
print >>sys.stderr, "Warning: couldn't submit {url}".format(url=url)
return None
def add_comment(self, comment):
if self.last_post:
self.last_post.add_comment(comment)
print '# comment added'
#############################################################################
if __name__ == "__main__":
# here is how to use it:
# url = '...'
# subreddit = '...'
# title = "..."
# comment = '...'
# r = RedBot()
# r.submit_link(url, subreddit, title)
# r.add_comment(comment)
pass
You can find the current version here.
Thanks to Bryce Boe, the maintainer of the reddit api, who kindly answered my questions.
Upload an image to imgur.com from Python
If you are familiar with reddit, you must have noticed that most images are hosted on imgur. I would like to upload several images from my computer and I want to collect their URLs on imgur. Let’s see how to do that.
Imgur has an API, this is what we’ll use. Anonymous upload is fine for my needs. For this you need to register and you get an API key. Under the examples there is a very simple Python code. When you execute it, pycurl prints the server’s XML response to the standard output. How to store that in a variable? From that XML we want to extract some data.
Here is an extended version of the uploader script:
#!/usr/bin/env python
import pycurl
import cStringIO
import untangle # XML parser
def upload_from_computer(image):
response = cStringIO.StringIO() # XML response is stored here
c = pycurl.Curl()
values = [
("key", your_api_key),
("image", (c.FORM_FILE, image))]
# OR: ("image", "http://example.com/example.jpg")]
# OR: ("image", "YOUR_BASE64_ENCODED_IMAGE_DATA")]
c.setopt(c.URL, "http://api.imgur.com/2/upload.xml")
c.setopt(c.HTTPPOST, values)
c.setopt(c.WRITEFUNCTION, response.write) # put the server's output in here
c.perform()
c.close()
return response.getvalue()
def process(xml):
o = untangle.parse(xml)
url = o.upload.links.original.cdata
delete_page = o.upload.links.delete_page.cdata
print 'url: ', url
print 'delete page:', delete_page
#############################################################################
if __name__ == "__main__":
img = '/tmp/something.jpg'
xml = upload_from_computer(img)
process(xml)
The tip for storing the XML output in a variable is from here. Untangle is a lightweight XML parser; more info here.
Python on reddit
I didn’t use reddit (wikipedia article) in the past but yesterday I visited its Python community and I found some really interesting links! So I think from now on I will check reddit regularly :)
Python-related communities on reddit:
- Python
- Python Educational (for beginners)
For a detailed list of subreddits, visit http://www.reddit.com/reddits/. Help, FAQ.
See this post for some notes about reddit.
Python Adventures 
Send a post to reddit from Python
Problem
How to send a post to reddit.com from a Python script? Motivation: when you send a post, you have to wait 8 minutes before you could send the next one. Imagine you have 10 posts to submit. It’d be nice to launch a script at night which would send everything till next morning.
Submit a post
Now I only show how to send one post. Batch processing is left as a future project.
The official Reddit API is here. There is a wrapper for it called reddit_api, which greatly simplifies its usage.
Install reddit_api:
Submit a post:
Submit a comment (update, 20111107)
Let’s see how to add a comment to a post. First, we need the URL of a post.
Example: http://www.reddit.com/r/thewalkingdead/comments/lkycy/that_look_on_the_kids_face/. Here, the last part of the URL is just garbage, the following URL is equivalent with it: http://www.reddit.com/r/thewalkingdead/comments/lkycy. The unique ID of the post is the last part: “lkycy”. Thus, this image can be accessed via this URL too: http://www.reddit.com/lkycy.
Now, let’s log in to reddit, fetch the post by its ID and add a comment.
def get_reddit_id(url): result = re.search('/comments/(.*?)/', url) return result.group(1) def add_comment(r, reddit_url): reddit_id = get_reddit_id(reddit_url) post = r.get_submission_by_id(reddit_id) comment = "first" # just to make reddit happy ;) post.add_comment(comment) print '# comment added:', comment def main(): r = reddit.Reddit(user_agent="my_cool_application") r.login(user="...", password="...") reddit_url = ... add_comment(r, reddit_url)