Archive

Posts Tagged ‘virtualenvwrapper’

Pew: A tool to manage multiple virtual environments written in pure Python

January 4, 2016 Leave a comment

Problem
You are fed up with virtualenv and virtualenvwrapper.

Solution
Use Pew, which is a new kid on the block. I collected my notes about it here, where I also provide two convenience scripts. I don’t want to copy all that here, so check out the GitHub link.

Advertisements

create a virtual environment easily

August 30, 2015 Leave a comment

Problem
I prefer to put my larger projects in a virtual environment (there are people who put every project in a virt. env. …). I keep my projects in Dropbox (thus they are available on all my machines), but the virt. env.’s are kept outside Dropbox since they can grow quite big (and they are easily reproducible).

For creating virt. env.’s, I use virtualenvwrapper, which (by default) puts virt. env.’s in the folder ~/.virtualenvs. Say I have a project in Dropbox, and I want to create a virt. env. for it. How to do it easily?

Solution
First, you need to know if your project is written in Python 2 or Python 3. Then, you need to use the mkvirtualenv command but I always need to look up its syntax. Solution: in the root folder of my project I want a script that will create a virt. env. for the project. Here is the script:

#!/usr/bin/env bash

# mk_venv.sh

# which Python version to use in the created virt. env. (2 or 3)
PYTHON_VER=2

a=`pwd`
f=${a##*/}
source `which virtualenvwrapper.sh` && mkvirtualenv -p `which python${PYTHON_VER}` $f

Just set PYTHON_VER and launch the script. It will figure out the name of the current folder and create a virt. env. with this name. For instance, if you have your project in ~/projects/stuff, then the virt. env. will be created in the folder ~/.virtualenvs/stuff.

Related links

Categories: bash, python Tags: ,

pyvenv: create virtual environmets for Python 3.4+

January 1, 2015 Leave a comment

For creating virtual environmets, I’ve used virtualenvwrapper so far. However, Python 3.4 contains the command pyvenv that does the same thing. Since it also installs pip in the virt. env., it can replace virtualenvwrapper.

I like to store my virtual environments in a dedicated folder, separated from the project directory. virtualenvwrapper, by default, stores the virt. env.’s in the ~/.virtualenvs folder. Since I got used to this folder, I will continue to keep my virt. env.’s in this folder.

pyvenv
Say we have our project folder here: ~/python/webapps/flasky_project. Create a virt. env. for this the following way:

pyvenv ~/.virtualenvs/flasky_project

It will create a Python 3 virt. env.

virtualenv / virtualenvwrapper
For the sake of completeness, I also write here how to create virt. env.’s with virtualenv and virtualenvwrapper:

# blog post: http://goo.gl/oEdtT3

# virtualenvwrapper for Python 3 or Python 2
mkvirtualenv -p `which python3` myenv3
mkvirtualenv -p `which python2` myenv2

# virtualenv for Python 3 or Python 2
virtualenv -p python3 myproject3
virtualenv -p python2 myproject2

# When the env. is created, activate it
# and launch the command python within.
# Verify if it's the correct version.

TL; DR
If you use Python 3.4+ and you need a virt. env., use the command “pyvenv“.

[django] virtualenvwrapper with Python 3

September 12, 2014 Leave a comment

Problem
Django 1.7 came out ten days ago. If you look at the updated poll tutorial, you will notice that it’s written for Python 3! So far I’ve used Python 2 only, so I thought I would redo this tutorial for two reasons: (1) learn about migrations, and (2) start using Python 3.

For my Django projects I use virtualenvwrapper to manage my virtual environments, so the first question was: how to create a Python 3 virt. env. with virtualenvwrapper?

Well, maybe my installation got corrupted somehow, but I ran into this problem:

$ mkvirtualenv -p `which python3` tmpenv
Running virtualenv with interpreter /usr/bin/python3
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 8, in <module>
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in <module>
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 53, in apport_excepthook
    if not enabled():
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 24, in enabled
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 8, in <module>
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in <module>
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'

Solution
I posted this question to reddit (see here) and I found the following solution:

First, install pip3:

sudo apt-get install python3-pip
sudo pip3 install virtualenvwrapper

Then, add the following lines to your .bashrc:

export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/workspace    # customize if needed
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv
source /usr/local/bin/virtualenvwrapper.sh

Try if it works for you. In my case, I had to edit the file “/usr/local/bin/virtualenv“. I changed its first line to

#!/usr/bin/env python3

The consequence of changing the first line was that creating Python 3 environments became the default, but at least it works now.

Create a Python 3 virt. env.:

mktmpenv

Create a Python 2 virt. env.:

mktmpenv -p `which python2`

If you use the command “virtualenv“, it will also create Python 3 environments by default. Thus, “virtualenv myproject3” will be a Python 3 env., while “virtualenv -p python2 myproject2” will be a Python 2 env.

[django] virtualenvwrapper + optimal project layout + requirements.txt

August 4, 2014 Leave a comment

Here we will see how to use virtualenvwrapper, how to create a requirements.txt file, and what is the optimal layout for a Django project.

virtualenvwrapper
Here I suppose you already know virtualenv. virtualenvwrapper, as its name suggests, is a wrapper for virtualenv, making your life even easier (especially when you have several projects).

First, install it globally:

sudo pip install virtualenvwrapper -U

I keep my Django projects in a dedicated folder, which is “$HOME/Dropbox/python/webapps” in my case. I put them to Dropbox, thus (1) I have a simple backup solution, and (2) I can work on a project on any machine of mine since the Dropbox client synchronizes everything. The path of the folder above is too long, so I have a symbolic link “$HOME/webapps” that points to “$HOME/Dropbox/python/webapps“.

Now, add the following lines to the end of your .bashrc file:

# virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/webapps
source /usr/local/bin/virtualenvwrapper.sh

Open a new terminal to activate the new settings.

The good thing is that virtual environments will be separated from our project directory. That is, instead of “$HOME/webapps“, virtual environments will be kept in the “$HOME/.virtualenvs” folder. A virtual env. can be quite large, but it won’t fill your Dropbox space since it’s kept outside.

At this time you might ask: “OK, my Django project is in Dropbox, but how can I work on it on two different machines if the virtual env. is outside of Dropbox?” That’s a good question. This is why we will use a “requirements.txt” file, which collects all the packages that are installed in the virtual environment. This file will be in the project folder (i.e., inside Dropbox). If you want to work on a project on a new machine, using the “requirements.txt” file you can re-create the virtual env., which will be identical to the one on your first machine. We will see it later in the “requirements.txt” section.

project layout
The 3rd chapter of the book “Two Scoops of Django” suggests a three-tiered approach. I like it, so let’s see it through an example. Let’s create a new Django project called “hello“. Enter the folder “$HOME/webapps” and create the project directory (it is the 1st tier). I always add the “_project” suffix to a project (later we will see a bash trick that relies on it, so let’s call the folder “hello_project“). We are here at the moment:

jabba@jabba-uplink:~$ cd
jabba@jabba-uplink:~$ cd webapps
jabba@jabba-uplink:~/webapps$ mkdir hello_project
jabba@jabba-uplink:~/webapps$ cd hello_project/
jabba@jabba-uplink:~/webapps/hello_project$ ls -al
total 8
drwxrwxr-x 2 jabba jabba 4096 Aug  4 16:04 .
drwxrwxr-x 7 jabba jabba 4096 Aug  4 16:04 ..

Now let’s create a virtual env. for the project:

jabba@jabba-uplink:~/webapps/hello_project$ mkvirtualenv hello_project
New python executable in hello_project/bin/python
Installing setuptools, pip...done.
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ ls -al
total 8
drwxrwxr-x 2 jabba jabba 4096 Aug  4 16:04 .
drwxrwxr-x 7 jabba jabba 4096 Aug  4 16:04 ..
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$

We use the “mkvirtualenv” command. The name of the virtual env. can be the same as the name of the project (actually, in our case they should be called the same — we will see later why). This way you can associate them easily. Notice two things: (1) the prompt has changed, indicating that the “hello_project” virtual env. has been activated (“mkvirtualenv” creates a virtual env. and also activates it), and (2) the current folder is still empty, i.e. the virtual env. was created somewhere else.

Where is the virtual env.? Go to “$HOME/.virtualenvs” and there you will see a subfolder “hello_project” that contains the virtual environment of our new Django project.

Extra stuff: the basic bash prompt is monochrome and thus not too appealing to the eyes. I have a solution that nicely colors the prompt, emphasizing activated virtual environments too. See the end of this blog post for more info. Here is a screenshot of the colorized prompt.

Now, let’s see how to activate and deactivate the virtual environment:

jabba@jabba-uplink:~/webapps/hello_project$ ls -al
total 8
drwxrwxr-x 2 jabba jabba 4096 Aug  4 16:04 .
drwxrwxr-x 7 jabba jabba 4096 Aug  4 16:04 ..
jabba@jabba-uplink:~/webapps/hello_project$ workon hello_project
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ ls -al
total 8
drwxrwxr-x 2 jabba jabba 4096 Aug  4 16:04 .
drwxrwxr-x 7 jabba jabba 4096 Aug  4 16:04 ..
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ deactivate 
jabba@jabba-uplink:~/webapps/hello_project$ ls -al
total 8
drwxrwxr-x 2 jabba jabba 4096 Aug  4 16:04 .
drwxrwxr-x 7 jabba jabba 4096 Aug  4 16:04 ..
jabba@jabba-uplink:~/webapps/hello_project$

That is, “workon PROJECT_NAME” activates the virtual environment whose name is PROJECT_NAME. Since we’ve chosen the same name for the Django project’s main folder (1st tier) and for the virtual environemnt, we don’t need to think “damn, what was the name of the corresponding virt. env.?”.

Let’s see what Python packages are installed in this virt. env.:

(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ pip freeze --local
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$

Nothing. That’s normal, we have a brand new virt. env. It’s time to create a basic Django application:

(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ pip install django
Downloading/unpacking django
  Using download cache from /home/jabba/.pip/download_cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fany%2FD%2FDjango%2FDjango-1.6.5-py2.py3-none-any.whl
Installing collected packages: django
Successfully installed django
Cleaning up...
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ django-admin.py startproject hello_project
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ pip freeze --local >requirements.txt
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ cat requirements.txt 
Django==1.6.5
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ mv hello_project/ hello
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ ls -al
total 16
drwxrwxr-x 3 jabba jabba 4096 Aug  4 16:32 .
drwxrwxr-x 7 jabba jabba 4096 Aug  4 16:04 ..
drwxrwxr-x 3 jabba jabba 4096 Aug  4 16:31 hello
-rw-rw-r-- 1 jabba jabba   14 Aug  4 16:32 requirements.txt
(hello_project)jabba@jabba-uplink:~/webapps/hello_project$ cd hello
(hello_project)jabba@jabba-uplink:~/webapps/hello_project/hello$ ls -al
total 16
drwxrwxr-x 3 jabba jabba 4096 Aug  4 16:31 .
drwxrwxr-x 3 jabba jabba 4096 Aug  4 16:32 ..
drwxrwxr-x 2 jabba jabba 4096 Aug  4 16:31 hello_project
-rwxrwxr-x 1 jabba jabba  256 Aug  4 16:31 manage.py
(hello_project)jabba@jabba-uplink:~/webapps/hello_project/hello$

First, we install Django. Whenever you work with a project, the first thing should be to activate the virtual env. The package django was installed inside the virt. env. With the command “pip freeze --local >requirements.txt” we create the “requirements.txt” file. Whenever you install or uninstall a package, don’t forget to update the “requirements.txt” file using the command above. The command “django-admin.py startproject hello_project” creates a project called “hello_project“. Next to the “requirements.txt” file you will have a “hello_project” subfolder. This folder is the 2nd tier in our three-tiered approach. This is actually the main folder of a Django project and it can be called anything. I prefer to remove the “_project” suffix from it, that’s why I renamed it to “hello“. Inside this folder you will have a file called “manage.py“. Next to “manage.py“, there is a subfolder also called “hello_project“. This “hello_project” folder is the 3rd tier and it contains –among others– the “settings.py” file, which is the main configuration file of a Django project. The current directory structure should look like this:

(hello_project)jabba@jabba-uplink:~/webapps$ tree
.
├── hello_project                     (1st tier)
│   ├── hello                         (2nd tier)
│   │   ├── hello_project             (3rd tier)
│   │   │   ├── __init__.py
│   │   │   ├── settings.py
│   │   │   ├── urls.py
│   │   │   └── wsgi.py
│   │   └── manage.py
│   └── requirements.txt
...

A Django project usually contains several applications. Each application is stored in a folder that is under the 2nd tier, i.e. next to “manage.py“. Normally, each application has a “urls.py” file too. I like to call these files “local”, meanwhile the files in the 3rd tier are “global”, since they contain the configuration of the whole Django project. This is why I prefer to add the “_project” suffix to the 3rd tier: I can see immediately which folder is the “global” folder with the main settings.

As seen before, I renamed the 2nd tier to “hello“. This way I can see easier where I am, in which folder. If I had three subfolders with the same name, I should always think a bit “Err, where am I now?”.

OK, you have a basic Django project that works but doesn’t do anything interesting. Well, let’s see what it produces:

(hello_project)jabba@jabba-uplink:~/webapps/hello_project/hello$ ./manage.py runserver

By default, it starts the project on port 8000. Open http://127.0.0.1:8000 in your browser. You should see a welcome page.

requirements.txt
I’ve already written a bit about “requirements.txt” above. To put it in a nutshell: always keep your “requirements.txt” up-to-date so that it contains the Python packages that are installed in the virt. env.

# create, update
pip freeze --local >requirements.txt    # 1st command

# install from
pip install -r requirements.txt         # 2nd command

Example: you create a project on your home machine. You install some packages and using the first command you update the “requirements.txt” file. Then, at your workplace, you want to polish a bit your pet project. Dropbox synchronized the project but the virt. env. is missing. Set up virtualenvwrapper, create a new virt. env. for the project (as explained at the beginning of this post), then use the second command to install the necessary packages. Voila, you are ready to continue developing your project.

virtualenvwrapper #2
I want to show you a trick how to activate/deactivate a virt. env. easily. When I want to work on a project, first I navigate to the project’s folder. Either I go to the 1st tier (in this example, “~/webapps/hello_project“) or the 2nd tier (“~/webapps/hello_project/hello“). Here, with the simple command “on“, I can activate the virt. env. Similarly, with the command “off” it can be deactivated. To have this functionality, add the following lines to your .bashrc file:

on_function()
{
    a=`pwd`
    f=${a##*/}
    workon $f
    if [ $? -ne 0 ]; then
        bak=`pwd` 
        a=`dirname $bak`
        f=${a##*/}
        workon $f
        cd $bak
    fi  
}   
alias on=on_function
alias off='deactivate'

The following condition must meet to make it work: the name of the 1st tier’s folder must be the same as the name of the virt. env. (here we called both of them “hello_project“).

Hmm, this post became a bit long. The next ones will be shorter :)

virtualenvwrapper (Part 2)

Here I describe how I use virtualenvwrapper.

I keep my virtual environments in the default location (~/.virtualenvs folder). My Python projects are kept in Dropbox. If a project (call it “stuff” for instance) has a virtual env., then I add the suffix “_project” to the projects’s folder name (resulting “stuff_project” in this example). This way I can see it immediately (without entering the folder) that it has an attached venv.

In the stuff_project folder create a requirements.txt file that contains all the modules that you installed in your venv. This way you can re-create your venv on another machine and continue to work on yout project.

Let’s see a concrete example. Say I want to create a new Django project:

$ cd $HOME/Dropbox/python/webapps
$ mkdir stuff_project
# ^ every project file will be in this folder
$ cd stuff_project
$ django-admin.py startproject stuff
# ^ create the Django project
$ mkvirtualenv stuff_project
# ^ venv is created AND activated
$ pip install django
$ pip install django-extensions
# ^ useful extensions
$ pip freeze --local >requirements.txt
# ^ packages installed in the venv are listed in this file

Since I put the project folder in Dropbox, it will be synced on all my machines. What to do if I want to work on this project on another machine of mine (let’s call it “machine #2”)? The virtual environment will be missing. But there is no need to worry. We have a requirements.txt file and from this we can populate a newly created venv. Install virtualenvwrapper on machine #2 and create an empty venv for the project:

# actions to perform on machine #2:
$ cd $HOME/Dropbox/python/webapps/stuff_project
$ mkvirtualenv stuff_project
# ^ the venv is created and activated
$ pip install -r requirements.txt
# ^ packages are installed in the venv

When you are done with the project, deactivate the venv with the “deactivate” command. Since it’s too long, I put an alias on it:

alias off='deactivate'

If you install new packages in the venv, don’t forget to update the requirements.txt file ( pip freeze >requirements.txt )!

virtualenvwrapper (Part 1)

I use virtualenv but so far it’s been enough for me. However, the time has come to step on to the next level: virtualenvwrapper. Virtualenvwrapper is just a wrapper around virtualenv, but it makes working with virtualenv so much easier.

Problem
I keep all my Python projects in Dropbox. Every project is located in a dedicated folder. If I use virtualenv, inside a project folder I need to create a subfolder for the the virtual environment (typically called “venv”). However, this folder can be really huge. In the case of a Flask project it can be up to 50-60 MB. Keeping it in Dropbox is a luxury. It would be great to move the virtual environments outside of Dropbox. My first idea was to put a symbolic link on the venv folder from the project folder but Dropbox doesn’t support symlinks :(

Solution
Let’s use virtualenvwrapper! Virtualenvwrapper was created with this idea: collect virtual environments in a separated folder, outside of your projects. In addition, it’s very easy to switch between virtual environments.

Here are some great resources on virtualenvwrapper:

For more info, refer to Bob’s blog (last link above), where he gives a very nice summary about the usage of virtualenvwrapper.

Categories: python Tags: ,