Archive

Posts Tagged ‘virtualenv’

Pew: A tool to manage multiple virtual environments written in pure Python

January 4, 2016 Leave a comment

Problem
You are fed up with virtualenv and virtualenvwrapper.

Solution
Use Pew, which is a new kid on the block. I collected my notes about it here, where I also provide two convenience scripts. I don’t want to copy all that here, so check out the GitHub link.

create a virtual environment easily

August 30, 2015 Leave a comment

Problem
I prefer to put my larger projects in a virtual environment (there are people who put every project in a virt. env. …). I keep my projects in Dropbox (thus they are available on all my machines), but the virt. env.’s are kept outside Dropbox since they can grow quite big (and they are easily reproducible).

For creating virt. env.’s, I use virtualenvwrapper, which (by default) puts virt. env.’s in the folder ~/.virtualenvs. Say I have a project in Dropbox, and I want to create a virt. env. for it. How to do it easily?

Solution
First, you need to know if your project is written in Python 2 or Python 3. Then, you need to use the mkvirtualenv command but I always need to look up its syntax. Solution: in the root folder of my project I want a script that will create a virt. env. for the project. Here is the script:

#!/usr/bin/env bash

# mk_venv.sh

# which Python version to use in the created virt. env. (2 or 3)
PYTHON_VER=2

a=`pwd`
f=${a##*/}
source `which virtualenvwrapper.sh` && mkvirtualenv -p `which python${PYTHON_VER}` $f

Just set PYTHON_VER and launch the script. It will figure out the name of the current folder and create a virt. env. with this name. For instance, if you have your project in ~/projects/stuff, then the virt. env. will be created in the folder ~/.virtualenvs/stuff.

Related links

Categories: bash, python Tags: ,

creating a Python 3 virt. env. on Ubuntu

Problem
Python 3 on Ubuntu is sometimes a total mess. I wanted to create a Python 3 virt. env., but I got this error:

$ virtualenv -p python3 venv
Running virtualenv with interpreter /usr/bin/python3
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 8, in 
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in 
    import re
  File "/usr/lib/python3.4/re.py", line 324, in 
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in 
    from copy_reg import *
ImportError: No module named 'copy_reg'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 53, in apport_excepthook
    if not enabled():
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 24, in enabled
    import re
  File "/usr/lib/python3.4/re.py", line 324, in 
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in 
    from copy_reg import *
ImportError: No module named 'copy_reg'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 8, in 
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in 
    import re
  File "/usr/lib/python3.4/re.py", line 324, in 
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in 
    from copy_reg import *
ImportError: No module named 'copy_reg'

Awesome! :(

Solution
I found a working solution here. The following command let me create a Python 3 virt. env.:

python3 -c 'import sys; del sys.argv[0]; s = sys.argv[0]; exec(open(s).read(), {"__file__": s, "__name__": "__main__"})' `which virtualenv` -p python3 venv 

Life on Arch / Manjaro is easier. There Python 3 has been the default for years.

pyvenv: create virtual environmets for Python 3.4+

January 1, 2015 Leave a comment

For creating virtual environmets, I’ve used virtualenvwrapper so far. However, Python 3.4 contains the command pyvenv that does the same thing. Since it also installs pip in the virt. env., it can replace virtualenvwrapper.

I like to store my virtual environments in a dedicated folder, separated from the project directory. virtualenvwrapper, by default, stores the virt. env.’s in the ~/.virtualenvs folder. Since I got used to this folder, I will continue to keep my virt. env.’s in this folder.

pyvenv
Say we have our project folder here: ~/python/webapps/flasky_project. Create a virt. env. for this the following way:

pyvenv ~/.virtualenvs/flasky_project

It will create a Python 3 virt. env.

virtualenv / virtualenvwrapper
For the sake of completeness, I also write here how to create virt. env.’s with virtualenv and virtualenvwrapper:

# blog post: http://goo.gl/oEdtT3

# virtualenvwrapper for Python 3 or Python 2
mkvirtualenv -p `which python3` myenv3
mkvirtualenv -p `which python2` myenv2

# virtualenv for Python 3 or Python 2
virtualenv -p python3 myproject3
virtualenv -p python2 myproject2

# When the env. is created, activate it
# and launch the command python within.
# Verify if it's the correct version.

TL; DR
If you use Python 3.4+ and you need a virt. env., use the command “pyvenv“.

[django] virtualenvwrapper with Python 3

September 12, 2014 Leave a comment

Problem
Django 1.7 came out ten days ago. If you look at the updated poll tutorial, you will notice that it’s written for Python 3! So far I’ve used Python 2 only, so I thought I would redo this tutorial for two reasons: (1) learn about migrations, and (2) start using Python 3.

For my Django projects I use virtualenvwrapper to manage my virtual environments, so the first question was: how to create a Python 3 virt. env. with virtualenvwrapper?

Well, maybe my installation got corrupted somehow, but I ran into this problem:

$ mkvirtualenv -p `which python3` tmpenv
Running virtualenv with interpreter /usr/bin/python3
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 8, in <module>
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in <module>
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 53, in apport_excepthook
    if not enabled():
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 24, in enabled
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 8, in <module>
    import base64
  File "/usr/lib/python3.4/base64.py", line 9, in <module>
    import re
  File "/usr/lib/python3.4/re.py", line 324, in <module>
    import copyreg
  File "/usr/local/lib/python2.7/dist-packages/copyreg.py", line 3, in <module>
    from copy_reg import *
ImportError: No module named 'copy_reg'

Solution
I posted this question to reddit (see here) and I found the following solution:

First, install pip3:

sudo apt-get install python3-pip
sudo pip3 install virtualenvwrapper

Then, add the following lines to your .bashrc:

export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/workspace    # customize if needed
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv
source /usr/local/bin/virtualenvwrapper.sh

Try if it works for you. In my case, I had to edit the file “/usr/local/bin/virtualenv“. I changed its first line to

#!/usr/bin/env python3

The consequence of changing the first line was that creating Python 3 environments became the default, but at least it works now.

Create a Python 3 virt. env.:

mktmpenv

Create a Python 2 virt. env.:

mktmpenv -p `which python2`

If you use the command “virtualenv“, it will also create Python 3 environments by default. Thus, “virtualenv myproject3” will be a Python 3 env., while “virtualenv -p python2 myproject2” will be a Python 2 env.

virtualenvwrapper (Part 2)

Here I describe how I use virtualenvwrapper.

I keep my virtual environments in the default location (~/.virtualenvs folder). My Python projects are kept in Dropbox. If a project (call it “stuff” for instance) has a virtual env., then I add the suffix “_project” to the projects’s folder name (resulting “stuff_project” in this example). This way I can see it immediately (without entering the folder) that it has an attached venv.

In the stuff_project folder create a requirements.txt file that contains all the modules that you installed in your venv. This way you can re-create your venv on another machine and continue to work on yout project.

Let’s see a concrete example. Say I want to create a new Django project:

$ cd $HOME/Dropbox/python/webapps
$ mkdir stuff_project
# ^ every project file will be in this folder
$ cd stuff_project
$ django-admin.py startproject stuff
# ^ create the Django project
$ mkvirtualenv stuff_project
# ^ venv is created AND activated
$ pip install django
$ pip install django-extensions
# ^ useful extensions
$ pip freeze --local >requirements.txt
# ^ packages installed in the venv are listed in this file

Since I put the project folder in Dropbox, it will be synced on all my machines. What to do if I want to work on this project on another machine of mine (let’s call it “machine #2”)? The virtual environment will be missing. But there is no need to worry. We have a requirements.txt file and from this we can populate a newly created venv. Install virtualenvwrapper on machine #2 and create an empty venv for the project:

# actions to perform on machine #2:
$ cd $HOME/Dropbox/python/webapps/stuff_project
$ mkvirtualenv stuff_project
# ^ the venv is created and activated
$ pip install -r requirements.txt
# ^ packages are installed in the venv

When you are done with the project, deactivate the venv with the “deactivate” command. Since it’s too long, I put an alias on it:

alias off='deactivate'

If you install new packages in the venv, don’t forget to update the requirements.txt file ( pip freeze >requirements.txt )!

virtualenvwrapper (Part 1)

I use virtualenv but so far it’s been enough for me. However, the time has come to step on to the next level: virtualenvwrapper. Virtualenvwrapper is just a wrapper around virtualenv, but it makes working with virtualenv so much easier.

Problem
I keep all my Python projects in Dropbox. Every project is located in a dedicated folder. If I use virtualenv, inside a project folder I need to create a subfolder for the the virtual environment (typically called “venv”). However, this folder can be really huge. In the case of a Flask project it can be up to 50-60 MB. Keeping it in Dropbox is a luxury. It would be great to move the virtual environments outside of Dropbox. My first idea was to put a symbolic link on the venv folder from the project folder but Dropbox doesn’t support symlinks :(

Solution
Let’s use virtualenvwrapper! Virtualenvwrapper was created with this idea: collect virtual environments in a separated folder, outside of your projects. In addition, it’s very easy to switch between virtual environments.

Here are some great resources on virtualenvwrapper:

For more info, refer to Bob’s blog (last link above), where he gives a very nice summary about the usage of virtualenvwrapper.

Categories: python Tags: ,

using virtualenv (Part 2)

September 21, 2013 Leave a comment

In Part 1 we saw how to use virtualenv.

Now let’s see how to colorize the bash prompt and how to activate a virtual environment easily.

Colorize the prompt
When a virtual env. is activated, the prompt changes. However, this change is not very visible because it’s not colorized. The name of the virtual env. should be printed with a different color, thus it would be visible immediately if an env. is activated or not.

Fortunately someone else also had this problem :) Here I found an excellent solution. I only changed the PS1 line the following way:

  PS1="${PYTHON_VIRTUALENV}${GREEN}\u@\h ${YELLOW}\w${COLOR_NONE} ${BRANCH}${PROMPT_SYMBOL} "

This way the prompt and the cursor are in one line. I made a fork, my slightly modified version is available here.

Update (20131001): I updated the script above to support light background too. Instructions are in the header in a comment.

Usage

Save the file above as ~/.bash_prompt and add the following line to the end of your ~/.bashrc:

source ~/.bash_prompt

The resulting prompt is way cooler than the default bash prompt, thus you can use it even if you don’t work with virtual environments!

See the end of the post for a screenshot.

Activating a virtual environment easily
The standard way for activating a virtual env. is to source the script “activate“:

jabba@jabba-uplink ~/python/mystuff $ ls -al
total 24
drwxrwxr-x   3 jabba jabba  4096 Sep 21 16:16 .
drwxrwxr-x 234 jabba jabba 12288 Sep 21 15:58 ..
-rwxrw-r--   1 jabba jabba    53 Sep 21 16:16 hello.py
drwxrwxr-x   6 jabba jabba  4096 Sep 21 16:13 venv
jabba@jabba-uplink ~/python/mystuff $ . venv/bin/activate
[venv] jabba@jabba-uplink ~/python/mystuff $

However, using the command “workon venv” would be much easier. I wanted to do it with an alias, but bash aliases do not accept arguments. There is a workaround: use functions instead.

Add the following lines to your ~/.bashrc:

func_workon()
{
if [[ -z "$1" ]]
then
    echo "Usage: workon <venv>"
else
    . $1/bin/activate
fi
}
alias workon=func_workon

alias workoff='deactivate'

Now you can activate a virtual env. much easier:

jabba@jabba-uplink ~/python/mystuff $ ls -al
total 24
drwxrwxr-x   3 jabba jabba  4096 Sep 21 16:16 .
drwxrwxr-x 234 jabba jabba 12288 Sep 21 15:58 ..
-rwxrw-r--   1 jabba jabba    53 Sep 21 16:16 hello.py
drwxrwxr-x   6 jabba jabba  4096 Sep 21 16:13 venv
jabba@jabba-uplink ~/python/mystuff $ workon venv
[venv] jabba@jabba-uplink ~/python/mystuff $

For deactivation use the command “deactivate“, or my alias “workoff“.

Screenshot
workon

Next step
There is virtualenvwrapper, which is “a set of extensions to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

I haven’t used it yet.

using virtualenv (Part 1)

September 21, 2013 Leave a comment

Here I won’t explain what virtualenv is. There are lots of tutorials on this topic (see the “virtualenv” subsection here for instance, or this SO thread).

This post is about the basic usage of virtualenv.

Step 0 (20131029)
Before doing anything, create the file ~/.pip/pip.conf and edit it:

[install]
download-cache = ~/.pip/download_cache

This way each version is downloaded only once.

Update (20140807): Maybe using a cache is not that good idea. It happened to me that a new version came out of a package that I wanted to install but because of the cache, it installed an older version! I didn’t understand why I still had an out-of-date version. So I removed the ~/.pip folder entirely. I don’t recommend using a cache.

Installation
Install virtualenv:

sudo pip install virtualenv

Trying out
Create a directory where you want to store your project:

$ cd ~/python
$ mkdir mystuff
$ cd mystuff

Let’s create here a virtual environment:

$ virtualenv venv

You will get a subdirectory called “venv“, which contains a clean Python virtual environment. Activate it with the following command:

$ . venv/bin/activate

Notice that the prompt changes: now it includes the name of the virtual environment. If you call the command “python” or “pip“, the binaries in the virtual environment will be called.

Running example:

jabba@jabba-uplink:~$ cd python
jabba@jabba-uplink:~/python$ mkdir mystuff
jabba@jabba-uplink:~/python$ cd mystuff/
jabba@jabba-uplink:~/python/mystuff$ ls -al
total 16
drwxrwxr-x   2 jabba jabba  4096 Sep 21 15:58 .
drwxrwxr-x 234 jabba jabba 12288 Sep 21 15:58 ..
jabba@jabba-uplink:~/python/mystuff$ virtualenv venv
...
jabba@jabba-uplink:~/python/mystuff$ ls -al
total 20
drwxrwxr-x   3 jabba jabba  4096 Sep 21 15:58 .
drwxrwxr-x 234 jabba jabba 12288 Sep 21 15:58 ..
drwxrwxr-x   6 jabba jabba  4096 Sep 21 15:58 venv
jabba@jabba-uplink:~/Dropbox/python/mystuff$ . venv/bin/activate
(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$ which python
/home/jabba/Dropbox/python/mystuff/venv/bin/python                                                     
(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$ which pip
/home/jabba/Dropbox/python/mystuff/venv/bin/pip                                                               

To stop using a virtual environment, use the command “deactivate“.

Example
Say I have the package “requests” globally installed (with “sudo pip install requests“). When I launch the python shell, this module can be imported.

However, when you activate a clean virtual environment, globally installed 3rd party packages are not available. They must be installed with the virtual environment’s “pip” command, and they will be installed inside the “venv” directory. This is the real advantage of virtualenv: virtual environments are completely isolated from each other and you can install different package versions, there won’t be any conflict.

Running example:

jabba@jabba-uplink:~/Dropbox/python/mystuff$ which python
/usr/bin/python
jabba@jabba-uplink:~/Dropbox/python/mystuff$ python
Python 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Persistent session history and tab completion are enabled.
>>> import requests
>>> 
jabba@jabba-uplink:~/Dropbox/python/mystuff$ . venv/bin/activate
(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$ which python
/home/jabba/Dropbox/python/mystuff/venv/bin/python
(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$ python
Python 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Persistent session history and tab completion are enabled.
>>> import requests
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named requests
>>> 
(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$ which pip
/home/jabba/Dropbox/python/mystuff/venv/bin/pip
(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$ pip install requests
Downloading/unpacking requests
  Downloading requests-1.2.3.tar.gz (348kB): 348kB downloaded
  Running setup.py egg_info for package requests
    
Installing collected packages: requests
  Running setup.py install for requests
    
Successfully installed requests
Cleaning up...
(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$ python
Python 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Persistent session history and tab completion are enabled.
>>> import requests
>>> 
(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$

Where to put the project files
Store your project files (in this example) in the “~/python/mystuff” directory. If the virtual environment is activated, then the modules installed in the virtual environment are available for your project:

(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$ ls -al
total 24
drwxrwxr-x   3 jabba jabba  4096 Sep 21 16:16 .
drwxrwxr-x 234 jabba jabba 12288 Sep 21 15:58 ..
-rwxrw-r--   1 jabba jabba    53 Sep 21 16:16 hello.py
drwxrwxr-x   6 jabba jabba  4096 Sep 21 16:13 venv
(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$ cat hello.py 
#!/usr/bin/env python

import requests
print "hello"
(venv)jabba@jabba-uplink:~/Dropbox/python/mystuff$ ./hello.py 
hello

We installed “requests” in the previous step, and it’s nicely available for the project. If you uninstall requests from the virtual environment with “pip uninstall requests” and try to run hello.py, you will get an import error message.

In Part 2 we will see how to colorize the prompt and how to activate a virtual environment easily.

Categories: python Tags:

The Python Ecosystem

January 7, 2012 Leave a comment

Here is an excellent introductory article to the Python ecosystem including packages, pip, virtualenv (and virtualenvwrapper), and some other important tools (e.g. fabric). I wish I had read it a long time ago.

Categories: python Tags: , , , ,