wiki:Documentation/vpy

Getting Started with a Virtual Python Installation

Working with and developing Python software is complicated by several factors:

  • Many users do not have the administrative privileges that are needed to install packages in the Python site packages directory, where Python packages are most easily installed
  • Python developers often need to manage many Python projects in the same development environment

The Python  virtualenv package resolves these issues by allowing users to generate a local Python installation. Thus, a user can create a Python installation into which they can install Python packages using standard tools like easy_install and pip.

Unfortunately, getting started with virtualenv is complicated by the fact that you need to install this package to create virtual Python installations. But, if you do not have administrative privileges, then you are stuck. This is the simplest process that I have seen:

hg clone http://bitbucket.org/ianb/virtualenv      # clone the virtualenv repository with mercurial
virtualenv/virtualenv.py <ENV>                     # create environment <ENV> with the virtualenv.py script

Unfortunately, the hg command may not be preinstalled on your machine.

The pyutilib.virtualenv package contains the vpy_install script, which can bootstrap the setup of virtualenv. This script extends the functionality of the virtualenv driver script to allow direct, and it customization of this initial environment. The vpy_install script can be easily downloaded from this wiki page, or it can be downloaded with commands like wget or curl:

wget http://doiop.com/pyutilib/vpy_install
vpy_install <ENV>

Using vpy_install

The basic usage of vpy_install mimics that of virtualenv:

vpy_install <ENV>

This creates a Python environment in the <ENV> directory. If omitted, the default directory is ./python (or C:\python on Windows). The environment consists of the following subdirectories:

  • admin - Administrative data for maintaining this distribution
  • bin - Scripts and executables
  • dist - Python packages that are not intended for development
  • include - Python header files
  • lib - Python libraries and installed packages
  • src - Python packages whose source files can be modified and used directly within this virtual Python installation.
  • Scripts - Python bin directory (used on MS Windows)

By default, vpy_install installs the virtualenv and setuptools packages. Thus, a user can install packages from the PyPI repository using easy_install:

<ENV>/bin/easy_install nose

This command installs the nose into this local Python installation.

Note: several packages -- particularly packages that have compiled CPython components like NumPy, SciPy, and MatPlotLib -- do not trivially install into virtual environments with easy_install. For those situations, a slightly more forceful approach usually works (the following example assumes a BASH shell):

export PYTHONPATH=<ENV>/lib/python*/site-packages
<ENV>/bin/easy_install --install-dir $PYTHONPATH scipy

The vpy_install command can use configuration files to specify additional Python packages that are installed during installation. For example, suppose that the configuration file config.ini contains:

;
; This INI file can be used by vpy_install to create a virtual
; Python installation.
;
[installer]
description=This is an example.
README="#
     # This is a README file created by the bootstrap script.
     #"

[nose]
pypi=nose

[pyutilib.subprocess]
root=https://software.sandia.gov/svn/public/pyutilib/pyutilib.subprocess
dev=True

Then the following command woudl create a virtual Python environment that contains both nose and pyutilib.subprocess:

vpy_install --config config.ini

The pyutilib.subprocess package is installed in python/src/pyutilib.subprocess, and this package is installed in this Python environment as an editable develop package.

Note that vpy_install does not require a user to install setuptools. The setuptools package is only required when the --preinstall option is specified. Thus, the default behavior of vpy_install (and other bootstrap scripts) does not require the installation of any third-party packages. The vpy_install script can be distributed independent of pyutilib.virtualenv, and it can generally be used as a stand-alone script.

Advanced Configuration Options

The vpy_install configuration file allows for a flexible specification of software packages that are incorporated into a virtual Python environment. The following examples illustrate different specifications that are supported by pyutilib.virtualenv.

PyPI Packages

The following specifies that the nose package will be installed from PyPI:

[nose]
pypi=nose

Subversion Software Packages

The following specifies the root directory of a subversion software repository that will be used:

[pyutilib.th]
root=https://software.sandia.gov/svn/public/pyutilib/pyutilib.th
dev=True

This specification assumes that the pyutilib.th project contains a trunk subdirectory (used for active development) and/or a releases or tags subdirectory (used for release snapshots). If the --trunk option is used with vpy_install, then the trunk subdirectory is installed; otherwise the latest releases or tags version is installed.

The dev configuration value is optional. If this is True, then this package is installed in the src directory, and it is installed with the develop distutils option. Otherwise, this package is installed in the dist directory, and it installed normally with distutils. Thus, developers can freely edit and test packages that are installed in the src directory.

Localizing Commands

On Windows, it is convenient to localize DOS CMD files can be localized to use a virtual Python environment. Localized files can be explicitly specified as follows:

[localize]
my_script.cmd=
foo.cmd=

This localization process replaces the string __VIRTUAL_ENV__ with the path to the virtual Python environment.

Auxilliary Directories

It is often convenient to bundle auxilliary directories into a virtual Python environment, especially when providing this to third-party users. However, this auxilliary information may not reside in a Python package. The following example illustrates the use of auxilliary data from the spot package:

[spot]
root=https://software.sandia.gov/svn/teva/spot/spot
install=False

[spot:auxdir]
doc=/doc/pub

The install option indicates that the spot package should not be installed. Instead, this declaration is used in the spot:auxdir declaration to specify the subversion directories that are installed in the virtual Python environment. If this environment is created in the <ENV> directory, then the spot/doc/pub directory is checked out as the <ENV>/doc directory.

Installer Options

The installer configuration section allows the user to specify a variety of global configuration options:

  • description - Description that is used when the the --help option is used (ignored by vpy_install)
  • default_dirname - The name of the default directory used to install the virtual environment (ignored by vpy_install, which uses python)
  • config - A string that contains configuration text that is used during installation
  • config_file - A filename or URL of a configuration file that is used during installation
  • README - Text for the README.txt file that is generated

Developing Python Packages in a Virtual Environment

If vpy_install is used with the --trunk option, then the packages in the src directory are trunk branches of Python packages that can be developed normally. Changes made in these packages should be immediately available within the virtual Python environment. However, when files or directories are created, these packages may need to be reinstalled. For example:

cd python/src/pyutilib.enum
../../bin/python setup.py develop

This reinstalls the pyutilib.enum package in with the develop option.

When the pyutilib.dev package is installed, the following utilities are available to assist the development of Python packages:

  • svn_update - Update all of the Python packages in the src directory
  • svn_status - Report the status of the Python packages in the src directory
  • svn_test - Apply the nose package to find and apply distutils tests.
    The general syntax is
    svn_test -p <package-to-cover> <dir1> <dir2> ...
    
    For example, the following command applied tests and compute coverage tests for the coopr package (and related directories):
    svn_test -p coopr coopr*
    
  • lbin - This script recurses up the current path, looking for the bin directories. If found, these directories are prepended to the PATH environment path.
  • lpython- This script recurses up the current path, looking for a bin directory that contains a Python executable. If found, this Python executable is executed, and either a Python shell is started or a Python script is executed.

Note that lbin and lpython are stand-alone scripts. These scripts can be placed in the user's bin directory and then used without direct reference to any Python virtual environment.