Creating a Virtualenv from a PEX

Created On:

Since PEX v2.1.25 PEX has supported a --venv flag when creating a PEX. The flag causes the created PEX to extract itself into a virtual environment on startup. This has two benefits:

The speed improvements are significant for tools that one might want to run repeatedly such as black or mypy.

Here is a benchmark of running black from a traditional PEX vs one built with the --venv flag.

$ pex 'black==20.8b1' --script black -o black.pex
$ pex 'black==20.8b1' --script black --venv -o black-venv.pex
$ hyperfine -w2 './black.pex --help' './black-venv.pex --help'
Benchmark #1: ./black.pex --help
  Time (mean ± σ):      1.046 s ±  0.009 s    [User: 866.0 ms, System: 160.4 ms]
  Range (min … max):    1.037 s …  1.068 s    10 runs

Benchmark #2: ./black-venv.pex --help
  Time (mean ± σ):     360.7 ms ±   4.0 ms    [User: 254.0 ms, System: 88.0 ms]
  Range (min … max):   356.5 ms … 367.4 ms    10 runs

Summary
  './black-venv.pex --help' ran
    2.90 ± 0.04 times faster than './black.pex --help'

The PEX built with the --venv flag is almost 3x faster to execute.

Where is the Virtualenv ?

By default the PEX will create a virtualenv in the ~/.pex/venvs directory similar to where PEX keeps its various caches. The full file path is determined by the hash of the PEX as well as the hash of the environment used to select the Python interpreter. Like everything in ~/.pex this can be safely deleted without issue and the next run of the PEX will recreate the folder.

Alternative Location for the Virtualenv

The default location of ~/.pex/venvs is hard to use since the full path of the virtualenv is composed of opaque file hashes and makes it hard to use the created virtualenv with other tools. Alternatively the virtualenv can be created in an arbitrary location using the ‘PEX Tools’ functionality.

When a PEX is created with the --venv flag, the created PEX has some extra PEX related code that can run when the PEX_TOOLS=1 environment variable is set.

The PEX can be created in the ./venv directory by invoking the PEX like so:

$ PEX_TOOLS=1 ./black-venv.pex venv ./venv

What’s in the Virtualenv?

The created virtualenv has everything a regular virtualenv would have plus a few PEX specific extras:

$ ls
PEX-INFO    __main__.py bin         include     lib         pex         pyvenv.cfg

Inside the bin directory we have the activation scripts as well as all of the console entry points from all of the wheels as well as our wheel console scripts.

$ ls bin
Activate.ps1  activate.csh  black         blackd        python3
activate      activate.fish black-primer  python        python3.9

It’s important to note that setuptools and pip are not installed into this virtual environment which makes them immutable.

We can directly execute a console script in the ./bin directory for an even larger performance boost than executing the PEX built with --venv.

$ hyperfine -w2 './venv/bin/black --help' './black-venv.pex --help' './black.pex --help'
Benchmark #1: ./venv/bin/black --help
  Time (mean ± σ):     187.8 ms ±   4.6 ms    [User: 151.0 ms, System: 32.7 ms]
  Range (min … max):   183.0 ms … 197.8 ms    15 runs

Benchmark #2: ./black-venv.pex --help
  Time (mean ± σ):     359.5 ms ±   5.6 ms    [User: 252.3 ms, System: 88.3 ms]
  Range (min … max):   351.9 ms … 367.4 ms    10 runs

Benchmark #3: ./black.pex --help
  Time (mean ± σ):      1.050 s ±  0.011 s    [User: 864.8 ms, System: 164.8 ms]
  Range (min … max):    1.033 s …  1.064 s    10 runs

Summary
  './venv/bin/black --help' ran
    1.91 ± 0.06 times faster than './black-venv.pex --help'
    5.59 ± 0.15 times faster than './black.pex --help'

There is also a PEX-INFO file in the root which is copied from the PEX that created the virtualenv.

{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.42",
    "platform": "macosx_11_0_x86_64",
    "version": [
      3,
      9,
      4
    ]
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "appdirs-1.4.4-py2.py3-none-any.whl": "b219560d4b28b017da90ce2c5484a2b6c62dfc52",
    "black-20.8b1-py3-none-any.whl": "efc41b1c13052ffc6f2e9c5144aedefeba0c4d41",
    "click-8.0.1-py3-none-any.whl": "60cbe72bab38df1f22792bc2d8b6b3a072d02dbf",
    "mypy_extensions-0.4.3-py2.py3-none-any.whl": "41374e5ada663fa33b6fc37950c079faba1a8f32",
    "pathspec-0.8.1-py2.py3-none-any.whl": "eb9e7328e69e63265e32837e52d572b28103f41a",
    "regex-2021.7.1-cp39-cp39-macosx_10_9_x86_64.whl": "356520db0bd32672b69f021d1e5f55f2a55b747e
",
    "toml-0.10.2-py2.py3-none-any.whl": "941913d720ad4816a848c11a218c9110f1978120",
    "typed_ast-1.4.3-cp39-cp39-macosx_10_9_x86_64.whl": "ff61103c4d1a9e3da01759d8a400ef02103f1d0
8",
    "typing_extensions-3.10.0.0-py3-none-any.whl": "2d21ea9841868bec222ae7bffdf3212ac3ad9c3e"
  },
  "emit_warnings": true,
  "entry_point": "black:patched_main",
  "ignore_errors": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_hash": "3bdf45072c53c0a39ca9398703cfccb01fb1ca4a",
  "pex_path": null,
  "pex_root": "/Users/zmanji/.pex",
  "requirements": [
    "black==20.8b1"
  ],
  "strip_pex_env": true,
  "unzip": false,
  "venv": true,
  "venv_bin_path": "false",
  "venv_copies": false,
  "zip_safe": true
}

Finally there is also a __main__.py at the root which makes the entire directory executable by Python. Pointing Python to this directory is equivalent to pointing Python to the PEX file itself except you get the speed benefits of the virtualenv.

$ python ./venv - --help
Re-execing from /Users/zmanji/.pyenv/versions/3.9.4/bin/python
Usage: python -m venv [OPTIONS] [SRC]...

  The uncompromising code formatter.

...

Conclusion

Creating a PEX with the --venv flag allows for a virtualenv to be implicitly or explicitly created from the PEX. Creating a virtualenv has less execution overhead than a plain PEX and the resulting virtualenv is immutable.