Packaging Multi Platform Python Applications

Created On: June 25 2021

When creating a Python program that many dependencies, it can become frustrating or complicated to deploy this program to multiple environments. Target environments might have different operating systems, CPU architectures or Python interpreters. Traditional deployment solutions such as dh-virtualenv would require creating multiple packages in order to deploy to a variety of environments. Creating a single executable that can be used across these environments can be very helpful.

To get a single executable that we can deploy to a large variety of environments, we can use PEX to solve this problem.

This blog post will explain how we can use PEX’s --platform flag to create a single executable that can run across different Python interpreter versions and different operating systems.

Why PEX ?

There are a multitude of ways to package and distribute an application written in Python. However PEX is unique compared to other alternatives because it supports the following:

Targeting Multiple Operating Systems: macOS and Linux.
Targeting Multiple Python Interpreters: PyPy, CPython, etc.
Producing a single artifact that works across all of the above.
Based on Python wheels, allowing for any kind of native dependency.

In general, if your application and all of the dependencies have wheels for all of your target environments, then it’s trivial to produce a PEX that allows for your application to work there.

Why is this useful?

Imagine the case where you have a tool that uses pygit2 to interact with a git repository. It might be useful to use this tool as apart of your build your development process. However you might need to target the following environments with your tool:

macOS for local development.
Debian stable bare metal machines.
Ubuntu LTS based Docker images.
Alpine based Docker image.

Within the above, you would need to support many CPython versions, since the CPython that ships with Debian Buster is 3.7.3, the version that ships with Ubuntu 20.04 is 3.8.5 and Alpine Docker images are currently based on 3.9.5.

Single Linux distribution tools like dh-virtualenv are not helpful in this scenario since we would still need to figure out how package our application for macOS and Alpine. A single PEX file that we could deploy to all of the above environments would greatly simplify distribution.

An Example

We can create a Python application call git-inspect that uses libgit2 to print out the commit id of HEAD in the repository.

[metadata]
name = git-inspect

[options]
py_modules =
  git_inspect
install_requires =
   pygit2 == 1.6.1

[options.entry_points]
console_scripts =
     git-inspect = git_inspect:main

setup.cfg

#!/usr/bin/env python3

import sys

import pygit2

def main():
    repo = pygit2.Repository(path=sys.argv[1])
    print(repo.head.peel())

if __name__ == "__main__":
    main()

git_inspect.py

We can also use the pip-compile command from the pip-tools project to give us the full dependency list for pygit2 == 1.6.1.

cffi==1.14.5
    # via pygit2
pycparser==2.20
    # via cffi
pygit2==1.6.1
    # via git-inspect (setup.py)

requirements.txt

Using PEX we can now build a PEX for our local environment:

$ pex --script git-inspect \
  -r requirements.txt . \
  --python-shebang '#!/usr/bin/env python3' \
  -o dist/out.pex

Running it produces a PEX file which runs fine on my MacBook:

$ ./dist/out.pex ~/code/pex
<pygit2.Object{commit:075da872288e8aa5405a454069cf633d578e578d}>

However running this on a Debian 10 host results in an error:

$ ./out.pex
Traceback (most recent call last):
  File "/home/zmanji/out.pex/.bootstrap/pex/pex.py", line 483, in execute
  File "/home/zmanji/out.pex/.bootstrap/pex/pex.py", line 139, in activate
  File "/home/zmanji/out.pex/.bootstrap/pex/pex.py", line 126, in _activate
  File "/home/zmanji/out.pex/.bootstrap/pex/environment.py", line 428, in activate
  File "/home/zmanji/out.pex/.bootstrap/pex/environment.py", line 784, in _activate
  File "/home/zmanji/out.pex/.bootstrap/pex/environment.py", line 608, in resolve
  File "/home/zmanji/out.pex/.bootstrap/pex/environment.py", line 629, in resolve_dists
  File "/home/zmanji/out.pex/.bootstrap/pex/environment.py", line 573, in _root_requirements_iter
pex.environment.ResolveError: A distribution for cffi could not be resolved in this environment.Found 1 distribution for cffi that do not apply:
1.) The wheel tags for cffi 1.14.5 are cp39-cp39-macosx_10_9_x86_64 which do not match the supported tags of DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.7', PythonIdentity('/usr/bin/python3', 'cp37', 'cp37m', 'manylinux_2_28_x86_64', (3, 7, 3)))):
...

This is because the PEX doesn’t contain wheels for Linux just for macOS. The wheels in the PEX can be seen in the PEX-INFO file.

{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.42",
    "platform": "macosx_11_0_x86_64",
    "version": [
      3,
      9,
      4
    ]
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "cffi-1.14.5-cp39-cp39-macosx_10_9_x86_64.whl": "89776038a89a4dfc0f24b0b274656ae011709cff",
    "git_inspect-0.0.0-py3-none-any.whl": "bd703f052a1b9eff5d194d197f6d9a3120c4695d",
    "pycparser-2.20-py2.py3-none-any.whl": "c1eaa0ece2823d3947bb56caed5b9fa8973c5e11",
    "pygit2-1.6.1-cp39-cp39-macosx_10_9_x86_64.whl": "a88d821366e60b44bc524cdaa8a9f2718ca510d1"
  },
  "emit_warnings": true,
  "entry_point": "git_inspect:main",
  "ignore_errors": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_hash": "7c5e0b6634b3959dd4267227a0dbd3cbefa3fd1a",
  "pex_path": null,
  "requirements": [
    "cffi==1.14.5",
    "git-inspect==0.0.0",
    "pycparser==2.20",
    "pygit2==1.6.1"
  ],
  "strip_pex_env": true,
  "unzip": false,
  "venv": false,
  "venv_bin_path": "False",
  "venv_copies": false,
  "zip_safe": true
}

PEX-INFO

To get a PEX file that can run on macOS and Linux the --platform argument to pex can be used to get wheels required for every target platform. The --platform argument accepts Python platform tags. By default PEX defaults to the platform of the interpreter that is running which in my case was cp39-cp39-macosx_10_9_x86_64. pex can be given many --platform arguments, to target CPython 3.6, 3.7, 3.8, 3.9 for macOS and Linux we can do:

$ pex --script git-inspect \
  -r requirements.txt . \
  --python-shebang '#!/usr/bin/env python3' \
  --platform macosx_11_0_x86_64-cp-39-cp39 \
  --platform macosx_11_0_x86_64-cp-38-cp38 \
  --platform macosx_11_0_x86_64-cp-37-cp37m \
  --platform macosx_11_0_x86_64-cp-36-cp36m \
  --platform manylinux2014_x86_64-cp-39-cp39 \
  --platform manylinux2014_x86_64-cp-38-cp38 \
  --platform manylinux2014_x86_64-cp-37-cp37m \
  --platform manylinux2014_x86_64-cp-36-cp36m \
  -o dist/out.pex

The PEX still works fine on my MacBook:

$ ./git_inspect.py ~/code/pex
<pygit2.Object{commit:075da872288e8aa5405a454069cf633d578e578d}>

Running this on a Debian 10 host also works:

$ ./out.pex ./pex
<pygit2.Object{commit:075da872288e8aa5405a454069cf633d578e578d}>

This works despite the fact that the Debian 10 host only has Python 3.7 and my MacBook has Python 3.9. The same PEX will work on any Mac or Linux host so long as it has CPython 3.6-3.9 installed.

The PEX-INFO file shows that all of the wheels are present for the target platforms:

{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.42",
    "platform": "macosx_11_0_x86_64",
    "version": [
      3,
      9,
      4
    ]
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "cached_property-1.5.2-py2.py3-none-any.whl": "96efe61d013fd9050cb22369407ea028e9cc9b2c",
    "cffi-1.14.5-cp36-cp36m-macosx_10_9_x86_64.whl": "55e5a5ff22a16068ae44212c0a88185b2e8c3f87",
    "cffi-1.14.5-cp36-cp36m-manylinux1_x86_64.whl": "2f061f0abdc3125831602a4a9cd43c1d47088143",
    "cffi-1.14.5-cp37-cp37m-macosx_10_9_x86_64.whl": "7721d95e00506cb199417ad52f33bb1e915c08b6",
    "cffi-1.14.5-cp37-cp37m-manylinux1_x86_64.whl": "7f2c48d8739abcd023275df9dc7e467b5515ccc6",
    "cffi-1.14.5-cp38-cp38-macosx_10_9_x86_64.whl": "64f6888ba9f8be79681ce952633e559175211ddd",
    "cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl": "7439b0775123c4cba0394c5a77aa2e3fce55c5a9",
    "cffi-1.14.5-cp39-cp39-macosx_10_9_x86_64.whl": "89776038a89a4dfc0f24b0b274656ae011709cff",
    "cffi-1.14.5-cp39-cp39-manylinux1_x86_64.whl": "80738ab38437ace103272942853e352d3f412274",
    "git_inspect-0.0.0-py3-none-any.whl": "bd703f052a1b9eff5d194d197f6d9a3120c4695d",
    "pycparser-2.20-py2.py3-none-any.whl": "c1eaa0ece2823d3947bb56caed5b9fa8973c5e11",
    "pygit2-1.6.1-cp36-cp36m-macosx_10_9_x86_64.whl": "0c8ed85f42aeaf377d99fefae68dfc184a93e37c",
    "pygit2-1.6.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "6fcb1d960dc5a008d802babf9a774d1e21655851",
    "pygit2-1.6.1-cp37-cp37m-macosx_10_9_x86_64.whl": "47ec25fa3b4654a612e841f9e9454b97dd07258e",
    "pygit2-1.6.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "1a370f1d0fc4ae433597becd8c7a08f5f5a3c9e8",
    "pygit2-1.6.1-cp38-cp38-macosx_10_9_x86_64.whl": "1f7e923e35f87c36bbae1615e02d30b6713f1701",
    "pygit2-1.6.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "d14961c955a7e104c2734a9e1e8606355092fe09",
    "pygit2-1.6.1-cp39-cp39-macosx_10_9_x86_64.whl": "a88d821366e60b44bc524cdaa8a9f2718ca510d1",
    "pygit2-1.6.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "0c4f1503bd8cf631712c47879cd25642b37c8d98"
  },
  "emit_warnings": true,
  "entry_point": "git_inspect:main",
  "ignore_errors": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_hash": "eab984c7ae563881fdc03c3baae2f64b16883631",
  "pex_path": null,
  "requirements": [
    "cffi==1.14.5",
    "git-inspect==0.0.0",
    "pycparser==2.20",
    "pygit2==1.6.1"
  ],
  "strip_pex_env": true,
  "unzip": false,
  "venv": false,
  "venv_bin_path": "False",
  "venv_copies": false,
  "zip_safe": true
}

A Note on CPython ABIs and PEX size

One issue with the approach above is the sheer size of the output PEX.

The PEX just for my MacBook is 1.5 MB whereas the PEX targeting all of our environments is 24 MB. This is because some of our dependencies, such as cffi and pygit2 do not target the stable CPython ABI but instead the CPython version specific ABI. This results in requiring a wheel for every target platform which results in a significantly larger PEX. Dependencies that target the stable ABI could have a single wheel that satisfies all platforms, resulting in smaller PEX files.