When creating a Python program that many dependencies, it can become frustrating or complicated to deploy this program to multiple environments. Target environments might have different operating systems, CPU architectures or Python interpreters. Traditional deployment solutions such as dh-virtualenv would require creating multiple packages in order to deploy to a variety of environments. Creating a single executable that can be used across these environments can be very helpful.
To get a single executable that we can deploy to a large variety of environments, we can use PEX to solve this problem.
This blog post will explain how we can use PEX’s --platform flag to create a single executable that can run across different Python interpreter versions and different operating systems.
Why PEX ?
There are a multitude of ways to package and distribute an application written in Python. However PEX is unique compared to other alternatives because it supports the following:
Targeting Multiple Operating Systems: macOS and Linux.
Targeting Multiple Python Interpreters: PyPy, CPython, etc.
Producing a single artifact that works across all of the above.
Based on Python wheels, allowing for any kind of native dependency.
In general, if your application and all of the dependencies have wheels for all of your target environments, then it’s trivial to produce a PEX that allows for your application to work there.
Why is this useful?
Imagine the case where you have a tool that uses pygit2 to interact with a git repository. It might be useful to use this tool as apart of your build your development process. However you might need to target the following environments with your tool:
macOS for local development.
Debian stable bare metal machines.
Ubuntu LTS based Docker images.
Alpine based Docker image.
Within the above, you would need to support many CPython versions, since the CPython that ships with Debian Buster is 3.7.3, the version that ships with Ubuntu 20.04 is 3.8.5 and Alpine Docker images are currently based on 3.9.5.
Single Linux distribution tools like dh-virtualenv are not helpful in this scenario since we would still need to figure out how package our application for macOS and Alpine. A single PEX file that we could deploy to all of the above environments would greatly simplify distribution.
We can create a Python application call git-inspect that uses libgit2 to print out the commit id of HEAD in the repository.
We can also use the pip-compile command from the pip-tools project to give us the full dependency list for pygit2 == 1.6.1.
Using PEX we can now build a PEX for our local environment:
Running it produces a PEX file which runs fine on my MacBook:
However running this on a Debian 10 host results in an error:
This is because the PEX doesn’t contain wheels for Linux just for macOS. The wheels in the PEX can be seen in the PEX-INFO file.
To get a PEX file that can run on macOS and Linux the --platform argument to pex can be used to get wheels required for every target platform. The --platform argument accepts Python platform tags. By default PEX defaults to the platform of the interpreter that is running which in my case was cp39-cp39-macosx_10_9_x86_64. pex can be given many --platform arguments, to target CPython 3.6, 3.7, 3.8, 3.9 for macOS and Linux we can do:
The PEX still works fine on my MacBook:
Running this on a Debian 10 host also works:
This works despite the fact that the Debian 10 host only has Python 3.7 and my MacBook has Python 3.9. The same PEX will work on any Mac or Linux host so long as it has CPython 3.6-3.9 installed.
The PEX-INFO file shows that all of the wheels are present for the target platforms:
A Note on CPython ABIs and PEX size
One issue with the approach above is the sheer size of the output PEX.
The PEX just for my MacBook is 1.5 MB whereas the PEX targeting all of our environments is 24 MB. This is because some of our dependencies, such as cffi and pygit2 do not target the stable CPython ABI but instead the CPython version specific ABI. This results in requiring a wheel for every target platform which results in a significantly larger PEX. Dependencies that target the stable ABI could have a single wheel that satisfies all platforms, resulting in smaller PEX files.