A utility for building wheels in reproducible environments

PEP 517 and PEP 518 improved Python packaging by allowing projects to specify which dependencies are needed to build a wheel. They specify a build-system.requires section in a pyproject.toml file which acts as a requirements.txt for build time dependencies. The benefit of specifying build time dependencies is projects other than setuptools can be used to build wheels.

Unfortunately there is no lock file equivalent of the dependencies in pyproject.toml meaning tools like pip need to re-resolve the dependencies every time a wheel needs to be built. As I have written before not only does this slow building the wheel, it also introduces non determinism. For example the following pyproject.toml section results in non determinism.

[build-system]
requires = ["setuptools>48", "wheel"]

Since these requirements don’t specify an exact version nor do they pin transitive dependencies, the versions used in the build will be the latest versions published on PyPI. It’s even possible to fail to build entirely because of backwards incompatible changes.

Introducing reproducible-wheel-builder

To fix the above issue I wrote reproducible-wheel-builder¹ a utility which combines pex and build to create reproducible environments for building wheels. Instead of relying on pip to re-resolve the build-system.requires for every build, pex and a Pex lockfile are used to create a reproducible virtual environment for the build. Then build uses that virtual environment to build the wheel. Since a lockfile is used, the build environment is reproducible.

It is distributed as a pex and can be downloaded from GitHub. Running it is as simple as executing main.pex.

$ ./main.pex --help
usage: __pex_executable__.py [-h] --lock LOCK --src SRC [--dist {wheel,sdist,editable}] --out OUT [--quiet] [--requires_config REQUIRES_CONFIG] [--build_config BUILD_CONFIG]

options:
  -h, --help            show this help message and exit
  --lock LOCK           Path to the pex lock file
  --src SRC             Path to the source directory to build
  --dist {wheel,sdist,editable}
                        Type of distribution to build
  --out OUT             Path to the output directory
  --quiet               Supress the output of the build backend
  --requires_config REQUIRES_CONFIG
                        JSON file of config to pass to build backend when getting requires
  --build_config BUILD_CONFIG
                        JSON file of config to pass to build backend when building

This can be used instead of pip wheel to build a project.

Example

gevent 21.1.2 does not have wheels for Python 3.10. Using pip wheel on the sdist results in the following error.

$ pip --verbose wheel ./gevent-21.1.2
Processing ./gevent-21.1.2
  ...

  Error compiling Cython file:
  -----------------------------------------------------------
  ...
  cdef load_traceback
  cdef Waiter
  cdef wait
  cdef iwait
  cdef reraise
  cpdef GEVENT_CONFIG
        ^
  ------------------------------------------------------------

  src/gevent/_gevent_cgreenlet.pxd:182:6: Variables cannot be declared with 'cpdef'. Use 'cdef' instead.
  Compiling src/gevent/greenlet.py because it changed.

  ...

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

This is because the pyproject.toml has the following build-system.requires section.

requires = [
     "setuptools >= 40.8.0",
     "wheel",
     "Cython >= 3.0a6",
     "cffi >= 1.12.3 ; platform_python_implementation == 'CPython'",
     "greenlet >= 0.4.17, < 2.0 ; platform_python_implementation == 'CPython'",
]

The Cython >= 3.0a6 requirement is problematic because Cython 3.0a8 has the following change:

Variables can no longer be declared with cpdef.

This can be overcome by using reproducible-wheel-builder with a pex lockfile that pins Cython to 3.0a7. Generating a pex lockfile for the dependencies can be done with the pex3 command.

$ pex3 lock create --pip-version 22.3 --resolver-version pip-2020-resolver \
  --no-build --indent 2 -o pex.lock \
  'setuptools >= 40.8.0' 'wheel' 'Cython == 3.0a7' 'cffi >= 1.12.3' 'greenlet >= 0.4.17, < 2.0'

With the pex.lock file and the main.pex of reproducible-wheel-builder a wheel can be built now.

$ ./main.pex --lock pex.lock --src ./gevent-21.1.2 --out ./out --quiet

The above command successfully runs and outputs a wheel under ./out.

Conclusion

PEP 517 and PEP 518 don’t specify a lockfile mechanism for specifying the build environment. This leads to non determinism and possibly failed builds when using pip wheel. This can be overcome with a pex lockfile of the build environment and passing it to reproducible-wheel-builder to do the build.