PEP 517 and PEP 518 improved Python packaging by allowing projects to specify which dependencies are needed to build a wheel. They specify a
build-system.requires section in a
pyproject.toml file which acts as a
requirements.txt for build time dependencies. The benefit of specifying build time dependencies is projects other than
setuptools can be used to build wheels.
Unfortunately there is no lock file equivalent of the dependencies in
pyproject.toml meaning tools like
pip need to re-resolve the dependencies every time a wheel needs to be built. As I have written before not only does this slow building the wheel, it also introduces non determinism. For example the following
pyproject.toml section results in non determinism.
Since these requirements don’t specify an exact version nor do they pin transitive dependencies, the versions used in the build will be the latest versions published on PyPI. It’s even possible to fail to build entirely because of backwards incompatible changes.
To fix the above issue I wrote
reproducible-wheel-builder1 a utility which combines
build to create reproducible environments for building wheels. Instead of relying on
pip to re-resolve the
build-system.requires for every build,
pex and a Pex lockfile are used to create a reproducible virtual environment for the build. Then
build uses that virtual environment to build the wheel. Since a lockfile is used, the build environment is reproducible.
It is distributed as a pex and can be downloaded from GitHub. Running it is as simple as executing
This can be used instead of
pip wheel to build a project.
gevent 21.1.2 does not have wheels for Python 3.10. Using
pip wheel on the sdist results in the following error.
This is because the
pyproject.toml has the following
Cython >= 3.0a6 requirement is problematic because Cython 3.0a8 has the following change:
Variables can no longer be declared with cpdef.
This can be overcome by using
reproducible-wheel-builder with a pex lockfile that pins
3.0a7. Generating a
pex lockfile for the dependencies can be done with the
pex.lock file and the
reproducible-wheel-builder a wheel can be built now.
The above command successfully runs and outputs a wheel under
PEP 517 and PEP 518 don’t specify a lockfile mechanism for specifying the build environment. This leads to non determinism and possibly failed builds when using
pip wheel. This can be overcome with a pex lockfile of the build environment and passing it to
reproducible-wheel-builder to do the build.
I am not good at naming things↩︎