How to use GitHub to host Python wheels
When using Python libraries that contain C extensions I have needed to build my own wheel instead of using an existing wheel on PyPI. The typical reason is to rebuild an existing release but with different flags to change the compiled C extension. This allows me to:
- Increase debug information to debug a crash.
- Have the wheel dynamically link against a system library instead of statically linking.
- Change compiler or linker flags to change optimization levels.
- Compile an older version of the wheel for a version of Python that was released after the library was released.
In a large enterprise environment it would be easy to upload the wheels to a custom PyPI mirror and configure it to prefer uploaded wheels over external wheels. However when working on a solo project or a hobby project something less cumbersome is required, especially when a single wheel needs to be rebuilt.
Instead of creating a custom PyPI mirror, it’s possible to host the wheels on a GitHub release and configure
pip to prefer those wheels over the wheels in PyPI. To make it happen there are two features needed to be combined:
- Leverage that wheels have a ‘build number’ and
setuptoolsallow setting the build number via
pipsupports combining using an index like PyPI with it’s own
--find-linksmechanism to supplement the index.
For this post I will be using
pyzstd as an example where I have rebuilt it with dynamic linking.
Setting a custom build number
To set a build number, the wheel needs to be built with the
bdist_wheel command. The
bdist_wheel command has a
--build-number flag which can be used to set the build number of the wheel.
For example with the
pyzstd wheel without the
--build-number set looks like so.
Where as setting the
1 when running
bdist_wheel will produce a name like so.
From the wheel specification the build number acts as a tie breaker.
Optional build number. Must start with a digit. Acts as a tie-breaker if two wheel file names are the same in all other respects (i.e. name, version, and other tags). Sort as an empty tuple if unspecified, else sort as a two-item tuple with the first item being the initial digits as an int, and the second item being the remainder of the tag as a str.
Adding this build number means the wheel with the build number will always be preferred over one without, which means it will be preferred over the wheels on PyPI. Further it’s better than changing the version number, since no code changes were made.
--find-links to discover the wheel
pip discover the wheels, assuming they are uploaded to a GitHub release page, is to pass the URL to the
--find-links flag. With
20.3.3 it’s possible to run the following command.
Note that in the current version of
pip, pip no longer prefers wheels found in
--find-links over PyPI due to this issue. To fix this the above command needs to have
Alternatively a newer version of
pip can be used in conjunction with
simpleindex doesn’t offer the library at all and pip is forced to pull the wheel from the
--find-links argument. This can be done by creating a
simpleindex configuration that looks like
simpleindex with the above configuration and pointing the latest
pip to it and passing
--find-links results in a successful install.
It’s possible to rebuild a Python library wheel with different flags as a different artifact without changing the version number by using the
--build-number flag of
bdist_wheel. After uploading the wheels to a GitHub release the wheels can be consumed easily with the
--find-links flag with