When using Python libraries that contain C extensions I have needed to build my own wheel instead of using an existing wheel on PyPI. The typical reason is to rebuild an existing release but with different flags to change the compiled C extension. This allows me to:

• Increase debug information to debug a crash.
• Have the wheel dynamically link against a system library instead of statically linking.
• Change compiler or linker flags to change optimization levels.
• Compile an older version of the wheel for a version of Python that was released after the library was released.

In a large enterprise environment it would be easy to upload the wheels to a custom PyPI mirror and configure it to prefer uploaded wheels over external wheels. However when working on a solo project or a hobby project something less cumbersome is required, especially when a single wheel needs to be rebuilt.

Instead of creating a custom PyPI mirror, it’s possible to host the wheels on a GitHub release and configure pip to prefer those wheels over the wheels in PyPI. To make it happen there are two features needed to be combined:

• Leverage that wheels have a ‘build number’ and setuptools allow setting the build number via bdist_wheel.
• pip supports combining using an index like PyPI with it’s own --find-links mechanism to supplement the index.

For this post I will be using pyzstd as an example where I have rebuilt it with dynamic linking.

Setting a custom build number

To set a build number, the wheel needs to be built with the bdist_wheel command. The bdist_wheel command has a --build-number flag which can be used to set the build number of the wheel.

For example with the pyzstd wheel without the --build-number set looks like so.

pyzstd-0.15.2-cp310-cp310-linux_x86_64.whl

Where as setting the --build-number to 1 when running bdist_wheel will produce a name like so.

pyzstd-0.15.2-1-cp310-cp310-linux_x86_64.whl

From the wheel specification the build number acts as a tie breaker.

Optional build number. Must start with a digit. Acts as a tie-breaker if two wheel file names are the same in all other respects (i.e. name, version, and other tags). Sort as an empty tuple if unspecified, else sort as a two-item tuple with the first item being the initial digits as an int, and the second item being the remainder of the tag as a str.

Adding this build number means the wheel with the build number will always be preferred over one without, which means it will be preferred over the wheels on PyPI. Further it’s better than changing the version number, since no code changes were made.

Using --find-links to discover the wheel

To have pip discover the wheels, assuming they are uploaded to a GitHub release page, is to pass the URL to the --find-links flag. With pip 20.3.3 it’s possible to run the following command.

$ pip install --no-cache --only-binary ':all:' --find-links 'https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7' 'pyzstd==0.15.2'

Looking in links: https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7
Collecting pyzstd==0.15.2
  Downloading https://github.com/zmanji/pyzstd-wheel-builder/releases/download/v0.0.7/pyzstd-0.15.2-1-cp310-cp310-linux_x86_64.whl (42 kB)
     |████████████████████████████████| 42 kB 5.1 MB/s
Installing collected packages: pyzstd
Successfully installed pyzstd-0.15.2

Note that in the current version of pip, pip no longer prefers wheels found in --find-links over PyPI due to this issue. To fix this the above command needs to have --no-index.

$ pip install --no-cache --only-binary ':all:' --find-links 'https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7' 'pyzstd==0.15.2' --no-index
Looking in links: https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7
Collecting pyzstd==0.15.2
  Downloading https://github.com/zmanji/pyzstd-wheel-builder/releases/download/v0.0.7/pyzstd-0.15.2-1-cp310-cp310-linux_x86_64.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.5/42.5 kB 5.1 MB/s eta 0:00:00
Installing collected packages: pyzstd
Successfully installed pyzstd-0.15.2

Alternatively a newer version of pip can be used in conjunction with simpleindex where simpleindex doesn’t offer the library at all and pip is forced to pull the wheel from the --find-links argument. This can be done by creating a simpleindex configuration that looks like

[routes."pyzstd"]
source = "path"
to = "/dev/null"

[routes."{project}"]
source = "http"
to = "https://pypi.org/simple/{project}/"

[server]
host = "127.0.0.1"
port = 8000

Running simpleindex with the above configuration and pointing the latest pip to it and passing --find-links results in a successful install.

$ pip install --no-cache --only-binary ':all:' -i 'http://127.0.0.1:8000' --find-links 'https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7' 'pyzstd==0.15.2'
Looking in indexes: http://127.0.0.1:8000
Looking in links: https://github.com/zmanji/pyzstd-wheel-builder/releases/tag/v0.0.7
Collecting pyzstd==0.15.2
  Downloading https://github.com/zmanji/pyzstd-wheel-builder/releases/download/v0.0.7/pyzstd-0.15.2-1-cp310-cp310-linux_x86_64.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.5/42.5 kB 4.3 MB/s eta 0:00:00
Installing collected packages: pyzstd
Successfully installed pyzstd-0.15.2

Conclusion

It’s possible to rebuild a Python library wheel with different flags as a different artifact without changing the version number by using the --build-number flag of bdist_wheel. After uploading the wheels to a GitHub release the wheels can be consumed easily with the --find-links flag with pip.