Packaging with setuptools

Tools for integrating with setuptools.

Integration with setuptools is a somewhat messy affair. By putting a cslug.CSlug in a package, we now have to coerce our package setup to do the following 5 things:

  1. (Re)build all cslug.CSlugs on setup.py build.

  2. Include C source code in sdists but exclude them from wheels.

  3. Include cslug type jsons and binariesNick-name for shared library. of the correct OS in wheels but exclude them from sdists.

  4. Mark cslug as a build-time dependency to be installed before trying to run setup.py build.

  5. Mark wheels as platform dependent but Python version independent.

Quick setup

For the short way to setup your package you may use the following recipe (note all steps are mandatory). Some of these steps are a little drastic so I recommend you make a backup, or a git commit, so you can easily fix any mess you make:

  • Copy this MANIFEST.in into the root of your project (or merge with yours if you already have one).

  • Similarly, copy/merge this pyproject.toml.

  • Add all your dependencies to the [build-system] requires list in the pyproject.toml (keep the current contents of that list). If you already specify your dependencies elsewhere remove them.

  • Copy the following lines of setup.py into your setup.py (replacing contains_slugs with the name of your package).

from cslug.building import (build_slugs, bdist_wheel, CSLUG_SUFFIX,
                            copy_requirements)

    install_requires=copy_requirements(),
    package_data={
        "contains_slugs": ["*" + CSLUG_SUFFIX, "*.json"],
    },
    cmdclass={
        "build": build_slugs("contains_slugs:deep_thought"),
        "bdist_wheel": bdist_wheel,
    },
)

  • Edit the "build": build_slugs("contains_slugs:deep_thought"), line in the setup.py, replace "contains_slugs:deep_thought" with the names of every cslug.CSlug in your package (see make() for the name format).

For the why and the how to customise it, see the rest of this page.

contains-slugs

Throughout this document we'll be using the contains_slugs package as an example. You can inspect its contents with the links listed below or download the full project here.

We'll start with the setup.py. Using a setuptools setup.py is currently the only supported way to use cslug. There is no way to use purely a setup.cfg or pyproject.toml or any of the alternatives to setuptools.

Apart from the cslug imports, the following lines are just generic setup.py and should already be familiar to you. (If it isn't, check out the Python packaging tutorial.)

from setuptools import setup, find_packages
from cslug.building import (build_slugs, bdist_wheel, CSLUG_SUFFIX,
                            copy_requirements)

setup(
    name="contains_slugs",
    version="1.0.0",
    author="Mr Slug",
    author_email="sluggy989@seaslugmail.com",
    url="www.toomanyslugs.com",
    packages=find_packages(),

Compile on setup.py build.

Whilst you rarely use it directly, setup.py build is called implicitly whenever you:

  • pip install . but not pip install -e .

  • pip wheel . but not python setup.py sdist.

  • pip install /path/to/source/distribution.tar.gz but not pip install /path/to/binary/distribution.whl.

To make package building as streamlined as possible, hook setup.py build to also call cslug.CSlug.make() on every cslug.CSlug. Calling cslug.building.make() is a convenient way to call make throughout a project but to attach it to setup.py build requires overloading the run() method of distutils.command.build.build. cslug.building.build_slugs() does exactly this. It creates a subclass of build, prepending make() to the run() method.

So finally, to create and register this custom build class, add the following option:

setup(
    ...
    cmdclass={
        "build": build_slugs("contains_slugs:deep_thought"),
    },
)

See make() for how the argument(s) to build_slugs should be formatted.

With this set up, you can now (re)compile all cslug components in a project with:

python setup.py build

Specify build-time dependencies

By putting a cslug import in our setup.py we've made cslug a build time dependency. And, because you need to import your project just to compile any cslug components, most or all of your run-time dependencies are now also build time dependencies.

The ability to specify build time dependencies is introduced by PEP 518. You require a pyproject.toml (a setup.cfg is not accepted).

If you don't like that your requirements are duplicated you may use copy_requirements() to extract your build time requirements and pass them to the install_requires option in your setup.py:

from cslug.building import copy_requirements

setup(
    ...
    install_requires=copy_requirements(),
    ...
)

If you have build time requirements that aren't runtime requirements you can exclude them with:

install_requires=copy_requirements(exclude=["build-time", "only", "packages"]),

setuptools, wheel and toml are excluded. If you want to include them or have other runtime-only dependencies then append them:

install_requires=copy_requirements() + ["toml", "some-other-library"],

Warning

pip builds packages in an isolated environment which ignores your currently installed packages. If you forget a build requirement in the toml file, but you will still a ModuleNotFoundError even if have it installed anyway.

Note

If you find the isolated build environment is maddeningly slow you can skip it in pip using the --no-build-isolation. But only once your sure it works without it.

When the build dependencies get noticed

Build dependencies, being new, has a few holes in it. They are ignored (usually resulting in ModuleNotFoundErrors) if you use:

python setup.py [bdist_wheel or install or anything else]

Using pip finds the build dependencies correctly so the solution is to the above is to:

pip install .

Or manually install the build dependencies before invoking any setup.py commands.

You also require modern versions of pip, setuptools and wheel. If you get ModuleNotFoundErrors for packages which are in your toml, trying upgrading those.

For sdists to work the pyproject.toml must be included in the sdist. See Data files for source distributions (sdists).

Data files for source distributions (sdists)

An sdist should include source code but exclude anything cslug-generated. It's also crucial that pyproject.toml is included or build time dependencies don't get propagated.

This all happens in the MANIFEST.in. You can just copy/paste this file directly into your project's root:

MANIFEST.in
recursive-include * *.c *.h
recursive-exclude * *.dll *.so *.dylib
recursive-exclude * *.json
include pyproject.toml

Data files for binary distributions (wheels)

A binary distribution is already compiled meaning that it doesn't need source code anymore. But it does need the compiled binariesNick-name for shared library. and type jsons.

We're back to the setup.py again, this time using the package_data argument.

from cslug.building import CSLUG_SUFFIX

setup(
    ...,
    include_package_data=False,
    package_data={
        "contains_slugs": ["*" + CSLUG_SUFFIX, "*.json"],
    },
)

Make sure that you do not use include_package_data=True. Using it causes all files to be collected, including source and junk files, rather than only those which are appropriate.

Note

The above filter would prevent binaries for the wrong platform from being collected but for some unfortunate caching. If you compile for multiple platforms which share a filesystem, run python setup.py clean between each switch. This problem is rectified in v0.4.0.

Platform specific wheel tag

Because a cslug package contains binariesNick-name for shared library. but those binaries do not use the Python ABI (i.e. don't include the line #include <Python.h> anywhere in the C code), a package is platform specific but independent of Python/ABI version. i.e. You can't compile on Linux and run on Windows but you can compile using Python 3.8 and run on Python 3.6.

We need to tell setuptools this, otherwise it incorrectly assumes that our packages are Pure Python which would eventually lead to pip installing binaries for the wrong operating system. Unfortunately setuptools is heavily geared towards Python extension modules exclusively and this is surprisingly fiddly. But cslug.building.bdist_wheel wraps it. Just add it as another command class in your setup.py:

from cslug.building import bdist_wheel

setup(
    ...
    cmdclass={
        "bdist_wheel": bdist_wheel,
    },
)

Now on running:

pip wheel --no-deps .

you should notice that the wheel produced has the name of your operating system in its filename.

Note

Building wheels requires the wheel package. If you get an error saying wheel isn't installed then just pip install wheel.