EP 10 — Public API and Backwards Compatibility Policy

Motivation

Docutils has a large user base and is used in production at several places (Python documentation, Linux kernel documentation, CMake documentation, readthedocs, ...). OTOH, Docutils has a version number below 1.0 (widely seen as an indicator of "beta" status of a project).

The current Docutils Project Policies section on version identifcation concentrates on the formal definition of the version specifier but leaves open what consists a "major change in the design or API".

The current backwards compatibility policy is a stub referencing PEP 387.

Rationale

Clearly defining how we will balance evolution with stability is important to both users and project developers.

People affected by changes in Docutils include:

Authors

writing or maintaining reStructuredText documents.

End-Users

of Docutils native front-end tools (optionally with 3rd-party drop-in extensions) or alternative tools using Docutils either as a library (Sphinx, …) or via the command line interface (build systems, Makefiles, scripts in other languages).

Developers

i.e. authors and maintainers of

• projects that use Docutils as a library (Sphinx, rsted, Leo, Pelican, ebookmaker, MyST, readthedocs, rinohtype, …),

• drop-in components (pycmark, rst2pdf, rst2beamer, …),

• alternative front-end tools,

• custom stylesheets (CSS style sheets, LaTeX styles, ODT styles), or

• re-implementations of the reStructuredText specification, e.g. Pandoc or Text-Restructured (prest).

A person may belong to more than one of these catgories.

Specification

Docutils public APIs are:

• the reStructuredText specification,

• the Docutils document structure (Docutils Document Tree),

• names, command-line arguments and behaviour of the "console_scripts" entry points (Front-end Tools),

• the core docutils Python package API:

  • the Docutils Publisher interface for programmatic use,

  • component interfaces as defined by the abstract base classes docutils.reader.Reader, docutils.writer.Writer, and docutils.transform.Transform,

• behaviour and names of all documented Python objects, [1]

• output templates and style sheets provided with the writers,

• the interface to custom stylesheets -- elements, macros and classes used by writers to represent doctree nodes in the output format.

Exemptions:

Python objects, stylesheets and templates can explicitly "opt-out" of the public API with a docstring noting that the object is provisional or internal.

All undocumented objects should be assumed to be internal. [2]

See also the API Reference Material for Client-Developers.

Backwards Compatibility

Beginning with version 1.0, Docutils will follow the rules of Semantic Versioning. All incompatible changes to the public APIs require increasing the major part of the version specifier. Backwards compatible changes can be done in minor releases.

Security Implications

If required, critical bug fixes may change the public API without advance warning.

How to Teach This

• Move the API specification and the backwards compatibility declaration to the Docutils Project Policies.

• Complete the API documentation and keep it up to date.

• Generate "docutils" package API documentation from the docstrings:

  • Fix/enhance/add docstrings to improve the output of pydoc.

  • Generate API documentation with Sphinx:

    • nicely format rST docstrings

    • include attribute docstrings (ignored by pydoc).

• Put the following text at a suitable place in the documentation:

  To find out if an object from the "docutils" package is safe to use, look up its docstring and the docstring of its parent(s) [3].

  If there is no documentation or the documentation says "provisional" or "internal", the name, behaviour, and existence of the object is not guaranteed to be stable.

  Code relying on non-public objects should be made robust using public alternatives. If there is a no such alternative or the required change would be a problem, contact the Docutils developers or file a feature request.

Rejected Ideas

• Use type annotations as an indication of status in the public API.

  • There is no known precedence for this approach.

  • Type annotations may be helpful also for non-public code.

• Use Calendar Versioning (CalVer).

  • Would be a break from current versioning without clear advantages.

• Allow breaking API changes in minor versions after prior announcement and a deprecation period.

  • Breaks the principle of least surprise.

• Enumerate all modules, classes, and functions that form the public API.

• Mark all private objects with a prefix underscore.

  • May needlessly break applications that use "internal" objects by the current name.

  • Too much work.

Open Issues

• Differentiate between "core API" and "extended API"?

  Cf. the Docutils Project Policies

  When Docutils reaches version 1.0, the major APIs will be considered frozen.

  The major number [...] may be incremented later if there is a major change in the design or API.

• Formalise the wording for docstrings for public/private/provisional (ideally this would be a single regex pattern)?

  • The keyword provisional is well defined. ✓

  • Use "private" or "internal"?

• Declare only objects included in the __all__ attribute of their parent objects as public rsp. explicitely list all public objects in __all__ attribute of their parents?

  This would hide private objects from pydoc help on the parent objects.

• Define a minimum deprecation time similar to Docbook? E.g.

  • A "major" release may contain backward-incompatible changes if:

    • the change was announced in the release notes for the previous version (major or minor) and

    • the change was announced in a release that occurred at least six months previously.

  By these rules, Docutils developers can announce, in release 5.1, for example, its plans to make a backward-incompatible change in release 6.0. Then, in 6.0, if it’s been at least six months since 5.1 was released, they can make that change.

References

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.