diff options
Diffstat (limited to 'Doc/library/annotationlib.rst')
| -rw-r--r-- | Doc/library/annotationlib.rst | 248 |
1 files changed, 222 insertions, 26 deletions
diff --git a/Doc/library/annotationlib.rst b/Doc/library/annotationlib.rst index 7946cd3a3ce..7dfc11449a6 100644 --- a/Doc/library/annotationlib.rst +++ b/Doc/library/annotationlib.rst @@ -40,7 +40,7 @@ The :func:`get_annotations` function is the main entry point for retrieving annotations. Given a function, class, or module, it returns an annotations dictionary in the requested format. This module also provides functionality for working directly with the :term:`annotate function` -that is used to evaluate annotations, such as :func:`get_annotate_function` +that is used to evaluate annotations, such as :func:`get_annotate_from_class_namespace` and :func:`call_annotate_function`, as well as the :func:`call_evaluate_function` function for working with :term:`evaluate functions <evaluate function>`. @@ -127,16 +127,27 @@ Classes Values are the result of evaluating the annotation expressions. - .. attribute:: FORWARDREF + .. attribute:: VALUE_WITH_FAKE_GLOBALS :value: 2 + Special value used to signal that an annotate function is being + evaluated in a special environment with fake globals. When passed this + value, annotate functions should either return the same value as for + the :attr:`Format.VALUE` format, or raise :exc:`NotImplementedError` + to signal that they do not support execution in this environment. + This format is only used internally and should not be passed to + the functions in this module. + + .. attribute:: FORWARDREF + :value: 3 + Values are real annotation values (as per :attr:`Format.VALUE` format) for defined values, and :class:`ForwardRef` proxies for undefined - values. Real objects may contain references to, :class:`ForwardRef` + values. Real objects may contain references to :class:`ForwardRef` proxy objects. .. attribute:: STRING - :value: 3 + :value: 4 Values are the text string of the annotation as it appears in the source code, up to modifications including, but not restricted to, @@ -144,17 +155,6 @@ Classes The exact values of these strings may change in future versions of Python. - .. attribute:: VALUE_WITH_FAKE_GLOBALS - :value: 4 - - Special value used to signal that an annotate function is being - evaluated in a special environment with fake globals. When passed this - value, annotate functions should either return the same value as for - the :attr:`Format.VALUE` format, or raise :exc:`NotImplementedError` - to signal that they do not support execution in this environment. - This format is only used internally and should not be passed to - the functions in this module. - .. versionadded:: 3.14 .. class:: ForwardRef @@ -172,14 +172,21 @@ Classes :class:`~ForwardRef`. The string may not be exactly equivalent to the original source. - .. method:: evaluate(*, owner=None, globals=None, locals=None, type_params=None) + .. method:: evaluate(*, owner=None, globals=None, locals=None, type_params=None, format=Format.VALUE) Evaluate the forward reference, returning its value. - This may throw an exception, such as :exc:`NameError`, if the forward + If the *format* argument is :attr:`~Format.VALUE` (the default), + this method may throw an exception, such as :exc:`NameError`, if the forward reference refers to a name that cannot be resolved. The arguments to this method can be used to provide bindings for names that would otherwise - be undefined. + be undefined. If the *format* argument is :attr:`~Format.FORWARDREF`, + the method will never throw an exception, but may return a :class:`~ForwardRef` + instance. For example, if the forward reference object contains the code + ``list[undefined]``, where ``undefined`` is a name that is not defined, + evaluating it with the :attr:`~Format.FORWARDREF` format will return + ``list[ForwardRef('undefined')]``. If the *format* argument is + :attr:`~Format.STRING`, the method will return :attr:`~ForwardRef.__forward_arg__`. The *owner* parameter provides the preferred mechanism for passing scope information to this method. The owner of a :class:`~ForwardRef` is the @@ -204,6 +211,10 @@ Classes means may not have any information about their scope, so passing arguments to this method may be necessary to evaluate them successfully. + If no *owner*, *globals*, *locals*, or *type_params* are provided and the + :class:`~ForwardRef` does not contain information about its origin, + empty globals and locals dictionaries are used. + .. versionadded:: 3.14 @@ -300,15 +311,13 @@ Functions .. versionadded:: 3.14 -.. function:: get_annotate_function(obj) +.. function:: get_annotate_from_class_namespace(namespace) - Retrieve the :term:`annotate function` for *obj*. Return :const:`!None` - if *obj* does not have an annotate function. *obj* may be a class, function, - module, or a namespace dictionary for a class. The last case is useful during - class creation, e.g. in the ``__new__`` method of a metaclass. - - This is usually equivalent to accessing the :attr:`~object.__annotate__` - attribute of *obj*, but access through this public function is preferred. + Retrieve the :term:`annotate function` from a class namespace dictionary *namespace*. + Return :const:`!None` if the namespace does not contain an annotate function. + This is primarily useful before the class has been fully created (e.g., in a metaclass); + after the class exists, the annotate function can be retrieved with ``cls.__annotate__``. + See :ref:`below <annotationlib-metaclass>` for an example using this function in a metaclass. .. versionadded:: 3.14 @@ -407,3 +416,190 @@ Functions .. versionadded:: 3.14 + +Recipes +------- + +.. _annotationlib-metaclass: + +Using annotations in a metaclass +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A :ref:`metaclass <metaclasses>` may want to inspect or even modify the annotations +in a class body during class creation. Doing so requires retrieving annotations +from the class namespace dictionary. For classes created with +``from __future__ import annotations``, the annotations will be in the ``__annotations__`` +key of the dictionary. For other classes with annotations, +:func:`get_annotate_from_class_namespace` can be used to get the +annotate function, and :func:`call_annotate_function` can be used to call it and +retrieve the annotations. Using the :attr:`~Format.FORWARDREF` format will usually +be best, because this allows the annotations to refer to names that cannot yet be +resolved when the class is created. + +To modify the annotations, it is best to create a wrapper annotate function +that calls the original annotate function, makes any necessary adjustments, and +returns the result. + +Below is an example of a metaclass that filters out all :class:`typing.ClassVar` +annotations from the class and puts them in a separate attribute: + +.. code-block:: python + + import annotationlib + import typing + + class ClassVarSeparator(type): + def __new__(mcls, name, bases, ns): + if "__annotations__" in ns: # from __future__ import annotations + annotations = ns["__annotations__"] + classvar_keys = { + key for key, value in annotations.items() + # Use string comparison for simplicity; a more robust solution + # could use annotationlib.ForwardRef.evaluate + if value.startswith("ClassVar") + } + classvars = {key: annotations[key] for key in classvar_keys} + ns["__annotations__"] = { + key: value for key, value in annotations.items() + if key not in classvar_keys + } + wrapped_annotate = None + elif annotate := annotationlib.get_annotate_from_class_namespace(ns): + annotations = annotationlib.call_annotate_function( + annotate, format=annotationlib.Format.FORWARDREF + ) + classvar_keys = { + key for key, value in annotations.items() + if typing.get_origin(value) is typing.ClassVar + } + classvars = {key: annotations[key] for key in classvar_keys} + + def wrapped_annotate(format): + annos = annotationlib.call_annotate_function(annotate, format, owner=typ) + return {key: value for key, value in annos.items() if key not in classvar_keys} + + else: # no annotations + classvars = {} + wrapped_annotate = None + typ = super().__new__(mcls, name, bases, ns) + + if wrapped_annotate is not None: + # Wrap the original __annotate__ with a wrapper that removes ClassVars + typ.__annotate__ = wrapped_annotate + typ.classvars = classvars # Store the ClassVars in a separate attribute + return typ + + +Limitations of the ``STRING`` format +------------------------------------ + +The :attr:`~Format.STRING` format is meant to approximate the source code +of the annotation, but the implementation strategy used means that it is not +always possible to recover the exact source code. + +First, the stringifier of course cannot recover any information that is not present in +the compiled code, including comments, whitespace, parenthesization, and operations that +get simplified by the compiler. + +Second, the stringifier can intercept almost all operations that involve names looked +up in some scope, but it cannot intercept operations that operate fully on constants. +As a corollary, this also means it is not safe to request the ``STRING`` format on +untrusted code: Python is powerful enough that it is possible to achieve arbitrary +code execution even with no access to any globals or builtins. For example: + +.. code-block:: pycon + + >>> def f(x: (1).__class__.__base__.__subclasses__()[-1].__init__.__builtins__["print"]("Hello world")): pass + ... + >>> annotationlib.get_annotations(f, format=annotationlib.Format.SOURCE) + Hello world + {'x': 'None'} + +.. note:: + This particular example works as of the time of writing, but it relies on + implementation details and is not guaranteed to work in the future. + +Among the different kinds of expressions that exist in Python, +as represented by the :mod:`ast` module, some expressions are supported, +meaning that the ``STRING`` format can generally recover the original source code; +others are unsupported, meaning that they may result in incorrect output or an error. + +The following are supported (sometimes with caveats): + +* :class:`ast.BinOp` +* :class:`ast.UnaryOp` + + * :class:`ast.Invert` (``~``), :class:`ast.UAdd` (``+``), and :class:`ast.USub` (``-``) are supported + * :class:`ast.Not` (``not``) is not supported + +* :class:`ast.Dict` (except when using ``**`` unpacking) +* :class:`ast.Set` +* :class:`ast.Compare` + + * :class:`ast.Eq` and :class:`ast.NotEq` are supported + * :class:`ast.Lt`, :class:`ast.LtE`, :class:`ast.Gt`, and :class:`ast.GtE` are supported, but the operand may be flipped + * :class:`ast.Is`, :class:`ast.IsNot`, :class:`ast.In`, and :class:`ast.NotIn` are not supported + +* :class:`ast.Call` (except when using ``**`` unpacking) +* :class:`ast.Constant` (though not the exact representation of the constant; for example, escape + sequences in strings are lost; hexadecimal numbers are converted to decimal) +* :class:`ast.Attribute` (assuming the value is not a constant) +* :class:`ast.Subscript` (assuming the value is not a constant) +* :class:`ast.Starred` (``*`` unpacking) +* :class:`ast.Name` +* :class:`ast.List` +* :class:`ast.Tuple` +* :class:`ast.Slice` + +The following are unsupported, but throw an informative error when encountered by the +stringifier: + +* :class:`ast.FormattedValue` (f-strings; error is not detected if conversion specifiers like ``!r`` + are used) +* :class:`ast.JoinedStr` (f-strings) + +The following are unsupported and result in incorrect output: + +* :class:`ast.BoolOp` (``and`` and ``or``) +* :class:`ast.IfExp` +* :class:`ast.Lambda` +* :class:`ast.ListComp` +* :class:`ast.SetComp` +* :class:`ast.DictComp` +* :class:`ast.GeneratorExp` + +The following are disallowed in annotation scopes and therefore not relevant: + +* :class:`ast.NamedExpr` (``:=``) +* :class:`ast.Await` +* :class:`ast.Yield` +* :class:`ast.YieldFrom` + + +Limitations of the ``FORWARDREF`` format +---------------------------------------- + +The :attr:`~Format.FORWARDREF` format aims to produce real values as much +as possible, with anything that cannot be resolved replaced with +:class:`ForwardRef` objects. It is affected by broadly the same Limitations +as the :attr:`~Format.STRING` format: annotations that perform operations on +literals or that use unsupported expression types may raise exceptions when +evaluated using the :attr:`~Format.FORWARDREF` format. + +Below are a few examples of the behavior with unsupported expressions: + +.. code-block:: pycon + + >>> from annotationlib import get_annotations, Format + >>> def zerodiv(x: 1 / 0): ... + >>> get_annotations(zerodiv, format=Format.STRING) + Traceback (most recent call last): + ... + ZeroDivisionError: division by zero + >>> get_annotations(zerodiv, format=Format.FORWARDREF) + Traceback (most recent call last): + ... + ZeroDivisionError: division by zero + >>> def ifexp(x: 1 if y else 0): ... + >>> get_annotations(ifexp, format=Format.STRING) + {'x': '1'} |