diff options
Diffstat (limited to 'Doc/c-api')
| -rw-r--r-- | Doc/c-api/allocation.rst | 132 | ||||
| -rw-r--r-- | Doc/c-api/code.rst | 2 | ||||
| -rw-r--r-- | Doc/c-api/function.rst | 2 | ||||
| -rw-r--r-- | Doc/c-api/gcsupport.rst | 61 | ||||
| -rw-r--r-- | Doc/c-api/init.rst | 229 | ||||
| -rw-r--r-- | Doc/c-api/init_config.rst | 2 | ||||
| -rw-r--r-- | Doc/c-api/intro.rst | 53 | ||||
| -rw-r--r-- | Doc/c-api/lifecycle.dot | 156 | ||||
| -rw-r--r-- | Doc/c-api/lifecycle.dot.css | 21 | ||||
| -rw-r--r-- | Doc/c-api/lifecycle.dot.pdf | bin | 0 -> 19328 bytes | |||
| -rw-r--r-- | Doc/c-api/lifecycle.dot.svg | 374 | ||||
| -rw-r--r-- | Doc/c-api/lifecycle.rst | 273 | ||||
| -rw-r--r-- | Doc/c-api/memory.rst | 18 | ||||
| -rw-r--r-- | Doc/c-api/module.rst | 42 | ||||
| -rw-r--r-- | Doc/c-api/objimpl.rst | 1 | ||||
| -rw-r--r-- | Doc/c-api/sys.rst | 51 | ||||
| -rw-r--r-- | Doc/c-api/type.rst | 25 | ||||
| -rw-r--r-- | Doc/c-api/typeobj.rst | 467 | ||||
| -rw-r--r-- | Doc/c-api/unicode.rst | 36 |
19 files changed, 1588 insertions, 357 deletions
diff --git a/Doc/c-api/allocation.rst b/Doc/c-api/allocation.rst index 7cbc99ad145..59d913a0462 100644 --- a/Doc/c-api/allocation.rst +++ b/Doc/c-api/allocation.rst @@ -16,7 +16,20 @@ Allocating Objects on the Heap Initialize a newly allocated object *op* with its type and initial reference. Returns the initialized object. Other fields of the object are - not affected. + not initialized. Despite its name, this function is unrelated to the + object's :meth:`~object.__init__` method (:c:member:`~PyTypeObject.tp_init` + slot). Specifically, this function does **not** call the object's + :meth:`!__init__` method. + + In general, consider this function to be a low-level routine. Use + :c:member:`~PyTypeObject.tp_alloc` where possible. + For implementing :c:member:`!tp_alloc` for your type, prefer + :c:func:`PyType_GenericAlloc` or :c:func:`PyObject_New`. + + .. note:: + + This function only initializes the object's memory corresponding to the + initial :c:type:`PyObject` structure. It does not zero the rest. .. c:function:: PyVarObject* PyObject_InitVar(PyVarObject *op, PyTypeObject *type, Py_ssize_t size) @@ -24,38 +37,107 @@ Allocating Objects on the Heap This does everything :c:func:`PyObject_Init` does, and also initializes the length information for a variable-size object. + .. note:: + + This function only initializes some of the object's memory. It does not + zero the rest. + .. c:macro:: PyObject_New(TYPE, typeobj) - Allocate a new Python object using the C structure type *TYPE* - and the Python type object *typeobj* (``PyTypeObject*``). - Fields not defined by the Python object header are not initialized. - The caller will own the only reference to the object - (i.e. its reference count will be one). - The size of the memory allocation is determined from the - :c:member:`~PyTypeObject.tp_basicsize` field of the type object. + Allocates a new Python object using the C structure type *TYPE* and the + Python type object *typeobj* (``PyTypeObject*``) by calling + :c:func:`PyObject_Malloc` to allocate memory and initializing it like + :c:func:`PyObject_Init`. The caller will own the only reference to the + object (i.e. its reference count will be one). + + Avoid calling this directly to allocate memory for an object; call the type's + :c:member:`~PyTypeObject.tp_alloc` slot instead. + + When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot, + :c:func:`PyType_GenericAlloc` is preferred over a custom function that + simply calls this macro. + + This macro does not call :c:member:`~PyTypeObject.tp_alloc`, + :c:member:`~PyTypeObject.tp_new` (:meth:`~object.__new__`), or + :c:member:`~PyTypeObject.tp_init` (:meth:`~object.__init__`). + + This cannot be used for objects with :c:macro:`Py_TPFLAGS_HAVE_GC` set in + :c:member:`~PyTypeObject.tp_flags`; use :c:macro:`PyObject_GC_New` instead. + + Memory allocated by this macro must be freed with :c:func:`PyObject_Free` + (usually called via the object's :c:member:`~PyTypeObject.tp_free` slot). + + .. note:: + + The returned memory is not guaranteed to have been completely zeroed + before it was initialized. + + .. note:: + + This macro does not construct a fully initialized object of the given + type; it merely allocates memory and prepares it for further + initialization by :c:member:`~PyTypeObject.tp_init`. To construct a + fully initialized object, call *typeobj* instead. For example:: - Note that this function is unsuitable if *typeobj* has - :c:macro:`Py_TPFLAGS_HAVE_GC` set. For such objects, - use :c:func:`PyObject_GC_New` instead. + PyObject *foo = PyObject_CallNoArgs((PyObject *)&PyFoo_Type); + + .. seealso:: + + * :c:func:`PyObject_Free` + * :c:macro:`PyObject_GC_New` + * :c:func:`PyType_GenericAlloc` + * :c:member:`~PyTypeObject.tp_alloc` .. c:macro:: PyObject_NewVar(TYPE, typeobj, size) - Allocate a new Python object using the C structure type *TYPE* and the - Python type object *typeobj* (``PyTypeObject*``). - Fields not defined by the Python object header - are not initialized. The allocated memory allows for the *TYPE* structure - plus *size* (``Py_ssize_t``) fields of the size - given by the :c:member:`~PyTypeObject.tp_itemsize` field of - *typeobj*. This is useful for implementing objects like tuples, which are - able to determine their size at construction time. Embedding the array of - fields into the same allocation decreases the number of allocations, - improving the memory management efficiency. + Like :c:macro:`PyObject_New` except: + + * It allocates enough memory for the *TYPE* structure plus *size* + (``Py_ssize_t``) fields of the size given by the + :c:member:`~PyTypeObject.tp_itemsize` field of *typeobj*. + * The memory is initialized like :c:func:`PyObject_InitVar`. + + This is useful for implementing objects like tuples, which are able to + determine their size at construction time. Embedding the array of fields + into the same allocation decreases the number of allocations, improving the + memory management efficiency. + + Avoid calling this directly to allocate memory for an object; call the type's + :c:member:`~PyTypeObject.tp_alloc` slot instead. + + When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot, + :c:func:`PyType_GenericAlloc` is preferred over a custom function that + simply calls this macro. + + This cannot be used for objects with :c:macro:`Py_TPFLAGS_HAVE_GC` set in + :c:member:`~PyTypeObject.tp_flags`; use :c:macro:`PyObject_GC_NewVar` + instead. + + Memory allocated by this function must be freed with :c:func:`PyObject_Free` + (usually called via the object's :c:member:`~PyTypeObject.tp_free` slot). + + .. note:: + + The returned memory is not guaranteed to have been completely zeroed + before it was initialized. + + .. note:: + + This macro does not construct a fully initialized object of the given + type; it merely allocates memory and prepares it for further + initialization by :c:member:`~PyTypeObject.tp_init`. To construct a + fully initialized object, call *typeobj* instead. For example:: + + PyObject *list_instance = PyObject_CallNoArgs((PyObject *)&PyList_Type); + + .. seealso:: - Note that this function is unsuitable if *typeobj* has - :c:macro:`Py_TPFLAGS_HAVE_GC` set. For such objects, - use :c:func:`PyObject_GC_NewVar` instead. + * :c:func:`PyObject_Free` + * :c:macro:`PyObject_GC_NewVar` + * :c:func:`PyType_GenericAlloc` + * :c:member:`~PyTypeObject.tp_alloc` .. c:function:: void PyObject_Del(void *op) @@ -71,6 +153,6 @@ Allocating Objects on the Heap .. seealso:: - :c:func:`PyModule_Create` + :ref:`moduleobjects` To allocate and create extension modules. diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst index 6eae24b38fa..42594f063b0 100644 --- a/Doc/c-api/code.rst +++ b/Doc/c-api/code.rst @@ -182,7 +182,7 @@ bound into a function. Type of a code object watcher callback function. If *event* is ``PY_CODE_EVENT_CREATE``, then the callback is invoked - after `co` has been fully initialized. Otherwise, the callback is invoked + after *co* has been fully initialized. Otherwise, the callback is invoked before the destruction of *co* takes place, so the prior state of *co* can be inspected. diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst index 58792edeed2..63b78f67767 100644 --- a/Doc/c-api/function.rst +++ b/Doc/c-api/function.rst @@ -169,7 +169,7 @@ There are a few functions specific to Python functions. unpredictable effects, including infinite recursion. If *event* is ``PyFunction_EVENT_CREATE``, then the callback is invoked - after `func` has been fully initialized. Otherwise, the callback is invoked + after *func* has been fully initialized. Otherwise, the callback is invoked before the modification to *func* takes place, so the prior state of *func* can be inspected. The runtime is permitted to optimize away the creation of function objects when possible. In such cases no event will be emitted. diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst index d1f0982b818..f6fa52b36c5 100644 --- a/Doc/c-api/gcsupport.rst +++ b/Doc/c-api/gcsupport.rst @@ -57,11 +57,49 @@ rules: Analogous to :c:macro:`PyObject_New` but for container objects with the :c:macro:`Py_TPFLAGS_HAVE_GC` flag set. + Do not call this directly to allocate memory for an object; call the type's + :c:member:`~PyTypeObject.tp_alloc` slot instead. + + When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot, + :c:func:`PyType_GenericAlloc` is preferred over a custom function that + simply calls this macro. + + Memory allocated by this macro must be freed with + :c:func:`PyObject_GC_Del` (usually called via the object's + :c:member:`~PyTypeObject.tp_free` slot). + + .. seealso:: + + * :c:func:`PyObject_GC_Del` + * :c:macro:`PyObject_New` + * :c:func:`PyType_GenericAlloc` + * :c:member:`~PyTypeObject.tp_alloc` + + .. c:macro:: PyObject_GC_NewVar(TYPE, typeobj, size) Analogous to :c:macro:`PyObject_NewVar` but for container objects with the :c:macro:`Py_TPFLAGS_HAVE_GC` flag set. + Do not call this directly to allocate memory for an object; call the type's + :c:member:`~PyTypeObject.tp_alloc` slot instead. + + When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot, + :c:func:`PyType_GenericAlloc` is preferred over a custom function that + simply calls this macro. + + Memory allocated by this macro must be freed with + :c:func:`PyObject_GC_Del` (usually called via the object's + :c:member:`~PyTypeObject.tp_free` slot). + + .. seealso:: + + * :c:func:`PyObject_GC_Del` + * :c:macro:`PyObject_NewVar` + * :c:func:`PyType_GenericAlloc` + * :c:member:`~PyTypeObject.tp_alloc` + + .. c:function:: PyObject* PyUnstable_Object_GC_NewWithExtraData(PyTypeObject *type, size_t extra_size) Analogous to :c:macro:`PyObject_GC_New` but allocates *extra_size* @@ -73,6 +111,10 @@ rules: The extra data will be deallocated with the object, but otherwise it is not managed by Python. + Memory allocated by this function must be freed with + :c:func:`PyObject_GC_Del` (usually called via the object's + :c:member:`~PyTypeObject.tp_free` slot). + .. warning:: The function is marked as unstable because the final mechanism for reserving extra data after an instance is not yet decided. @@ -136,6 +178,21 @@ rules: Releases memory allocated to an object using :c:macro:`PyObject_GC_New` or :c:macro:`PyObject_GC_NewVar`. + Do not call this directly to free an object's memory; call the type's + :c:member:`~PyTypeObject.tp_free` slot instead. + + Do not use this for memory allocated by :c:macro:`PyObject_New`, + :c:macro:`PyObject_NewVar`, or related allocation functions; use + :c:func:`PyObject_Free` instead. + + .. seealso:: + + * :c:func:`PyObject_Free` is the non-GC equivalent of this function. + * :c:macro:`PyObject_GC_New` + * :c:macro:`PyObject_GC_NewVar` + * :c:func:`PyType_GenericAlloc` + * :c:member:`~PyTypeObject.tp_free` + .. c:function:: void PyObject_GC_UnTrack(void *op) @@ -180,9 +237,9 @@ provided. In order to use this macro, the :c:member:`~PyTypeObject.tp_traverse` must name its arguments exactly *visit* and *arg*: -.. c:function:: void Py_VISIT(PyObject *o) +.. c:macro:: Py_VISIT(o) - If *o* is not ``NULL``, call the *visit* callback, with arguments *o* + If the :c:expr:`PyObject *` *o* is not ``NULL``, call the *visit* callback, with arguments *o* and *arg*. If *visit* returns a non-zero value, then return it. Using this macro, :c:member:`~PyTypeObject.tp_traverse` handlers look like:: diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 52f64a61006..9c866438b48 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -77,10 +77,7 @@ The following functions can be safely called before Python is initialized: Despite their apparent similarity to some of the functions listed above, the following functions **should not be called** before the interpreter has - been initialized: :c:func:`Py_EncodeLocale`, :c:func:`Py_GetPath`, - :c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, - :c:func:`Py_GetProgramFullPath`, :c:func:`Py_GetPythonHome`, - :c:func:`Py_GetProgramName`, :c:func:`PyEval_InitThreads`, and + been initialized: :c:func:`Py_EncodeLocale`, :c:func:`PyEval_InitThreads`, and :c:func:`Py_RunMain`. @@ -145,9 +142,6 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. :c:member:`PyConfig.pathconfig_warnings` should be used instead, see :ref:`Python Initialization Configuration <init-config>`. - Suppress error messages when calculating the module search path in - :c:func:`Py_GetPath`. - Private flag used by ``_freeze_module`` and ``frozenmain`` programs. .. deprecated-removed:: 3.12 3.15 @@ -203,7 +197,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-i` option. - .. deprecated:: 3.12 + .. deprecated-removed:: 3.12 3.15 .. c:var:: int Py_IsolatedFlag @@ -586,7 +580,6 @@ Process-wide parameters .. index:: single: Py_Initialize() single: main() - single: Py_GetPath() This API is kept for backward compatibility: setting :c:member:`PyConfig.program_name` should be used instead, see :ref:`Python @@ -596,7 +589,7 @@ Process-wide parameters the first time, if it is called at all. It tells the interpreter the value of the ``argv[0]`` argument to the :c:func:`main` function of the program (converted to wide characters). - This is used by :c:func:`Py_GetPath` and some other functions below to find + This is used by some other functions below to find the Python run-time libraries relative to the interpreter executable. The default value is ``'python'``. The argument should point to a zero-terminated wide character string in static storage whose contents will not @@ -609,146 +602,6 @@ Process-wide parameters .. deprecated-removed:: 3.11 3.15 -.. c:function:: wchar_t* Py_GetProgramName() - - Return the program name set with :c:member:`PyConfig.program_name`, or the default. - The returned string points into static storage; the caller should not modify its - value. - - This function should not be called before :c:func:`Py_Initialize`, otherwise - it returns ``NULL``. - - .. versionchanged:: 3.10 - It now returns ``NULL`` if called before :c:func:`Py_Initialize`. - - .. deprecated-removed:: 3.13 3.15 - Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>` - (:data:`sys.executable`) instead. - - -.. c:function:: wchar_t* Py_GetPrefix() - - Return the *prefix* for installed platform-independent files. This is derived - through a number of complicated rules from the program name set with - :c:member:`PyConfig.program_name` and some environment variables; for example, if the - program name is ``'/usr/local/bin/python'``, the prefix is ``'/usr/local'``. The - returned string points into static storage; the caller should not modify its - value. This corresponds to the :makevar:`prefix` variable in the top-level - :file:`Makefile` and the :option:`--prefix` argument to the :program:`configure` - script at build time. The value is available to Python code as ``sys.base_prefix``. - It is only useful on Unix. See also the next function. - - This function should not be called before :c:func:`Py_Initialize`, otherwise - it returns ``NULL``. - - .. versionchanged:: 3.10 - It now returns ``NULL`` if called before :c:func:`Py_Initialize`. - - .. deprecated-removed:: 3.13 3.15 - Use :c:func:`PyConfig_Get("base_prefix") <PyConfig_Get>` - (:data:`sys.base_prefix`) instead. Use :c:func:`PyConfig_Get("prefix") - <PyConfig_Get>` (:data:`sys.prefix`) if :ref:`virtual environments - <venv-def>` need to be handled. - - -.. c:function:: wchar_t* Py_GetExecPrefix() - - Return the *exec-prefix* for installed platform-*dependent* files. This is - derived through a number of complicated rules from the program name set with - :c:member:`PyConfig.program_name` and some environment variables; for example, if the - program name is ``'/usr/local/bin/python'``, the exec-prefix is - ``'/usr/local'``. The returned string points into static storage; the caller - should not modify its value. This corresponds to the :makevar:`exec_prefix` - variable in the top-level :file:`Makefile` and the ``--exec-prefix`` - argument to the :program:`configure` script at build time. The value is - available to Python code as ``sys.base_exec_prefix``. It is only useful on - Unix. - - Background: The exec-prefix differs from the prefix when platform dependent - files (such as executables and shared libraries) are installed in a different - directory tree. In a typical installation, platform dependent files may be - installed in the :file:`/usr/local/plat` subtree while platform independent may - be installed in :file:`/usr/local`. - - Generally speaking, a platform is a combination of hardware and software - families, e.g. Sparc machines running the Solaris 2.x operating system are - considered the same platform, but Intel machines running Solaris 2.x are another - platform, and Intel machines running Linux are yet another platform. Different - major revisions of the same operating system generally also form different - platforms. Non-Unix operating systems are a different story; the installation - strategies on those systems are so different that the prefix and exec-prefix are - meaningless, and set to the empty string. Note that compiled Python bytecode - files are platform independent (but not independent from the Python version by - which they were compiled!). - - System administrators will know how to configure the :program:`mount` or - :program:`automount` programs to share :file:`/usr/local` between platforms - while having :file:`/usr/local/plat` be a different filesystem for each - platform. - - This function should not be called before :c:func:`Py_Initialize`, otherwise - it returns ``NULL``. - - .. versionchanged:: 3.10 - It now returns ``NULL`` if called before :c:func:`Py_Initialize`. - - .. deprecated-removed:: 3.13 3.15 - Use :c:func:`PyConfig_Get("base_exec_prefix") <PyConfig_Get>` - (:data:`sys.base_exec_prefix`) instead. Use - :c:func:`PyConfig_Get("exec_prefix") <PyConfig_Get>` - (:data:`sys.exec_prefix`) if :ref:`virtual environments <venv-def>` need - to be handled. - -.. c:function:: wchar_t* Py_GetProgramFullPath() - - .. index:: - single: executable (in module sys) - - Return the full program name of the Python executable; this is computed as a - side-effect of deriving the default module search path from the program name - (set by :c:member:`PyConfig.program_name`). The returned string points into - static storage; the caller should not modify its value. The value is available - to Python code as ``sys.executable``. - - This function should not be called before :c:func:`Py_Initialize`, otherwise - it returns ``NULL``. - - .. versionchanged:: 3.10 - It now returns ``NULL`` if called before :c:func:`Py_Initialize`. - - .. deprecated-removed:: 3.13 3.15 - Use :c:func:`PyConfig_Get("executable") <PyConfig_Get>` - (:data:`sys.executable`) instead. - - -.. c:function:: wchar_t* Py_GetPath() - - .. index:: - triple: module; search; path - single: path (in module sys) - - Return the default module search path; this is computed from the program name - (set by :c:member:`PyConfig.program_name`) and some environment variables. - The returned string consists of a series of directory names separated by a - platform dependent delimiter character. The delimiter character is ``':'`` - on Unix and macOS, ``';'`` on Windows. The returned string points into - static storage; the caller should not modify its value. The list - :data:`sys.path` is initialized with this value on interpreter startup; it - can be (and usually is) modified later to change the search path for loading - modules. - - This function should not be called before :c:func:`Py_Initialize`, otherwise - it returns ``NULL``. - - .. XXX should give the exact rules - - .. versionchanged:: 3.10 - It now returns ``NULL`` if called before :c:func:`Py_Initialize`. - - .. deprecated-removed:: 3.13 3.15 - Use :c:func:`PyConfig_Get("module_search_paths") <PyConfig_Get>` - (:data:`sys.path`) instead. - .. c:function:: const char* Py_GetVersion() Return the version of this Python interpreter. This is a string that looks @@ -919,23 +772,6 @@ Process-wide parameters .. deprecated-removed:: 3.11 3.15 -.. c:function:: wchar_t* Py_GetPythonHome() - - Return the default "home", that is, the value set by - :c:member:`PyConfig.home`, or the value of the :envvar:`PYTHONHOME` - environment variable if it is set. - - This function should not be called before :c:func:`Py_Initialize`, otherwise - it returns ``NULL``. - - .. versionchanged:: 3.10 - It now returns ``NULL`` if called before :c:func:`Py_Initialize`. - - .. deprecated-removed:: 3.13 3.15 - Use :c:func:`PyConfig_Get("home") <PyConfig_Get>` or the - :envvar:`PYTHONHOME` environment variable instead. - - .. _threads: Thread State and the Global Interpreter Lock @@ -1083,8 +919,36 @@ Note that the ``PyGILState_*`` functions assume there is only one global interpreter (created automatically by :c:func:`Py_Initialize`). Python supports the creation of additional interpreters (using :c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the -``PyGILState_*`` API is unsupported. +``PyGILState_*`` API is unsupported. This is because :c:func:`PyGILState_Ensure` +and similar functions default to :term:`attaching <attached thread state>` a +:term:`thread state` for the main interpreter, meaning that the thread can't safely +interact with the calling subinterpreter. + +Supporting subinterpreters in non-Python threads +------------------------------------------------ + +If you would like to support subinterpreters with non-Python created threads, you +must use the ``PyThreadState_*`` API instead of the traditional ``PyGILState_*`` +API. + +In particular, you must store the interpreter state from the calling +function and pass it to :c:func:`PyThreadState_New`, which will ensure that +the :term:`thread state` is targeting the correct interpreter:: + + /* The return value of PyInterpreterState_Get() from the + function that created this thread. */ + PyInterpreterState *interp = ThreadData->interp; + PyThreadState *tstate = PyThreadState_New(interp); + PyThreadState_Swap(tstate); + + /* GIL of the subinterpreter is now held. + Perform Python actions here. */ + result = CallSomeFunction(); + /* evaluate result or handle exception */ + /* Destroy the thread state. No Python API allowed beyond this point. */ + PyThreadState_Clear(tstate); + PyThreadState_DeleteCurrent(); .. _fork-and-threads: @@ -1261,6 +1125,10 @@ code, or when embedding the Python interpreter: .. seealso: :c:func:`PyEval_ReleaseThread` + .. note:: + Similar to :c:func:`PyGILState_Ensure`, this function will hang the + thread if the runtime is finalizing. + The following functions use thread-local storage, and are not compatible with sub-interpreters: @@ -1287,10 +1155,10 @@ with sub-interpreters: When the function returns, there will be an :term:`attached thread state` and the thread will be able to call arbitrary Python code. Failure is a fatal error. - .. note:: - Calling this function from a thread when the runtime is finalizing will - hang the thread until the program exits, even if the thread was not - created by Python. Refer to + .. warning:: + Calling this function when the runtime is finalizing is unsafe. Doing + so will either hang the thread until the program ends, or fully crash + the interpreter in rare cases. Refer to :ref:`cautions-regarding-runtime-finalization` for more details. .. versionchanged:: 3.14 @@ -1307,7 +1175,6 @@ with sub-interpreters: Every call to :c:func:`PyGILState_Ensure` must be matched by a call to :c:func:`PyGILState_Release` on the same thread. - .. c:function:: PyThreadState* PyGILState_GetThisThreadState() Get the :term:`attached thread state` for this thread. May return ``NULL`` if no @@ -1315,20 +1182,30 @@ with sub-interpreters: always has such a thread-state, even if no auto-thread-state call has been made on the main thread. This is mainly a helper/diagnostic function. - .. seealso: :c:func:`PyThreadState_Get`` + .. note:: + This function does not account for :term:`thread states <thread state>` created + by something other than :c:func:`PyGILState_Ensure` (such as :c:func:`PyThreadState_New`). + Prefer :c:func:`PyThreadState_Get` or :c:func:`PyThreadState_GetUnchecked` + for most cases. + .. seealso: :c:func:`PyThreadState_Get`` .. c:function:: int PyGILState_Check() Return ``1`` if the current thread is holding the :term:`GIL` and ``0`` otherwise. This function can be called from any thread at any time. - Only if it has had its Python thread state initialized and currently is - holding the :term:`GIL` will it return ``1``. + Only if it has had its :term:`thread state <attached thread state>` initialized + via :c:func:`PyGILState_Ensure` will it return ``1``. This is mainly a helper/diagnostic function. It can be useful for example in callback contexts or memory allocation functions when knowing that the :term:`GIL` is locked can allow the caller to perform sensitive actions or otherwise behave differently. + .. note:: + If the current Python process has ever created a subinterpreter, this + function will *always* return ``1``. Prefer :c:func:`PyThreadState_GetUnchecked` + for most cases. + .. versionadded:: 3.4 diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index e1931655618..4fd10224262 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -2111,7 +2111,7 @@ initialization:: /* Specify sys.path explicitly */ /* If you want to modify the default set of paths, finish - initialization first and then use PySys_GetObject("path") */ + initialization first and then use PySys_GetAttrString("path") */ config.module_search_paths_set = 1; status = PyWideStringList_Append(&config.module_search_paths, L"/path/to/stdlib"); diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst index 2bad0bab224..41856922110 100644 --- a/Doc/c-api/intro.rst +++ b/Doc/c-api/intro.rst @@ -127,7 +127,7 @@ complete listing. item defined in the module file. Example:: static struct PyModuleDef spam_module = { - PyModuleDef_HEAD_INIT, + .m_base = PyModuleDef_HEAD_INIT, .m_name = "spam", ... }; @@ -135,7 +135,7 @@ complete listing. PyMODINIT_FUNC PyInit_spam(void) { - return PyModule_Create(&spam_module); + return PyModuleDef_Init(&spam_module); } @@ -779,20 +779,11 @@ found along :envvar:`PATH`.) The user can override this behavior by setting the environment variable :envvar:`PYTHONHOME`, or insert additional directories in front of the standard path by setting :envvar:`PYTHONPATH`. -.. index:: - single: Py_GetPath (C function) - single: Py_GetPrefix (C function) - single: Py_GetExecPrefix (C function) - single: Py_GetProgramFullPath (C function) - The embedding application can steer the search by setting :c:member:`PyConfig.program_name` *before* calling :c:func:`Py_InitializeFromConfig`. Note that :envvar:`PYTHONHOME` still overrides this and :envvar:`PYTHONPATH` is still -inserted in front of the standard path. An application that requires total -control has to provide its own implementation of :c:func:`Py_GetPath`, -:c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, and -:c:func:`Py_GetProgramFullPath` (all defined in :file:`Modules/getpath.c`). +inserted in front of the standard path. .. index:: single: Py_IsInitialized (C function) @@ -847,3 +838,41 @@ after every statement run by the interpreter.) Please refer to :file:`Misc/SpecialBuilds.txt` in the Python source distribution for more detailed information. + + +.. _c-api-tools: + +Recommended third party tools +============================= + +The following third party tools offer both simpler and more sophisticated +approaches to creating C, C++ and Rust extensions for Python: + +* `Cython <https://cython.org/>`_ +* `cffi <https://cffi.readthedocs.io>`_ +* `HPy <https://hpyproject.org/>`_ +* `nanobind <https://github.com/wjakob/nanobind>`_ (C++) +* `Numba <https://numba.pydata.org/>`_ +* `pybind11 <https://pybind11.readthedocs.io/>`_ (C++) +* `PyO3 <https://pyo3.rs/>`_ (Rust) +* `SWIG <https://www.swig.org>`_ + +Using tools such as these can help avoid writing code that is tightly bound to +a particular version of CPython, avoid reference counting errors, and focus +more on your own code than on using the CPython API. In general, new versions +of Python can be supported by updating the tool, and your code will often use +newer and more efficient APIs automatically. Some tools also support compiling +for other implementations of Python from a single set of sources. + +These projects are not supported by the same people who maintain Python, and +issues need to be raised with the projects directly. Remember to check that the +project is still maintained and supported, as the list above may become +outdated. + +.. seealso:: + + `Python Packaging User Guide: Binary Extensions <https://packaging.python.org/guides/packaging-binary-extensions/>`_ + The Python Packaging User Guide not only covers several available + tools that simplify the creation of binary extensions, but also + discusses the various reasons why creating an extension module may be + desirable in the first place. diff --git a/Doc/c-api/lifecycle.dot b/Doc/c-api/lifecycle.dot new file mode 100644 index 00000000000..dca9f87e9e0 --- /dev/null +++ b/Doc/c-api/lifecycle.dot @@ -0,0 +1,156 @@ +digraph "Life Events" { + graph [ + fontnames="svg" + fontsize=12.0 + id="life_events_graph" + layout="dot" + margin="0,0" + ranksep=0.25 + stylesheet="lifecycle.dot.css" + ] + node [ + fontname="Courier" + fontsize=12.0 + ] + edge [ + fontname="Times-Italic" + fontsize=12.0 + ] + + "start" [fontname="Times-Italic" shape=plain label=< start > style=invis] + { + rank="same" + "tp_new" [href="typeobj.html#c.PyTypeObject.tp_new" target="_top"] + "tp_alloc" [href="typeobj.html#c.PyTypeObject.tp_alloc" target="_top"] + } + "tp_init" [href="typeobj.html#c.PyTypeObject.tp_init" target="_top"] + "reachable" [fontname="Times-Italic" shape=box] + "tp_traverse" [ + href="typeobj.html#c.PyTypeObject.tp_traverse" + ordering="in" + target="_top" + ] + "finalized?" [ + fontname="Times-Italic" + label=<marked as<br/>finalized?> + ordering="in" + shape=diamond + tooltip="marked as finalized?" + ] + "tp_finalize" [ + href="typeobj.html#c.PyTypeObject.tp_finalize" + ordering="in" + target="_top" + ] + "tp_clear" [href="typeobj.html#c.PyTypeObject.tp_clear" target="_top"] + "uncollectable" [ + fontname="Times-Italic" + label=<uncollectable<br/>(leaked)> + shape=box + tooltip="uncollectable (leaked)" + ] + "tp_dealloc" [ + href="typeobj.html#c.PyTypeObject.tp_dealloc" + ordering="in" + target="_top" + ] + "tp_free" [href="typeobj.html#c.PyTypeObject.tp_free" target="_top"] + + "start" -> "tp_new" [ + label=< type call > + ] + "tp_new" -> "tp_alloc" [ + label=< direct call > arrowhead=empty + labeltooltip="tp_new to tp_alloc: direct call" + tooltip="tp_new to tp_alloc: direct call" + ] + "tp_new" -> "tp_init" [tooltip="tp_new to tp_init"] + "tp_init" -> "reachable" [tooltip="tp_init to reachable"] + "reachable" -> "tp_traverse" [ + dir="back" + label=< not in a <br/> cyclic <br/> isolate > + labeltooltip="tp_traverse to reachable: not in a cyclic isolate" + tooltip="tp_traverse to reachable: not in a cyclic isolate" + ] + "reachable" -> "tp_traverse" [ + label=< periodic <br/> cyclic isolate <br/> detection > + labeltooltip="reachable to tp_traverse: periodic cyclic isolate detection" + tooltip="reachable to tp_traverse: periodic cyclic isolate detection" + ] + "reachable" -> "tp_init" [tooltip="reachable to tp_init"] + "reachable" -> "tp_finalize" [ + dir="back" + label=< resurrected <br/> (maybe remove <br/> finalized mark) > + labeltooltip="tp_finalize to reachable: resurrected (maybe remove finalized mark)" + tooltip="tp_finalize to reachable: resurrected (maybe remove finalized mark)" + ] + "tp_traverse" -> "finalized?" [ + label=< cyclic <br/> isolate > + labeltooltip="tp_traverse to finalized?: cyclic isolate" + tooltip="tp_traverse to finalized?: cyclic isolate" + ] + "reachable" -> "finalized?" [ + label=< no refs > + labeltooltip="reachable to finalized?: no refs" + tooltip="reachable to finalized?: no refs" + ] + "finalized?" -> "tp_finalize" [ + label=< no (mark <br/> as finalized) > + labeltooltip="finalized? to tp_finalize: no (mark as finalized)" + tooltip="finalized? to tp_finalize: no (mark as finalized)" + ] + "finalized?" -> "tp_clear" [ + label=< yes > + labeltooltip="finalized? to tp_clear: yes" + tooltip="finalized? to tp_clear: yes" + ] + "tp_finalize" -> "tp_clear" [ + label=< no refs or <br/> cyclic isolate > + labeltooltip="tp_finalize to tp_clear: no refs or cyclic isolate" + tooltip="tp_finalize to tp_clear: no refs or cyclic isolate" + ] + "tp_finalize" -> "tp_dealloc" [ + arrowtail=empty + dir="back" + href="lifecycle.html#c.PyObject_CallFinalizerFromDealloc" + style=dashed + label=< recommended<br/> call (see<br/> explanation)> + labeltooltip="tp_dealloc to tp_finalize: recommended call (see explanation)" + target="_top" + tooltip="tp_dealloc to tp_finalize: recommended call (see explanation)" + ] + "tp_finalize" -> "tp_dealloc" [ + label=< no refs > + labeltooltip="tp_finalize to tp_dealloc: no refs" + tooltip="tp_finalize to tp_dealloc: no refs" + ] + "tp_clear" -> "tp_dealloc" [ + label=< no refs > + labeltooltip="tp_clear to tp_dealloc: no refs" + tooltip="tp_clear to tp_dealloc: no refs" + ] + "tp_clear" -> "uncollectable" [ + label=< cyclic <br/> isolate > + labeltooltip="tp_clear to uncollectable: cyclic isolate" + tooltip="tp_clear to uncollectable: cyclic isolate" + ] + "uncollectable" -> "tp_dealloc" [ + style=invis + tooltip="uncollectable to tp_dealloc" + ] + "reachable" -> "uncollectable" [ + label=< cyclic <br/> isolate <br/> (no GC <br/> support) > + labeltooltip="reachable to uncollectable: cyclic isolate (no GC support)" + tooltip="reachable to uncollectable: cyclic isolate (no GC support)" + ] + "reachable" -> "tp_dealloc" [ + label=< no refs> + labeltooltip="reachable to tp_dealloc: no refs" + ] + "tp_dealloc" -> "tp_free" [ + arrowhead=empty + label=< direct call > + labeltooltip="tp_dealloc to tp_free: direct call" + tooltip="tp_dealloc to tp_free: direct call" + ] +} diff --git a/Doc/c-api/lifecycle.dot.css b/Doc/c-api/lifecycle.dot.css new file mode 100644 index 00000000000..3abf95b74da --- /dev/null +++ b/Doc/c-api/lifecycle.dot.css @@ -0,0 +1,21 @@ +#life_events_graph { + --svg-fgcolor: currentcolor; + --svg-bgcolor: transparent; +} +#life_events_graph a { + color: inherit; +} +#life_events_graph [stroke="black"] { + stroke: var(--svg-fgcolor); +} +#life_events_graph text, +#life_events_graph [fill="black"] { + fill: var(--svg-fgcolor); +} +#life_events_graph [fill="white"] { + fill: var(--svg-bgcolor); +} +#life_events_graph [fill="none"] { + /* On links, setting fill will make the entire shape clickable */ + fill: var(--svg-bgcolor); +} diff --git a/Doc/c-api/lifecycle.dot.pdf b/Doc/c-api/lifecycle.dot.pdf Binary files differnew file mode 100644 index 00000000000..ed5b5039c83 --- /dev/null +++ b/Doc/c-api/lifecycle.dot.pdf diff --git a/Doc/c-api/lifecycle.dot.svg b/Doc/c-api/lifecycle.dot.svg new file mode 100644 index 00000000000..7ace27dfcba --- /dev/null +++ b/Doc/c-api/lifecycle.dot.svg @@ -0,0 +1,374 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<?xml-stylesheet href="lifecycle.dot.css" type="text/css"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" + "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by graphviz version 12.2.0 (0) + --> +<!-- Title: Life Events Pages: 1 --> +<svg width="465pt" height="845pt" + viewBox="0.00 0.00 465.30 845.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> +<g id="life_events_graph" class="graph" transform="scale(1 1) rotate(0) translate(4 841)"> +<title>Life Events</title> +<polygon fill="white" stroke="none" points="-4,4 -4,-841 461.3,-841 461.3,4 -4,4"/> +<!-- start --> +<!-- tp_new --> +<g id="life_events_graph_node2" class="node"> +<title>tp_new</title> +<g id="a_life_events_graph_node2"><a xlink:href="typeobj.html#c.PyTypeObject.tp_new" xlink:title="tp_new" target="_top"> +<ellipse fill="none" stroke="black" cx="192.8" cy="-772.5" rx="38.8" ry="18"/> +<text text-anchor="middle" x="192.8" y="-768.23" font-family="monospace,monospace" font-size="12.00">tp_new</text> +</a> +</g> +</g> +<!-- start->tp_new --> +<g id="life_events_graph_edge1" class="edge"> +<title>start->tp_new</title> +<g id="a_life_events_graph_edge1"><a xlink:title="start to tp_new: type call"> +<path fill="none" stroke="black" d="M192.8,-822.95C192.8,-817.85 192.8,-810.09 192.8,-802.22"/> +<polygon fill="black" stroke="black" points="196.3,-802.42 192.8,-792.42 189.3,-802.42 196.3,-802.42"/> +</a> +</g> +<g id="a_life_events_graph_edge1-label"><a xlink:title="start to tp_new: type call"> +<text text-anchor="start" x="192.8" y="-802.35" font-family="serif,serif" font-style="italic" font-size="12.00">    type call  </text> +</a> +</g> +</g> +<!-- tp_alloc --> +<g id="life_events_graph_node3" class="node"> +<title>tp_alloc</title> +<g id="a_life_events_graph_node3"><a xlink:href="typeobj.html#c.PyTypeObject.tp_alloc" xlink:title="tp_alloc" target="_top"> +<ellipse fill="none" stroke="black" cx="373.8" cy="-772.5" rx="48.34" ry="18"/> +<text text-anchor="middle" x="373.8" y="-768.23" font-family="monospace,monospace" font-size="12.00">tp_alloc</text> +</a> +</g> +</g> +<!-- tp_new->tp_alloc --> +<g id="life_events_graph_edge2" class="edge"> +<title>tp_new->tp_alloc</title> +<g id="a_life_events_graph_edge2"><a xlink:title="tp_new to tp_alloc: direct call"> +<path fill="none" stroke="black" d="M232.07,-772.5C256,-772.5 287.05,-772.5 313.98,-772.5"/> +<polygon fill="none" stroke="black" points="313.73,-776 323.73,-772.5 313.73,-769 313.73,-776"/> +</a> +</g> +<g id="a_life_events_graph_edge2-label"><a xlink:title="tp_new to tp_alloc: direct call"> +<text text-anchor="start" x="240.65" y="-778.35" font-family="serif,serif" font-style="italic" font-size="12.00">  direct call  </text> +</a> +</g> +</g> +<!-- tp_init --> +<g id="life_events_graph_node4" class="node"> +<title>tp_init</title> +<g id="a_life_events_graph_node4"><a xlink:href="typeobj.html#c.PyTypeObject.tp_init" xlink:title="tp_init" target="_top"> +<ellipse fill="none" stroke="black" cx="192.8" cy="-717.5" rx="43.57" ry="18"/> +<text text-anchor="middle" x="192.8" y="-713.23" font-family="monospace,monospace" font-size="12.00">tp_init</text> +</a> +</g> +</g> +<!-- tp_new->tp_init --> +<g id="life_events_graph_edge3" class="edge"> +<title>tp_new->tp_init</title> +<g id="a_life_events_graph_edge3"><a xlink:title="tp_new to tp_init"> +<path fill="none" stroke="black" d="M192.8,-754.15C192.8,-751.83 192.8,-749.42 192.8,-746.98"/> +<polygon fill="black" stroke="black" points="196.3,-747.23 192.8,-737.23 189.3,-747.23 196.3,-747.23"/> +</a> +</g> +</g> +<!-- reachable --> +<g id="life_events_graph_node5" class="node"> +<title>reachable</title> +<polygon fill="none" stroke="black" points="230.8,-680.5 154.8,-680.5 154.8,-644.5 230.8,-644.5 230.8,-680.5"/> +<text text-anchor="middle" x="192.8" y="-658.23" font-family="serif,serif" font-style="italic" font-size="12.00">reachable</text> +</g> +<!-- tp_init->reachable --> +<g id="life_events_graph_edge4" class="edge"> +<title>tp_init->reachable</title> +<g id="a_life_events_graph_edge4"><a xlink:title="tp_init to reachable"> +<path fill="none" stroke="black" d="M186.44,-699.44C186.24,-697.12 186.11,-694.69 186.07,-692.24"/> +<polygon fill="black" stroke="black" points="189.56,-692.51 186.37,-682.41 182.56,-692.29 189.56,-692.51"/> +</a> +</g> +</g> +<!-- reachable->tp_init --> +<g id="life_events_graph_edge7" class="edge"> +<title>reachable->tp_init</title> +<g id="a_life_events_graph_edge7"><a xlink:title="reachable to tp_init"> +<path fill="none" stroke="black" d="M199.18,-680.89C199.37,-683.22 199.49,-685.65 199.53,-688.11"/> +<polygon fill="black" stroke="black" points="196.04,-687.81 199.2,-697.93 203.04,-688.05 196.04,-687.81"/> +</a> +</g> +</g> +<!-- tp_traverse --> +<g id="life_events_graph_node6" class="node"> +<title>tp_traverse</title> +<g id="a_life_events_graph_node6"><a xlink:href="typeobj.html#c.PyTypeObject.tp_traverse" xlink:title="tp_traverse" target="_top"> +<ellipse fill="none" stroke="black" cx="136.8" cy="-565.75" rx="62.65" ry="18"/> +<text text-anchor="middle" x="136.8" y="-561.48" font-family="monospace,monospace" font-size="12.00">tp_traverse</text> +</a> +</g> +</g> +<!-- reachable->tp_traverse --> +<g id="life_events_graph_edge5" class="edge"> +<title>reachable->tp_traverse</title> +<g id="a_life_events_graph_edge5"><a xlink:title="tp_traverse to reachable: not in a cyclic isolate"> +<path fill="none" stroke="black" d="M143.43,-658.77C108.3,-655.68 65.38,-649.16 54.05,-635.5 41.91,-620.88 42.8,-608.07 54.05,-592.75 60.55,-583.89 70.07,-577.97 80.37,-574.03"/> +<polygon fill="black" stroke="black" points="142.76,-662.23 153.01,-659.54 143.32,-655.25 142.76,-662.23"/> +</a> +</g> +<g id="a_life_events_graph_edge5-label"><a xlink:title="tp_traverse to reachable: not in a cyclic isolate"> +<text text-anchor="start" x="54.05" y="-624.1" font-family="serif,serif" font-style="italic" font-size="12.00">  not in a  </text> +<text text-anchor="start" x="59.67" y="-609.85" font-family="serif,serif" font-style="italic" font-size="12.00">  cyclic  </text> +<text text-anchor="start" x="57.05" y="-595.6" font-family="serif,serif" font-style="italic" font-size="12.00">  isolate  </text> +</a> +</g> +</g> +<!-- reachable->tp_traverse --> +<g id="life_events_graph_edge6" class="edge"> +<title>reachable->tp_traverse</title> +<g id="a_life_events_graph_edge6"><a xlink:title="reachable to tp_traverse: periodic cyclic isolate detection"> +<path fill="none" stroke="black" d="M154.41,-650.07C147.94,-646.44 142.04,-641.69 138.05,-635.5 130.52,-623.82 129.57,-608.56 130.79,-595.38"/> +<polygon fill="black" stroke="black" points="134.25,-595.91 132.17,-585.52 127.31,-594.94 134.25,-595.91"/> +</a> +</g> +<g id="a_life_events_graph_edge6-label"><a xlink:title="reachable to tp_traverse: periodic cyclic isolate detection"> +<text text-anchor="start" x="154.17" y="-624.1" font-family="serif,serif" font-style="italic" font-size="12.00">  periodic  </text> +<text text-anchor="start" x="138.05" y="-609.85" font-family="serif,serif" font-style="italic" font-size="12.00">  cyclic isolate   </text> +<text text-anchor="start" x="151.17" y="-595.6" font-family="serif,serif" font-style="italic" font-size="12.00">  detection  </text> +</a> +</g> +</g> +<!-- finalized? --> +<g id="life_events_graph_node7" class="node"> +<title>finalized?</title> +<g id="a_life_events_graph_node7"><a xlink:title="marked as finalized?"> +<polygon fill="none" stroke="black" points="191.8,-487 112.05,-450.5 191.8,-414 271.55,-450.5 191.8,-487"/> +<text text-anchor="start" x="159.92" y="-453.35" font-family="serif,serif" font-style="italic" font-size="12.00">marked as</text> +<text text-anchor="start" x="162.92" y="-439.1" font-family="serif,serif" font-style="italic" font-size="12.00">finalized?</text> +</a> +</g> +</g> +<!-- reachable->finalized? --> +<g id="life_events_graph_edge10" class="edge"> +<title>reachable->finalized?</title> +<g id="a_life_events_graph_edge10"><a xlink:title="reachable to finalized?: no refs"> +<path fill="none" stroke="black" d="M227.72,-644.32C230.51,-641.73 232.96,-638.8 234.8,-635.5 244.04,-618.9 235.48,-611.74 234.8,-592.75 233.24,-549.67 243.64,-536.1 227.8,-496 226.37,-492.38 224.53,-488.82 222.45,-485.4"/> +<polygon fill="black" stroke="black" points="225.47,-483.62 216.91,-477.39 219.72,-487.61 225.47,-483.62"/> +</a> +</g> +<g id="a_life_events_graph_edge10-label"><a xlink:title="reachable to finalized?: no refs"> +<text text-anchor="start" x="236.45" y="-561.48" font-family="serif,serif" font-style="italic" font-size="12.00">  no refs  </text> +</a> +</g> +</g> +<!-- tp_finalize --> +<g id="life_events_graph_node8" class="node"> +<title>tp_finalize</title> +<g id="a_life_events_graph_node8"><a xlink:href="typeobj.html#c.PyTypeObject.tp_finalize" xlink:title="tp_finalize" target="_top"> +<ellipse fill="none" stroke="black" cx="122.8" cy="-321" rx="62.65" ry="18"/> +<text text-anchor="middle" x="122.8" y="-316.73" font-family="monospace,monospace" font-size="12.00">tp_finalize</text> +</a> +</g> +</g> +<!-- reachable->tp_finalize --> +<g id="life_events_graph_edge8" class="edge"> +<title>reachable->tp_finalize</title> +<g id="a_life_events_graph_edge8"><a xlink:title="tp_finalize to reachable: resurrected (maybe remove finalized mark)"> +<path fill="none" stroke="black" d="M142.86,-659.6C103.8,-656.96 53.97,-650.63 40.8,-635.5 -37.32,-545.75 69.61,-390.31 109.14,-338.99"/> +<polygon fill="black" stroke="black" points="142.62,-663.09 152.82,-660.2 143.05,-656.11 142.62,-663.09"/> +</a> +</g> +<g id="a_life_events_graph_edge8-label"><a xlink:title="tp_finalize to reachable: resurrected (maybe remove finalized mark)"> +<text text-anchor="start" x="33.68" y="-527.35" font-family="serif,serif" font-style="italic" font-size="12.00">  resurrected  </text> +<text text-anchor="start" x="22.43" y="-513.1" font-family="serif,serif" font-style="italic" font-size="12.00">  (maybe remove  </text> +<text text-anchor="start" x="23.18" y="-498.85" font-family="serif,serif" font-style="italic" font-size="12.00">  finalized mark)  </text> +</a> +</g> +</g> +<!-- uncollectable --> +<g id="life_events_graph_node10" class="node"> +<title>uncollectable</title> +<g id="a_life_events_graph_node10"><a xlink:title="uncollectable (leaked)"> +<polygon fill="none" stroke="black" points="371.92,-159.75 275.67,-159.75 275.67,-123.25 371.92,-123.25 371.92,-159.75"/> +<text text-anchor="start" x="283.67" y="-144.35" font-family="serif,serif" font-style="italic" font-size="12.00">uncollectable</text> +<text text-anchor="start" x="299.42" y="-130.1" font-family="serif,serif" font-style="italic" font-size="12.00">(leaked)</text> +</a> +</g> +</g> +<!-- reachable->uncollectable --> +<g id="life_events_graph_edge19" class="edge"> +<title>reachable->uncollectable</title> +<g id="a_life_events_graph_edge19"><a xlink:title="reachable to uncollectable: cyclic isolate (no GC support)"> +<path fill="none" stroke="black" d="M231.2,-652.03C270.79,-639.69 326.8,-613.9 326.8,-566.75 326.8,-566.75 326.8,-566.75 326.8,-237.5 326.8,-215.3 325.97,-190.2 325.18,-171.37"/> +<polygon fill="black" stroke="black" points="328.68,-171.35 324.75,-161.52 321.69,-171.66 328.68,-171.35"/> +</a> +</g> +<g id="a_life_events_graph_edge19-label"><a xlink:title="reachable to uncollectable: cyclic isolate (no GC support)"> +<text text-anchor="start" x="335.05" y="-393.6" font-family="serif,serif" font-style="italic" font-size="12.00">  cyclic  </text> +<text text-anchor="start" x="332.42" y="-379.35" font-family="serif,serif" font-style="italic" font-size="12.00">  isolate  </text> +<text text-anchor="start" x="331.3" y="-365.1" font-family="serif,serif" font-style="italic" font-size="12.00">  (no GC  </text> +<text text-anchor="start" x="326.8" y="-350.85" font-family="serif,serif" font-style="italic" font-size="12.00">  support)  </text> +</a> +</g> +</g> +<!-- tp_dealloc --> +<g id="life_events_graph_node11" class="node"> +<title>tp_dealloc</title> +<g id="a_life_events_graph_node11"><a xlink:href="typeobj.html#c.PyTypeObject.tp_dealloc" xlink:title="tp_dealloc" target="_top"> +<ellipse fill="none" stroke="black" cx="200.8" cy="-86.25" rx="57.88" ry="18"/> +<text text-anchor="middle" x="200.8" y="-81.97" font-family="monospace,monospace" font-size="12.00">tp_dealloc</text> +</a> +</g> +</g> +<!-- reachable->tp_dealloc --> +<g id="life_events_graph_edge20" class="edge"> +<title>reachable->tp_dealloc</title> +<path fill="none" stroke="black" d="M231.23,-661.18C293.08,-658.43 407.8,-643.03 407.8,-566.75 407.8,-566.75 407.8,-566.75 407.8,-140.5 407.8,-111.22 329.12,-97.8 268.77,-91.82"/> +<polygon fill="black" stroke="black" points="269.15,-88.34 258.87,-90.89 268.5,-95.31 269.15,-88.34"/> +<g id="a_life_events_graph_edge20-label"><a xlink:title="reachable to tp_dealloc: no refs"> +<text text-anchor="start" x="407.8" y="-316.73" font-family="serif,serif" font-style="italic" font-size="12.00">  no refs</text> +</a> +</g> +</g> +<!-- tp_traverse->finalized? --> +<g id="life_events_graph_edge9" class="edge"> +<title>tp_traverse->finalized?</title> +<g id="a_life_events_graph_edge9"><a xlink:title="tp_traverse to finalized?: cyclic isolate"> +<path fill="none" stroke="black" d="M145.15,-547.55C152.4,-532.62 163.18,-510.43 172.55,-491.13"/> +<polygon fill="black" stroke="black" points="175.7,-492.66 176.92,-482.14 169.4,-489.61 175.7,-492.66"/> +</a> +</g> +<g id="a_life_events_graph_edge9-label"><a xlink:title="tp_traverse to finalized?: cyclic isolate"> +<text text-anchor="start" x="171.85" y="-520.23" font-family="serif,serif" font-style="italic" font-size="12.00">  cyclic  </text> +<text text-anchor="start" x="169.22" y="-505.98" font-family="serif,serif" font-style="italic" font-size="12.00">  isolate  </text> +</a> +</g> +</g> +<!-- finalized?->tp_finalize --> +<g id="life_events_graph_edge11" class="edge"> +<title>finalized?->tp_finalize</title> +<g id="a_life_events_graph_edge11"><a xlink:title="finalized? to tp_finalize: no (mark as finalized)"> +<path fill="none" stroke="black" d="M172.89,-422.6C169.14,-416.89 165.34,-410.82 162.05,-405 151.89,-387.08 141.99,-366.11 134.68,-349.73"/> +<polygon fill="black" stroke="black" points="137.89,-348.35 130.66,-340.61 131.48,-351.17 137.89,-348.35"/> +</a> +</g> +<g id="a_life_events_graph_edge11-label"><a xlink:title="finalized? to tp_finalize: no (mark as finalized)"> +<text text-anchor="start" x="170.67" y="-379.35" font-family="serif,serif" font-style="italic" font-size="12.00">  no (mark  </text> +<text text-anchor="start" x="162.05" y="-365.1" font-family="serif,serif" font-style="italic" font-size="12.00">  as finalized)  </text> +</a> +</g> +</g> +<!-- tp_clear --> +<g id="life_events_graph_node9" class="node"> +<title>tp_clear</title> +<g id="a_life_events_graph_node9"><a xlink:href="typeobj.html#c.PyTypeObject.tp_clear" xlink:title="tp_clear" target="_top"> +<ellipse fill="none" stroke="black" cx="222.8" cy="-238.5" rx="48.34" ry="18"/> +<text text-anchor="middle" x="222.8" y="-234.22" font-family="monospace,monospace" font-size="12.00">tp_clear</text> +</a> +</g> +</g> +<!-- finalized?->tp_clear --> +<g id="life_events_graph_edge12" class="edge"> +<title>finalized?->tp_clear</title> +<g id="a_life_events_graph_edge12"><a xlink:title="finalized? to tp_clear: yes"> +<path fill="none" stroke="black" d="M227.56,-430.1C236.46,-423.41 244.86,-415.02 249.8,-405 277.22,-349.39 274.06,-322.55 249.8,-265.5 249.7,-265.27 249.6,-265.04 249.49,-264.81"/> +<polygon fill="black" stroke="black" points="252.41,-262.88 243.93,-256.52 246.6,-266.78 252.41,-262.88"/> +</a> +</g> +<g id="a_life_events_graph_edge12-label"><a xlink:title="finalized? to tp_clear: yes"> +<text text-anchor="start" x="269.2" y="-316.73" font-family="serif,serif" font-style="italic" font-size="12.00">  yes  </text> +</a> +</g> +</g> +<!-- tp_finalize->tp_clear --> +<g id="life_events_graph_edge13" class="edge"> +<title>tp_finalize->tp_clear</title> +<g id="a_life_events_graph_edge13"><a xlink:title="tp_finalize to tp_clear: no refs or cyclic isolate"> +<path fill="none" stroke="black" d="M130.02,-302.72C135.75,-290.85 144.8,-275.49 156.8,-265.5 161.95,-261.21 167.9,-257.57 174.07,-254.49"/> +<polygon fill="black" stroke="black" points="175.46,-257.71 183.18,-250.45 172.62,-251.31 175.46,-257.71"/> +</a> +</g> +<g id="a_life_events_graph_edge13-label"><a xlink:title="tp_finalize to tp_clear: no refs or cyclic isolate"> +<text text-anchor="start" x="164.3" y="-282.6" font-family="serif,serif" font-style="italic" font-size="12.00">  no refs or   </text> +<text text-anchor="start" x="156.8" y="-268.35" font-family="serif,serif" font-style="italic" font-size="12.00">  cyclic isolate  </text> +</a> +</g> +</g> +<!-- tp_finalize->tp_dealloc --> +<g id="life_events_graph_edge14" class="edge"> +<title>tp_finalize->tp_dealloc</title> +<g id="a_life_events_graph_edge14"><a xlink:href="lifecycle.html#c.PyObject_CallFinalizerFromDealloc" xlink:title="tp_dealloc to tp_finalize: recommended call (see explanation)" target="_top"> +<path fill="none" stroke="black" stroke-dasharray="5,2" d="M85.85,-298.52C42.09,-270.18 -21.4,-218.11 7.8,-168.75 36.22,-120.7 99.95,-100.97 146.42,-92.87"/> +<polygon fill="none" stroke="black" points="83.78,-301.35 94.11,-303.72 87.52,-295.43 83.78,-301.35"/> +</a> +</g> +<g id="a_life_events_graph_edge14-label"><a xlink:href="lifecycle.html#c.PyObject_CallFinalizerFromDealloc" xlink:title="tp_dealloc to tp_finalize: recommended call (see explanation)" target="_top"> +<text text-anchor="start" x="7.8" y="-200.1" font-family="serif,serif" font-style="italic" font-size="12.00">  recommended</text> +<text text-anchor="start" x="25.8" y="-185.85" font-family="serif,serif" font-style="italic" font-size="12.00">  call (see</text> +<text text-anchor="start" x="13.05" y="-171.6" font-family="serif,serif" font-style="italic" font-size="12.00">  explanation)</text> +</a> +</g> +</g> +<!-- tp_finalize->tp_dealloc --> +<g id="life_events_graph_edge15" class="edge"> +<title>tp_finalize->tp_dealloc</title> +<g id="a_life_events_graph_edge15"><a xlink:title="tp_finalize to tp_dealloc: no refs"> +<path fill="none" stroke="black" d="M123.03,-302.58C123.95,-273.77 128.08,-214.78 146.05,-168.75 153.95,-148.5 167.56,-128.2 179.24,-112.92"/> +<polygon fill="black" stroke="black" points="181.81,-115.32 185.25,-105.3 176.31,-110.98 181.81,-115.32"/> +</a> +</g> +<g id="a_life_events_graph_edge15-label"><a xlink:title="tp_finalize to tp_dealloc: no refs"> +<text text-anchor="start" x="146.05" y="-185.85" font-family="serif,serif" font-style="italic" font-size="12.00">   no refs  </text> +</a> +</g> +</g> +<!-- tp_clear->uncollectable --> +<g id="life_events_graph_edge17" class="edge"> +<title>tp_clear->uncollectable</title> +<g id="a_life_events_graph_edge17"><a xlink:title="tp_clear to uncollectable: cyclic isolate"> +<path fill="none" stroke="black" d="M227.75,-220.38C232.99,-205 242.67,-182.74 258.05,-168.75 260.43,-166.58 263.02,-164.58 265.74,-162.73"/> +<polygon fill="black" stroke="black" points="267.27,-165.89 274.12,-157.81 263.73,-159.86 267.27,-165.89"/> +</a> +</g> +<g id="a_life_events_graph_edge17-label"><a xlink:title="tp_clear to uncollectable: cyclic isolate"> +<text text-anchor="start" x="260.67" y="-192.97" font-family="serif,serif" font-style="italic" font-size="12.00">  cyclic  </text> +<text text-anchor="start" x="258.05" y="-178.72" font-family="serif,serif" font-style="italic" font-size="12.00">  isolate  </text> +</a> +</g> +</g> +<!-- tp_clear->tp_dealloc --> +<g id="life_events_graph_edge16" class="edge"> +<title>tp_clear->tp_dealloc</title> +<g id="a_life_events_graph_edge16"><a xlink:title="tp_clear to tp_dealloc: no refs"> +<path fill="none" stroke="black" d="M219.7,-220.24C216.92,-204.51 212.83,-180.61 209.8,-159.75 207.7,-145.34 205.67,-129.26 204.07,-115.92"/> +<polygon fill="black" stroke="black" points="207.56,-115.59 202.91,-106.07 200.61,-116.41 207.56,-115.59"/> +</a> +</g> +<g id="a_life_events_graph_edge16-label"><a xlink:title="tp_clear to tp_dealloc: no refs"> +<text text-anchor="start" x="209.8" y="-137.22" font-family="serif,serif" font-style="italic" font-size="12.00">  no refs  </text> +</a> +</g> +</g> +<!-- uncollectable->tp_dealloc --> +<!-- tp_free --> +<g id="life_events_graph_node12" class="node"> +<title>tp_free</title> +<g id="a_life_events_graph_node12"><a xlink:href="typeobj.html#c.PyTypeObject.tp_free" xlink:title="tp_free" target="_top"> +<ellipse fill="none" stroke="black" cx="200.8" cy="-18" rx="43.57" ry="18"/> +<text text-anchor="middle" x="200.8" y="-13.72" font-family="monospace,monospace" font-size="12.00">tp_free</text> +</a> +</g> +</g> +<!-- tp_dealloc->tp_free --> +<g id="life_events_graph_edge21" class="edge"> +<title>tp_dealloc->tp_free</title> +<g id="a_life_events_graph_edge21"><a xlink:title="tp_dealloc to tp_free: direct call"> +<path fill="none" stroke="black" d="M200.8,-67.84C200.8,-61.63 200.8,-54.46 200.8,-47.56"/> +<polygon fill="none" stroke="black" points="204.3,-47.57 200.8,-37.57 197.3,-47.57 204.3,-47.57"/> +</a> +</g> +<g id="a_life_events_graph_edge21-label"><a xlink:title="tp_dealloc to tp_free: direct call"> +<text text-anchor="start" x="200.8" y="-47.85" font-family="serif,serif" font-style="italic" font-size="12.00">    direct call  </text> +</a> +</g> +</g> +</g> +</svg> diff --git a/Doc/c-api/lifecycle.rst b/Doc/c-api/lifecycle.rst new file mode 100644 index 00000000000..0e2ffc096ca --- /dev/null +++ b/Doc/c-api/lifecycle.rst @@ -0,0 +1,273 @@ +.. highlight:: c + +.. _life-cycle: + +Object Life Cycle +================= + +This section explains how a type's slots relate to each other throughout the +life of an object. It is not intended to be a complete canonical reference for +the slots; instead, refer to the slot-specific documentation in +:ref:`type-structs` for details about a particular slot. + + +Life Events +----------- + +The figure below illustrates the order of events that can occur throughout an +object's life. An arrow from *A* to *B* indicates that event *B* can occur +after event *A* has occurred, with the arrow's label indicating the condition +that must be true for *B* to occur after *A*. + +.. only:: html and not epub + + .. raw:: html + + <style type="text/css"> + + .. raw:: html + :file: lifecycle.dot.css + + .. raw:: html + + </style> + + .. raw:: html + :file: lifecycle.dot.svg + + .. raw:: html + + <script> + (() => { + const g = document.getElementById('life_events_graph'); + const title = g.querySelector(':scope > title'); + title.id = 'life-events-graph-title'; + const svg = g.closest('svg'); + svg.role = 'img'; + svg.setAttribute('aria-describedby', + 'life-events-graph-description'); + svg.setAttribute('aria-labelledby', 'life-events-graph-title'); + })(); + </script> + +.. only:: epub or not (html or latex) + + .. image:: lifecycle.dot.svg + :align: center + :class: invert-in-dark-mode + :alt: Diagram showing events in an object's life. Explained in detail + below. + +.. only:: latex + + .. image:: lifecycle.dot.pdf + :align: center + :class: invert-in-dark-mode + :alt: Diagram showing events in an object's life. Explained in detail + below. + +.. container:: + :name: life-events-graph-description + + Explanation: + + * When a new object is constructed by calling its type: + + #. :c:member:`~PyTypeObject.tp_new` is called to create a new object. + #. :c:member:`~PyTypeObject.tp_alloc` is directly called by + :c:member:`~PyTypeObject.tp_new` to allocate the memory for the new + object. + #. :c:member:`~PyTypeObject.tp_init` initializes the newly created object. + :c:member:`!tp_init` can be called again to re-initialize an object, if + desired. The :c:member:`!tp_init` call can also be skipped entirely, + for example by Python code calling :py:meth:`~object.__new__`. + + * After :c:member:`!tp_init` completes, the object is ready to use. + * Some time after the last reference to an object is removed: + + #. If an object is not marked as *finalized*, it might be finalized by + marking it as *finalized* and calling its + :c:member:`~PyTypeObject.tp_finalize` function. Python does + *not* finalize an object when the last reference to it is deleted; use + :c:func:`PyObject_CallFinalizerFromDealloc` to ensure that + :c:member:`~PyTypeObject.tp_finalize` is always called. + #. If the object is marked as finalized, + :c:member:`~PyTypeObject.tp_clear` might be called by the garbage collector + to clear references held by the object. It is *not* called when the + object's reference count reaches zero. + #. :c:member:`~PyTypeObject.tp_dealloc` is called to destroy the object. + To avoid code duplication, :c:member:`~PyTypeObject.tp_dealloc` typically + calls into :c:member:`~PyTypeObject.tp_clear` to free up the object's + references. + #. When :c:member:`~PyTypeObject.tp_dealloc` finishes object destruction, + it directly calls :c:member:`~PyTypeObject.tp_free` (usually set to + :c:func:`PyObject_Free` or :c:func:`PyObject_GC_Del` automatically as + appropriate for the type) to deallocate the memory. + + * The :c:member:`~PyTypeObject.tp_finalize` function is permitted to add a + reference to the object if desired. If it does, the object is + *resurrected*, preventing its pending destruction. (Only + :c:member:`!tp_finalize` is allowed to resurrect an object; + :c:member:`~PyTypeObject.tp_clear` and + :c:member:`~PyTypeObject.tp_dealloc` cannot without calling into + :c:member:`!tp_finalize`.) Resurrecting an object may + or may not cause the object's *finalized* mark to be removed. Currently, + Python does not remove the *finalized* mark from a resurrected object if + it supports garbage collection (i.e., the :c:macro:`Py_TPFLAGS_HAVE_GC` + flag is set) but does remove the mark if the object does not support + garbage collection; either or both of these behaviors may change in the + future. + * :c:member:`~PyTypeObject.tp_dealloc` can optionally call + :c:member:`~PyTypeObject.tp_finalize` via + :c:func:`PyObject_CallFinalizerFromDealloc` if it wishes to reuse that + code to help with object destruction. This is recommended because it + guarantees that :c:member:`!tp_finalize` is always called before + destruction. See the :c:member:`~PyTypeObject.tp_dealloc` documentation + for example code. + * If the object is a member of a :term:`cyclic isolate` and either + :c:member:`~PyTypeObject.tp_clear` fails to break the reference cycle or + the cyclic isolate is not detected (perhaps :func:`gc.disable` was called, + or the :c:macro:`Py_TPFLAGS_HAVE_GC` flag was erroneously omitted in one + of the involved types), the objects remain indefinitely uncollectable + (they "leak"). See :data:`gc.garbage`. + + If the object is marked as supporting garbage collection (the + :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set in + :c:member:`~PyTypeObject.tp_flags`), the following events are also possible: + + * The garbage collector occasionally calls + :c:member:`~PyTypeObject.tp_traverse` to identify :term:`cyclic isolates + <cyclic isolate>`. + * When the garbage collector discovers a :term:`cyclic isolate`, it + finalizes one of the objects in the group by marking it as *finalized* and + calling its :c:member:`~PyTypeObject.tp_finalize` function, if it has one. + This repeats until the cyclic isolate doesn't exist or all of the objects + have been finalized. + * :c:member:`~PyTypeObject.tp_finalize` is permitted to resurrect the object + by adding a reference from outside the :term:`cyclic isolate`. The new + reference causes the group of objects to no longer form a cyclic isolate + (the reference cycle may still exist, but if it does the objects are no + longer isolated). + * When the garbage collector discovers a :term:`cyclic isolate` and all of + the objects in the group have already been marked as *finalized*, the + garbage collector clears one or more of the uncleared objects in the group + (possibly concurrently) by calling each's + :c:member:`~PyTypeObject.tp_clear` function. This repeats as long as the + cyclic isolate still exists and not all of the objects have been cleared. + + +Cyclic Isolate Destruction +-------------------------- + +Listed below are the stages of life of a hypothetical :term:`cyclic isolate` +that continues to exist after each member object is finalized or cleared. It +is a memory leak if a cyclic isolate progresses through all of these stages; it should +vanish once all objects are cleared, if not sooner. A cyclic isolate can +vanish either because the reference cycle is broken or because the objects are +no longer isolated due to finalizer resurrection (see +:c:member:`~PyTypeObject.tp_finalize`). + +0. **Reachable** (not yet a cyclic isolate): All objects are in their normal, + reachable state. A reference cycle could exist, but an external reference + means the objects are not yet isolated. +#. **Unreachable but consistent:** The final reference from outside the cyclic + group of objects has been removed, causing the objects to become isolated + (thus a cyclic isolate is born). None of the group's objects have been + finalized or cleared yet. The cyclic isolate remains at this stage until + some future run of the garbage collector (not necessarily the next run + because the next run might not scan every object). +#. **Mix of finalized and not finalized:** Objects in a cyclic isolate are + finalized one at a time, which means that there is a period of time when the + cyclic isolate is composed of a mix of finalized and non-finalized objects. + Finalization order is unspecified, so it can appear random. A finalized + object must behave in a sane manner when non-finalized objects interact with + it, and a non-finalized object must be able to tolerate the finalization of + an arbitrary subset of its referents. +#. **All finalized:** All objects in a cyclic isolate are finalized before any + of them are cleared. +#. **Mix of finalized and cleared:** The objects can be cleared serially or + concurrently (but with the :term:`GIL` held); either way, some will finish + before others. A finalized object must be able to tolerate the clearing of + a subset of its referents. :pep:`442` calls this stage "cyclic trash". +#. **Leaked:** If a cyclic isolate still exists after all objects in the group + have been finalized and cleared, then the objects remain indefinitely + uncollectable (see :data:`gc.garbage`). It is a bug if a cyclic isolate + reaches this stage---it means the :c:member:`~PyTypeObject.tp_clear` methods + of the participating objects have failed to break the reference cycle as + required. + +If :c:member:`~PyTypeObject.tp_clear` did not exist, then Python would have no +way to safely break a reference cycle. Simply destroying an object in a cyclic +isolate would result in a dangling pointer, triggering undefined behavior when +an object referencing the destroyed object is itself destroyed. The clearing +step makes object destruction a two-phase process: first +:c:member:`~PyTypeObject.tp_clear` is called to partially destroy the objects +enough to detangle them from each other, then +:c:member:`~PyTypeObject.tp_dealloc` is called to complete the destruction. + +Unlike clearing, finalization is not a phase of destruction. A finalized +object must still behave properly by continuing to fulfill its design +contracts. An object's finalizer is allowed to execute arbitrary Python code, +and is even allowed to prevent the impending destruction by adding a reference. +The finalizer is only related to destruction by call order---if it runs, it runs +before destruction, which starts with :c:member:`~PyTypeObject.tp_clear` (if +called) and concludes with :c:member:`~PyTypeObject.tp_dealloc`. + +The finalization step is not necessary to safely reclaim the objects in a +cyclic isolate, but its existence makes it easier to design types that behave +in a sane manner when objects are cleared. Clearing an object might +necessarily leave it in a broken, partially destroyed state---it might be +unsafe to call any of the cleared object's methods or access any of its +attributes. With finalization, only finalized objects can possibly interact +with cleared objects; non-finalized objects are guaranteed to interact with +only non-cleared (but potentially finalized) objects. + +To summarize the possible interactions: + +* A non-finalized object might have references to or from non-finalized and + finalized objects, but not to or from cleared objects. +* A finalized object might have references to or from non-finalized, finalized, + and cleared objects. +* A cleared object might have references to or from finalized and cleared + objects, but not to or from non-finalized objects. + +Without any reference cycles, an object can be simply destroyed once its last +reference is deleted; the finalization and clearing steps are not necessary to +safely reclaim unused objects. However, it can be useful to automatically call +:c:member:`~PyTypeObject.tp_finalize` and :c:member:`~PyTypeObject.tp_clear` +before destruction anyway because type design is simplified when all objects +always experience the same series of events regardless of whether they +participated in a cyclic isolate. Python currently only calls +:c:member:`~PyTypeObject.tp_finalize` and :c:member:`~PyTypeObject.tp_clear` as +needed to destroy a cyclic isolate; this may change in a future version. + + +Functions +--------- + +To allocate and free memory, see :ref:`allocating-objects`. + + +.. c:function:: void PyObject_CallFinalizer(PyObject *op) + + Finalizes the object as described in :c:member:`~PyTypeObject.tp_finalize`. + Call this function (or :c:func:`PyObject_CallFinalizerFromDealloc`) instead + of calling :c:member:`~PyTypeObject.tp_finalize` directly because this + function may deduplicate multiple calls to :c:member:`!tp_finalize`. + Currently, calls are only deduplicated if the type supports garbage + collection (i.e., the :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set); this may + change in the future. + + +.. c:function:: int PyObject_CallFinalizerFromDealloc(PyObject *op) + + Same as :c:func:`PyObject_CallFinalizer` but meant to be called at the + beginning of the object's destructor (:c:member:`~PyTypeObject.tp_dealloc`). + There must not be any references to the object. If the object's finalizer + resurrects the object, this function returns -1; no further destruction + should happen. Otherwise, this function returns 0 and destruction can + continue normally. + + .. seealso:: + + :c:member:`~PyTypeObject.tp_dealloc` for example code. diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 64ae35daa70..61fa49f8681 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -376,6 +376,24 @@ The :ref:`default object allocator <default-memory-allocators>` uses the If *p* is ``NULL``, no operation is performed. + Do not call this directly to free an object's memory; call the type's + :c:member:`~PyTypeObject.tp_free` slot instead. + + Do not use this for memory allocated by :c:macro:`PyObject_GC_New` or + :c:macro:`PyObject_GC_NewVar`; use :c:func:`PyObject_GC_Del` instead. + + .. seealso:: + + * :c:func:`PyObject_GC_Del` is the equivalent of this function for memory + allocated by types that support garbage collection. + * :c:func:`PyObject_Malloc` + * :c:func:`PyObject_Realloc` + * :c:func:`PyObject_Calloc` + * :c:macro:`PyObject_New` + * :c:macro:`PyObject_NewVar` + * :c:func:`PyType_GenericAlloc` + * :c:member:`~PyTypeObject.tp_free` + .. _default-memory-allocators: diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index f7f4d37d4c7..710135dca89 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -288,22 +288,40 @@ An alternate way to specify extensions is to request "multi-phase initialization Extension modules created this way behave more like Python modules: the initialization is split between the *creation phase*, when the module object is created, and the *execution phase*, when it is populated. -The distinction is similar to the :py:meth:`!__new__` and :py:meth:`!__init__` methods -of classes. +The distinction is similar to the :py:meth:`~object.__new__` and +:py:meth:`~object.__init__` methods of classes. Unlike modules created using single-phase initialization, these modules are not -singletons: if the *sys.modules* entry is removed and the module is re-imported, -a new module object is created, and the old module is subject to normal garbage -collection -- as with Python modules. -By default, multiple modules created from the same definition should be -independent: changes to one should not affect the others. -This means that all state should be specific to the module object (using e.g. -using :c:func:`PyModule_GetState`), or its contents (such as the module's -:attr:`~object.__dict__` or individual classes created with :c:func:`PyType_FromSpec`). +singletons. +For example, if the :py:attr:`sys.modules` entry is removed and the module +is re-imported, a new module object is created, and typically populated with +fresh method and type objects. +The old module is subject to normal garbage collection. +This mirrors the behavior of pure-Python modules. + +Additional module instances may be created in +:ref:`sub-interpreters <sub-interpreter-support>` +or after after Python runtime reinitialization +(:c:func:`Py_Finalize` and :c:func:`Py_Initialize`). +In these cases, sharing Python objects between module instances would likely +cause crashes or undefined behavior. + +To avoid such issues, each instance of an extension module should +be *isolated*: changes to one instance should not implicitly affect the others, +and all state, including references to Python objects, should be specific to +a particular module instance. +See :ref:`isolating-extensions-howto` for more details and a practical guide. + +A simpler way to avoid these issues is +:ref:`raising an error on repeated initialization <isolating-extensions-optout>`. All modules created using multi-phase initialization are expected to support -:ref:`sub-interpreters <sub-interpreter-support>`. Making sure multiple modules -are independent is typically enough to achieve this. +:ref:`sub-interpreters <sub-interpreter-support>`, or otherwise explicitly +signal a lack of support. +This is usually achieved by isolation or blocking repeated initialization, +as above. +A module may also be limited to the main interpreter using +the :c:data:`Py_mod_multiple_interpreters` slot. To request multi-phase initialization, the initialization function (PyInit_modulename) returns a :c:type:`PyModuleDef` instance with non-empty diff --git a/Doc/c-api/objimpl.rst b/Doc/c-api/objimpl.rst index 8bd8c107c98..83de4248039 100644 --- a/Doc/c-api/objimpl.rst +++ b/Doc/c-api/objimpl.rst @@ -12,6 +12,7 @@ object types. .. toctree:: allocation.rst + lifecycle.rst structures.rst typeobj.rst gcsupport.rst diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index b3c89800e38..b34936dd55e 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -258,10 +258,57 @@ These are utility functions that make functionality from the :mod:`sys` module accessible to C code. They all work with the current interpreter thread's :mod:`sys` module's dict, which is contained in the internal thread state structure. +.. c:function:: PyObject *PySys_GetAttr(PyObject *name) + + Get the attribute *name* of the :mod:`sys` module. + Return a :term:`strong reference`. + Raise :exc:`RuntimeError` and return ``NULL`` if it does not exist or + if the :mod:`sys` module cannot be found. + + If the non-existing object should not be treated as a failure, you can use + :c:func:`PySys_GetOptionalAttr` instead. + + .. versionadded:: next + +.. c:function:: PyObject *PySys_GetAttrString(const char *name) + + This is the same as :c:func:`PySys_GetAttr`, but *name* is + specified as a :c:expr:`const char*` UTF-8 encoded bytes string, + rather than a :c:expr:`PyObject*`. + + If the non-existing object should not be treated as a failure, you can use + :c:func:`PySys_GetOptionalAttrString` instead. + + .. versionadded:: next + +.. c:function:: int PySys_GetOptionalAttr(PyObject *name, PyObject **result) + + Variant of :c:func:`PySys_GetAttr` which doesn't raise + exception if the object does not exist. + + * Set *\*result* to a new :term:`strong reference` to the object and + return ``1`` if the object exists. + * Set *\*result* to ``NULL`` and return ``0`` without setting an exception + if the object does not exist. + * Set an exception, set *\*result* to ``NULL``, and return ``-1``, + if an error occurred. + + .. versionadded:: next + +.. c:function:: int PySys_GetOptionalAttrString(const char *name, PyObject **result) + + This is the same as :c:func:`PySys_GetOptionalAttr`, but *name* is + specified as a :c:expr:`const char*` UTF-8 encoded bytes string, + rather than a :c:expr:`PyObject*`. + + .. versionadded:: next + .. c:function:: PyObject *PySys_GetObject(const char *name) - Return the object *name* from the :mod:`sys` module or ``NULL`` if it does - not exist, without setting an exception. + Similar to :c:func:`PySys_GetAttrString`, but return a :term:`borrowed + reference` and return ``NULL`` *without* setting exception on failure. + + Preserves exception that was set before the call. .. c:function:: int PySys_SetObject(const char *name, PyObject *v) diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst index ec2867b0ce0..2176b8e492f 100644 --- a/Doc/c-api/type.rst +++ b/Doc/c-api/type.rst @@ -151,14 +151,29 @@ Type Objects .. c:function:: PyObject* PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems) - Generic handler for the :c:member:`~PyTypeObject.tp_alloc` slot of a type object. Use - Python's default memory allocation mechanism to allocate a new instance and - initialize all its contents to ``NULL``. + Generic handler for the :c:member:`~PyTypeObject.tp_alloc` slot of a type + object. Uses Python's default memory allocation mechanism to allocate memory + for a new instance, zeros the memory, then initializes the memory as if by + calling :c:func:`PyObject_Init` or :c:func:`PyObject_InitVar`. + + Do not call this directly to allocate memory for an object; call the type's + :c:member:`~PyTypeObject.tp_alloc` slot instead. + + For types that support garbage collection (i.e., the + :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set), this function behaves like + :c:macro:`PyObject_GC_New` or :c:macro:`PyObject_GC_NewVar` (except the + memory is guaranteed to be zeroed before initialization), and should be + paired with :c:func:`PyObject_GC_Del` in :c:member:`~PyTypeObject.tp_free`. + Otherwise, it behaves like :c:macro:`PyObject_New` or + :c:macro:`PyObject_NewVar` (except the memory is guaranteed to be zeroed + before initialization) and should be paired with :c:func:`PyObject_Free` in + :c:member:`~PyTypeObject.tp_free`. .. c:function:: PyObject* PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds) - Generic handler for the :c:member:`~PyTypeObject.tp_new` slot of a type object. Create a - new instance using the type's :c:member:`~PyTypeObject.tp_alloc` slot. + Generic handler for the :c:member:`~PyTypeObject.tp_new` slot of a type + object. Creates a new instance using the type's + :c:member:`~PyTypeObject.tp_alloc` slot and returns the resulting object. .. c:function:: int PyType_Ready(PyTypeObject *type) diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 3b9f07778d5..91046c0e6f1 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -79,7 +79,7 @@ Quick Reference | :c:member:`~PyTypeObject.tp_setattro` | :c:type:`setattrofunc` | __setattr__, | X | X | | G | | | | __delattr__ | | | | | +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+ - | :c:member:`~PyTypeObject.tp_as_buffer` | :c:type:`PyBufferProcs` * | | | | | % | + | :c:member:`~PyTypeObject.tp_as_buffer` | :c:type:`PyBufferProcs` * | :ref:`sub-slots` | | | | % | +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+ | :c:member:`~PyTypeObject.tp_flags` | unsigned long | | X | X | | ? | +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+ @@ -325,9 +325,10 @@ sub-slots +---------------------------------------------------------+-----------------------------------+---------------+ | | +---------------------------------------------------------+-----------------------------------+---------------+ - | :c:member:`~PyBufferProcs.bf_getbuffer` | :c:func:`getbufferproc` | | + | :c:member:`~PyBufferProcs.bf_getbuffer` | :c:func:`getbufferproc` | __buffer__ | +---------------------------------------------------------+-----------------------------------+---------------+ - | :c:member:`~PyBufferProcs.bf_releasebuffer` | :c:func:`releasebufferproc` | | + | :c:member:`~PyBufferProcs.bf_releasebuffer` | :c:func:`releasebufferproc` | __release_\ | + | | | buffer\__ | +---------------------------------------------------------+-----------------------------------+---------------+ .. _slot-typedefs-table: @@ -676,77 +677,122 @@ and :c:data:`PyType_Type` effectively act as defaults.) .. c:member:: destructor PyTypeObject.tp_dealloc - A pointer to the instance destructor function. This function must be defined - unless the type guarantees that its instances will never be deallocated (as is - the case for the singletons ``None`` and ``Ellipsis``). The function signature is:: + A pointer to the instance destructor function. The function signature is:: void tp_dealloc(PyObject *self); - The destructor function is called by the :c:func:`Py_DECREF` and - :c:func:`Py_XDECREF` macros when the new reference count is zero. At this point, - the instance is still in existence, but there are no references to it. The - destructor function should free all references which the instance owns, free all - memory buffers owned by the instance (using the freeing function corresponding - to the allocation function used to allocate the buffer), and call the type's - :c:member:`~PyTypeObject.tp_free` function. If the type is not subtypable - (doesn't have the :c:macro:`Py_TPFLAGS_BASETYPE` flag bit set), it is - permissible to call the object deallocator directly instead of via - :c:member:`~PyTypeObject.tp_free`. The object deallocator should be the one used to allocate the - instance; this is normally :c:func:`PyObject_Free` if the instance was allocated - using :c:macro:`PyObject_New` or :c:macro:`PyObject_NewVar`, or - :c:func:`PyObject_GC_Del` if the instance was allocated using - :c:macro:`PyObject_GC_New` or :c:macro:`PyObject_GC_NewVar`. - - If the type supports garbage collection (has the :c:macro:`Py_TPFLAGS_HAVE_GC` - flag bit set), the destructor should call :c:func:`PyObject_GC_UnTrack` + The destructor function should remove all references which the instance owns + (e.g., call :c:func:`Py_CLEAR`), free all memory buffers owned by the + instance, and call the type's :c:member:`~PyTypeObject.tp_free` function to + free the object itself. + + No guarantees are made about when an object is destroyed, except: + + * Python will destroy an object immediately or some time after the final + reference to the object is deleted, unless its finalizer + (:c:member:`~PyTypeObject.tp_finalize`) subsequently resurrects the + object. + * An object will not be destroyed while it is being automatically finalized + (:c:member:`~PyTypeObject.tp_finalize`) or automatically cleared + (:c:member:`~PyTypeObject.tp_clear`). + + CPython currently destroys an object immediately from :c:func:`Py_DECREF` + when the new reference count is zero, but this may change in a future + version. + + It is recommended to call :c:func:`PyObject_CallFinalizerFromDealloc` at the + beginning of :c:member:`!tp_dealloc` to guarantee that the object is always + finalized before destruction. + + If the type supports garbage collection (the :c:macro:`Py_TPFLAGS_HAVE_GC` + flag is set), the destructor should call :c:func:`PyObject_GC_UnTrack` before clearing any member fields. - .. code-block:: c + It is permissible to call :c:member:`~PyTypeObject.tp_clear` from + :c:member:`!tp_dealloc` to reduce code duplication and to guarantee that the + object is always cleared before destruction. Beware that + :c:member:`!tp_clear` might have already been called. - static void - foo_dealloc(PyObject *op) - { + If the type is heap allocated (:c:macro:`Py_TPFLAGS_HEAPTYPE`), the + deallocator should release the owned reference to its type object (via + :c:func:`Py_DECREF`) after calling the type deallocator. See the example + code below.:: + + static void + foo_dealloc(PyObject *op) + { foo_object *self = (foo_object *) op; PyObject_GC_UnTrack(self); Py_CLEAR(self->ref); Py_TYPE(self)->tp_free(self); - } + } + + :c:member:`!tp_dealloc` must leave the exception status unchanged. If it + needs to call something that might raise an exception, the exception state + must be backed up first and restored later (after logging any exceptions + with :c:func:`PyErr_WriteUnraisable`). + + Example:: - Finally, if the type is heap allocated (:c:macro:`Py_TPFLAGS_HEAPTYPE`), the - deallocator should release the owned reference to its type object - (via :c:func:`Py_DECREF`) after - calling the type deallocator. In order to avoid dangling pointers, the - recommended way to achieve this is: + static void + foo_dealloc(PyObject *self) + { + PyObject *exc = PyErr_GetRaisedException(); - .. code-block:: c + if (PyObject_CallFinalizerFromDealloc(self) < 0) { + // self was resurrected. + goto done; + } - static void - foo_dealloc(PyObject *op) - { - PyTypeObject *tp = Py_TYPE(op); - // free references and buffers here - tp->tp_free(op); - Py_DECREF(tp); - } + PyTypeObject *tp = Py_TYPE(self); - .. warning:: + if (tp->tp_flags & Py_TPFLAGS_HAVE_GC) { + PyObject_GC_UnTrack(self); + } - In a garbage collected Python, :c:member:`!tp_dealloc` may be called from - any Python thread, not just the thread which created the object (if the - object becomes part of a refcount cycle, that cycle might be collected by - a garbage collection on any thread). This is not a problem for Python - API calls, since the thread on which :c:member:`!tp_dealloc` is called - with an :term:`attached thread state`. However, if the object being - destroyed in turn destroys objects from some other C or C++ library, care - should be taken to ensure that destroying those objects on the thread - which called :c:member:`!tp_dealloc` will not violate any assumptions of - the library. + // Optional, but convenient to avoid code duplication. + if (tp->tp_clear && tp->tp_clear(self) < 0) { + PyErr_WriteUnraisable(self); + } + + // Any additional destruction goes here. + + tp->tp_free(self); + self = NULL; // In case PyErr_WriteUnraisable() is called below. + + if (tp->tp_flags & Py_TPFLAGS_HEAPTYPE) { + Py_CLEAR(tp); + } + + done: + // Optional, if something was called that might have raised an + // exception. + if (PyErr_Occurred()) { + PyErr_WriteUnraisable(self); + } + PyErr_SetRaisedException(exc); + } + + :c:member:`!tp_dealloc` may be called from + any Python thread, not just the thread which created the object (if the + object becomes part of a refcount cycle, that cycle might be collected by + a garbage collection on any thread). This is not a problem for Python + API calls, since the thread on which :c:member:`!tp_dealloc` is called + with an :term:`attached thread state`. However, if the object being + destroyed in turn destroys objects from some other C library, care + should be taken to ensure that destroying those objects on the thread + which called :c:member:`!tp_dealloc` will not violate any assumptions of + the library. **Inheritance:** This field is inherited by subtypes. + .. seealso:: + + :ref:`life-cycle` for details about how this slot relates to other slots. + .. c:member:: Py_ssize_t PyTypeObject.tp_vectorcall_offset @@ -1137,11 +1183,11 @@ and :c:data:`PyType_Type` effectively act as defaults.) .. c:macro:: Py_TPFLAGS_HAVE_GC This bit is set when the object supports garbage collection. If this bit - is set, instances must be created using :c:macro:`PyObject_GC_New` and - destroyed using :c:func:`PyObject_GC_Del`. More information in section - :ref:`supporting-cycle-detection`. This bit also implies that the - GC-related fields :c:member:`~PyTypeObject.tp_traverse` and :c:member:`~PyTypeObject.tp_clear` are present in - the type object. + is set, memory for new instances (see :c:member:`~PyTypeObject.tp_alloc`) + must be allocated using :c:macro:`PyObject_GC_New` or + :c:func:`PyType_GenericAlloc` and deallocated (see + :c:member:`~PyTypeObject.tp_free`) using :c:func:`PyObject_GC_Del`. More + information in section :ref:`supporting-cycle-detection`. **Inheritance:** @@ -1192,7 +1238,7 @@ and :c:data:`PyType_Type` effectively act as defaults.) .. c:macro:: Py_TPFLAGS_MANAGED_DICT - This bit indicates that instances of the class have a `~object.__dict__` + This bit indicates that instances of the class have a :attr:`~object.__dict__` attribute, and that the space for the dictionary is managed by the VM. If this flag is set, :c:macro:`Py_TPFLAGS_HAVE_GC` should also be set. @@ -1478,6 +1524,11 @@ and :c:data:`PyType_Type` effectively act as defaults.) heap-allocated superclass). If they do not, the type object may not be garbage-collected. + .. note:: + + The :c:member:`~PyTypeObject.tp_traverse` function can be called from any + thread. + .. versionchanged:: 3.9 Heap-allocated types are expected to visit ``Py_TYPE(self)`` in @@ -1497,20 +1548,101 @@ and :c:data:`PyType_Type` effectively act as defaults.) .. c:member:: inquiry PyTypeObject.tp_clear - An optional pointer to a clear function for the garbage collector. This is only - used if the :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit is set. The signature is:: + An optional pointer to a clear function. The signature is:: int tp_clear(PyObject *); - The :c:member:`~PyTypeObject.tp_clear` member function is used to break reference cycles in cyclic - garbage detected by the garbage collector. Taken together, all :c:member:`~PyTypeObject.tp_clear` - functions in the system must combine to break all reference cycles. This is - subtle, and if in any doubt supply a :c:member:`~PyTypeObject.tp_clear` function. For example, - the tuple type does not implement a :c:member:`~PyTypeObject.tp_clear` function, because it's - possible to prove that no reference cycle can be composed entirely of tuples. - Therefore the :c:member:`~PyTypeObject.tp_clear` functions of other types must be sufficient to - break any cycle containing a tuple. This isn't immediately obvious, and there's - rarely a good reason to avoid implementing :c:member:`~PyTypeObject.tp_clear`. + The purpose of this function is to break reference cycles that are causing a + :term:`cyclic isolate` so that the objects can be safely destroyed. A + cleared object is a partially destroyed object; the object is not obligated + to satisfy design invariants held during normal use. + + :c:member:`!tp_clear` does not need to delete references to objects that + can't participate in reference cycles, such as Python strings or Python + integers. However, it may be convenient to clear all references, and write + the type's :c:member:`~PyTypeObject.tp_dealloc` function to invoke + :c:member:`!tp_clear` to avoid code duplication. (Beware that + :c:member:`!tp_clear` might have already been called. Prefer calling + idempotent functions like :c:func:`Py_CLEAR`.) + + Any non-trivial cleanup should be performed in + :c:member:`~PyTypeObject.tp_finalize` instead of :c:member:`!tp_clear`. + + .. note:: + + If :c:member:`!tp_clear` fails to break a reference cycle then the + objects in the :term:`cyclic isolate` may remain indefinitely + uncollectable ("leak"). See :data:`gc.garbage`. + + .. note:: + + Referents (direct and indirect) might have already been cleared; they are + not guaranteed to be in a consistent state. + + .. note:: + + The :c:member:`~PyTypeObject.tp_clear` function can be called from any + thread. + + .. note:: + + An object is not guaranteed to be automatically cleared before its + destructor (:c:member:`~PyTypeObject.tp_dealloc`) is called. + + This function differs from the destructor + (:c:member:`~PyTypeObject.tp_dealloc`) in the following ways: + + * The purpose of clearing an object is to remove references to other objects + that might participate in a reference cycle. The purpose of the + destructor, on the other hand, is a superset: it must release *all* + resources it owns, including references to objects that cannot participate + in a reference cycle (e.g., integers) as well as the object's own memory + (by calling :c:member:`~PyTypeObject.tp_free`). + * When :c:member:`!tp_clear` is called, other objects might still hold + references to the object being cleared. Because of this, + :c:member:`!tp_clear` must not deallocate the object's own memory + (:c:member:`~PyTypeObject.tp_free`). The destructor, on the other hand, + is only called when no (strong) references exist, and as such, must + safely destroy the object itself by deallocating it. + * :c:member:`!tp_clear` might never be automatically called. An object's + destructor, on the other hand, will be automatically called some time + after the object becomes unreachable (i.e., either there are no references + to the object or the object is a member of a :term:`cyclic isolate`). + + No guarantees are made about when, if, or how often Python automatically + clears an object, except: + + * Python will not automatically clear an object if it is reachable, i.e., + there is a reference to it and it is not a member of a :term:`cyclic + isolate`. + * Python will not automatically clear an object if it has not been + automatically finalized (see :c:member:`~PyTypeObject.tp_finalize`). (If + the finalizer resurrected the object, the object may or may not be + automatically finalized again before it is cleared.) + * If an object is a member of a :term:`cyclic isolate`, Python will not + automatically clear it if any member of the cyclic isolate has not yet + been automatically finalized (:c:member:`~PyTypeObject.tp_finalize`). + * Python will not destroy an object until after any automatic calls to its + :c:member:`!tp_clear` function have returned. This ensures that the act + of breaking a reference cycle does not invalidate the ``self`` pointer + while :c:member:`!tp_clear` is still executing. + * Python will not automatically call :c:member:`!tp_clear` multiple times + concurrently. + + CPython currently only automatically clears objects as needed to break + reference cycles in a :term:`cyclic isolate`, but future versions might + clear objects regularly before their destruction. + + Taken together, all :c:member:`~PyTypeObject.tp_clear` functions in the + system must combine to break all reference cycles. This is subtle, and if + in any doubt supply a :c:member:`~PyTypeObject.tp_clear` function. For + example, the tuple type does not implement a + :c:member:`~PyTypeObject.tp_clear` function, because it's possible to prove + that no reference cycle can be composed entirely of tuples. Therefore the + :c:member:`~PyTypeObject.tp_clear` functions of other types are responsible + for breaking any cycle containing a tuple. This isn't immediately obvious, + and there's rarely a good reason to avoid implementing + :c:member:`~PyTypeObject.tp_clear`. Implementations of :c:member:`~PyTypeObject.tp_clear` should drop the instance's references to those of its members that may be Python objects, and set its pointers to those @@ -1545,18 +1677,6 @@ and :c:data:`PyType_Type` effectively act as defaults.) PyObject_ClearManagedDict((PyObject*)self); - Note that :c:member:`~PyTypeObject.tp_clear` is not *always* called - before an instance is deallocated. For example, when reference counting - is enough to determine that an object is no longer used, the cyclic garbage - collector is not involved and :c:member:`~PyTypeObject.tp_dealloc` is - called directly. - - Because the goal of :c:member:`~PyTypeObject.tp_clear` functions is to break reference cycles, - it's not necessary to clear contained objects like Python strings or Python - integers, which can't participate in reference cycles. On the other hand, it may - be convenient to clear all contained Python objects, and write the type's - :c:member:`~PyTypeObject.tp_dealloc` function to invoke :c:member:`~PyTypeObject.tp_clear`. - More information about Python's garbage collection scheme can be found in section :ref:`supporting-cycle-detection`. @@ -1569,6 +1689,10 @@ and :c:data:`PyType_Type` effectively act as defaults.) :c:member:`~PyTypeObject.tp_clear` are all inherited from the base type if they are all zero in the subtype. + .. seealso:: + + :ref:`life-cycle` for details about how this slot relates to other slots. + .. c:member:: richcmpfunc PyTypeObject.tp_richcompare @@ -1945,18 +2069,17 @@ and :c:data:`PyType_Type` effectively act as defaults.) **Inheritance:** - This field is inherited by static subtypes, but not by dynamic - subtypes (subtypes created by a class statement). + Static subtypes inherit this slot, which will be + :c:func:`PyType_GenericAlloc` if inherited from :class:`object`. + + :ref:`Heap subtypes <heap-types>` do not inherit this slot. **Default:** - For dynamic subtypes, this field is always set to - :c:func:`PyType_GenericAlloc`, to force a standard heap - allocation strategy. + For heap subtypes, this field is always set to + :c:func:`PyType_GenericAlloc`. - For static subtypes, :c:data:`PyBaseObject_Type` uses - :c:func:`PyType_GenericAlloc`. That is the recommended value - for all statically defined types. + For static subtypes, this slot is inherited (see above). .. c:member:: newfunc PyTypeObject.tp_new @@ -2004,20 +2127,27 @@ and :c:data:`PyType_Type` effectively act as defaults.) void tp_free(void *self); - An initializer that is compatible with this signature is :c:func:`PyObject_Free`. + This function must free the memory allocated by + :c:member:`~PyTypeObject.tp_alloc`. **Inheritance:** - This field is inherited by static subtypes, but not by dynamic - subtypes (subtypes created by a class statement) + Static subtypes inherit this slot, which will be :c:func:`PyObject_Free` if + inherited from :class:`object`. Exception: If the type supports garbage + collection (i.e., the :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set in + :c:member:`~PyTypeObject.tp_flags`) and it would inherit + :c:func:`PyObject_Free`, then this slot is not inherited but instead defaults + to :c:func:`PyObject_GC_Del`. + + :ref:`Heap subtypes <heap-types>` do not inherit this slot. **Default:** - In dynamic subtypes, this field is set to a deallocator suitable to - match :c:func:`PyType_GenericAlloc` and the value of the - :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit. + For :ref:`heap subtypes <heap-types>`, this slot defaults to a deallocator suitable to match + :c:func:`PyType_GenericAlloc` and the value of the + :c:macro:`Py_TPFLAGS_HAVE_GC` flag. - For static subtypes, :c:data:`PyBaseObject_Type` uses :c:func:`PyObject_Free`. + For static subtypes, this slot is inherited (see above). .. c:member:: inquiry PyTypeObject.tp_is_gc @@ -2144,29 +2274,138 @@ and :c:data:`PyType_Type` effectively act as defaults.) .. c:member:: destructor PyTypeObject.tp_finalize - An optional pointer to an instance finalization function. Its signature is:: + An optional pointer to an instance finalization function. This is the C + implementation of the :meth:`~object.__del__` special method. Its signature + is:: void tp_finalize(PyObject *self); - If :c:member:`~PyTypeObject.tp_finalize` is set, the interpreter calls it once when - finalizing an instance. It is called either from the garbage - collector (if the instance is part of an isolated reference cycle) or - just before the object is deallocated. Either way, it is guaranteed - to be called before attempting to break reference cycles, ensuring - that it finds the object in a sane state. + The primary purpose of finalization is to perform any non-trivial cleanup + that must be performed before the object is destroyed, while the object and + any other objects it directly or indirectly references are still in a + consistent state. The finalizer is allowed to execute + arbitrary Python code. + + Before Python automatically finalizes an object, some of the object's direct + or indirect referents might have themselves been automatically finalized. + However, none of the referents will have been automatically cleared + (:c:member:`~PyTypeObject.tp_clear`) yet. + + Other non-finalized objects might still be using a finalized object, so the + finalizer must leave the object in a sane state (e.g., invariants are still + met). + + .. note:: + + After Python automatically finalizes an object, Python might start + automatically clearing (:c:member:`~PyTypeObject.tp_clear`) the object + and its referents (direct and indirect). Cleared objects are not + guaranteed to be in a consistent state; a finalized object must be able + to tolerate cleared referents. + + .. note:: + + An object is not guaranteed to be automatically finalized before its + destructor (:c:member:`~PyTypeObject.tp_dealloc`) is called. It is + recommended to call :c:func:`PyObject_CallFinalizerFromDealloc` at the + beginning of :c:member:`!tp_dealloc` to guarantee that the object is + always finalized before destruction. + + .. note:: + + The :c:member:`~PyTypeObject.tp_finalize` function can be called from any + thread, although the :term:`GIL` will be held. - :c:member:`~PyTypeObject.tp_finalize` should not mutate the current exception status; - therefore, a recommended way to write a non-trivial finalizer is:: + .. note:: + + The :c:member:`!tp_finalize` function can be called during shutdown, + after some global variables have been deleted. See the documentation of + the :meth:`~object.__del__` method for details. + + When Python finalizes an object, it behaves like the following algorithm: + + #. Python might mark the object as *finalized*. Currently, Python always + marks objects whose type supports garbage collection (i.e., the + :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set in + :c:member:`~PyTypeObject.tp_flags`) and never marks other types of + objects; this might change in a future version. + #. If the object is not marked as *finalized* and its + :c:member:`!tp_finalize` finalizer function is non-``NULL``, the + finalizer function is called. + #. If the finalizer function was called and the finalizer made the object + reachable (i.e., there is a reference to the object and it is not a + member of a :term:`cyclic isolate`), then the finalizer is said to have + *resurrected* the object. It is unspecified whether the finalizer can + also resurrect the object by adding a new reference to the object that + does not make it reachable, i.e., the object is (still) a member of a + cyclic isolate. + #. If the finalizer resurrected the object, the object's pending destruction + is canceled and the object's *finalized* mark might be removed if + present. Currently, Python never removes the *finalized* mark; this + might change in a future version. + + *Automatic finalization* refers to any finalization performed by Python + except via calls to :c:func:`PyObject_CallFinalizer` or + :c:func:`PyObject_CallFinalizerFromDealloc`. No guarantees are made about + when, if, or how often an object is automatically finalized, except: + + * Python will not automatically finalize an object if it is reachable, i.e., + there is a reference to it and it is not a member of a :term:`cyclic + isolate`. + * Python will not automatically finalize an object if finalizing it would + not mark the object as *finalized*. Currently, this applies to objects + whose type does not support garbage collection, i.e., the + :c:macro:`Py_TPFLAGS_HAVE_GC` flag is not set. Such objects can still be + manually finalized by calling :c:func:`PyObject_CallFinalizer` or + :c:func:`PyObject_CallFinalizerFromDealloc`. + * Python will not automatically finalize any two members of a :term:`cyclic + isolate` concurrently. + * Python will not automatically finalize an object after it has + automatically cleared (:c:member:`~PyTypeObject.tp_clear`) the object. + * If an object is a member of a :term:`cyclic isolate`, Python will not + automatically finalize it after automatically clearing (see + :c:member:`~PyTypeObject.tp_clear`) any other member. + * Python will automatically finalize every member of a :term:`cyclic + isolate` before it automatically clears (see + :c:member:`~PyTypeObject.tp_clear`) any of them. + * If Python is going to automatically clear an object + (:c:member:`~PyTypeObject.tp_clear`), it will automatically finalize the + object first. + + Python currently only automatically finalizes objects that are members of a + :term:`cyclic isolate`, but future versions might finalize objects regularly + before their destruction. + + To manually finalize an object, do not call this function directly; call + :c:func:`PyObject_CallFinalizer` or + :c:func:`PyObject_CallFinalizerFromDealloc` instead. + + :c:member:`~PyTypeObject.tp_finalize` should leave the current exception + status unchanged. The recommended way to write a non-trivial finalizer is + to back up the exception at the beginning by calling + :c:func:`PyErr_GetRaisedException` and restore the exception at the end by + calling :c:func:`PyErr_SetRaisedException`. If an exception is encountered + in the middle of the finalizer, log and clear it with + :c:func:`PyErr_WriteUnraisable` or :c:func:`PyErr_FormatUnraisable`. For + example:: static void - local_finalize(PyObject *self) + foo_finalize(PyObject *self) { - /* Save the current exception, if any. */ + // Save the current exception, if any. PyObject *exc = PyErr_GetRaisedException(); - /* ... */ + // ... - /* Restore the saved exception. */ + if (do_something_that_might_raise() != success_indicator) { + PyErr_WriteUnraisable(self); + goto done; + } + + done: + // Restore the saved exception. This silently discards any exception + // raised above, so be sure to call PyErr_WriteUnraisable first if + // necessary. PyErr_SetRaisedException(exc); } @@ -2182,7 +2421,13 @@ and :c:data:`PyType_Type` effectively act as defaults.) :c:macro:`Py_TPFLAGS_HAVE_FINALIZE` flags bit in order for this field to be used. This is no longer required. - .. seealso:: "Safe object finalization" (:pep:`442`) + .. seealso:: + + * :pep:`442`: "Safe object finalization" + * :ref:`life-cycle` for details about how this slot relates to other + slots. + * :c:func:`PyObject_CallFinalizer` + * :c:func:`PyObject_CallFinalizerFromDealloc` .. c:member:: vectorcallfunc PyTypeObject.tp_vectorcall diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index 95987e872ce..45f50ba5f97 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -645,6 +645,17 @@ APIs: difference being that it decrements the reference count of *right* by one. +.. c:function:: PyObject* PyUnicode_BuildEncodingMap(PyObject* string) + + Return a mapping suitable for decoding a custom single-byte encoding. + Given a Unicode string *string* of up to 256 characters representing an encoding + table, returns either a compact internal mapping object or a dictionary + mapping character ordinals to byte values. Raises a :exc:`TypeError` and + return ``NULL`` on invalid input. + + .. versionadded:: 3.2 + + .. c:function:: const char* PyUnicode_GetDefaultEncoding(void) Return the name of the default string encoding, ``"utf-8"``. @@ -1450,10 +1461,6 @@ the user settings on the machine running the codec. .. versionadded:: 3.3 -Methods & Slots -""""""""""""""" - - .. _unicodemethodsandslots: Methods and Slot Functions @@ -1715,10 +1722,6 @@ They all return ``NULL`` or ``-1`` if an exception occurs. from user input, prefer calling :c:func:`PyUnicode_FromString` and :c:func:`PyUnicode_InternInPlace` directly. - .. impl-detail:: - - Strings interned this way are made :term:`immortal`. - .. c:function:: unsigned int PyUnicode_CHECK_INTERNED(PyObject *str) @@ -1795,9 +1798,24 @@ object. See also :c:func:`PyUnicodeWriter_DecodeUTF8Stateful`. +.. c:function:: int PyUnicodeWriter_WriteASCII(PyUnicodeWriter *writer, const char *str, Py_ssize_t size) + + Write the ASCII string *str* into *writer*. + + *size* is the string length in bytes. If *size* is equal to ``-1``, call + ``strlen(str)`` to get the string length. + + *str* must only contain ASCII characters. The behavior is undefined if + *str* contains non-ASCII characters. + + On success, return ``0``. + On error, set an exception, leave the writer unchanged, and return ``-1``. + + .. versionadded:: next + .. c:function:: int PyUnicodeWriter_WriteWideChar(PyUnicodeWriter *writer, const wchar_t *str, Py_ssize_t size) - Writer the wide string *str* into *writer*. + Write the wide string *str* into *writer*. *size* is a number of wide characters. If *size* is equal to ``-1``, call ``wcslen(str)`` to get the string length. |
