aboutsummaryrefslogtreecommitdiffstatshomepage
path: root/Python/pystate.c
Commit message (Collapse)AuthorAge
* gh-91048: Refactor and optimize remote debugging module (#134652)Pablo Galindo Salgado13 days
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Completely refactor Modules/_remote_debugging_module.c with improved code organization, replacing scattered reference counting and error handling with centralized goto error paths. This cleanup improves maintainability and reduces code duplication throughout the module while preserving the same external API. Implement memory page caching optimization in Python/remote_debug.h to avoid repeated reads of the same memory regions during debugging operations. The cache stores previously read memory pages and reuses them for subsequent reads, significantly reducing system calls and improving performance. Add code object caching mechanism with a new code_object_generation field in the interpreter state that tracks when code object caches need invalidation. This allows efficient reuse of parsed code object metadata and eliminates redundant processing of the same code objects across debugging sessions. Optimize memory operations by replacing multiple individual structure copies with single bulk reads for the same data structures. This reduces the number of memory operations and system calls required to gather debugging information from the target process. Update Makefile.pre.in to include Python/remote_debug.h in the headers list, ensuring that changes to the remote debugging header force proper recompilation of dependent modules and maintain build consistency across the codebase. Also, make the module compatible with the free threading build as an extra :) Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* gh-131185: Use a proper thread-local for cached thread states (gh-132510)Peter Bierma2025-05-21
| | | | | Switches over to a _Py_thread_local in place of autoTssKey, and also fixes a few other checks regarding PyGILState_Ensure after finalization. Note that this doesn't fix concurrent use of PyGILState_Ensure with Py_Finalize; I'm pretty sure zapthreads doesn't work at all, and that needs to be fixed seperately.
* Simplify interp_look_up_id() (#134257)Victor Stinner2025-05-19
| | | | Don't use PyInterpreterState_GetID() but get directly the interpreter 'id' member which cannot fail.
* gh-134144: Fix use-after-free in zapthreads() (#134145)b-pass2025-05-18
|
* GH-133231: Changes to executor management to support proposed `sys._jit` ↵Mark Shannon2025-05-04
| | | | | | | | module (GH-133287) * Track the current executor, not the previous one, on the thread-state. * Batch executors for deallocation to avoid having to constantly incref executors; this is an ad-hoc form of deferred reference counting.
* GH-124715: Move trashcan mechanism into `Py_Dealloc` (GH-132280)Mark Shannon2025-04-30
|
* GH-132508: Use tagged integers on the evaluation stack for the last ↵Mark Shannon2025-04-29
| | | | instruction offset (GH-132545)
* gh-133079: Remove Py_C_RECURSION_LIMIT & PyThreadState.c_recursion_remaining ↵Petr Viktorin2025-04-29
| | | | | | | | (GH-133080) Both were added in 3.13, are undocumented, and don't make sense in 3.14 due to changes in the stack overflow detection machinery (gh-112282). PEP 387 exception for skipping a deprecation period: https://github.com/python/steering-council/issues/288
* gh-132775: Drop PyUnstable_InterpreterState_GetMainModule() (gh-132978)Eric Snow2025-04-28
| | | | | We replace it with _Py_GetMainModule(), and add _Py_CheckMainModule(), but both in the internal-only C-API. We also add _PyImport_GetModulesRef(), which is the equivalent of _PyImport_GetModules(), but which increfs before the lock is released. This is used by a later change related to pickle and handling __main__.
* gh-132399: ensure correct alignment of `PyInterpreterState` (#132428)Bénédikt Tran2025-04-19
|
* gh-131238: Add pycore_interpframe_structs.h header (#131553)Victor Stinner2025-03-21
| | | | Add an explicit include to pycore_interpframe_structs.h in pycore_runtime_structs.h to fix a dependency cycle.
* gh-131238: Remove includes from pycore_interp.h (#131495)Victor Stinner2025-03-20
| | | Remove also now unused includes in C files.
* gh-131238: Remove more includes from pycore_interp.h (#131480)Victor Stinner2025-03-19
|
* gh-131238: Remove many includes from pycore_interp.h (#131472)Victor Stinner2025-03-19
|
* gh-131238: Convert pycore_pystate.h static inline to functions (#131352)Victor Stinner2025-03-17
| | | | | | | | Convert static inline functions to functions: * _Py_IsMainThread() * _PyInterpreterState_Main() * _Py_IsMainInterpreterFinalizing() * _Py_GetMainConfig()
* GH-131238: Core header refactor (GH-131250)Mark Shannon2025-03-17
| | | | | * Moves most structs in pycore_ header files into pycore_structs.h and pycore_runtime_structs.h * Removes many cross-header dependencies
* gh-124878: Fix race conditions during interpreter finalization (#130649)Sam Gross2025-03-06
| | | | | | | | | | | | | | | | | | | | The PyThreadState field gains a reference count field to avoid issues with PyThreadState being a dangling pointer to freed memory. The refcount starts with a value of two: one reference is owned by the interpreter's linked list of thread states and one reference is owned by the OS thread. The reference count is decremented when the thread state is removed from the interpreter's linked list and before the OS thread calls `PyThread_hang_thread()`. The thread that decrements it to zero frees the `PyThreadState` memory. The `holds_gil` field is moved out of the `_status` bit field, to avoid a data race where on thread calls `PyThreadState_Clear()`, modifying the `_status` bit field while the OS thread reads `holds_gil` when attempting to acquire the GIL. The `PyThreadState.state` field now has `_Py_THREAD_SHUTTING_DOWN` as a possible value. This corresponds to the `_PyThreadState_MustExit()` check. This avoids race conditions in the free threading build when checking `_PyThreadState_MustExit()`.
* GH-127705: better double free message. (GH-130785)Mark Shannon2025-03-05
| | | | | * Add location information when accessing already closed stackref * Add #def option to track closed stackrefs to provide precise information for use after free and double frees.
* gh-130091: Reorder `_PyThreadState_Attach` to avoid data race (gh-130092)Sam Gross2025-02-27
| | | | | | | | | | | | This moves `tstate_activate()` down to avoid a data race in the free threading build on the `_PyRuntime`'s thread-local `autoTSSkey`. This key is deleted during runtime finalization, which may happen concurrently with a call to `_PyThreadState_Attach`. The earlier `tstate_try/wait_attach` ensures that the thread is blocked before it attempts to access the deleted `autoTSSkey`. This fixes a TSAN reported data race in `test_threading.test_import_from_another_thread`.
* gh-130421: Fix data race on timebase initialization (gh-130592)Sam Gross2025-02-27
| | | | | | Windows and macOS require precomputing a "timebase" in order to convert OS timestamps into nanoseconds. Retrieve and compute this value during runtime initialization to avoid data races when accessing the time.
* GH-130396: Use computed stack limits on linux (GH-130398)Mark Shannon2025-02-25
| | | | | | | | | | | * Implement C recursion protection with limit pointers for Linux, MacOS and Windows * Remove calls to PyOS_CheckStack * Add stack protection to parser * Make tests more robust to low stacks * Improve error messages for stack overflow
* GH-91079: Revert "GH-91079: Implement C stack limits using addresses, not ↵Petr Viktorin2025-02-24
| | | | | | | | | counters. (GH-130007)" for now (GH130413) Revert "GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)" for now Unfortunatlely, the change broke some buildbots. This reverts commit 2498c22fa0a2b560491bc503fa676585c1a603d0.
* gh-111924: Fix data races when swapping allocators (gh-130287)Sam Gross2025-02-20
| | | | | | | | | | | | | | | CPython current temporarily changes `PYMEM_DOMAIN_RAW` to the default allocator during initialization and shutdown. The motivation is to ensure that core runtime structures are allocated and freed using the same allocator. However, modifying the current allocator changes global state and is not thread-safe even with the GIL. Other threads may be allocating or freeing objects use PYMEM_DOMAIN_RAW; they are not required to hold the GIL to call PyMem_RawMalloc/PyMem_RawFree. This adds new internal-only functions like `_PyMem_DefaultRawMalloc` that aren't affected by calls to `PyMem_SetAllocator()`, so they're appropriate for Python runtime initialization and finalization. Use these calls in places where we previously swapped to the default raw allocator.
* GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)Mark Shannon2025-02-19
| | | | | | | | | | | | * Implement C recursion protection with limit pointers * Remove calls to PyOS_CheckStack * Add stack protection to parser * Make tests more robust to low stacks * Improve error messages for stack overflow
* gh-128002: use per threads tasks linked list in asyncio (#128869)Kumar Aditya2025-02-06
| | | Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* GH-126599: Remove the PyOptimizer API (GH-129194)Brandt Bucher2025-01-28
|
* GH-91048: Add utils for capturing async call stack for asyncio programs and ↵Yury Selivanov2025-01-22
| | | | | | | | | | | enable profiling (#124640) Signed-off-by: Pablo Galindo <pablogsal@gmail.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com> Co-authored-by: Kumar Aditya <kumaraditya@python.org> Co-authored-by: Łukasz Langa <lukasz@langa.pl> Co-authored-by: Savannah Ostrowski <savannahostrowski@gmail.com> Co-authored-by: Jacob Coffee <jacob@z7x.org> Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
* gh-129033: Remove _PyInterpreterState_SetConfig() function (#129048)Victor Stinner2025-01-20
| | | | | | Remove _PyInterpreterState_GetConfigCopy() and _PyInterpreterState_SetConfig() private functions. PEP 741 "Python Configuration C API" added a better public C API: PyConfig_Get() and PyConfig_Set().
* gh-128360: Add `_Py_AssertHoldsTstate` as assertion for holding a thread ↵Peter Bierma2025-01-20
| | | | | state (#128361) Co-authored-by: Kumar Aditya <kumaraditya@python.org>
* GH-127705: Add debug mode for `_PyStackRef`s inspired by HPy debug mode ↵Mark Shannon2024-12-20
| | | | (GH-128121)
* gh-115999: Enable specialization of `CALL` instructions in free-threaded ↵mpage2024-12-03
| | | | | | | | | | | | | | | | | | | | | | builds (#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.
* gh-109746: Make _thread.start_new_thread delete state of new thread on its ↵Radislav Chugunov2024-11-22
| | | | | | | | | | startup failure (GH-109761) If Python fails to start newly created thread due to failure of underlying PyThread_start_new_thread() call, its state should be removed from interpreter' thread states list to avoid its double cleanup. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* gh-114940: Add _Py_FOR_EACH_TSTATE_UNLOCKED(), and Friends (gh-127077)Eric Snow2024-11-21
| | | This is a precursor to the actual fix for gh-114940, where we will change these macros to use the new lock. This change is almost entirely mechanical; the exceptions are the loops in codeobject.c and ceval.c, which now hold the "head" lock. Note that almost all of the uses of _Py_FOR_EACH_TSTATE_UNLOCKED() here will change to _Py_FOR_EACH_TSTATE_BEGIN() once we add the new per-interpreter lock.
* gh-121058: Warn if `PyThreadState_Clear` is called with an exception set ↵Peter Bierma2024-11-20
| | | | (gh-121343)
* gh-126914: Store the Preallocated Thread State's Pointer in a ↵Eric Snow2024-11-19
| | | | | PyInterpreterState Field (gh-126989) This approach eliminates the originally reported race. It also gets rid of the deadlock reported in gh-96071, so we can remove the workaround added then.
* gh-126986: Drop _PyInterpreterState_FailIfNotRunning() (gh-126988)Eric Snow2024-11-19
| | | We replace it with _PyErr_SetInterpreterAlreadyRunning().
* gh-76785: Minor Cleanup of "Cross-interpreter" Code (gh-126457)Eric Snow2024-11-07
| | | | | | | | The primary objective here is to allow some later changes to be cleaner. Mostly this involves renaming things and moving a few things around. * CrossInterpreterData -> XIData * crossinterpdatafunc -> xidatafunc * split out pycore_crossinterp_data_registry.h * add _PyXIData_lookup_t
* gh-115999: Implement thread-local bytecode and enable specialization for ↵mpage2024-11-04
| | | | | | | | | `BINARY_OP` (#123926) Each thread specializes a thread-local copy of the bytecode, created on the first RESUME, in free-threaded builds. All copies of the bytecode for a code object are stored in the co_tlbc array on the code object. Threads reserve a globally unique index identifying its copy of the bytecode in all co_tlbc arrays at thread creation and release the index at thread destruction. The first entry in every co_tlbc array always points to the "main" copy of the bytecode that is stored at the end of the code object. This ensures that no bytecode is copied for programs that do not use threads. Thread-local bytecode can be disabled at runtime by providing either -X tlbc=0 or PYTHON_TLBC=0. Disabling thread-local bytecode also disables specialization. Concurrent modifications to the bytecode made by the specializing interpreter and instrumentation use atomics, with specialization taking care not to overwrite an instruction that was instrumented concurrently.
* gh-125286: Share the Main Refchain With Legacy Interpreters (gh-125709)Eric Snow2024-10-23
| | | | | They used to be shared, before 3.12. Returning to sharing them resolves a failure on Py_TRACE_REFS builds. Co-authored-by: Petr Viktorin <encukou@gmail.com>
* gh-125604: Move _Py_AuditHookEntry, etc. Out of pycore_runtime.h (gh-125605)Eric Snow2024-10-18
| | | | | | | | | | | | This is essentially a cleanup, moving a handful of API declarations to the header files where they fit best, creating new ones when needed. We do the following: * add pycore_debug_offsets.h and move _Py_DebugOffsets, etc. there * inline struct _getargs_runtime_state and struct _gilstate_runtime_state in _PyRuntimeState * move struct _reftracer_runtime_state to the existing pycore_object_state.h * add pycore_audit.h and move to it _Py_AuditHookEntry , _PySys_Audit(), and _PySys_ClearAuditHooks * add audit.h and cpython/audit.h and move the existing audit-related API there *move the perfmap/trampoline API from cpython/sysmodule.h to cpython/ceval.h, and remove the now-empty cpython/sysmodule.h
* gh-124218: Use per-thread refcounts for code objects (#125216)Sam Gross2024-10-15
| | | | | | | Use per-thread refcounting for the reference from function objects to their corresponding code object. This can be a source of contention when frequently creating nested functions. Deferred refcounting alone isn't a great fit here because these references are on the heap and may be modified by other libraries.
* gh-111924: use atomics for interp id refcounting (#125321)Kumar Aditya2024-10-12
|
* gh-116750: Add clear_tool_id function to unregister events and callbacks ↵Tian Gao2024-10-01
| | | | (#124568)
* gh-124218: Refactor per-thread reference counting (#124844)Sam Gross2024-10-01
| | | | | | | Currently, we only use per-thread reference counting for heap type objects and the naming reflects that. We will extend it to a few additional types in an upcoming change to avoid scaling bottlenecks when creating nested functions. Rename some of the files and functions in preparation for this change.
* GH-123516: Improve JIT memory consumption by invalidating cold executors ↵Savannah Ostrowski2024-09-27
| | | | | (GH-124443) Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* gh-119333: Add C api to have contextvar enter/exit callbacks (#119335)Jason Fried2024-09-23
| | | Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
* Remove comment from pystate created in 2003 (#123259)Anthony Shaw2024-08-24
|
* Add debug offsets for free threaded builds (#123041)Pablo Galindo Salgado2024-08-15
|
* gh-122697: Fix free-threading memory leaks at shutdown (#122703)Sam Gross2024-08-08
| | | | | | | | | | | | | | | We were not properly accounting for interpreter memory leaks at shutdown and had two sources of leaks: * Objects that use deferred reference counting and were reachable via static types outlive the final GC. We now disable deferred reference counting on all objects if we are calling the GC due to interpreter shutdown. * `_PyMem_FreeDelayed` did not properly check for interpreter shutdown so we had some memory blocks that were enqueued to be freed, but never actually freed. * `_PyType_FinalizeIdPool` wasn't called at interpreter shutdown.
* gh-122417: Implement per-thread heap type refcounts (#122418)Sam Gross2024-08-06
| | | | | | | The free-threaded build partially stores heap type reference counts in distributed manner in per-thread arrays. This avoids reference count contention when creating or destroying instances. Co-authored-by: Ken Jin <kenjin@python.org>
generated by cgit v1.2.3 (git 2.39.1) at 2025-06-07 20:51:28 +0000