aboutsummaryrefslogtreecommitdiffstatshomepage
path: root/Lib/test/_test_multiprocessing.py
Commit message (Collapse)AuthorAge
* gh-134381: Fix RuntimeError when starting not-yet started Thread after fork ↵Jiucheng(Oliver)10 days
| | | | (gh-134514)
* gh-133744: Fix multiprocessing interrupt test: add an event (#133746)Victor Stinner2025-05-12
| | | | Add an event to synchronize the parent process with the child process: wait until the child process starts sleeping.
* gh-133017: Improve error message for invalid typecodes in ↵Tomas R.2025-05-09
| | | | multiprocessing.{Array,Value} (GH-133252)
* gh-132719: Skip multiprocessing test if missing ctypes (#132846)Victor Stinner2025-04-23
| | | | Skip test_rlock_locked_2processes() if multiprocessing.sharedctypes is missing (need ctypes).
* gh-132719: Skip multiprocessing test if missing ctypes (#132842)Victor Stinner2025-04-23
| | | | Skip test_lock_locked_2processes() if multiprocessing.sharedctypes is missing (need ctypes).
* gh-131913: multiprocessing: add interrupt for POSIX (GH-132453)pulkin2025-04-22
| | | | | | * multiprocessing: interrupt Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* gh-132719: Fix AMD64 FreeBSD14/15 3.x failures - ↵Duprat2025-04-21
| | | | | `test_rlock_locked_2processes` used an unknown `Value` (GH-132774) * Fix creation of resvariable
* gh-132561: Fix the public `multiprocessing.SemLock.locked` method (#132586)Duprat2025-04-17
| | | Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
* gh-115942: Add `locked` to several multiprocessing locks (#115944)sobolevn2025-04-08
| | | | Co-authored-by: mpage <mpage@cs.stanford.edu> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
* gh-130954: Fix multiprocessing test_notify_n (#130955)Sam Gross2025-03-08
| | | | | | The test could deadlock trying join on the worker processes. Apply the same technique as gh-130933. Join the process before the test ends in `test_notify` as well.
* gh-110206: Fix multiprocessing test_notify_all (#130933)Sam Gross2025-03-07
| | | | | | | | | | | | | | | | | | | | | | | | The test could deadlock trying join on the worker processes due to a combination of behaviors: * The use of `assertReachesEventually` did not ensure that workers actually woken.release() because the SyncManager's Semaphore does not implement get_value. * This mean that the test could finish and the variable "sleeping" would got out of scope and be collected. This unregisters the proxy leading to failures in the worker or possibly the manager. * The subsequent call to `p.join()` during cleanUp therefore never finished. This takes two approaches to fix this: 1) Use woken.acquire() to ensure that the workers actually finish calling woken.release() 2) At the end of the test, wait until the workers are finished, while `cond`, `sleeping`, and `woken` are still valid.
* gh-130730: Fix multiprocessing test_active_children() (#130837)Victor Stinner2025-03-04
| | | | Replace a sleep with an event: sleep is not a reliable synchronization primitive.
* gh-130737: Fix multiprocessing test_notify() (#130797)Victor Stinner2025-03-03
| | | | | | | | Replace hardcoded delay (100 ms) with a loop awaiting until a condition is true: replace assertReturnsIfImplemented() with assertReachesEventually(). Use sleeping_retry() in assertReachesEventually() to tolerate slow buildbots and raise an exception on timeout (30 seconds).
* gh-129401: Test repr rlock failing randomly (#129959)Duprat2025-03-01
| | | | Fix and simplify a test of `test_repr_rlock` about multiprocessing.RLock primitive.
* gh-129948: Add `set()` to `multiprocessing.managers.SyncManager` (#129949)mingyu2025-02-23
| | | | | | | | The SyncManager provided support for various data structures such as dict, list, and queue, but oddly, not set. This introduces support for set by defining SetProxy and registering it with SyncManager. --- Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-71339: Use new assertion methods in the multiprocessing tests (GH-128847)Serhiy Storchaka2025-01-15
|
* gh-127586: multiprocessing.Pool does not properly restore blocked signals ↵Stephen Hansen2024-12-27
| | | | | | | | | | | | | | (try 2) (GH-128011) Correct pthread_sigmask in resource_tracker to restore old signals Using SIG_UNBLOCK to remove blocked "ignored signals" may accidentally cause side effects if the calling parent already had said signals blocked to begin with and did not intend to unblock them when creating a pool. Use SIG_SETMASK instead with the previous mask of blocked signals to restore the original blocked set. Co-authored-by: Peter Bierma <zintensitydev@gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* Revert "gh-127586: properly restore blocked signals in resource_tracker.py ↵Hugo van Kemenade2024-12-16
| | | | | (GH-127587)" (#127983) This reverts commit 46006a1b355f75d06c10e7b8086912c483b34487.
* gh-127586: properly restore blocked signals in resource_tracker.py (GH-127587)Stephen Hansen2024-12-15
| | | | | | | | | | | | | | * Correct pthread_sigmask in resource_tracker to restore old signals Using SIG_UNBLOCK to remove blocked "ignored signals" may accidentally cause side effects if the calling parent already had said signals blocked to begin with and did not intend to unblock them when creating a pool. Use SIG_SETMASK instead with the previous mask of blocked signals to restore the original blocked set. * Adding resource_tracker blocked signals test Co-authored-by: Peter Bierma <zintensitydev@gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-97514: Authenticate the forkserver control socket. (GH-99309)Gregory P. Smith2024-11-20
| | | | | | | | | | | | | | | | | | | This adds authentication to the forkserver control socket. In the past only filesystem permissions protected this socket from code injection into the forkserver process by limiting access to the same UID, which didn't exist when Linux abstract namespace sockets were used (see issue) meaning that any process in the same system network namespace could inject code. We've since stopped using abstract namespace sockets by default, but protecting our control sockets regardless of type is a good idea. This reuses the HMAC based shared key auth already used by `multiprocessing.connection` sockets for other purposes. Doing this is useful so that filesystem permissions are not relied upon and trust isn't implied by default between all processes running as the same UID with access to the unix socket. ### pyperformance benchmarks No significant changes. Including `concurrent_imap` which exercises `multiprocessing.Pool.imap` in that suite. ### Microbenchmarks This does _slightly_ slow down forkserver use. How much so appears to depend on the platform. Modern platforms and simple platforms are less impacted. This PR adds additional IPC round trips to the control socket to tell forkserver to spawn a new process. Systems with potentially high latency IPC are naturally impacted more. Typically a 1-4% slowdown on a very targeted process creation microbenchmark, with a worst case overloaded system slowdown of 20%. No evidence that these slowdowns appear in practical sense. See the PR for details.
* gh-126417: validate ABC methods on multiprocessing proxy types (#126454)Stephen Morton2024-11-10
| | | | | Checks that appropriate dunder __ methods exist on the dict and list proxy types. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* gh-117378: Only run the new multiprocessing SysPath test when appropriate ↵Gregory P. Smith2024-11-10
| | | | | | | | | (GH-126635) The first version had it running two forkserver and one spawn tests underneath each of the _fork, _forkserver, and _spawn test suites that build off the generic one. This adds to the existing complexity of the multiprocessing test suite by offering BaseTestCase classes another attribute to control which suites they are invoked under. Practicality vs purity here. :/ Net result: we don't over-run the new test and their internal logic is simplified.
* gh-117378: Fix multiprocessing forkserver preload sys.path inheritance. ↵Gregory P. Smith2024-11-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | (GH-126538) gh-117378: Fix multiprocessing forkserver preload sys.path inheritance. `sys.path` was not properly being sent from the parent process when launching the multiprocessing forkserver process to preload imports. This bug has been there since the forkserver start method was introduced in Python 3.4. It was always _supposed_ to inherit `sys.path` the same way the spawn method does. Observable behavior change: A `''` value in `sys.path` will now be replaced in the forkserver's `sys.path` with an absolute pathname `os.path.abspath(os.getcwd())` saved at the time that `multiprocessing` was imported in the parent process as it already was when using the spawn start method. **This will only be observable during forkserver preload imports**. The code invoked before calling things in another process already correctly sets `sys.path`. Which is likely why this went unnoticed for so long as a mere performance issue in some configurations. A workaround for the bug on impacted Pythons is to set PYTHONPATH in the environment before multiprocessing's forkserver process was started. Not perfect as that is then inherited by other children, etc, but likely good enough for many people's purposes. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
* gh-125679: multiprocessing Lock and RLock - fix invalid representation ↵Duprat2024-11-07
| | | | string on MacOSX. (#125680)
* gh-126417: Register multiprocessing proxy types to an appropriate ↵Stephen Morton2024-11-05
| | | | collections.abc class (#126419)
* gh-125620: Skip check_resource_tracker_death on NetBSD due to long wait for ↵Furkan Onder2024-10-17
| | | | | SIGKILL process termination (GH-125621) * Skip test_resource_tracker_sigkill on NetBSD
* gh-84559: Change the multiprocessing start method default to `forkserver` ↵Gregory P. Smith2024-09-26
| | | | | | | | (GH-101556) Change the default multiprocessing start method away from fork to forkserver or spawn on the remaining platforms where it was fork. See the issue for context. This makes the default far more thread safe (other than for people spawning threads at import time... - don't do that!). Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
* gh-120012: clarify the behaviour of `multiprocessing.Queue.empty` on closed ↵Bénédikt Tran2024-06-13
| | | | | | | | queues. (GH-120102) * improve doc for `multiprocessing.Queue.empty` * add tests for checking emptiness of queues Co-authored-by: Gregory P. Smith <greg@krypto.org>
* gh-103134: Update multiprocessing.managers.ListProxy and DictProxy (GH-103133)Roy Hyunjin Han2024-05-20
|
* gh-119064: Use os_helper.FakePath instead of pathlib.Path in tests (GH-119065)Serhiy Storchaka2024-05-16
|
* gh-83856: Honor atexit for all multiprocessing start methods (GH-114279)Tian Gao2024-05-03
| | | | | Use atexit for all multiprocessing start methods to cleanup. See the GH-114279 PR discussion and related issue for details as to why.
* gh-118413: Fix test_release_task_refs on free-threaded build (#118494)Sam Gross2024-05-02
| | | | | | The `time.sleep()` call should happen before the GC to give the worker threads time to clean-up their remaining references to objs. Additionally, use `support.gc_collect()` instead of `gc.collect()` just in case the extra GC calls matter.
* gh-118413: Temporarily skip `test_release_task_refs` in free-threaded builds ↵Sam Gross2024-05-01
| | | | (#118491)
* gh-116622: Test updates for Android (#117299)Malcolm Smith2024-04-05
| | | | | | | - re-enable test_fcntl_64_bit on Linux aarch64, but disable it on all Android ABIs - use support.setswitchinterval in all relevant tests - skip test_fma_zero_result on Android x86_64 - accept EACCES when calling os.get_terminal_size on Android
* gh-88118: Fix some test_multiprocessing flakiness. (#116434)Gregory P. Smith2024-03-06
| | | | | | | | | | | | | | | | | | Fix some test_multiprocessing flakiness. Potentially introduced by https://github.com/python/cpython/pull/25845 not joining that thread likely leads to recently observed "environment changed" logically passing but overall failing tests seen on some buildbots similar to: ``` 1 test altered the execution environment (env changed): test.test_multiprocessing_fork.test_processes 2 re-run tests: test.test_multiprocessing_fork.test_processes test.test_multiprocessing_forkserver.test_processes ```
* bpo-43952: Fix multiprocessing Listener authkey bug (GH-25845)Miguel Brito2024-02-27
| | | | Listener.accept() no longer hangs when authkey is an empty bytes object.
* gh-115886: Handle embedded null characters in shared memory name (GH-115887)Serhiy Storchaka2024-02-25
| | | | shm_open() and shm_unlink() now check for embedded null characters in the name and raise an error instead of silently truncating it.
* gh-104090: Add exit code to multiprocessing ResourceTracker (GH-115410)Petr Viktorin2024-02-21
| | | | | | | | | | | | | This builds on https://github.com/python/cpython/pull/106807, which adds a return code to ResourceTracker, to make future debugging easier. Testing this “in situ” proved difficult, since the global ResourceTracker is involved in test infrastructure. So, the tests here create a new instance and feed it fake data. --------- Co-authored-by: Yonatan Bitton <yonatan.bitton@perception-point.io> Co-authored-by: Yonatan Bitton <bityob@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org>
* gh-89240: Enable multiprocessing on Windows to use large process pools ↵Steve Dower2024-02-13
| | | | | | | (GH-107873) We add _winapi.BatchedWaitForMultipleObjects to wait for larger numbers of handles. This is an internal module, hence undocumented, and should be used with caution. Check the docstring for info before using BatchedWaitForMultipleObjects.
* gh-113205: test_multiprocessing.test_terminate: Give tasks a chance to start ↵Serhiy Storchaka2024-01-24
| | | | (GH-114249)
* gh-113205: test_multiprocessing.test_terminate: Test the API on threadpools ↵Petr Viktorin2024-01-17
| | | | | | | | | | | | | | | | (#114186) gh-113205: test_multiprocessing.test_terminate: Test the API works on threadpools Threads can't be forced to terminate (without potentially corrupting too much state), so the expected behaviour of `ThreadPool.terminate` is to wait for the currently executing tasks to finish. The entire test was skipped in GH-110848 (0e9c364f4ac18a2237bdbac702b96bcf8ef9cb09). Instead of skipping it entirely, we should ensure the API eventually succeeds: use a shorter timeout. For the record: on my machine, when the test is un-skipped, the task manages to start in about 1.5% cases.
* gh-113421: Fix multiprocessing logger for "%(filename)s" (GH-113423)Xu Song2023-12-24
|
* gh-82300: Add track parameter to multiprocessing.shared_memory (#110778)pan3242023-12-05
| | | | | | | | | | Add a track parameter to shared memory to allow resource tracking via the side-launched resource tracker process to be disabled on platforms that use it (POSIX). This allows people who do not want automated cleanup at process exit because they are using the shared memory with processes not participating in Python's resource tracking to use the shared_memory API. Co-authored-by: Łukasz Langa <lukasz@langa.pl> Co-authored-by: Guido van Rossum <gvanrossum@gmail.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr> Co-authored-by: Gregory P. Smith <greg@krypto.org>
* GH-110829: Ensure Thread.join() joins the OS thread (#110848)Antoine Pitrou2023-11-04
| | | | | | | Joining a thread now ensures the underlying OS thread has exited. This is required for safer fork() in multi-threaded processes. --------- Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
* gh-111284: Make multiprocessing tests with threads faster and more reliable ↵Serhiy Storchaka2023-10-30
| | | | (GH-111285)
* gh-110662: multiprocessing test_async_timeout() increase timeout (#110663)Victor Stinner2023-10-11
| | | | Increase timeout from 1 second to 30 seconds, if not longer. The important part is that apply_async() takes longer than TIMEOUT2.
* gh-110666: Fix multiprocessing test_terminate() elapsed (#110667)Victor Stinner2023-10-11
| | | | | | multiprocessing test_terminate() and test_wait_socket_slow() no longer test the CI performance: no longer check maximum elapsed time. Add CLOCK_RES constant: tolerate a difference of 100 ms.
* gh-110656: Fix logging test_post_fork_child_no_deadlock() if ASAN (#110657)Victor Stinner2023-10-11
| | | | | Skip test_post_fork_child_no_deadlock() if Python is built with ASAN. Add support.HAVE_ASAN_FORK_BUG.
* gh-109840: Fix multiprocessing test_waitfor_timeout() (#110428)Victor Stinner2023-10-05
| | | | Don't measure the CI performance: don't fail if cond.wait_for() takes longer than 1 second on a slow CI.
* gh-110036: multiprocessing Popen.terminate() catches PermissionError (#110037)Victor Stinner2023-09-29
| | | | | | On Windows, multiprocessing Popen.terminate() now catchs PermissionError and get the process exit code. If the process is still running, raise again the PermissionError. Otherwise, the process terminated as expected: store its exit code.
generated by cgit v1.2.3 (git 2.39.1) at 2025-06-02 21:57:04 +0000