aboutsummaryrefslogtreecommitdiffstatshomepage
path: root/Python/clinic/Python-tokenize.c.h
Commit message (Collapse)AuthorAge
* gh-107603: Argument Clinic: Only include pycore_gc.h if needed (#108726)Victor Stinner2023-08-31
| | | | | | | | | | | | | | | | | | | | Argument Clinic now only includes pycore_gc.h if PyGC_Head is needed, and only includes pycore_runtime.h if _Py_ID() is needed. * Add 'condition' optional argument to Clinic.add_include(). * deprecate_keyword_use() includes pycore_runtime.h when using the _PyID() function. * Fix rendering of includes: comments start at the column 35. * Mark PC/clinic/_wmimodule.cpp.h and "Objects/stringlib/clinic/*.h.h" header files as generated in .gitattributes. Effects: * 42 header files generated by AC no longer include the internal C API, instead of 4 header files before. For example, Modules/clinic/_abc.c.h no longer includes the internal C API. * Fix _testclinic_depr.c.h: it now always includes pycore_runtime.h to get _Py_ID().
* gh-105069: Add a readline-like callable to the tokenizer to consume input ↵Pablo Galindo Salgado2023-05-30
| | | | iteratively (#105070)
* gh-102856: Python tokenizer implementation for PEP 701 (#104323)Marta Gómez Macías2023-05-21
| | | | | | | | | | | This commit replaces the Python implementation of the tokenize module with an implementation that reuses the real C tokenizer via a private extension module. The tokenize module now implements a compatibility layer that transforms tokens from the C tokenizer into Python tokenize tokens for backward compatibility. As the C tokenizer does not emit some tokens that the Python tokenizer provides (such as comments and non-semantic newlines), a new special mode has been added to the C tokenizer mode that currently is only used via the extension module that exposes it to the Python layer. This new mode forces the C tokenizer to emit these new extra tokens and add the appropriate metadata that is needed to match the old Python implementation. Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
* gh-90928: Improve static initialization of keywords tuple in AC (#95907)Erlend E. Aasland2022-08-13
|
* gh-90928: Statically Initialize the Keywords Tuple in Clinic-Generated Code ↵Eric Snow2022-08-11
| | | | | | | | | | | | | | | | (gh-95860) We only statically initialize for core code and builtin modules. Extension modules still create the tuple at runtime. We'll solve that part of interpreter isolation separately. This change includes generated code. The non-generated changes are in: * Tools/clinic/clinic.py * Python/getargs.c * Include/cpython/modsupport.h * Makefile.pre.in (re-generate global strings after running clinic) * very minor tweaks to Modules/_codecsmodule.c and Python/Python-tokenize.c All other changes are generated code (clinic, global strings).
* Add tests for the C tokenizer and expose it as a private module (GH-27924)Pablo Galindo Salgado2021-08-24