summaryrefslogtreecommitdiffstatshomepage
path: root/py/parse.c
Commit message (Collapse)AuthorAge
...
* py: Create str/bytes objects in the parser, not the compiler.Damien George2017-02-24
| | | | | | | | | | | | | | | | | | Previous to this patch any non-interned str/bytes objects would create a special parse node that held a copy of the str/bytes data. Then in the compiler this data would be turned into a str/bytes object. This actually lead to 2 copies of the data, one in the parse node and one in the object. The parse node's copy of the data would be freed at the end of the compile stage but nevertheless it meant that the peak memory usage of the parse/compile stage was higher than it needed to be (by an amount equal to the number of bytes in all the non-interned str/bytes objects). This patch changes the behaviour so that str/bytes objects are created directly in the parser and the object stored in a const-object parse node (which already exists for bignum, float and complex const objects). This reduces peak RAM usage of the parse/compile stage, simplifies the parser and compiler, and reduces code size by about 170 bytes on Thumb2 archs, and by about 300 bytes on Xtensa archs.
* py/parse: Allow parser/compiler consts to be bignums.Damien George2017-02-24
| | | | | | | | | | | This patch allows uPy consts to be bignums, eg: X = const(1 << 100) The infrastructure for consts to be a bignum (rather than restricted to small integers) has been in place for a while, ever since constant folding was upgraded to allow bignums. It just required a small change (in this patch) to enable it.
* py/grammar: Group no-compile grammar rules together to shrink tables.Damien George2017-02-16
| | | | | | | | | | | | | | | | | | | | | | Grammar rules have 2 variants: ones that are attached to a specific compile function which is called to compile that grammar node, and ones that don't have a compile function and are instead just inspected to see what form they take. In the compiler there is a table of all grammar rules, with each entry having a pointer to the associated compile function. Those rules with no compile function have a null pointer. There are 120 such rules, so that's 120 words of essentially wasted code space. By grouping together the compile vs no-compile rules we can put all the no-compile rules at the end of the list of rules, and then we don't need to store the null pointers. We just have a truncated table and it's guaranteed that when indexing this table we only index the first half, the half with populated pointers. This patch implements such a grouping by having a specific macro for the compile vs no-compile grammar rules (DEF_RULE vs DEF_RULE_NC). It saves around 460 bytes of code on 32-bit archs.
* py/parse: Refactor code to remove assert(0)'s.Damien George2017-01-17
| | | | | This helps to improve code coverage. Note that most of the changes in this patch are just de-denting the cases of the switch statements.
* py/parse: Add code to fold logical constants in or/and/not operations.Damien George2016-11-15
| | | | Adds about 200 bytes to the code size when constant folding is enabled.
* py/parse: Make mp_parse_node_new_leaf an inline function.Damien George2016-11-15
| | | | | | It is split into 2 functions, one to make small ints and the other to make a non-small-int leaf node. This reduces code size by 32 bytes on bare-arm, 64 bytes on unix (x64-64) and 144 bytes on stmhal.
* py/parse: Move function to check for const parse node to parse.[ch].Damien George2016-11-15
|
* py: Fix wrong assumption that m_renew will not move if shrinkingColin Hogben2016-11-02
| | | | | | | | | | | | | | | | In both parse.c and qstr.c, an internal chunking allocator tidies up by calling m_renew to shrink an allocated chunk to the size used, and assumes that the chunk will not move. However, when MICROPY_ENABLE_GC is false, m_renew calls the system realloc, which does not guarantee this behaviour. Environments where realloc may return a different pointer include: (1) mbed-os with MBED_HEAP_STATS_ENABLED (which adds a wrapper around malloc & friends; this is where I was hit by the bug); (2) valgrind on linux (how I diagnosed it). The fix is to call m_renew_maybe with allow_move=false.
* py/parse: Only replace constants that are standalone identifiers.Damien George2016-09-23
| | | | | | This fixes constant substitution so that only standalone identifiers are replaced with their constant value (if they have one). I.e. don't replace NAME in expressions like obj.NAME or NAME = expr.
* py/parse: Treat constants that start with underscore as private.Damien George2016-06-06
| | | | | | | | Assignments of the form "_id = const(value)" are treated as private (following a similar CPython convention) and code is no longer emitted for the assignment to a global variable. See issue #2111.
* py: Declare constant data as properly constant.Damien George2016-05-20
| | | | | Otherwise some compilers (eg without optimisation) will put this read-only data in RAM instead of ROM.
* py/parse: Add uerrno to list of modules to look for constants in.Damien George2016-05-10
|
* py: Simplify "and" action within parser by making ident-rules explicit.Damien George2016-04-14
| | | | | | | | | | Most grammar rules can optimise to the identity if they only have a single argument, saving a lot of RAM building the parse tree. Previous to this patch, whether a given grammar rule could be optimised was defined (mostly implicitly) by a complicated set of logic rules. With this patch the definition is always specified explicitly by using "and_ident" in the rule definition in the grammar. This simplifies the logic of the parser, making it a bit smaller and faster. RAM usage in unaffected.
* py: Fix constant folding and inline-asm to work with new async grammar.Damien George2016-04-13
|
* py/parse: When looking up consts, check they exist before checking type.Damien George2016-03-19
|
* py/parse: Use m_renew_maybe to ensure that memory is shrunk in-place.Damien George2016-02-23
| | | | | | | The chunks of memory that the parser allocates contain parse nodes and are pointed to from many places, so these chunks cannot be relocated by the memory manager. This patch makes it so that when a chunk is shrunk to fit, it is not relocated.
* py: unary_op enum type fix, and a cast to remove clang warningAntonin ENFRUN2016-01-12
|
* py/parse: Include unistd.h for ssize_t definition.Damien George2016-01-08
| | | | In some cases ssize_t is not defined by already included headers.
* py/parse: Improve constant folding to operate on small and big ints.Damien George2016-01-07
| | | | | | | | | | | | | | | Constant folding in the parser can now operate on big ints, whatever their representation. This is now possible because the parser can create parse nodes holding arbitrary objects. For the case of small ints the folding is still efficient in RAM because the folded small int is stored inplace in the parse node. Adds 48 bytes to code size on Thumb2 architecture. Helps reduce heap usage because more constants can be computed at compile time, leading to a smaller parse tree, and most importantly means that the constants don't have to be computed at runtime (perhaps more than once). Parser will now be a little slower when folding due to calls to runtime to do the arithmetic.
* py/parse: Optimise away parse node that's just parenthesis around expr.Damien George2016-01-07
| | | | | | | | | | Before this patch, (x+y)*z would be parsed to a tree that contained a redundant identity parse node corresponding to the parenthesis. With this patch such nodes are optimised away, which reduces memory requirements for expressions with parenthesis, and simplifies the compiler because it doesn't need to handle this identity case. A parenthesis parse node is still needed for tuples.
* py: Add MICROPY_ENABLE_COMPILER and MICROPY_PY_BUILTINS_EVAL_EXEC opts.Damien George2015-12-18
| | | | | | | | | | | | MICROPY_ENABLE_COMPILER can be used to enable/disable the entire compiler, which is useful when only loading of pre-compiled bytecode is supported. It is enabled by default. MICROPY_PY_BUILTINS_EVAL_EXEC controls support of eval and exec builtin functions. By default they are only included if MICROPY_ENABLE_COMPILER is enabled. Disabling both options saves about 40k of code size on 32-bit x86.
* py/parse: Replace mp_int_t/mp_uint_t with size_t etc, where appropriate.Damien George2015-12-17
|
* py: Add support for 64-bit NaN-boxing object model, on 32-bit machine.Damien George2015-11-29
| | | | | | | | | | | | | To use, put the following in mpconfigport.h: #define MICROPY_OBJ_REPR (MICROPY_OBJ_REPR_D) #define MICROPY_FLOAT_IMPL (MICROPY_FLOAT_IMPL_DOUBLE) typedef int64_t mp_int_t; typedef uint64_t mp_uint_t; #define UINT_FMT "%llu" #define INT_FMT "%lld" Currently does not work with native emitter enabled.
* py: Wrap all obj-ptr conversions in MP_OBJ_TO_PTR/MP_OBJ_FROM_PTR.Damien George2015-11-29
| | | | | | | | | This allows the mp_obj_t type to be configured to something other than a pointer-sized primitive type. This patch also includes additional changes to allow the code to compile when sizeof(mp_uint_t) != sizeof(void*), such as using size_t instead of mp_uint_t, and various casts.
* py: Add MP_ROM_* macros and mp_rom_* types and use them.Damien George2015-11-29
|
* py: Implement default and star args for lambdas.Damien George2015-11-17
|
* py/parse: Make parser error handling cleaner, less spaghetti-like.Damien George2015-10-12
|
* py: Move constant folding from compiler to parser.Damien George2015-10-12
| | | | | | | | | | It makes much more sense to do constant folding in the parser while the parse tree is being built. This eliminates the need to create parse nodes that will just be folded away. The code is slightly simpler and a bit smaller as well. Constant folding now has a configuration option, MICROPY_COMP_CONST_FOLDING, which is enabled by default.
* py/parse: Factor logic when creating parse node from and-rule.Damien George2015-10-08
|
* py: Allocate parse nodes in chunks to reduce fragmentation and RAM use.Damien George2015-10-02
| | | | | | | | With this patch parse nodes are allocated sequentially in chunks. This reduces fragmentation of the heap and prevents waste at the end of individually allocated parse nodes. Saves roughly 20% of RAM during parse stage.
* unix-cpy: Remove unix-cpy. It's no longer needed.Damien George2015-08-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | unix-cpy was originally written to get semantic equivalent with CPython without writing functional tests. When writing the initial implementation of uPy it was a long way between lexer and functional tests, so the half-way test was to make sure that the bytecode was correct. The idea was that if the uPy bytecode matched CPython 1-1 then uPy would be proper Python if the bytecodes acted correctly. And having matching bytecode meant that it was less likely to miss some deep subtlety in the Python semantics that would require an architectural change later on. But that is all history and it no longer makes sense to retain the ability to output CPython bytecode, because: 1. It outputs CPython 3.3 compatible bytecode. CPython's bytecode changes from version to version, and seems to have changed quite a bit in 3.5. There's no point in changing the bytecode output to match CPython anymore. 2. uPy and CPy do different optimisations to the bytecode which makes it harder to match. 3. The bytecode tests are not run. They were never part of Travis and are not run locally anymore. 4. The EMIT_CPYTHON option needs a lot of extra source code which adds heaps of noise, especially in compile.c. 5. Now that there is an extensive test suite (which tests functionality) there is no need to match the bytecode. Some very subtle behaviour is tested with the test suite and passing these tests is a much better way to stay Python-language compliant, rather than trying to match CPy bytecode.
* py/parse: Fix handling of empty input so it raises an exception.Damien George2015-07-24
|
* py/parse: De-duplicate and simplify code for parser "or" rule.Damien George2015-07-24
|
* py: Improve allocation policy of qstr data.Damien George2015-07-14
| | | | | | | | | | | | | | | | | | | | Previous to this patch all interned strings lived in their own malloc'd chunk. On average this wastes N/2 bytes per interned string, where N is the number-of-bytes for a quanta of the memory allocator (16 bytes on 32 bit archs). With this patch interned strings are concatenated into the same malloc'd chunk when possible. Such chunks are enlarged inplace when possible, and shrunk to fit when a new chunk is needed. RAM savings with this patch are highly varied, but should always show an improvement (unless only 3 or 4 strings are interned). New version typically uses about 70% of previous memory for the qstr data, and can lead to savings of around 10% of total memory footprint of a running script. Costs about 120 bytes code size on Thumb2 archs (depends on how many calls to gc_realloc are made).
* py: Clean up some bits and pieces in parser, grammar.Damien George2015-04-21
|
* py: Update parse.c&mpconfig.h to reflect rename of mp_lexer_show_token.nhtshot2015-02-23
| | | | | This function is only used when DEBUG_PRINTERS and USE_RULE_NAME are enabled.
* py: Expose compile.c:list_get as mp_parse_node_extract_list.Damien George2015-02-13
|
* py: Initialise variables in mp_parse correctly, to satisfy gcc warning.Damien George2015-02-08
|
* py: Parse big-int/float/imag constants directly in parser.Damien George2015-02-08
| | | | | | | | | Previous to this patch, a big-int, float or imag constant was interned (made into a qstr) and then parsed at runtime to create an object each time it was needed. This is wasteful in RAM and not efficient. Now, these constants are parsed straight away in the parser and turned into objects. This allows constants with large numbers of digits (so addresses issue #1103) and takes us a step closer to #722.
* py: Protect mp_parse and mp_compile with nlr push/pop block.Damien George2015-02-07
| | | | | | | | | | To enable parsing constants more efficiently, mp_parse should be allowed to raise an exception, and mp_compile can already raise a MemoryError. So these functions need to be protected by an nlr push/pop block. This patch adds that feature in all places. This allows to simplify how mp_parse and mp_compile are called: they now raise an exception if they have an error and so explicit checking is not needed anymore.
* py: Be more machine-portable with size of bit fields.Damien George2015-01-24
|
* py, unix, stmhal: Allow to compile with -Wshadow.Damien George2015-01-20
| | | | See issue #699.
* py, unix: Allow to compile with -Wsign-compare.Damien George2015-01-16
| | | | See issue #699.
* py: Add "default" to switches to allow better code flow analysis.Damien George2015-01-14
| | | | | This helps compiler produce smaller code. Saves 124 bytes on stmhal and bare-arm.
* py: Never intern data of large string/bytes object; add relevant tests.Damien George2015-01-13
| | | | | | | | | | Previously to this patch all constant string/bytes objects were interned by the compiler, and this lead to crashes when the qstr was too long (noticeable now that qstr length storage defaults to 1 byte). With this patch, long string/bytes objects are never interned, and are referenced directly as constant objects within generated code using load_const_obj.
* py: Move to guarded includes, everywhere in py/ core.Damien George2015-01-01
| | | | Addresses issue #1022.
* py: Remove unnecessary RULE_none and PN_none from parser.Damien George2014-12-20
|
* py: Add blank and ident flags to grammar rules to simplify parser.Damien George2014-12-20
| | | | This saves around 100 bytes code space on stmhal, more on unix.
* py: Save a few code bytes in parser; make vars local where possible.Damien George2014-12-20
|
* py: Optimise lexer by exposing lexer type.Damien George2014-12-05
| | | | | | | | | mp_lexer_t type is exposed, mp_token_t type is removed, and simple lexer functions (like checking current token kind) are now inlined. This saves 784 bytes ROM on 32-bit unix, 348 bytes on stmhal, and 460 bytes on bare-arm. It also saves a tiny bit of RAM since mp_lexer_t is a bit smaller. Also will run a bit more efficiently.
generated by cgit v1.2.3 (git 2.39.1) at 2025-05-03 18:35:33 +0000