diff options
author | Damien George <damien.p.george@gmail.com> | 2017-08-13 21:24:32 +1000 |
---|---|---|
committer | Damien George <damien.p.george@gmail.com> | 2017-08-13 21:24:32 +1000 |
commit | b084ab966e2d4d18c9b2be8317541313f0e2586f (patch) | |
tree | ee073405e02d4d23c1bdb1e85b0c5a9842e0010b /py | |
parent | 72912c753a31301efb7d6b2f35dd1e3e64b98cd4 (diff) | |
parent | 2c2fc070ecca654b0e9a80e1d9a342d46c6f0cdb (diff) | |
download | micropython-b084ab966e2d4d18c9b2be8317541313f0e2586f.tar.gz micropython-b084ab966e2d4d18c9b2be8317541313f0e2586f.zip |
Merge tag 'v1.8' into parse-bytecode
First general release of ESP8266 port, and support for frozen bytecode
This release marks the first general release of official ESP8266 support
within the MicroPython code base. The ESP8266 port has many improvements
and additions, including: websocket and webrepl modules, deep-sleep mode,
reading on UART, enhanced I2C support, enhanced network configuration,
full sequence of start-up scripts (built-in _boot.py, boot.py and
main.py), improved filesystem support with automatic flash-size detection
as well as documentation and a tutorial.
Known issues with ESP8266 port are:
- Basic SSL support is available but it works only with relatively short
data lengths (a few kilobytes).
- WebREPL over SSL is not supported.
- Only 5 or so WebREPL sessions in succession can be started (including
file transfer sessions), after that the board requires a hard reset.
- File transfer over WebREPL has issues with large files (above a few
tens of kilobytes).
- By design, only one concurrent WebREPL connection is supported.
- Soft reset doesn't reset the network stack.
In addition to ESP8266 support, this release brings frozen bytecode
which allows to compile bytecode offline and link it into the firmware,
completely eliminating the need for compilation (and the associated
RAM usage) at runtime. Basic async/await syntax is now supported, and
qstrs are now auto-generated in the build system.
A detailed list of changes is given below.
README:
- explicitly point to required dependencies section
- promote "docs" and "tests" to "major components"
- mention support "async" keyword from Python 3.5
ACKNOWLEDGEMENTS:
- add list of 842 backers from the ESP8266 campaign
py core:
- modbuiltins: add comment about setting "_" special var
- add async/await/async for/async with syntax
- fix constant folding and inline-asm to work with new async grammar
- emitglue: move typedef of mp_raw_code_t from .c to .h file
- emitglue: make mp_raw_code_t* arguments constant pointers
- makeqstrdata: factor out some code to functions that can be reused
- add ability to have frozen persistent bytecode from .mpy files
- makeqstrdata: add more names for escaped chars and esc non-printable
- simplify "and" action within parser by making ident-rules explicit
- makeqstrdata: fix rendering of qstrs that have non-printable ASCII
- makeqstrdata: add special case to handle \n qstr
- declare help, input, open builtins in core
- map: change hash-table allocation policy to be less aggressive
- makeqstrdefs: add script to automate extraction of qstr from sources
- frozenmod: pass the source name of the frozen module to the lexer
- rework QSTR extraction to work in simple and obvious way
- divide "split" and "cat" phases of qstr extraction for better efficiency
- fix bug passing a string as a keyword arg in a dict
- move call_function_*_protected() functions to py/ for reuse
- makeqstrdefs.py: windows compatibility
- obj.h: when constructing a small-int cast to mp_uint_t for bit-shift
- emitnative: use MP_OBJ_NEW_SMALL_INT instead of manual bit shifting
- vm: "yield from" didn't handle MP_OBJ_STOP_ITERATION optimization
- modio: rename module name to "uio" for consistency with other modules
- modcollections: rename module name have "u" prefix for consistency
extmod:
- add initial framebuf module
- add generic machine.I2C class, with bit-bang I2C
- machine_i2c: fix I2C reading by sending ack/nack at end of byte
- machine_i2c: implement I2C memory reading/writing
- moduos_dupterm: don't swallow exceptions in dupterm's read()/write()
- modlwip: lwip_tcp_receive(): Properly handle EOF for non-blocking sock
- modlwip: more debug messages for various edge conditions
- fsusermount: in mount/mkfs, deregister VFS object on error
- modlwip: lwip_tcp_send(): full error handling
- modlwip: lwip_tcp_receive(): full error handling
- modlwip: add ability to run callback on "recv" and "accept" events
- machine_i2c: allow mp_hal_pin_obj_t to be any type, not a ptr
- modlwip: protect recv/accept Python callback against exceptions
- modlwip: workaround esp8266 sendto issue where 1 is returned
- modlwip: add print_pcbs() debug function
- modwebsocket: handle CLOSE control frame
- modussl: throw Python exceptions in case of errors
- modussl: support server-side SSL sockets
- modussl: SSL_OK from ssl_read() means "no user data so far"
- modwebrepl: module to handle WebREPL protocol
- modwebrepl: initial implementation of "get file" operation
- modwebrepl: keep reading data when there's something to read
- modwebrepl: GET_FILE: Send length-prefix chunk with one write()
- modwebrepl: add rate-limiting workaround for broken network drivers
- modwebrepl: set debugging by default to off
- modwebrepl: add support for password
- modlwip: implement sendall() method for TCP sockets
- fix typo of macro that detects if float is enabled
lib:
- utils/printf: rework overriding printer of DEBUG_printf()
- utils/pyexec: condition-out GC calls from pyexec
- utils/pyhelp: extract implementation of help(obj) to a library function
- axtls: update to the latest upstream, fix reported MacOSX build issue
drivers:
- add SSD1306 OLED driver, with I2C and SPI interfaces
tools:
- mpy-tool.py: new tool to work with .mpy files; currently it can freeze them
- mpy-tool.py: add support for Python 2.7
tests:
- add 6 tests for async await/for/with
- add .exp files for async tests, so they can run with Python 3.4
- fix dict1.py so it doesn't rely on the order of dict elems
- extmod: move split-on-empty-match tests to a separate test file
- add testcase for yielding from a stopped generator
- run-bench-tests: process tests in alphabetical order
- update for _io/_collections module having been renamed
minimal port:
- add example of frozen persistent bytecode (.mpy file)
- disable async/await syntax
unix port:
- build with MICROPY_PY_UHASHLIB_SHA1 if already building with axTLS
- make sure build dir exists before accessing it for freezing upip
- Makefile: make install more compatible (BSD, etc.)
windows port:
- msvc: implement automatic qstr generation using makeqstrdefs
stmhal port:
- implement basic C-level pin HAL
- use new generic I2C object in machine module
- enable framebuf module
- properly handle RTS/CTS flow control for buf/unbuf transfers
- add Makefile option FROZEN_MPY_DIR to support frozen bytecode
- for frozen bytecode generation, add dependency of qstr file
- l4: add CMSIS files to support STM32L476
- l4: add basic STM32L4xx HAL files
- l4: adapt UART HAL to avoid 64-bit integer division
- update HALCOMMITS due to change to hal
- l4: add board definition files for STM32L476DISC
- l4: add line to Makefile for building L4 series
- l4: modify adc.c to add support for STM32L4 series
- l4: modify uart.c to support L4 MCU
- l4: modify usbd_conf.c to support L4 MCU
- l4: modify rtc.c to support L4 MCU
- l4: modify timer.c to support L4 MCU
- fix machine.unique_id() function to work for all MCUs
- l4: modify mphalport to support L4 MCU
- l4: modify flash.c and storage.c to support L4 MCU
- l4: add support for external interrupts/events
- accel: raise an exception if the accel couldn't be initialised
- use pyhelp_print_obj function
- change i2c.scan() method to scan addresses 0x08-0x77
- fix typo of macro that detects if float is enabled
cc3200 port:
- use pyhelp_print_obj function
- change i2c.scan() method to scan addresses 0x08-0x77
- disable async/await syntax
esp8266 port:
- enable websocket module
- modnetwork: .config(): Check interface whose config is requested
- modnetwork: .config(): Add "authmode" param
- modnetwork: .config(): Add "password" param (W/O)
- add Python modules for initial configuration
- scripts/inisetup.py: set WPA/WPA2 AP mode with a predefined password
- protect modpyb.h header file from multiple inclusions
- implement basic C-level pin HAL
- switch from using custom I2C driver to generic extmod one
- enable framebuf module
- in callback helpers, pop nlr_buf on successful call
- esp_mphal: don't swallow exceptions in dupterm's read()/write()
- esp_mphal: call_dupterm_read(): Fix order of deactivating on EOF
- remove pin_id field from C pin object
- add dummy entries for non-existing pins to simplify pin logic
- add hard IRQ callbacks for pin change on GPIO0-15
- separate 1-wire timing funcs from Python module to save iRAM
- switch integer arith routines to BootROM
- scripts/port_diag.py: module to collect diagnostic info
- enable input() builtin
- scripts/_boot: mount block device on "" instead of "/"
- moduos: add uos.mkdir function
- modmachine: add reset_cause() function
- adapt port to use new auto-qstr generation
- moduos: add dupterm_notify() function
- esp_mphal: protect dupterm_task_handler() from recursive exec
- README.md: update feature list for current state of affairs
- modesp: allow esp.deepsleep to take 2nd arg for RF wake opt
- scripts/flashbdev: use all available space in 1MB FlashROM for FS
- modesp: add flash_size() function
- scripts: don't try to create filesystem on 512KB devices or less
- modnetwork: .config(): Add "channel" param
- scripts/flashbdev: disable debug output/checks
- scripts/_boot: print notice when initial setup is executed
- scripts/flashbdev: correct bootloader flash size to match real size
- modnetwork: .config(): Add "hidden ESSID" param
- implement basic deep-sleep capabilities
- add uart_rx_wait and uart_rx_char functions
- implement UART.read functionality
- uart: remove obsolete UART rx buffering code
- esp_mphal: remove mp_hal_feed_watchdog
- convert mp_hal_pin_obj_t from pin ptr to simple integer
- allow GPIO16 to be used as a pin in the uPy pin HAL
- change software SPI driver to use general pin HAL
- scripts/websocket_helper: module encapsulating handshake sequences
- scripts/websocket_helper: disable debug output
- scripts/webrepl: webREPL based on C-level websocket object
- scripts/webrepl: convert to persistent daemon
- scripts/webrepl: don't start on import
- scripts/webrepl: allow to override port
- scripts/webrepl: print connection address
- scripts/webrepl: print client address for incoming connections
- scripts/flashbdev: use all available Flash for filesystem
- scripts/webrepl: add "ws://" to "daemon started at" message
- Makefile: add target to build axTLS
- esp8266.ld: put axTLS to FlashROM
- Makefile: override abort() when building axtls
- axtls_helpers: helper/wrapper functions for axTLS
- tests/onewire.py: don't run test on import
- Makefile: support linking with axTLS built from source
- Makefile: enable "ussl" module; axTLS should be built first using "make axtls"
- move pyb.info() function to esp module and remove pyb module
- move onewire.py, neopixel.py drivers from tests/ to scripts/
- scripts/onewire.py: simplify and improve 1-wire driver
- scripts/neopixel.py: remove test function from neopixel driver
- set suitable values for axtls's RT_MAX_PLAIN_LENGTH & RT_EXTRA
- README: add recently required step of 'make axtls'
- modnetwork: make WLAN.ifconfig() read/write
- help: implement help() builtin
- help: add cheatsheet for basic WiFi configuration
- enable WebREPL file transfer rate limiting
- enable webrepl module
- scripts: move all of initial setup to inisetup module
- scripts/inisetup: create default boot.py in filesystem
- scripts/webrepl: connection ack prompt is now printed by modwebrepl
- scripts/webrepl: switch to using _webrepl object wrapper
- scripts/webrepl: add "first connection" mode to setup password
- README: mention WebREPL
- Makefile: be sure to pass cross-compiling AR when building axtls
- scripts/webrepl_setup: reject too short passwords
- change platform name from ESP8266 to esp8266
- modnetwork: remove deprecated wifi_mode()
- esp_mphal: add ets_esf_free_bufs(), etc. functions
- modesp: add esf_free_bufs() debugging function
- modesp: add malloc() and free() functions
- scripts/inisetup: enable WebREPL auto-start on boot
- modnetwork: remove .mac() method, move to .config("mac")
- scripts/inisetup: update for nic.mac() method being gone
- scripts/neopixel.py: swap red and green in pixel accessor
- modpybpin: use enum+array instead of struct for parsing args
- modpybpin: use None instead of PULL_NONE for no-pull config
- modpybpin: make pin.irq() methods take keyword args
- tutorial: mention that esptool is available via pip
- scripts/ntptime: add simple NTP client
docs:
- esp: enumerate flash access functions
- machine: start to update for esp8266 port
- speed_python: clarify/generalize "Buffers" subsection
- speed_python: generalize "Floating point" subsection
- speed_python: add many more details on memoryviews
- machine: reset_cause() has been implemented for esp8266
- topindex.html: esp8266: Enable quickref/general on the main page
- esp8266/general: start "General information" for esp8266
- esp8266: add info about using deep-sleep mode to quickref
- esp8266/tutorial: add tutorial placeholder page
- ustruct: there's no complete "struct" module, only "ustruct" subset
- esp8266: include usocket module reference
- library/usocket: add link to CPython's socket module
- usocket: socket-specific exceptions are for WiPy only
- usocket: socket.IPPROTO_SEC is WiPy-specific
- usocket: describe address format once at the beginning
- network: esp8266: scan(): add note that bssid is bytes object
- library: group MicroPython-specific modules under separate heading
- library/index: move WiPy "micro-libraries" under corresponding heading
- library/index: esp8266 has the same set of stdlibs as pyboard/unix
- module "time" is actually "utime"
- library/index: make single section for "micro-ified" modules
- library/index: order sections from the most to least standard modules
- Makefile: default BUILDDIR based on MICROPY_PORT
- library: "os" module is actually "uos"
- fix uos and utime heading underlines to be the correct length
- library/utime: add more time functions for unix and esp8266 ports
- utime: describe time() peculiarities in MicroPython
- utime: describe sleep() peculiarities in MicroPython
- library: consistently use admonitions for CPython differences
- ubinascii: clean up grammar
- network: esp8266: Add wlan.ifconfig() method
- network: esp8266: Describe wlan.config() method
- make the short port names in the port/version sidebar lowercase
- esp8266: update quickref to reflect changes to 1-wire and NeoPixel
- library/utime: elaborate on epochs and calendar time maintenance
- esp8266/quickref: add info about WebREPL
- ustruct: document pack_into(), unpack_from()
- sys: document sys.modules
- ustruct: fix argument formatting
- sys: remove port-specific details from description of stdin/out/err
- sys: clarify description of sys.exit()
- sys: clean up print_exception() description
- sys: describe sys.implementation
- sys: describe sys.maxsize
- sys: describe sys.platform is port-neutral manner
- add _collections module reference
- add _io module reference
- _io and _collections were renamed to have standard "u" prefix
- library/machine.I2C: update to reflect ESP8266 implementation
- esp8266/general: webREPL is described in quickref for now
- esp8266/general: add more points to "Multitude of boards" section
- esp8266/general: fix list formatting
- esp8266/general: add techspec section
- esp8266/general: add "Boot process" section
- esp8266_contents: reference general and tutorial docs
- network: esp8266: MAC address is set via .config() method
- esp8266: add ESP8266 tutorial
- machine: generalize docs from just WiPy to other ports
- machine: move WiPy-specific hardware details to its general reference
- machine: more generic description of sleep's, WiPy details to its genref
- machine: idle() description generalization
- library/machine.Pin: update pin docs to reflect ESP8266 support
- esp8266/tutorial: update pins tutorial to reflect changes in API
examples:
- http_server_ssl.py: add HTTPS server example
Diffstat (limited to 'py')
-rw-r--r-- | py/bc.c | 1 | ||||
-rw-r--r-- | py/builtin.h | 6 | ||||
-rw-r--r-- | py/builtinimport.c | 21 | ||||
-rw-r--r-- | py/compile.c | 238 | ||||
-rw-r--r-- | py/emitglue.c | 30 | ||||
-rw-r--r-- | py/emitglue.h | 26 | ||||
-rw-r--r-- | py/emitnative.c | 2 | ||||
-rw-r--r-- | py/frozenmod.c | 65 | ||||
-rw-r--r-- | py/frozenmod.h | 8 | ||||
-rw-r--r-- | py/grammar.h | 119 | ||||
-rw-r--r-- | py/lexer.c | 4 | ||||
-rw-r--r-- | py/lexer.h | 4 | ||||
-rw-r--r-- | py/makeqstrdata.py | 72 | ||||
-rw-r--r-- | py/makeqstrdefs.py | 109 | ||||
-rw-r--r-- | py/map.c | 29 | ||||
-rw-r--r-- | py/mkrules.mk | 30 | ||||
-rw-r--r-- | py/modbuiltins.c | 4 | ||||
-rw-r--r-- | py/modcollections.c | 4 | ||||
-rw-r--r-- | py/modio.c | 4 | ||||
-rw-r--r-- | py/mpconfig.h | 29 | ||||
-rw-r--r-- | py/obj.h | 7 | ||||
-rw-r--r-- | py/objexcept.c | 3 | ||||
-rw-r--r-- | py/objgenerator.c | 1 | ||||
-rw-r--r-- | py/objmodule.c | 10 | ||||
-rw-r--r-- | py/objstr.c | 2 | ||||
-rw-r--r-- | py/parse.c | 94 | ||||
-rw-r--r-- | py/py.mk | 19 | ||||
-rw-r--r-- | py/qstr.c | 19 | ||||
-rw-r--r-- | py/qstr.h | 4 | ||||
-rw-r--r-- | py/qstrdefs.h | 721 | ||||
-rw-r--r-- | py/repl.c | 3 | ||||
-rw-r--r-- | py/runtime.c | 14 | ||||
-rw-r--r-- | py/runtime.h | 3 | ||||
-rw-r--r-- | py/runtime_utils.c | 50 | ||||
-rw-r--r-- | py/vm.c | 3 |
35 files changed, 813 insertions, 945 deletions
@@ -171,6 +171,7 @@ void mp_setup_code_state(mp_code_state *code_state, mp_obj_fun_bc_t *self, size_ const mp_obj_t *arg_names = (const mp_obj_t*)code_state->const_table; for (size_t i = 0; i < n_kw; i++) { + // the keys in kwargs are expected to be qstr objects mp_obj_t wanted_arg_name = kwargs[2 * i]; for (size_t j = 0; j < n_pos_args + n_kwonly_args; j++) { if (wanted_arg_name == arg_names[j]) { diff --git a/py/builtin.h b/py/builtin.h index e3e68e1519..9d6e424091 100644 --- a/py/builtin.h +++ b/py/builtin.h @@ -71,6 +71,10 @@ MP_DECLARE_CONST_FUN_OBJ(mp_builtin_repr_obj); MP_DECLARE_CONST_FUN_OBJ(mp_builtin_round_obj); MP_DECLARE_CONST_FUN_OBJ(mp_builtin_sorted_obj); MP_DECLARE_CONST_FUN_OBJ(mp_builtin_sum_obj); +// Defined by a port, but declared here for simplicity +MP_DECLARE_CONST_FUN_OBJ(mp_builtin_help_obj); +MP_DECLARE_CONST_FUN_OBJ(mp_builtin_input_obj); +MP_DECLARE_CONST_FUN_OBJ(mp_builtin_open_obj); MP_DECLARE_CONST_FUN_OBJ(mp_namedtuple_obj); @@ -106,6 +110,8 @@ extern const mp_obj_module_t mp_module_ussl; extern const mp_obj_module_t mp_module_machine; extern const mp_obj_module_t mp_module_lwip; extern const mp_obj_module_t mp_module_websocket; +extern const mp_obj_module_t mp_module_webrepl; +extern const mp_obj_module_t mp_module_framebuf; // extmod functions MP_DECLARE_CONST_FUN_OBJ(pyb_mount_obj); diff --git a/py/builtinimport.c b/py/builtinimport.c index ec79357cb7..0e4dce6430 100644 --- a/py/builtinimport.c +++ b/py/builtinimport.c @@ -144,7 +144,7 @@ STATIC void do_load_from_lexer(mp_obj_t module_obj, mp_lexer_t *lex, const char } #endif -#if MICROPY_PERSISTENT_CODE_LOAD +#if MICROPY_PERSISTENT_CODE_LOAD || MICROPY_MODULE_FROZEN_MPY STATIC void do_execute_raw_code(mp_obj_t module_obj, mp_raw_code_t *raw_code) { #if MICROPY_PY___FILE__ // TODO @@ -182,8 +182,9 @@ STATIC void do_execute_raw_code(mp_obj_t module_obj, mp_raw_code_t *raw_code) { #endif STATIC void do_load(mp_obj_t module_obj, vstr_t *file) { - // create the lexer + #if MICROPY_PERSISTENT_CODE_LOAD || MICROPY_ENABLE_COMPILER char *file_str = vstr_null_terminated_str(file); + #endif #if MICROPY_PERSISTENT_CODE_LOAD if (file_str[file->len - 3] == 'm') { @@ -340,8 +341,9 @@ mp_obj_t mp_builtin___import__(size_t n_args, const mp_obj_t *args) { DEBUG_printf("Module not yet loaded\n"); #if MICROPY_MODULE_FROZEN - mp_lexer_t *lex = mp_find_frozen_module(mod_str, mod_len); - if (lex != NULL) { + void *frozen_data; + int frozen_type = mp_find_frozen_module(mod_str, mod_len, &frozen_data); + if (frozen_type != MP_FROZEN_NONE) { module_obj = mp_obj_new_module(module_name_qstr); // if args[3] (fromtuple) has magic value False, set up // this module for command-line "-m" option (set module's @@ -351,7 +353,16 @@ mp_obj_t mp_builtin___import__(size_t n_args, const mp_obj_t *args) { mp_obj_module_t *o = MP_OBJ_TO_PTR(module_obj); mp_obj_dict_store(MP_OBJ_FROM_PTR(o->globals), MP_OBJ_NEW_QSTR(MP_QSTR___name__), MP_OBJ_NEW_QSTR(MP_QSTR___main__)); } - do_load_from_lexer(module_obj, lex, mod_str); + #if MICROPY_MODULE_FROZEN_STR + if (frozen_type == MP_FROZEN_STR) { + do_load_from_lexer(module_obj, frozen_data, mod_str); + } + #endif + #if MICROPY_MODULE_FROZEN_MPY + if (frozen_type == MP_FROZEN_MPY) { + do_execute_raw_code(module_obj, frozen_data); + } + #endif return module_obj; } #endif diff --git a/py/compile.c b/py/compile.c index ae91455e0e..d40d8a1ff5 100644 --- a/py/compile.c +++ b/py/compile.c @@ -318,14 +318,14 @@ STATIC void c_if_cond(compiler_t *comp, mp_parse_node_t pn, bool jump_if, int la typedef enum { ASSIGN_STORE, ASSIGN_AUG_LOAD, ASSIGN_AUG_STORE } assign_kind_t; STATIC void c_assign(compiler_t *comp, mp_parse_node_t pn, assign_kind_t kind); -STATIC void c_assign_power(compiler_t *comp, mp_parse_node_struct_t *pns, assign_kind_t assign_kind) { +STATIC void c_assign_atom_expr(compiler_t *comp, mp_parse_node_struct_t *pns, assign_kind_t assign_kind) { if (assign_kind != ASSIGN_AUG_STORE) { compile_node(comp, pns->nodes[0]); } if (MP_PARSE_NODE_IS_STRUCT(pns->nodes[1])) { mp_parse_node_struct_t *pns1 = (mp_parse_node_struct_t*)pns->nodes[1]; - if (MP_PARSE_NODE_STRUCT_KIND(pns1) == PN_power_trailers) { + if (MP_PARSE_NODE_STRUCT_KIND(pns1) == PN_atom_expr_trailers) { int n = MP_PARSE_NODE_STRUCT_NUM_NODES(pns1); if (assign_kind != ASSIGN_AUG_STORE) { for (int i = 0; i < n - 1; i++) { @@ -366,10 +366,6 @@ STATIC void c_assign_power(compiler_t *comp, mp_parse_node_struct_t *pns, assign goto cannot_assign; } - if (!MP_PARSE_NODE_IS_NULL(pns->nodes[2])) { - goto cannot_assign; - } - return; cannot_assign: @@ -440,9 +436,9 @@ STATIC void c_assign(compiler_t *comp, mp_parse_node_t pn, assign_kind_t assign_ // pn must be a struct mp_parse_node_struct_t *pns = (mp_parse_node_struct_t*)pn; switch (MP_PARSE_NODE_STRUCT_KIND(pns)) { - case PN_power: + case PN_atom_expr_normal: // lhs is an index or attribute - c_assign_power(comp, pns, assign_kind); + c_assign_atom_expr(comp, pns, assign_kind); break; case PN_testlist_star_expr: @@ -818,11 +814,19 @@ STATIC void compile_decorated(compiler_t *comp, mp_parse_node_struct_t *pns) { } } - // compile the body (funcdef or classdef) and get its name + // compile the body (funcdef, async funcdef or classdef) and get its name mp_parse_node_struct_t *pns_body = (mp_parse_node_struct_t*)pns->nodes[1]; qstr body_name = 0; if (MP_PARSE_NODE_STRUCT_KIND(pns_body) == PN_funcdef) { body_name = compile_funcdef_helper(comp, pns_body, emit_options); + #if MICROPY_PY_ASYNC_AWAIT + } else if (MP_PARSE_NODE_STRUCT_KIND(pns_body) == PN_async_funcdef) { + assert(MP_PARSE_NODE_IS_STRUCT(pns_body->nodes[0])); + mp_parse_node_struct_t *pns0 = (mp_parse_node_struct_t*)pns_body->nodes[0]; + body_name = compile_funcdef_helper(comp, pns0, emit_options); + scope_t *fscope = (scope_t*)pns0->nodes[4]; + fscope->scope_flags |= MP_SCOPE_FLAG_GENERATOR; + #endif } else { assert(MP_PARSE_NODE_STRUCT_KIND(pns_body) == PN_classdef); // should be body_name = compile_classdef_helper(comp, pns_body, emit_options); @@ -846,14 +850,14 @@ STATIC void compile_funcdef(compiler_t *comp, mp_parse_node_struct_t *pns) { STATIC void c_del_stmt(compiler_t *comp, mp_parse_node_t pn) { if (MP_PARSE_NODE_IS_ID(pn)) { compile_delete_id(comp, MP_PARSE_NODE_LEAF_ARG(pn)); - } else if (MP_PARSE_NODE_IS_STRUCT_KIND(pn, PN_power)) { + } else if (MP_PARSE_NODE_IS_STRUCT_KIND(pn, PN_atom_expr_normal)) { mp_parse_node_struct_t *pns = (mp_parse_node_struct_t*)pn; - compile_node(comp, pns->nodes[0]); // base of the power node + compile_node(comp, pns->nodes[0]); // base of the atom_expr_normal node if (MP_PARSE_NODE_IS_STRUCT(pns->nodes[1])) { mp_parse_node_struct_t *pns1 = (mp_parse_node_struct_t*)pns->nodes[1]; - if (MP_PARSE_NODE_STRUCT_KIND(pns1) == PN_power_trailers) { + if (MP_PARSE_NODE_STRUCT_KIND(pns1) == PN_atom_expr_trailers) { int n = MP_PARSE_NODE_STRUCT_NUM_NODES(pns1); for (int i = 0; i < n - 1; i++) { compile_node(comp, pns1->nodes[i]); @@ -874,9 +878,6 @@ STATIC void c_del_stmt(compiler_t *comp, mp_parse_node_t pn) { goto cannot_delete; } - if (!MP_PARSE_NODE_IS_NULL(pns->nodes[2])) { - goto cannot_delete; - } } else if (MP_PARSE_NODE_IS_STRUCT_KIND(pn, PN_atom_paren)) { pn = ((mp_parse_node_struct_t*)pn)->nodes[0]; if (MP_PARSE_NODE_IS_NULL(pn)) { @@ -1397,12 +1398,11 @@ STATIC void compile_for_stmt(compiler_t *comp, mp_parse_node_struct_t *pns) { // this bit optimises: for <x> in range(...), turning it into an explicitly incremented variable // this is actually slower, but uses no heap memory // for viper it will be much, much faster - if (/*comp->scope_cur->emit_options == MP_EMIT_OPT_VIPER &&*/ MP_PARSE_NODE_IS_ID(pns->nodes[0]) && MP_PARSE_NODE_IS_STRUCT_KIND(pns->nodes[1], PN_power)) { + if (/*comp->scope_cur->emit_options == MP_EMIT_OPT_VIPER &&*/ MP_PARSE_NODE_IS_ID(pns->nodes[0]) && MP_PARSE_NODE_IS_STRUCT_KIND(pns->nodes[1], PN_atom_expr_normal)) { mp_parse_node_struct_t *pns_it = (mp_parse_node_struct_t*)pns->nodes[1]; - if (MP_PARSE_NODE_IS_ID(pns_it->nodes[0]) + if (MP_PARSE_NODE_IS_ID(pns_it->nodes[0]) && MP_PARSE_NODE_LEAF_ARG(pns_it->nodes[0]) == MP_QSTR_range - && MP_PARSE_NODE_IS_STRUCT_KIND(pns_it->nodes[1], PN_trailer_paren) - && MP_PARSE_NODE_IS_NULL(pns_it->nodes[2])) { + && MP_PARSE_NODE_IS_STRUCT_KIND(pns_it->nodes[1], PN_trailer_paren)) { mp_parse_node_t pn_range_args = ((mp_parse_node_struct_t*)pns_it->nodes[1])->nodes[0]; mp_parse_node_t *args; int n_args = mp_parse_node_extract_list(&pn_range_args, PN_arglist, &args); @@ -1661,6 +1661,177 @@ STATIC void compile_with_stmt(compiler_t *comp, mp_parse_node_struct_t *pns) { compile_with_stmt_helper(comp, n, nodes, pns->nodes[1]); } +STATIC void compile_yield_from(compiler_t *comp) { + EMIT(get_iter); + EMIT_ARG(load_const_tok, MP_TOKEN_KW_NONE); + EMIT(yield_from); +} + +#if MICROPY_PY_ASYNC_AWAIT +STATIC void compile_await_object_method(compiler_t *comp, qstr method) { + EMIT_ARG(load_method, method); + EMIT_ARG(call_method, 0, 0, 0); + compile_yield_from(comp); +} + +STATIC void compile_async_for_stmt(compiler_t *comp, mp_parse_node_struct_t *pns) { + // comp->break_label |= MP_EMIT_BREAK_FROM_FOR; + + qstr context = MP_PARSE_NODE_LEAF_ARG(pns->nodes[1]); + uint while_else_label = comp_next_label(comp); + uint try_exception_label = comp_next_label(comp); + uint try_else_label = comp_next_label(comp); + uint try_finally_label = comp_next_label(comp); + + compile_node(comp, pns->nodes[1]); // iterator + compile_await_object_method(comp, MP_QSTR___aiter__); + compile_store_id(comp, context); + + START_BREAK_CONTINUE_BLOCK + + EMIT_ARG(label_assign, continue_label); + + EMIT_ARG(setup_except, try_exception_label); + compile_increase_except_level(comp); + + compile_load_id(comp, context); + compile_await_object_method(comp, MP_QSTR___anext__); + c_assign(comp, pns->nodes[0], ASSIGN_STORE); // variable + EMIT(pop_block); + EMIT_ARG(jump, try_else_label); + + EMIT_ARG(label_assign, try_exception_label); + EMIT(start_except_handler); + EMIT(dup_top); + EMIT_LOAD_GLOBAL(MP_QSTR_StopAsyncIteration); + EMIT_ARG(binary_op, MP_BINARY_OP_EXCEPTION_MATCH); + EMIT_ARG(pop_jump_if, false, try_finally_label); + EMIT(pop_top); + EMIT(pop_top); + EMIT(pop_top); + EMIT(pop_except); + EMIT_ARG(jump, while_else_label); + + EMIT_ARG(label_assign, try_finally_label); + EMIT_ARG(adjust_stack_size, 3); + compile_decrease_except_level(comp); + EMIT(end_finally); + EMIT(end_except_handler); + + EMIT_ARG(label_assign, try_else_label); + compile_node(comp, pns->nodes[2]); // body + + EMIT_ARG(jump, continue_label); + // break/continue apply to outer loop (if any) in the else block + END_BREAK_CONTINUE_BLOCK + + EMIT_ARG(label_assign, while_else_label); + compile_node(comp, pns->nodes[3]); // else + + EMIT_ARG(label_assign, break_label); +} + +STATIC void compile_async_with_stmt_helper(compiler_t *comp, int n, mp_parse_node_t *nodes, mp_parse_node_t body) { + if (n == 0) { + // no more pre-bits, compile the body of the with + compile_node(comp, body); + } else { + uint try_exception_label = comp_next_label(comp); + uint no_reraise_label = comp_next_label(comp); + uint try_else_label = comp_next_label(comp); + uint end_label = comp_next_label(comp); + qstr context; + + if (MP_PARSE_NODE_IS_STRUCT_KIND(nodes[0], PN_with_item)) { + // this pre-bit is of the form "a as b" + mp_parse_node_struct_t *pns = (mp_parse_node_struct_t*)nodes[0]; + compile_node(comp, pns->nodes[0]); + context = MP_PARSE_NODE_LEAF_ARG(pns->nodes[0]); + compile_store_id(comp, context); + compile_load_id(comp, context); + compile_await_object_method(comp, MP_QSTR___aenter__); + c_assign(comp, pns->nodes[1], ASSIGN_STORE); + } else { + // this pre-bit is just an expression + compile_node(comp, nodes[0]); + context = MP_PARSE_NODE_LEAF_ARG(nodes[0]); + compile_store_id(comp, context); + compile_load_id(comp, context); + compile_await_object_method(comp, MP_QSTR___aenter__); + EMIT(pop_top); + } + + compile_load_id(comp, context); + EMIT_ARG(load_method, MP_QSTR___aexit__); + + EMIT_ARG(setup_except, try_exception_label); + compile_increase_except_level(comp); + // compile additional pre-bits and the body + compile_async_with_stmt_helper(comp, n - 1, nodes + 1, body); + // finish this with block + EMIT(pop_block); + EMIT_ARG(jump, try_else_label); // jump over exception handler + + EMIT_ARG(label_assign, try_exception_label); // start of exception handler + EMIT(start_except_handler); + EMIT(rot_three); + EMIT(rot_two); + EMIT_ARG(call_method, 3, 0, 0); + compile_yield_from(comp); + EMIT_ARG(pop_jump_if, true, no_reraise_label); + EMIT_ARG(raise_varargs, 0); + + EMIT_ARG(label_assign, no_reraise_label); + EMIT(pop_except); + EMIT_ARG(jump, end_label); + + EMIT_ARG(adjust_stack_size, 5); + compile_decrease_except_level(comp); + EMIT(end_finally); + EMIT(end_except_handler); + + EMIT_ARG(label_assign, try_else_label); // start of try-else handler + EMIT_ARG(load_const_tok, MP_TOKEN_KW_NONE); + EMIT(dup_top); + EMIT(dup_top); + EMIT_ARG(call_method, 3, 0, 0); + compile_yield_from(comp); + EMIT(pop_top); + + EMIT_ARG(label_assign, end_label); + + } +} + +STATIC void compile_async_with_stmt(compiler_t *comp, mp_parse_node_struct_t *pns) { + // get the nodes for the pre-bit of the with (the a as b, c as d, ... bit) + mp_parse_node_t *nodes; + int n = mp_parse_node_extract_list(&pns->nodes[0], PN_with_stmt_list, &nodes); + assert(n > 0); + + // compile in a nested fashion + compile_async_with_stmt_helper(comp, n, nodes, pns->nodes[1]); +} + +STATIC void compile_async_stmt(compiler_t *comp, mp_parse_node_struct_t *pns) { + assert(MP_PARSE_NODE_IS_STRUCT(pns->nodes[0])); + mp_parse_node_struct_t *pns0 = (mp_parse_node_struct_t*)pns->nodes[0]; + if (MP_PARSE_NODE_STRUCT_KIND(pns0) == PN_funcdef) { + // async def + compile_funcdef(comp, pns0); + scope_t *fscope = (scope_t*)pns0->nodes[4]; + fscope->scope_flags |= MP_SCOPE_FLAG_GENERATOR; + } else if (MP_PARSE_NODE_STRUCT_KIND(pns0) == PN_for_stmt) { + // async for + compile_async_for_stmt(comp, pns0); + } else { + // async with + assert(MP_PARSE_NODE_STRUCT_KIND(pns0) == PN_with_stmt); + compile_async_with_stmt(comp, pns0); + } +} +#endif + STATIC void compile_expr_stmt(compiler_t *comp, mp_parse_node_struct_t *pns) { if (MP_PARSE_NODE_IS_NULL(pns->nodes[1])) { if (comp->is_repl && comp->scope_cur->kind == SCOPE_MODULE) { @@ -1967,15 +2138,16 @@ STATIC void compile_factor_2(compiler_t *comp, mp_parse_node_struct_t *pns) { } } -STATIC void compile_power(compiler_t *comp, mp_parse_node_struct_t *pns) { +STATIC void compile_atom_expr_normal(compiler_t *comp, mp_parse_node_struct_t *pns) { // this is to handle special super() call comp->func_arg_is_super = MP_PARSE_NODE_IS_ID(pns->nodes[0]) && MP_PARSE_NODE_LEAF_ARG(pns->nodes[0]) == MP_QSTR_super; compile_generic_all_nodes(comp, pns); +} - if (!MP_PARSE_NODE_IS_NULL(pns->nodes[2])) { - EMIT_ARG(binary_op, MP_BINARY_OP_POWER); - } +STATIC void compile_power(compiler_t *comp, mp_parse_node_struct_t *pns) { + compile_generic_all_nodes(comp, pns); // 2 nodes, arguments of power + EMIT_ARG(binary_op, MP_BINARY_OP_POWER); } STATIC void compile_trailer_paren_helper(compiler_t *comp, mp_parse_node_t pn_arglist, bool is_method_call, int n_positional_extra) { @@ -2076,7 +2248,7 @@ STATIC void compile_trailer_paren_helper(compiler_t *comp, mp_parse_node_t pn_ar } } -STATIC void compile_power_trailers(compiler_t *comp, mp_parse_node_struct_t *pns) { +STATIC void compile_atom_expr_trailers(compiler_t *comp, mp_parse_node_struct_t *pns) { int num_nodes = MP_PARSE_NODE_STRUCT_NUM_NODES(pns); for (int i = 0; i < num_nodes; i++) { if (i + 1 < num_nodes && MP_PARSE_NODE_IS_STRUCT_KIND(pns->nodes[i], PN_trailer_period) && MP_PARSE_NODE_IS_STRUCT_KIND(pns->nodes[i + 1], PN_trailer_paren)) { @@ -2431,15 +2603,24 @@ STATIC void compile_yield_expr(compiler_t *comp, mp_parse_node_struct_t *pns) { } else if (MP_PARSE_NODE_IS_STRUCT_KIND(pns->nodes[0], PN_yield_arg_from)) { pns = (mp_parse_node_struct_t*)pns->nodes[0]; compile_node(comp, pns->nodes[0]); - EMIT(get_iter); - EMIT_ARG(load_const_tok, MP_TOKEN_KW_NONE); - EMIT(yield_from); + compile_yield_from(comp); } else { compile_node(comp, pns->nodes[0]); EMIT(yield_value); } } +#if MICROPY_PY_ASYNC_AWAIT +STATIC void compile_atom_expr_await(compiler_t *comp, mp_parse_node_struct_t *pns) { + if (comp->scope_cur->kind != SCOPE_FUNCTION && comp->scope_cur->kind != SCOPE_LAMBDA) { + compile_syntax_error(comp, (mp_parse_node_t)pns, "'await' outside function"); + return; + } + compile_atom_expr_normal(comp, pns); + compile_yield_from(comp); +} +#endif + STATIC void compile_string(compiler_t *comp, mp_parse_node_struct_t *pns) { // only create and load the actual str object on the last pass if (comp->pass != MP_PASS_EMIT) { @@ -2995,7 +3176,7 @@ STATIC void compile_scope_inline_asm(compiler_t *comp, scope_t *scope, pass_kind goto not_an_instruction; } pns2 = (mp_parse_node_struct_t*)pns2->nodes[0]; - if (MP_PARSE_NODE_STRUCT_KIND(pns2) != PN_power) { + if (MP_PARSE_NODE_STRUCT_KIND(pns2) != PN_atom_expr_normal) { goto not_an_instruction; } if (!MP_PARSE_NODE_IS_ID(pns2->nodes[0])) { @@ -3004,7 +3185,6 @@ STATIC void compile_scope_inline_asm(compiler_t *comp, scope_t *scope, pass_kind if (!MP_PARSE_NODE_IS_STRUCT_KIND(pns2->nodes[1], PN_trailer_paren)) { goto not_an_instruction; } - assert(MP_PARSE_NODE_IS_NULL(pns2->nodes[2])); // parse node looks like an instruction // get instruction name and args diff --git a/py/emitglue.c b/py/emitglue.c index e327d9d0cb..b710371177 100644 --- a/py/emitglue.c +++ b/py/emitglue.c @@ -49,28 +49,6 @@ mp_uint_t mp_verbose_flag = 0; #endif -struct _mp_raw_code_t { - mp_raw_code_kind_t kind : 3; - mp_uint_t scope_flags : 7; - mp_uint_t n_pos_args : 11; - union { - struct { - const byte *bytecode; - const mp_uint_t *const_table; - #if MICROPY_PERSISTENT_CODE_SAVE - mp_uint_t bc_len; - uint16_t n_obj; - uint16_t n_raw_code; - #endif - } u_byte; - struct { - void *fun_data; - const mp_uint_t *const_table; - mp_uint_t type_sig; // for viper, compressed as 2-bit types; ret is MSB, then arg0, arg1, etc - } u_native; - } data; -}; - mp_raw_code_t *mp_emit_glue_new_raw_code(void) { mp_raw_code_t *rc = m_new0(mp_raw_code_t, 1); rc->kind = MP_CODE_RESERVED; @@ -136,7 +114,7 @@ void mp_emit_glue_assign_native(mp_raw_code_t *rc, mp_raw_code_kind_t kind, void } #endif -mp_obj_t mp_make_function_from_raw_code(mp_raw_code_t *rc, mp_obj_t def_args, mp_obj_t def_kw_args) { +mp_obj_t mp_make_function_from_raw_code(const mp_raw_code_t *rc, mp_obj_t def_args, mp_obj_t def_kw_args) { DEBUG_OP_printf("make_function_from_raw_code %p\n", rc); assert(rc != NULL); @@ -180,7 +158,7 @@ mp_obj_t mp_make_function_from_raw_code(mp_raw_code_t *rc, mp_obj_t def_args, mp return fun; } -mp_obj_t mp_make_closure_from_raw_code(mp_raw_code_t *rc, mp_uint_t n_closed_over, const mp_obj_t *args) { +mp_obj_t mp_make_closure_from_raw_code(const mp_raw_code_t *rc, mp_uint_t n_closed_over, const mp_obj_t *args) { DEBUG_OP_printf("make_closure_from_raw_code %p " UINT_FMT " %p\n", rc, n_closed_over, args); // make function object mp_obj_t ffun; @@ -195,7 +173,7 @@ mp_obj_t mp_make_closure_from_raw_code(mp_raw_code_t *rc, mp_uint_t n_closed_ove return mp_obj_new_closure(ffun, n_closed_over & 0xff, args + ((n_closed_over >> 7) & 2)); } -#if MICROPY_PERSISTENT_CODE +#if MICROPY_PERSISTENT_CODE_LOAD || MICROPY_PERSISTENT_CODE_SAVE #include "py/smallint.h" @@ -252,7 +230,7 @@ STATIC void extract_prelude(const byte **ip, const byte **ip2, bytecode_prelude_ } } -#endif // MICROPY_PERSISTENT_CODE +#endif // MICROPY_PERSISTENT_CODE_LOAD || MICROPY_PERSISTENT_CODE_SAVE #if MICROPY_PERSISTENT_CODE_LOAD diff --git a/py/emitglue.h b/py/emitglue.h index b31e8dbb22..f5618577d5 100644 --- a/py/emitglue.h +++ b/py/emitglue.h @@ -39,7 +39,27 @@ typedef enum { MP_CODE_NATIVE_ASM, } mp_raw_code_kind_t; -typedef struct _mp_raw_code_t mp_raw_code_t; +typedef struct _mp_raw_code_t { + mp_raw_code_kind_t kind : 3; + mp_uint_t scope_flags : 7; + mp_uint_t n_pos_args : 11; + union { + struct { + const byte *bytecode; + const mp_uint_t *const_table; + #if MICROPY_PERSISTENT_CODE_SAVE + mp_uint_t bc_len; + uint16_t n_obj; + uint16_t n_raw_code; + #endif + } u_byte; + struct { + void *fun_data; + const mp_uint_t *const_table; + mp_uint_t type_sig; // for viper, compressed as 2-bit types; ret is MSB, then arg0, arg1, etc + } u_native; + } data; +} mp_raw_code_t; mp_raw_code_t *mp_emit_glue_new_raw_code(void); @@ -51,8 +71,8 @@ void mp_emit_glue_assign_bytecode(mp_raw_code_t *rc, const byte *code, mp_uint_t mp_uint_t scope_flags); void mp_emit_glue_assign_native(mp_raw_code_t *rc, mp_raw_code_kind_t kind, void *fun_data, mp_uint_t fun_len, const mp_uint_t *const_table, mp_uint_t n_pos_args, mp_uint_t scope_flags, mp_uint_t type_sig); -mp_obj_t mp_make_function_from_raw_code(mp_raw_code_t *rc, mp_obj_t def_args, mp_obj_t def_kw_args); -mp_obj_t mp_make_closure_from_raw_code(mp_raw_code_t *rc, mp_uint_t n_closed_over, const mp_obj_t *args); +mp_obj_t mp_make_function_from_raw_code(const mp_raw_code_t *rc, mp_obj_t def_args, mp_obj_t def_kw_args); +mp_obj_t mp_make_closure_from_raw_code(const mp_raw_code_t *rc, mp_uint_t n_closed_over, const mp_obj_t *args); #if MICROPY_PERSISTENT_CODE_LOAD typedef struct _mp_reader_t { diff --git a/py/emitnative.c b/py/emitnative.c index a03ab36ed5..9adaabc11c 100644 --- a/py/emitnative.c +++ b/py/emitnative.c @@ -1208,7 +1208,7 @@ STATIC void emit_get_stack_pointer_to_reg_for_pop(emit_t *emit, mp_uint_t reg_de break; case VTYPE_INT: case VTYPE_UINT: - ASM_MOV_IMM_TO_LOCAL_USING(emit->as, (si->data.u_imm << 1) | 1, emit->stack_start + emit->stack_size - 1 - i, reg_dest); + ASM_MOV_IMM_TO_LOCAL_USING(emit->as, (uintptr_t)MP_OBJ_NEW_SMALL_INT(si->data.u_imm), emit->stack_start + emit->stack_size - 1 - i, reg_dest); si->vtype = VTYPE_PYOBJ; break; default: diff --git a/py/frozenmod.c b/py/frozenmod.c index 6b76bf662b..18beb0f8e4 100644 --- a/py/frozenmod.c +++ b/py/frozenmod.c @@ -4,6 +4,7 @@ * The MIT License (MIT) * * Copyright (c) 2015 Paul Sokolovsky + * Copyright (c) 2016 Damien P. George * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to deal @@ -30,7 +31,7 @@ #include "py/lexer.h" #include "py/frozenmod.h" -#if MICROPY_MODULE_FROZEN +#if MICROPY_MODULE_FROZEN_STR #ifndef MICROPY_MODULE_FROZEN_LEXER #define MICROPY_MODULE_FROZEN_LEXER mp_lexer_new_from_str_len @@ -38,24 +39,68 @@ mp_lexer_t *MICROPY_MODULE_FROZEN_LEXER(qstr src_name, const char *str, mp_uint_t len, mp_uint_t free_len); #endif -extern const char mp_frozen_names[]; -extern const uint32_t mp_frozen_sizes[]; -extern const char mp_frozen_content[]; +extern const char mp_frozen_str_names[]; +extern const uint32_t mp_frozen_str_sizes[]; +extern const char mp_frozen_str_content[]; -mp_lexer_t *mp_find_frozen_module(const char *str, int len) { - const char *name = mp_frozen_names; +STATIC mp_lexer_t *mp_find_frozen_str(const char *str, size_t len) { + const char *name = mp_frozen_str_names; size_t offset = 0; for (int i = 0; *name != 0; i++) { - int l = strlen(name); + size_t l = strlen(name); if (l == len && !memcmp(str, name, l)) { - mp_lexer_t *lex = MICROPY_MODULE_FROZEN_LEXER(MP_QSTR_, mp_frozen_content + offset, mp_frozen_sizes[i], 0); + qstr source = qstr_from_strn(name, l); + mp_lexer_t *lex = MICROPY_MODULE_FROZEN_LEXER(source, mp_frozen_str_content + offset, mp_frozen_str_sizes[i], 0); return lex; } name += l + 1; - offset += mp_frozen_sizes[i] + 1; + offset += mp_frozen_str_sizes[i] + 1; + } + return NULL; +} + +#endif + +#if MICROPY_MODULE_FROZEN_MPY + +#include "py/emitglue.h" + +extern const char mp_frozen_mpy_names[]; +extern const mp_raw_code_t *const mp_frozen_mpy_content[]; + +STATIC const mp_raw_code_t *mp_find_frozen_mpy(const char *str, size_t len) { + const char *name = mp_frozen_mpy_names; + for (size_t i = 0; *name != 0; i++) { + size_t l = strlen(name); + if (l == len && !memcmp(str, name, l)) { + return mp_frozen_mpy_content[i]; + } + name += l + 1; } return NULL; } -#endif // MICROPY_MODULE_FROZEN +#endif + +#if MICROPY_MODULE_FROZEN + +int mp_find_frozen_module(const char *str, size_t len, void **data) { + #if MICROPY_MODULE_FROZEN_STR + mp_lexer_t *lex = mp_find_frozen_str(str, len); + if (lex != NULL) { + *data = lex; + return MP_FROZEN_STR; + } + #endif + #if MICROPY_MODULE_FROZEN_MPY + const mp_raw_code_t *rc = mp_find_frozen_mpy(str, len); + if (rc != NULL) { + *data = (void*)rc; + return MP_FROZEN_MPY; + } + #endif + return MP_FROZEN_NONE; +} + +#endif diff --git a/py/frozenmod.h b/py/frozenmod.h index 67caced14d..a1638d2293 100644 --- a/py/frozenmod.h +++ b/py/frozenmod.h @@ -24,4 +24,10 @@ * THE SOFTWARE. */ -mp_lexer_t *mp_find_frozen_module(const char *str, int len); +enum { + MP_FROZEN_NONE, + MP_FROZEN_STR, + MP_FROZEN_MPY, +}; + +int mp_find_frozen_module(const char *str, size_t len, void **data); diff --git a/py/grammar.h b/py/grammar.h index 89a5a06537..dd21d193a1 100644 --- a/py/grammar.h +++ b/py/grammar.h @@ -38,16 +38,17 @@ // eval_input: testlist NEWLINE* ENDMARKER DEF_RULE(single_input, nc, or(3), tok(NEWLINE), rule(simple_stmt), rule(compound_stmt)) -DEF_RULE(file_input, c(generic_all_nodes), and(1), opt_rule(file_input_2)) +DEF_RULE(file_input, c(generic_all_nodes), and_ident(1), opt_rule(file_input_2)) DEF_RULE(file_input_2, c(generic_all_nodes), one_or_more, rule(file_input_3)) DEF_RULE(file_input_3, nc, or(2), tok(NEWLINE), rule(stmt)) -DEF_RULE(eval_input, nc, and(2), rule(testlist), opt_rule(eval_input_2)) +DEF_RULE(eval_input, nc, and_ident(2), rule(testlist), opt_rule(eval_input_2)) DEF_RULE(eval_input_2, nc, and(1), tok(NEWLINE)) // decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE // decorators: decorator+ -// decorated: decorators (classdef | funcdef) +// decorated: decorators (classdef | funcdef | async_funcdef) // funcdef: 'def' NAME parameters ['->' test] ':' suite +// async_funcdef: 'async' funcdef // parameters: '(' [typedargslist] ')' // typedargslist: tfpdef ['=' test] (',' tfpdef ['=' test])* [',' ['*' [tfpdef] (',' tfpdef ['=' test])* [',' '**' tfpdef] | '**' tfpdef]] | '*' [tfpdef] (',' tfpdef ['=' test])* [',' '**' tfpdef] | '**' tfpdef // tfpdef: NAME [':' test] @@ -56,27 +57,32 @@ DEF_RULE(eval_input_2, nc, and(1), tok(NEWLINE)) DEF_RULE(decorator, nc, and(4), tok(DEL_AT), rule(dotted_name), opt_rule(trailer_paren), tok(NEWLINE)) DEF_RULE(decorators, nc, one_or_more, rule(decorator)) -DEF_RULE(decorated, c(decorated), and(2), rule(decorators), rule(decorated_body)) +DEF_RULE(decorated, c(decorated), and_ident(2), rule(decorators), rule(decorated_body)) +#if MICROPY_PY_ASYNC_AWAIT +DEF_RULE(decorated_body, nc, or(3), rule(classdef), rule(funcdef), rule(async_funcdef)) +DEF_RULE(async_funcdef, nc, and(2), tok(KW_ASYNC), rule(funcdef)) +#else DEF_RULE(decorated_body, nc, or(2), rule(classdef), rule(funcdef)) -DEF_RULE(funcdef, c(funcdef), blank | and(8), tok(KW_DEF), tok(NAME), tok(DEL_PAREN_OPEN), opt_rule(typedargslist), tok(DEL_PAREN_CLOSE), opt_rule(funcdefrettype), tok(DEL_COLON), rule(suite)) -DEF_RULE(funcdefrettype, nc, ident | and(2), tok(DEL_MINUS_MORE), rule(test)) +#endif +DEF_RULE(funcdef, c(funcdef), and_blank(8), tok(KW_DEF), tok(NAME), tok(DEL_PAREN_OPEN), opt_rule(typedargslist), tok(DEL_PAREN_CLOSE), opt_rule(funcdefrettype), tok(DEL_COLON), rule(suite)) +DEF_RULE(funcdefrettype, nc, and_ident(2), tok(DEL_MINUS_MORE), rule(test)) // note: typedargslist lets through more than is allowed, compiler does further checks DEF_RULE(typedargslist, nc, list_with_end, rule(typedargslist_item), tok(DEL_COMMA)) DEF_RULE(typedargslist_item, nc, or(3), rule(typedargslist_name), rule(typedargslist_star), rule(typedargslist_dbl_star)) -DEF_RULE(typedargslist_name, nc, ident | and(3), tok(NAME), opt_rule(typedargslist_colon), opt_rule(typedargslist_equal)) +DEF_RULE(typedargslist_name, nc, and_ident(3), tok(NAME), opt_rule(typedargslist_colon), opt_rule(typedargslist_equal)) DEF_RULE(typedargslist_star, nc, and(2), tok(OP_STAR), opt_rule(tfpdef)) DEF_RULE(typedargslist_dbl_star, nc, and(3), tok(OP_DBL_STAR), tok(NAME), opt_rule(typedargslist_colon)) -DEF_RULE(typedargslist_colon, nc, ident | and(2), tok(DEL_COLON), rule(test)) -DEF_RULE(typedargslist_equal, nc, ident | and(2), tok(DEL_EQUAL), rule(test)) +DEF_RULE(typedargslist_colon, nc, and_ident(2), tok(DEL_COLON), rule(test)) +DEF_RULE(typedargslist_equal, nc, and_ident(2), tok(DEL_EQUAL), rule(test)) DEF_RULE(tfpdef, nc, and(2), tok(NAME), opt_rule(typedargslist_colon)) // note: varargslist lets through more than is allowed, compiler does further checks DEF_RULE(varargslist, nc, list_with_end, rule(varargslist_item), tok(DEL_COMMA)) DEF_RULE(varargslist_item, nc, or(3), rule(varargslist_name), rule(varargslist_star), rule(varargslist_dbl_star)) -DEF_RULE(varargslist_name, nc, ident | and(2), tok(NAME), opt_rule(varargslist_equal)) +DEF_RULE(varargslist_name, nc, and_ident(2), tok(NAME), opt_rule(varargslist_equal)) DEF_RULE(varargslist_star, nc, and(2), tok(OP_STAR), opt_rule(vfpdef)) DEF_RULE(varargslist_dbl_star, nc, and(2), tok(OP_DBL_STAR), tok(NAME)) -DEF_RULE(varargslist_equal, nc, ident | and(2), tok(DEL_EQUAL), rule(test)) -DEF_RULE(vfpdef, nc, ident | and(1), tok(NAME)) +DEF_RULE(varargslist_equal, nc, and_ident(2), tok(DEL_EQUAL), rule(test)) +DEF_RULE(vfpdef, nc, and_ident(1), tok(NAME)) // stmt: compound_stmt | simple_stmt @@ -84,7 +90,7 @@ DEF_RULE(stmt, nc, or(2), rule(compound_stmt), rule(simple_stmt)) // simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE -DEF_RULE(simple_stmt, nc, and(2), rule(simple_stmt_2), tok(NEWLINE)) +DEF_RULE(simple_stmt, nc, and_ident(2), rule(simple_stmt_2), tok(NEWLINE)) DEF_RULE(simple_stmt_2, c(generic_all_nodes), list_with_end, rule(small_stmt), tok(DEL_SEMICOLON)) // small_stmt: expr_stmt | del_stmt | pass_stmt | flow_stmt | import_stmt | global_stmt | nonlocal_stmt | assert_stmt @@ -96,9 +102,9 @@ DEF_RULE(simple_stmt_2, c(generic_all_nodes), list_with_end, rule(small_stmt), t DEF_RULE(small_stmt, nc, or(8), rule(del_stmt), rule(pass_stmt), rule(flow_stmt), rule(import_stmt), rule(global_stmt), rule(nonlocal_stmt), rule(assert_stmt), rule(expr_stmt)) DEF_RULE(expr_stmt, c(expr_stmt), and(2), rule(testlist_star_expr), opt_rule(expr_stmt_2)) DEF_RULE(expr_stmt_2, nc, or(2), rule(expr_stmt_augassign), rule(expr_stmt_assign_list)) -DEF_RULE(expr_stmt_augassign, nc, and(2), rule(augassign), rule(expr_stmt_6)) +DEF_RULE(expr_stmt_augassign, nc, and_ident(2), rule(augassign), rule(expr_stmt_6)) DEF_RULE(expr_stmt_assign_list, nc, one_or_more, rule(expr_stmt_assign)) -DEF_RULE(expr_stmt_assign, nc, ident | and(2), tok(DEL_EQUAL), rule(expr_stmt_6)) +DEF_RULE(expr_stmt_assign, nc, and_ident(2), tok(DEL_EQUAL), rule(expr_stmt_6)) DEF_RULE(expr_stmt_6, nc, or(2), rule(yield_expr), rule(testlist_star_expr)) DEF_RULE(testlist_star_expr, c(generic_tuple), list_with_end, rule(testlist_star_expr_2), tok(DEL_COMMA)) DEF_RULE(testlist_star_expr_2, nc, or(2), rule(star_expr), rule(test)) @@ -121,8 +127,8 @@ DEF_RULE(continue_stmt, c(continue_stmt), and(1), tok(KW_CONTINUE)) DEF_RULE(return_stmt, c(return_stmt), and(2), tok(KW_RETURN), opt_rule(testlist)) DEF_RULE(yield_stmt, c(yield_stmt), and(1), rule(yield_expr)) DEF_RULE(raise_stmt, c(raise_stmt), and(2), tok(KW_RAISE), opt_rule(raise_stmt_arg)) -DEF_RULE(raise_stmt_arg, nc, and(2), rule(test), opt_rule(raise_stmt_from)) -DEF_RULE(raise_stmt_from, nc, ident | and(2), tok(KW_FROM), rule(test)) +DEF_RULE(raise_stmt_arg, nc, and_ident(2), rule(test), opt_rule(raise_stmt_from)) +DEF_RULE(raise_stmt_from, nc, and_ident(2), tok(KW_FROM), rule(test)) // import_stmt: import_name | import_from // import_name: 'import' dotted_as_names @@ -140,14 +146,14 @@ DEF_RULE(import_stmt, nc, or(2), rule(import_name), rule(import_from)) DEF_RULE(import_name, c(import_name), and(2), tok(KW_IMPORT), rule(dotted_as_names)) DEF_RULE(import_from, c(import_from), and(4), tok(KW_FROM), rule(import_from_2), tok(KW_IMPORT), rule(import_from_3)) DEF_RULE(import_from_2, nc, or(2), rule(dotted_name), rule(import_from_2b)) -DEF_RULE(import_from_2b, nc, and(2), rule(one_or_more_period_or_ellipsis), opt_rule(dotted_name)) +DEF_RULE(import_from_2b, nc, and_ident(2), rule(one_or_more_period_or_ellipsis), opt_rule(dotted_name)) DEF_RULE(import_from_3, nc, or(3), tok(OP_STAR), rule(import_as_names_paren), rule(import_as_names)) -DEF_RULE(import_as_names_paren, nc, ident | and(3), tok(DEL_PAREN_OPEN), rule(import_as_names), tok(DEL_PAREN_CLOSE)) +DEF_RULE(import_as_names_paren, nc, and_ident(3), tok(DEL_PAREN_OPEN), rule(import_as_names), tok(DEL_PAREN_CLOSE)) DEF_RULE(one_or_more_period_or_ellipsis, nc, one_or_more, rule(period_or_ellipsis)) DEF_RULE(period_or_ellipsis, nc, or(2), tok(DEL_PERIOD), tok(ELLIPSIS)) DEF_RULE(import_as_name, nc, and(2), tok(NAME), opt_rule(as_name)) -DEF_RULE(dotted_as_name, nc, and(2), rule(dotted_name), opt_rule(as_name)) -DEF_RULE(as_name, nc, ident | and(2), tok(KW_AS), tok(NAME)) +DEF_RULE(dotted_as_name, nc, and_ident(2), rule(dotted_name), opt_rule(as_name)) +DEF_RULE(as_name, nc, and_ident(2), tok(KW_AS), tok(NAME)) DEF_RULE(import_as_names, nc, list_with_end, rule(import_as_name), tok(DEL_COMMA)) DEF_RULE(dotted_as_names, nc, list, rule(dotted_as_name), tok(DEL_COMMA)) DEF_RULE(dotted_name, nc, list, tok(NAME), tok(DEL_PERIOD)) @@ -155,9 +161,9 @@ DEF_RULE(global_stmt, c(global_stmt), and(2), tok(KW_GLOBAL), rule(name_list)) DEF_RULE(nonlocal_stmt, c(nonlocal_stmt), and(2), tok(KW_NONLOCAL), rule(name_list)) DEF_RULE(name_list, nc, list, tok(NAME), tok(DEL_COMMA)) DEF_RULE(assert_stmt, c(assert_stmt), and(3), tok(KW_ASSERT), rule(test), opt_rule(assert_stmt_extra)) -DEF_RULE(assert_stmt_extra, nc, ident | and(2), tok(DEL_COMMA), rule(test)) +DEF_RULE(assert_stmt_extra, nc, and_ident(2), tok(DEL_COMMA), rule(test)) -// compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated +// compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated | async_stmt // if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite] // while_stmt: 'while' test ':' suite ['else' ':' suite] // for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite] @@ -167,8 +173,15 @@ DEF_RULE(assert_stmt_extra, nc, ident | and(2), tok(DEL_COMMA), rule(test)) // with_stmt: 'with' with_item (',' with_item)* ':' suite // with_item: test ['as' expr] // suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT +// async_stmt: 'async' (funcdef | with_stmt | for_stmt) +#if MICROPY_PY_ASYNC_AWAIT +DEF_RULE(compound_stmt, nc, or(9), rule(if_stmt), rule(while_stmt), rule(for_stmt), rule(try_stmt), rule(with_stmt), rule(funcdef), rule(classdef), rule(decorated), rule(async_stmt)) +DEF_RULE(async_stmt, c(async_stmt), and(2), tok(KW_ASYNC), rule(async_stmt_2)) +DEF_RULE(async_stmt_2, nc, or(3), rule(funcdef), rule(with_stmt), rule(for_stmt)) +#else DEF_RULE(compound_stmt, nc, or(8), rule(if_stmt), rule(while_stmt), rule(for_stmt), rule(try_stmt), rule(with_stmt), rule(funcdef), rule(classdef), rule(decorated)) +#endif DEF_RULE(if_stmt, c(if_stmt), and(6), tok(KW_IF), rule(test), tok(DEL_COLON), rule(suite), opt_rule(if_stmt_elif_list), opt_rule(else_stmt)) DEF_RULE(if_stmt_elif_list, nc, one_or_more, rule(if_stmt_elif)) DEF_RULE(if_stmt_elif, nc, and(4), tok(KW_ELIF), rule(test), tok(DEL_COLON), rule(suite)) @@ -176,18 +189,18 @@ DEF_RULE(while_stmt, c(while_stmt), and(5), tok(KW_WHILE), rule(test), tok(DEL_C DEF_RULE(for_stmt, c(for_stmt), and(7), tok(KW_FOR), rule(exprlist), tok(KW_IN), rule(testlist), tok(DEL_COLON), rule(suite), opt_rule(else_stmt)) DEF_RULE(try_stmt, c(try_stmt), and(4), tok(KW_TRY), tok(DEL_COLON), rule(suite), rule(try_stmt_2)) DEF_RULE(try_stmt_2, nc, or(2), rule(try_stmt_except_and_more), rule(try_stmt_finally)) -DEF_RULE(try_stmt_except_and_more, nc, and(3), rule(try_stmt_except_list), opt_rule(else_stmt), opt_rule(try_stmt_finally)) +DEF_RULE(try_stmt_except_and_more, nc, and_ident(3), rule(try_stmt_except_list), opt_rule(else_stmt), opt_rule(try_stmt_finally)) DEF_RULE(try_stmt_except, nc, and(4), tok(KW_EXCEPT), opt_rule(try_stmt_as_name), tok(DEL_COLON), rule(suite)) -DEF_RULE(try_stmt_as_name, nc, and(2), rule(test), opt_rule(as_name)) +DEF_RULE(try_stmt_as_name, nc, and_ident(2), rule(test), opt_rule(as_name)) DEF_RULE(try_stmt_except_list, nc, one_or_more, rule(try_stmt_except)) DEF_RULE(try_stmt_finally, nc, and(3), tok(KW_FINALLY), tok(DEL_COLON), rule(suite)) -DEF_RULE(else_stmt, nc, ident | and(3), tok(KW_ELSE), tok(DEL_COLON), rule(suite)) +DEF_RULE(else_stmt, nc, and_ident(3), tok(KW_ELSE), tok(DEL_COLON), rule(suite)) DEF_RULE(with_stmt, c(with_stmt), and(4), tok(KW_WITH), rule(with_stmt_list), tok(DEL_COLON), rule(suite)) DEF_RULE(with_stmt_list, nc, list, rule(with_item), tok(DEL_COMMA)) -DEF_RULE(with_item, nc, and(2), rule(test), opt_rule(with_item_as)) -DEF_RULE(with_item_as, nc, ident | and(2), tok(KW_AS), rule(expr)) +DEF_RULE(with_item, nc, and_ident(2), rule(test), opt_rule(with_item_as)) +DEF_RULE(with_item_as, nc, and_ident(2), tok(KW_AS), rule(expr)) DEF_RULE(suite, nc, or(2), rule(suite_block), rule(simple_stmt)) -DEF_RULE(suite_block, nc, and(4), tok(NEWLINE), tok(INDENT), rule(suite_block_stmts), tok(DEDENT)) +DEF_RULE(suite_block, nc, and_ident(4), tok(NEWLINE), tok(INDENT), rule(suite_block_stmts), tok(DEDENT)) DEF_RULE(suite_block_stmts, c(generic_all_nodes), one_or_more, rule(stmt)) // test: or_test ['if' or_test 'else' test] | lambdef @@ -196,11 +209,11 @@ DEF_RULE(suite_block_stmts, c(generic_all_nodes), one_or_more, rule(stmt)) // lambdef_nocond: 'lambda' [varargslist] ':' test_nocond DEF_RULE(test, nc, or(2), rule(lambdef), rule(test_if_expr)) -DEF_RULE(test_if_expr, c(test_if_expr), and(2), rule(or_test), opt_rule(test_if_else)) +DEF_RULE(test_if_expr, c(test_if_expr), and_ident(2), rule(or_test), opt_rule(test_if_else)) DEF_RULE(test_if_else, nc, and(4), tok(KW_IF), rule(or_test), tok(KW_ELSE), rule(test)) DEF_RULE(test_nocond, nc, or(2), rule(lambdef_nocond), rule(or_test)) -DEF_RULE(lambdef, c(lambdef), blank | and(4), tok(KW_LAMBDA), opt_rule(varargslist), tok(DEL_COLON), rule(test)) -DEF_RULE(lambdef_nocond, c(lambdef), blank | and(4), tok(KW_LAMBDA), opt_rule(varargslist), tok(DEL_COLON), rule(test_nocond)) +DEF_RULE(lambdef, c(lambdef), and_blank(4), tok(KW_LAMBDA), opt_rule(varargslist), tok(DEL_COLON), rule(test)) +DEF_RULE(lambdef_nocond, c(lambdef), and_blank(4), tok(KW_LAMBDA), opt_rule(varargslist), tok(DEL_COLON), rule(test_nocond)) // or_test: and_test ('or' and_test)* // and_test: not_test ('and' not_test)* @@ -215,7 +228,8 @@ DEF_RULE(lambdef_nocond, c(lambdef), blank | and(4), tok(KW_LAMBDA), opt_rule(va // arith_expr: term (('+'|'-') term)* // term: factor (('*'|'/'|'%'|'//') factor)* // factor: ('+'|'-'|'~') factor | power -// power: atom trailer* ['**' factor] +// power: atom_expr ['**' factor] +// atom_expr: 'await' atom trailer* | atom trailer* DEF_RULE(or_test, c(or_test), list, rule(and_test), tok(KW_OR)) DEF_RULE(and_test, c(and_test), list, rule(not_test), tok(KW_AND)) @@ -237,11 +251,18 @@ DEF_RULE(arith_op, nc, or(2), tok(OP_PLUS), tok(OP_MINUS)) DEF_RULE(term, c(term), list, rule(factor), rule(term_op)) DEF_RULE(term_op, nc, or(4), tok(OP_STAR), tok(OP_SLASH), tok(OP_PERCENT), tok(OP_DBL_SLASH)) DEF_RULE(factor, nc, or(2), rule(factor_2), rule(power)) -DEF_RULE(factor_2, c(factor_2), and(2), rule(factor_op), rule(factor)) +DEF_RULE(factor_2, c(factor_2), and_ident(2), rule(factor_op), rule(factor)) DEF_RULE(factor_op, nc, or(3), tok(OP_PLUS), tok(OP_MINUS), tok(OP_TILDE)) -DEF_RULE(power, c(power), and(3), rule(atom), opt_rule(power_trailers), opt_rule(power_dbl_star)) -DEF_RULE(power_trailers, c(power_trailers), one_or_more, rule(trailer)) -DEF_RULE(power_dbl_star, nc, ident | and(2), tok(OP_DBL_STAR), rule(factor)) +DEF_RULE(power, c(power), and_ident(2), rule(atom_expr), opt_rule(power_dbl_star)) +#if MICROPY_PY_ASYNC_AWAIT +DEF_RULE(atom_expr, nc, or(2), rule(atom_expr_await), rule(atom_expr_normal)) +DEF_RULE(atom_expr_await, c(atom_expr_await), and(3), tok(KW_AWAIT), rule(atom), opt_rule(atom_expr_trailers)) +#else +DEF_RULE(atom_expr, nc, or(1), rule(atom_expr_normal)) +#endif +DEF_RULE(atom_expr_normal, c(atom_expr_normal), and_ident(2), rule(atom), opt_rule(atom_expr_trailers)) +DEF_RULE(atom_expr_trailers, c(atom_expr_trailers), one_or_more, rule(trailer)) +DEF_RULE(power_dbl_star, nc, and_ident(2), tok(OP_DBL_STAR), rule(factor)) // atom: '(' [yield_expr|testlist_comp] ')' | '[' [testlist_comp] ']' | '{' [dictorsetmaker] '}' | NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False' // testlist_comp: (test|star_expr) ( comp_for | (',' (test|star_expr))* [','] ) @@ -253,10 +274,10 @@ DEF_RULE(atom_paren, c(atom_paren), and(3), tok(DEL_PAREN_OPEN), opt_rule(atom_2 DEF_RULE(atom_2b, nc, or(2), rule(yield_expr), rule(testlist_comp)) DEF_RULE(atom_bracket, c(atom_bracket), and(3), tok(DEL_BRACKET_OPEN), opt_rule(testlist_comp), tok(DEL_BRACKET_CLOSE)) DEF_RULE(atom_brace, c(atom_brace), and(3), tok(DEL_BRACE_OPEN), opt_rule(dictorsetmaker), tok(DEL_BRACE_CLOSE)) -DEF_RULE(testlist_comp, nc, and(2), rule(testlist_comp_2), opt_rule(testlist_comp_3)) +DEF_RULE(testlist_comp, nc, and_ident(2), rule(testlist_comp_2), opt_rule(testlist_comp_3)) DEF_RULE(testlist_comp_2, nc, or(2), rule(star_expr), rule(test)) DEF_RULE(testlist_comp_3, nc, or(2), rule(comp_for), rule(testlist_comp_3b)) -DEF_RULE(testlist_comp_3b, nc, ident | and(2), tok(DEL_COMMA), opt_rule(testlist_comp_3c)) +DEF_RULE(testlist_comp_3b, nc, and_ident(2), tok(DEL_COMMA), opt_rule(testlist_comp_3c)) DEF_RULE(testlist_comp_3c, nc, list_with_end, rule(testlist_comp_2), tok(DEL_COMMA)) DEF_RULE(trailer, nc, or(3), rule(trailer_paren), rule(trailer_bracket), rule(trailer_period)) DEF_RULE(trailer_paren, c(trailer_paren), and(3), tok(DEL_PAREN_OPEN), opt_rule(arglist), tok(DEL_PAREN_CLOSE)) @@ -270,11 +291,11 @@ DEF_RULE(trailer_period, c(trailer_period), and(2), tok(DEL_PERIOD), tok(NAME)) #if MICROPY_PY_BUILTINS_SLICE DEF_RULE(subscriptlist, c(generic_tuple), list_with_end, rule(subscript), tok(DEL_COMMA)) DEF_RULE(subscript, nc, or(2), rule(subscript_3), rule(subscript_2)) -DEF_RULE(subscript_2, c(subscript_2), and(2), rule(test), opt_rule(subscript_3)) +DEF_RULE(subscript_2, c(subscript_2), and_ident(2), rule(test), opt_rule(subscript_3)) DEF_RULE(subscript_3, c(subscript_3), and(2), tok(DEL_COLON), opt_rule(subscript_3b)) DEF_RULE(subscript_3b, nc, or(2), rule(subscript_3c), rule(subscript_3d)) DEF_RULE(subscript_3c, nc, and(2), tok(DEL_COLON), opt_rule(test)) -DEF_RULE(subscript_3d, nc, and(2), rule(test), opt_rule(sliceop)) +DEF_RULE(subscript_3d, nc, and_ident(2), rule(test), opt_rule(sliceop)) DEF_RULE(sliceop, nc, and(2), tok(DEL_COLON), opt_rule(test)) #else DEF_RULE(subscriptlist, c(generic_tuple), list_with_end, rule(test), tok(DEL_COMMA)) @@ -288,10 +309,10 @@ DEF_RULE(exprlist, nc, list_with_end, rule(exprlist_2), tok(DEL_COMMA)) DEF_RULE(exprlist_2, nc, or(2), rule(star_expr), rule(expr)) DEF_RULE(testlist, c(generic_tuple), list_with_end, rule(test), tok(DEL_COMMA)) // TODO dictorsetmaker lets through more than is allowed -DEF_RULE(dictorsetmaker, nc, and(2), rule(dictorsetmaker_item), opt_rule(dictorsetmaker_tail)) +DEF_RULE(dictorsetmaker, nc, and_ident(2), rule(dictorsetmaker_item), opt_rule(dictorsetmaker_tail)) #if MICROPY_PY_BUILTINS_SET -DEF_RULE(dictorsetmaker_item, c(dictorsetmaker_item), and(2), rule(test), opt_rule(dictorsetmaker_colon)) -DEF_RULE(dictorsetmaker_colon, nc, ident | and(2), tok(DEL_COLON), rule(test)) +DEF_RULE(dictorsetmaker_item, c(dictorsetmaker_item), and_ident(2), rule(test), opt_rule(dictorsetmaker_colon)) +DEF_RULE(dictorsetmaker_colon, nc, and_ident(2), tok(DEL_COLON), rule(test)) #else DEF_RULE(dictorsetmaker_item, c(dictorsetmaker_item), and(3), rule(test), tok(DEL_COLON), rule(test)) #endif @@ -301,8 +322,8 @@ DEF_RULE(dictorsetmaker_list2, nc, list_with_end, rule(dictorsetmaker_item), tok // classdef: 'class' NAME ['(' [arglist] ')'] ':' suite -DEF_RULE(classdef, c(classdef), blank | and(5), tok(KW_CLASS), tok(NAME), opt_rule(classdef_2), tok(DEL_COLON), rule(suite)) -DEF_RULE(classdef_2, nc, ident | and(3), tok(DEL_PAREN_OPEN), opt_rule(arglist), tok(DEL_PAREN_CLOSE)) +DEF_RULE(classdef, c(classdef), and_blank(5), tok(KW_CLASS), tok(NAME), opt_rule(classdef_2), tok(DEL_COLON), rule(suite)) +DEF_RULE(classdef_2, nc, and_ident(3), tok(DEL_PAREN_OPEN), opt_rule(arglist), tok(DEL_PAREN_CLOSE)) // arglist: (argument ',')* (argument [','] | '*' test (',' argument)* [',' '**' test] | '**' test) @@ -319,11 +340,11 @@ DEF_RULE(arglist_dbl_star, nc, and(2), tok(OP_DBL_STAR), rule(test)) // comp_for: 'for' exprlist 'in' or_test [comp_iter] // comp_if: 'if' test_nocond [comp_iter] -DEF_RULE(argument, nc, and(2), rule(test), opt_rule(argument_2)) +DEF_RULE(argument, nc, and_ident(2), rule(test), opt_rule(argument_2)) DEF_RULE(argument_2, nc, or(2), rule(comp_for), rule(argument_3)) -DEF_RULE(argument_3, nc, ident | and(2), tok(DEL_EQUAL), rule(test)) +DEF_RULE(argument_3, nc, and_ident(2), tok(DEL_EQUAL), rule(test)) DEF_RULE(comp_iter, nc, or(2), rule(comp_for), rule(comp_if)) -DEF_RULE(comp_for, nc, blank | and(5), tok(KW_FOR), rule(exprlist), tok(KW_IN), rule(or_test), opt_rule(comp_iter)) +DEF_RULE(comp_for, nc, and_blank(5), tok(KW_FOR), rule(exprlist), tok(KW_IN), rule(or_test), opt_rule(comp_iter)) DEF_RULE(comp_if, nc, and(3), tok(KW_IF), rule(test_nocond), opt_rule(comp_iter)) // # not used in grammar, but may appear in "node" passed from Parser to Compiler diff --git a/py/lexer.c b/py/lexer.c index 76abedd451..1639740d34 100644 --- a/py/lexer.c +++ b/py/lexer.c @@ -234,6 +234,10 @@ STATIC const char *tok_kw[] = { "and", "as", "assert", + #if MICROPY_PY_ASYNC_AWAIT + "async", + "await", + #endif "break", "class", "continue", diff --git a/py/lexer.h b/py/lexer.h index 36d1e99d23..463be5fffc 100644 --- a/py/lexer.h +++ b/py/lexer.h @@ -63,6 +63,10 @@ typedef enum _mp_token_kind_t { MP_TOKEN_KW_AND, MP_TOKEN_KW_AS, MP_TOKEN_KW_ASSERT, + #if MICROPY_PY_ASYNC_AWAIT + MP_TOKEN_KW_ASYNC, + MP_TOKEN_KW_AWAIT, + #endif MP_TOKEN_KW_BREAK, MP_TOKEN_KW_CLASS, MP_TOKEN_KW_CONTINUE, diff --git a/py/makeqstrdata.py b/py/makeqstrdata.py index e8adb0cbb6..c00ec1eb28 100644 --- a/py/makeqstrdata.py +++ b/py/makeqstrdata.py @@ -9,11 +9,15 @@ from __future__ import print_function import re import sys -# codepoint2name is different in Python 2 to Python 3 +# Python 2/3 compatibility: +# - iterating through bytes is different +# - codepoint2name lives in a different module import platform if platform.python_version_tuple()[0] == '2': + ord_bytes = ord from htmlentitydefs import codepoint2name elif platform.python_version_tuple()[0] == '3': + ord_bytes = lambda x:x from html.entities import codepoint2name codepoint2name[ord('-')] = 'hyphen'; @@ -23,6 +27,7 @@ codepoint2name[ord('\'')] = 'squot' codepoint2name[ord(',')] = 'comma' codepoint2name[ord('.')] = 'dot' codepoint2name[ord(':')] = 'colon' +codepoint2name[ord(';')] = 'semicolon' codepoint2name[ord('/')] = 'slash' codepoint2name[ord('%')] = 'percent' codepoint2name[ord('#')] = 'hash' @@ -36,6 +41,13 @@ codepoint2name[ord('*')] = 'star' codepoint2name[ord('!')] = 'bang' codepoint2name[ord('\\')] = 'backslash' codepoint2name[ord('+')] = 'plus' +codepoint2name[ord('$')] = 'dollar' +codepoint2name[ord('=')] = 'equals' +codepoint2name[ord('?')] = 'question' +codepoint2name[ord('@')] = 'at_sign' +codepoint2name[ord('^')] = 'caret' +codepoint2name[ord('|')] = 'pipe' +codepoint2name[ord('~')] = 'tilde' # this must match the equivalent function in qstr.c def compute_hash(qstr, bytes_hash): @@ -45,7 +57,17 @@ def compute_hash(qstr, bytes_hash): # Make sure that valid hash is never zero, zero means "hash not computed" return (hash & ((1 << (8 * bytes_hash)) - 1)) or 1 -def do_work(infiles): +def qstr_escape(qst): + def esc_char(m): + c = ord(m.group(0)) + try: + name = codepoint2name[c] + except KeyError: + name = '0x%02x' % c + return "_" + name + '_' + return re.sub(r'[^A-Za-z0-9_]', esc_char, qst) + +def parse_input_headers(infiles): # read the qstrs in from the input files qcfgs = {} qstrs = {} @@ -71,7 +93,13 @@ def do_work(infiles): # get the qstr value qstr = match.group(1) - ident = re.sub(r'[^A-Za-z0-9_]', lambda s: "_" + codepoint2name[ord(s.group(0))] + "_", qstr) + + # special case to specify control characters + if qstr == '\\n': + qstr = '\n' + + # work out the corresponding qstr name + ident = qstr_escape(qstr) # don't add duplicates if ident in qstrs: @@ -84,10 +112,30 @@ def do_work(infiles): sys.stderr.write("ERROR: Empty preprocessor output - check for errors above\n") sys.exit(1) + return qcfgs, qstrs + +def make_bytes(cfg_bytes_len, cfg_bytes_hash, qstr): + qhash = compute_hash(qstr, cfg_bytes_hash) + if all(32 <= ord(c) <= 126 and c != '\\' for c in qstr): + # qstr is all printable ASCII so render it as-is (for easier debugging) + qlen = len(qstr) + qdata = qstr + else: + # qstr contains non-printable codes so render entire thing as hex pairs + qbytes = qstr.encode('utf8') + qlen = len(qbytes) + qdata = ''.join(('\\x%02x' % ord_bytes(b)) for b in qbytes) + if qlen >= (1 << (8 * cfg_bytes_len)): + print('qstr is too long:', qstr) + assert False + qlen_str = ('\\x%02x' * cfg_bytes_len) % tuple(((qlen >> (8 * i)) & 0xff) for i in range(cfg_bytes_len)) + qhash_str = ('\\x%02x' * cfg_bytes_hash) % tuple(((qhash >> (8 * i)) & 0xff) for i in range(cfg_bytes_hash)) + return '(const byte*)"%s%s" "%s"' % (qhash_str, qlen_str, qdata) + +def print_qstr_data(qcfgs, qstrs): # get config variables cfg_bytes_len = int(qcfgs['BYTES_IN_LEN']) cfg_bytes_hash = int(qcfgs['BYTES_IN_HASH']) - cfg_max_len = 1 << (8 * cfg_bytes_len) # print out the starter of the generated C header file print('// This file was automatically generated by makeqstrdata.py') @@ -98,16 +146,12 @@ def do_work(infiles): # go through each qstr and print it out for order, ident, qstr in sorted(qstrs.values(), key=lambda x: x[0]): - qhash = compute_hash(qstr, cfg_bytes_hash) - # Calculate len of str, taking escapes into account - qlen = len(qstr.replace("\\\\", "-").replace("\\", "")) - qdata = qstr.replace('"', '\\"') - if qlen >= cfg_max_len: - print('qstr is too long:', qstr) - assert False - qlen_str = ('\\x%02x' * cfg_bytes_len) % tuple(((qlen >> (8 * i)) & 0xff) for i in range(cfg_bytes_len)) - qhash_str = ('\\x%02x' * cfg_bytes_hash) % tuple(((qhash >> (8 * i)) & 0xff) for i in range(cfg_bytes_hash)) - print('QDEF(MP_QSTR_%s, (const byte*)"%s%s" "%s")' % (ident, qhash_str, qlen_str, qdata)) + qbytes = make_bytes(cfg_bytes_len, cfg_bytes_hash, qstr) + print('QDEF(MP_QSTR_%s, %s)' % (ident, qbytes)) + +def do_work(infiles): + qcfgs, qstrs = parse_input_headers(infiles) + print_qstr_data(qcfgs, qstrs) if __name__ == "__main__": do_work(sys.argv[1:]) diff --git a/py/makeqstrdefs.py b/py/makeqstrdefs.py new file mode 100644 index 0000000000..194d901d26 --- /dev/null +++ b/py/makeqstrdefs.py @@ -0,0 +1,109 @@ +""" +This script processes the output from the C preprocessor and extracts all +qstr. Each qstr is transformed into a qstr definition of the form 'Q(...)'. + +This script works with Python 2.6, 2.7, 3.3 and 3.4. +""" + +import re +import argparse +import os + +# Blacklist of qstrings that are specially handled in further +# processing and should be ignored +QSTRING_BLACK_LIST = {'NULL', 'number_of', } + + +def write_out(fname, output): + if output: + for m, r in [("/", "__"), ("\\", "__"), (":", "@"), ("..", "@@")]: + fname = fname.replace(m, r) + with open(args.output_dir + "/" + fname + ".qstr", "w") as f: + f.write("\n".join(output) + "\n") + +def process_file(f): + output = [] + last_fname = None + for line in f: + # match gcc-like output (# n "file") and msvc-like output (#line n "file") + if line and (line[0:2] == "# " or line[0:5] == "#line"): + m = re.match(r"#[line]*\s\d+\s\"([^\"]+)\"", line) + assert m is not None + fname = m.group(1) + if fname[0] == "/" or not fname.endswith(".c"): + continue + if fname != last_fname: + write_out(last_fname, output) + output = [] + last_fname = fname + continue + for match in re.findall(r'MP_QSTR_[_a-zA-Z0-9]+', line): + name = match.replace('MP_QSTR_', '') + if name not in QSTRING_BLACK_LIST: + output.append('Q(' + name + ')') + + write_out(last_fname, output) + return "" + + +def cat_together(): + import glob + import hashlib + hasher = hashlib.md5() + all_lines = [] + outf = open(args.output_dir + "/out", "wb") + for fname in glob.glob(args.output_dir + "/*.qstr"): + with open(fname, "rb") as f: + lines = f.readlines() + all_lines += lines + all_lines.sort() + all_lines = b"\n".join(all_lines) + outf.write(all_lines) + outf.close() + hasher.update(all_lines) + new_hash = hasher.hexdigest() + #print(new_hash) + old_hash = None + try: + with open(args.output_file + ".hash") as f: + old_hash = f.read() + except IOError: + pass + if old_hash != new_hash: + print("QSTR updated") + try: + # rename below might fail if file exists + os.remove(args.output_file) + except: + pass + os.rename(args.output_dir + "/out", args.output_file) + with open(args.output_file + ".hash", "w") as f: + f.write(new_hash) + else: + print("QSTR not updated") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description='Generates qstr definitions from a specified source') + + parser.add_argument('command', + help='Command (split/cat)') + parser.add_argument('input_filename', + help='Name of the input file (when not specified, the script reads standard input)') + parser.add_argument('output_dir', + help='Output directory to store individual qstr files') + parser.add_argument('output_file', + help='Name of the output file with collected qstrs') + + args = parser.parse_args() + try: + os.makedirs(args.output_dir) + except OSError: + pass + + if args.command == "split": + with open(args.input_filename) as infile: + process_file(infile) + + if args.command == "cat": + cat_together() @@ -45,19 +45,26 @@ const mp_map_t mp_const_empty_map = { .table = NULL, }; -// approximatelly doubling primes; made with Mathematica command: Table[Prime[Floor[(1.7)^n]], {n, 3, 24}] -// prefixed with zero for the empty case. -STATIC uint32_t doubling_primes[] = {0, 7, 19, 43, 89, 179, 347, 647, 1229, 2297, 4243, 7829, 14347, 26017, 47149, 84947, 152443, 273253, 488399, 869927, 1547173, 2745121, 4861607}; - -STATIC mp_uint_t get_doubling_prime_greater_or_equal_to(mp_uint_t x) { - for (size_t i = 0; i < MP_ARRAY_SIZE(doubling_primes); i++) { - if (doubling_primes[i] >= x) { - return doubling_primes[i]; +// This table of sizes is used to control the growth of hash tables. +// The first set of sizes are chosen so the allocation fits exactly in a +// 4-word GC block, and it's not so important for these small values to be +// prime. The latter sizes are prime and increase at an increasing rate. +STATIC uint16_t hash_allocation_sizes[] = { + 0, 2, 4, 6, 8, 10, 12, // +2 + 17, 23, 29, 37, 47, 59, 73, // *1.25 + 97, 127, 167, 223, 293, 389, 521, 691, 919, 1223, 1627, 2161, // *1.33 + 3229, 4831, 7243, 10861, 16273, 24407, 36607, 54907, // *1.5 +}; + +STATIC mp_uint_t get_hash_alloc_greater_or_equal_to(mp_uint_t x) { + for (size_t i = 0; i < MP_ARRAY_SIZE(hash_allocation_sizes); i++) { + if (hash_allocation_sizes[i] >= x) { + return hash_allocation_sizes[i]; } } // ran out of primes in the table! // return something sensible, at least make it odd - return x | 1; + return (x + x / 2) | 1; } /******************************************************************************/ @@ -118,7 +125,7 @@ void mp_map_clear(mp_map_t *map) { STATIC void mp_map_rehash(mp_map_t *map) { mp_uint_t old_alloc = map->alloc; - mp_uint_t new_alloc = get_doubling_prime_greater_or_equal_to(map->alloc + 1); + mp_uint_t new_alloc = get_hash_alloc_greater_or_equal_to(map->alloc + 1); mp_map_elem_t *old_table = map->table; mp_map_elem_t *new_table = m_new0(mp_map_elem_t, new_alloc); // If we reach this point, table resizing succeeded, now we can edit the old map. @@ -298,7 +305,7 @@ void mp_set_init(mp_set_t *set, mp_uint_t n) { STATIC void mp_set_rehash(mp_set_t *set) { mp_uint_t old_alloc = set->alloc; mp_obj_t *old_table = set->table; - set->alloc = get_doubling_prime_greater_or_equal_to(set->alloc + 1); + set->alloc = get_hash_alloc_greater_or_equal_to(set->alloc + 1); set->used = 0; set->table = m_new0(mp_obj_t, set->alloc); for (mp_uint_t i = 0; i < old_alloc; i++) { diff --git a/py/mkrules.mk b/py/mkrules.mk index 9cbef1ac5a..3ed4afec19 100644 --- a/py/mkrules.mk +++ b/py/mkrules.mk @@ -46,21 +46,47 @@ vpath %.c . $(TOP) $(BUILD)/%.o: %.c $(call compile_c) +# List all native flags since the current build system doesn't have +# the micropython configuration available. However, these flags are +# needed to extract all qstrings +QSTR_GEN_EXTRA_CFLAGS += -D__QSTR_EXTRACT -DN_X64 -DN_X86 -DN_THUMB -DN_ARM +QSTR_GEN_EXTRA_CFLAGS += -I$(BUILD)/tmp + +vpath %.c . $(TOP) + $(BUILD)/%.pp: %.c $(ECHO) "PreProcess $<" $(Q)$(CC) $(CFLAGS) -E -Wp,-C,-dD,-dI -o $@ $< -# The following rule uses | to create an order only prereuisite. Order only +# The following rule uses | to create an order only prerequisite. Order only # prerequisites only get built if they don't exist. They don't cause timestamp # checking to be performed. # # We don't know which source files actually need the generated.h (since # it is #included from str.h). The compiler generated dependencies will cause # the right .o's to get recompiled if the generated.h file changes. Adding -# an order-only dependendency to all of the .o's will cause the generated .h +# an order-only dependency to all of the .o's will cause the generated .h # to get built before we try to compile any of them. $(OBJ): | $(HEADER_BUILD)/qstrdefs.generated.h $(HEADER_BUILD)/mpversion.h +$(HEADER_BUILD)/qstr.i.last: $(SRC_QSTR) | $(HEADER_BUILD)/mpversion.h + $(ECHO) "GEN $@" + $(Q)if [ "$?" = "" ]; then \ + echo "QSTR Looks like -B used, trying to emulate"; \ + $(CPP) $(QSTR_GEN_EXTRA_CFLAGS) $(CFLAGS) $^ >$(HEADER_BUILD)/qstr.i.last; \ + else \ + $(CPP) $(QSTR_GEN_EXTRA_CFLAGS) $(CFLAGS) $? >$(HEADER_BUILD)/qstr.i.last; \ + fi + +$(HEADER_BUILD)/qstr.split: $(HEADER_BUILD)/qstr.i.last + $(ECHO) "GEN $@" + $(Q)$(PYTHON) $(PY_SRC)/makeqstrdefs.py split $(HEADER_BUILD)/qstr.i.last $(HEADER_BUILD)/qstr $(QSTR_DEFS_COLLECTED) + $(Q)touch $@ + +$(QSTR_DEFS_COLLECTED): $(HEADER_BUILD)/qstr.split + $(ECHO) "GEN $@" + $(Q)$(PYTHON) $(PY_SRC)/makeqstrdefs.py cat $(HEADER_BUILD)/qstr.i.last $(HEADER_BUILD)/qstr $(QSTR_DEFS_COLLECTED) + # $(sort $(var)) removes duplicates # # The net effect of this, is it causes the objects to depend on the diff --git a/py/modbuiltins.c b/py/modbuiltins.c index 859cb11116..87446f7fae 100644 --- a/py/modbuiltins.c +++ b/py/modbuiltins.c @@ -439,6 +439,7 @@ STATIC mp_obj_t mp_builtin___repl_print__(mp_obj_t o) { mp_print_str(&mp_plat_print, "\n"); #endif #if MICROPY_CAN_OVERRIDE_BUILTINS + // Set "_" special variable mp_obj_t dest[2] = {MP_OBJ_SENTINEL, o}; mp_type_module.attr(MP_OBJ_FROM_PTR(&mp_module_builtins), MP_QSTR__, dest); #endif @@ -703,6 +704,9 @@ STATIC const mp_rom_map_elem_t mp_module_builtins_globals_table[] = { { MP_ROM_QSTR(MP_QSTR_OSError), MP_ROM_PTR(&mp_type_OSError) }, { MP_ROM_QSTR(MP_QSTR_OverflowError), MP_ROM_PTR(&mp_type_OverflowError) }, { MP_ROM_QSTR(MP_QSTR_RuntimeError), MP_ROM_PTR(&mp_type_RuntimeError) }, + #if MICROPY_PY_ASYNC_AWAIT + { MP_ROM_QSTR(MP_QSTR_StopAsyncIteration), MP_ROM_PTR(&mp_type_StopAsyncIteration) }, + #endif { MP_ROM_QSTR(MP_QSTR_StopIteration), MP_ROM_PTR(&mp_type_StopIteration) }, { MP_ROM_QSTR(MP_QSTR_SyntaxError), MP_ROM_PTR(&mp_type_SyntaxError) }, { MP_ROM_QSTR(MP_QSTR_SystemExit), MP_ROM_PTR(&mp_type_SystemExit) }, diff --git a/py/modcollections.c b/py/modcollections.c index e43de184a7..dceaa203de 100644 --- a/py/modcollections.c +++ b/py/modcollections.c @@ -29,7 +29,7 @@ #if MICROPY_PY_COLLECTIONS STATIC const mp_rom_map_elem_t mp_module_collections_globals_table[] = { - { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR__collections) }, + { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_ucollections) }, { MP_ROM_QSTR(MP_QSTR_namedtuple), MP_ROM_PTR(&mp_namedtuple_obj) }, #if MICROPY_PY_COLLECTIONS_ORDEREDDICT { MP_ROM_QSTR(MP_QSTR_OrderedDict), MP_ROM_PTR(&mp_type_ordereddict) }, @@ -40,7 +40,7 @@ STATIC MP_DEFINE_CONST_DICT(mp_module_collections_globals, mp_module_collections const mp_obj_module_t mp_module_collections = { .base = { &mp_type_module }, - .name = MP_QSTR__collections, + .name = MP_QSTR_ucollections, .globals = (mp_obj_dict_t*)&mp_module_collections_globals, }; diff --git a/py/modio.c b/py/modio.c index 96805d2911..423315081f 100644 --- a/py/modio.c +++ b/py/modio.c @@ -124,7 +124,7 @@ STATIC const mp_obj_type_t bufwriter_type = { #endif // MICROPY_PY_IO_BUFFEREDWRITER STATIC const mp_rom_map_elem_t mp_module_io_globals_table[] = { - { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR__io) }, + { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_uio) }, // Note: mp_builtin_open_obj should be defined by port, it's not // part of the core. { MP_ROM_QSTR(MP_QSTR_open), MP_ROM_PTR(&mp_builtin_open_obj) }, @@ -147,7 +147,7 @@ STATIC MP_DEFINE_CONST_DICT(mp_module_io_globals, mp_module_io_globals_table); const mp_obj_module_t mp_module_io = { .base = { &mp_type_module }, - .name = MP_QSTR__io, + .name = MP_QSTR_uio, .globals = (mp_obj_dict_t*)&mp_module_io_globals, }; diff --git a/py/mpconfig.h b/py/mpconfig.h index 6c2db2f1fc..42ef19b72c 100644 --- a/py/mpconfig.h +++ b/py/mpconfig.h @@ -234,7 +234,7 @@ // Whether generated code can persist independently of the VM/runtime instance // This is enabled automatically when needed by other features #ifndef MICROPY_PERSISTENT_CODE -#define MICROPY_PERSISTENT_CODE (MICROPY_PERSISTENT_CODE_LOAD || MICROPY_PERSISTENT_CODE_SAVE) +#define MICROPY_PERSISTENT_CODE (MICROPY_PERSISTENT_CODE_LOAD || MICROPY_PERSISTENT_CODE_SAVE || MICROPY_MODULE_FROZEN_MPY) #endif // Whether to emit x64 native code @@ -526,9 +526,19 @@ typedef double mp_float_t; #define MICROPY_MODULE_WEAK_LINKS (0) #endif -// Whether frozen modules are supported +// Whether frozen modules are supported in the form of strings +#ifndef MICROPY_MODULE_FROZEN_STR +#define MICROPY_MODULE_FROZEN_STR (0) +#endif + +// Whether frozen modules are supported in the form of .mpy files +#ifndef MICROPY_MODULE_FROZEN_MPY +#define MICROPY_MODULE_FROZEN_MPY (0) +#endif + +// Convenience macro for whether frozen modules are supported #ifndef MICROPY_MODULE_FROZEN -#define MICROPY_MODULE_FROZEN (0) +#define MICROPY_MODULE_FROZEN (MICROPY_MODULE_FROZEN_STR || MICROPY_MODULE_FROZEN_MPY) #endif // Whether you can override builtins in the builtins module @@ -564,6 +574,11 @@ typedef double mp_float_t; #define MICROPY_PY_DESCRIPTORS (0) #endif +// Support for async/await/async for/async with +#ifndef MICROPY_PY_ASYNC_AWAIT +#define MICROPY_PY_ASYNC_AWAIT (1) +#endif + // Whether str object is proper unicode #ifndef MICROPY_PY_BUILTINS_STR_UNICODE #define MICROPY_PY_BUILTINS_STR_UNICODE (0) @@ -837,6 +852,10 @@ typedef double mp_float_t; #define MICROPY_PY_MACHINE (0) #endif +#ifndef MICROPY_PY_MACHINE_I2C +#define MICROPY_PY_MACHINE_I2C (0) +#endif + #ifndef MICROPY_PY_USSL #define MICROPY_PY_USSL (0) #endif @@ -845,6 +864,10 @@ typedef double mp_float_t; #define MICROPY_PY_WEBSOCKET (0) #endif +#ifndef MICROPY_PY_FRAMEBUF +#define MICROPY_PY_FRAMEBUF (0) +#endif + /*****************************************************************************/ /* Hooks for a port to add builtins */ @@ -83,7 +83,7 @@ typedef struct _mp_obj_base_t mp_obj_base_t; static inline bool MP_OBJ_IS_SMALL_INT(mp_const_obj_t o) { return ((((mp_int_t)(o)) & 1) != 0); } #define MP_OBJ_SMALL_INT_VALUE(o) (((mp_int_t)(o)) >> 1) -#define MP_OBJ_NEW_SMALL_INT(small_int) ((mp_obj_t)((((mp_int_t)(small_int)) << 1) | 1)) +#define MP_OBJ_NEW_SMALL_INT(small_int) ((mp_obj_t)((((mp_uint_t)(small_int)) << 1) | 1)) static inline bool MP_OBJ_IS_QSTR(mp_const_obj_t o) { return ((((mp_int_t)(o)) & 3) == 2); } @@ -109,7 +109,7 @@ static inline bool MP_OBJ_IS_OBJ(mp_const_obj_t o) static inline bool MP_OBJ_IS_SMALL_INT(mp_const_obj_t o) { return ((((mp_int_t)(o)) & 3) == 1); } #define MP_OBJ_SMALL_INT_VALUE(o) (((mp_int_t)(o)) >> 2) -#define MP_OBJ_NEW_SMALL_INT(small_int) ((mp_obj_t)((((mp_int_t)(small_int)) << 2) | 1)) +#define MP_OBJ_NEW_SMALL_INT(small_int) ((mp_obj_t)((((mp_uint_t)(small_int)) << 2) | 1)) static inline bool MP_OBJ_IS_QSTR(mp_const_obj_t o) { return ((((mp_int_t)(o)) & 3) == 3); } @@ -135,7 +135,7 @@ static inline bool MP_OBJ_IS_OBJ(mp_const_obj_t o) static inline bool MP_OBJ_IS_SMALL_INT(mp_const_obj_t o) { return ((((mp_int_t)(o)) & 1) != 0); } #define MP_OBJ_SMALL_INT_VALUE(o) (((mp_int_t)(o)) >> 1) -#define MP_OBJ_NEW_SMALL_INT(small_int) ((mp_obj_t)((((mp_int_t)(small_int)) << 1) | 1)) +#define MP_OBJ_NEW_SMALL_INT(small_int) ((mp_obj_t)((((mp_uint_t)(small_int)) << 1) | 1)) #define mp_const_float_e MP_ROM_PTR((mp_obj_t)(((0x402df854 & ~3) | 2) + 0x80800000)) #define mp_const_float_pi MP_ROM_PTR((mp_obj_t)(((0x40490fdb & ~3) | 2) + 0x80800000)) @@ -561,6 +561,7 @@ extern const mp_obj_type_t mp_type_OSError; extern const mp_obj_type_t mp_type_TimeoutError; extern const mp_obj_type_t mp_type_OverflowError; extern const mp_obj_type_t mp_type_RuntimeError; +extern const mp_obj_type_t mp_type_StopAsyncIteration; extern const mp_obj_type_t mp_type_StopIteration; extern const mp_obj_type_t mp_type_SyntaxError; extern const mp_obj_type_t mp_type_SystemExit; diff --git a/py/objexcept.c b/py/objexcept.c index d8aecb80f0..adf17b08d0 100644 --- a/py/objexcept.c +++ b/py/objexcept.c @@ -197,6 +197,9 @@ MP_DEFINE_EXCEPTION(KeyboardInterrupt, BaseException) MP_DEFINE_EXCEPTION(GeneratorExit, BaseException) MP_DEFINE_EXCEPTION(Exception, BaseException) MP_DEFINE_EXCEPTION_BASE(Exception) + #if MICROPY_PY_ASYNC_AWAIT + MP_DEFINE_EXCEPTION(StopAsyncIteration, Exception) + #endif MP_DEFINE_EXCEPTION(StopIteration, Exception) MP_DEFINE_EXCEPTION(ArithmeticError, Exception) MP_DEFINE_EXCEPTION_BASE(ArithmeticError) diff --git a/py/objgenerator.c b/py/objgenerator.c index 93df7ce13f..2480b0a4b8 100644 --- a/py/objgenerator.c +++ b/py/objgenerator.c @@ -99,6 +99,7 @@ mp_vm_return_kind_t mp_obj_gen_resume(mp_obj_t self_in, mp_obj_t send_value, mp_ assert(MP_OBJ_IS_TYPE(self_in, &mp_type_gen_instance)); mp_obj_gen_instance_t *self = MP_OBJ_TO_PTR(self_in); if (self->code_state.ip == 0) { + // Trying to resume already stopped generator *ret_val = MP_OBJ_STOP_ITERATION; return MP_VM_RETURN_NORMAL; } diff --git a/py/objmodule.c b/py/objmodule.c index 5fd7b82c5b..8c3cb85e67 100644 --- a/py/objmodule.c +++ b/py/objmodule.c @@ -137,10 +137,10 @@ STATIC const mp_rom_map_elem_t mp_builtin_module_table[] = { { MP_ROM_QSTR(MP_QSTR_array), MP_ROM_PTR(&mp_module_array) }, #endif #if MICROPY_PY_IO - { MP_ROM_QSTR(MP_QSTR__io), MP_ROM_PTR(&mp_module_io) }, + { MP_ROM_QSTR(MP_QSTR_uio), MP_ROM_PTR(&mp_module_io) }, #endif #if MICROPY_PY_COLLECTIONS - { MP_ROM_QSTR(MP_QSTR__collections), MP_ROM_PTR(&mp_module_collections) }, + { MP_ROM_QSTR(MP_QSTR_ucollections), MP_ROM_PTR(&mp_module_collections) }, #endif #if MICROPY_PY_STRUCT { MP_ROM_QSTR(MP_QSTR_ustruct), MP_ROM_PTR(&mp_module_ustruct) }, @@ -196,6 +196,12 @@ STATIC const mp_rom_map_elem_t mp_builtin_module_table[] = { #if MICROPY_PY_WEBSOCKET { MP_ROM_QSTR(MP_QSTR_websocket), MP_ROM_PTR(&mp_module_websocket) }, #endif +#if MICROPY_PY_WEBREPL + { MP_ROM_QSTR(MP_QSTR__webrepl), MP_ROM_PTR(&mp_module_webrepl) }, +#endif +#if MICROPY_PY_FRAMEBUF + { MP_ROM_QSTR(MP_QSTR_framebuf), MP_ROM_PTR(&mp_module_framebuf) }, +#endif // extra builtin modules as defined by a port MICROPY_PORT_BUILTIN_MODULES diff --git a/py/objstr.c b/py/objstr.c index 0c2d904035..d0d090b995 100644 --- a/py/objstr.c +++ b/py/objstr.c @@ -564,7 +564,7 @@ STATIC mp_obj_t str_splitlines(size_t n_args, const mp_obj_t *pos_args, mp_map_t mp_arg_parse_all(n_args - 1, pos_args + 1, kw_args, MP_ARRAY_SIZE(allowed_args), allowed_args, (mp_arg_val_t*)&args); - mp_obj_t new_args[2] = {pos_args[0], MP_OBJ_NEW_QSTR(MP_QSTR__backslash_n)}; + mp_obj_t new_args[2] = {pos_args[0], MP_OBJ_NEW_QSTR(MP_QSTR__0x0a_)}; return str_split_internal(2, new_args, SPLITLINES | (args.keepends.u_bool ? KEEP : 0)); } #endif diff --git a/py/parse.c b/py/parse.c index b57e82d459..3daa5ff83e 100644 --- a/py/parse.c +++ b/py/parse.c @@ -56,8 +56,6 @@ #define RULE_ARG_RULE (0x2000) #define RULE_ARG_OPT_RULE (0x3000) -#define ADD_BLANK_NODE(rule) ((rule->act & RULE_ACT_ADD_BLANK) != 0) - // (un)comment to use rule names; for debugging //#define USE_RULE_NAME (1) @@ -80,10 +78,10 @@ enum { RULE_const_object, // special node for a constant, generic Python object }; -#define ident (RULE_ACT_ALLOW_IDENT) -#define blank (RULE_ACT_ADD_BLANK) #define or(n) (RULE_ACT_OR | n) #define and(n) (RULE_ACT_AND | n) +#define and_ident(n) (RULE_ACT_AND | n | RULE_ACT_ALLOW_IDENT) +#define and_blank(n) (RULE_ACT_AND | n | RULE_ACT_ADD_BLANK) #define one_or_more (RULE_ACT_LIST | 2) #define list (RULE_ACT_LIST | 1) #define list_with_end (RULE_ACT_LIST | 3) @@ -563,11 +561,10 @@ STATIC bool fold_constants(parser_t *parser, const rule_t *rule, size_t num_args // this node is of the form <x> = <y> mp_parse_node_t pn0 = peek_result(parser, 1); if (MP_PARSE_NODE_IS_ID(pn0) - && MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_power) + && MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_atom_expr_normal) && MP_PARSE_NODE_IS_ID(((mp_parse_node_struct_t*)pn1)->nodes[0]) && MP_PARSE_NODE_LEAF_ARG(((mp_parse_node_struct_t*)pn1)->nodes[0]) == MP_QSTR_const && MP_PARSE_NODE_IS_STRUCT_KIND(((mp_parse_node_struct_t*)pn1)->nodes[1], RULE_trailer_paren) - && MP_PARSE_NODE_IS_NULL(((mp_parse_node_struct_t*)pn1)->nodes[2]) ) { // code to assign dynamic constants: id = const(value) @@ -599,13 +596,11 @@ STATIC bool fold_constants(parser_t *parser, const rule_t *rule, size_t num_args #endif #if MICROPY_COMP_MODULE_CONST - } else if (rule->rule_id == RULE_power) { - mp_parse_node_t pn0 = peek_result(parser, 2); - mp_parse_node_t pn1 = peek_result(parser, 1); - mp_parse_node_t pn2 = peek_result(parser, 0); + } else if (rule->rule_id == RULE_atom_expr_normal) { + mp_parse_node_t pn0 = peek_result(parser, 1); + mp_parse_node_t pn1 = peek_result(parser, 0); if (!(MP_PARSE_NODE_IS_ID(pn0) - && MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_trailer_period) - && MP_PARSE_NODE_IS_NULL(pn2))) { + && MP_PARSE_NODE_IS_STRUCT_KIND(pn1, RULE_trailer_period))) { return false; } // id1.id2 @@ -833,25 +828,6 @@ mp_parse_tree_t mp_parse(mp_lexer_t *lex, mp_parse_input_kind_t input_kind) { // matched the rule, so now build the corresponding parse_node - // count number of arguments for the parse_node - i = 0; - bool emit_rule = false; - for (size_t x = 0; x < n; ++x) { - if ((rule->arg[x] & RULE_ARG_KIND_MASK) == RULE_ARG_TOK) { - mp_token_kind_t tok_kind = rule->arg[x] & RULE_ARG_ARG_MASK; - if (tok_kind >= MP_TOKEN_NAME) { - emit_rule = true; - } - if (tok_kind == MP_TOKEN_NAME) { - // only tokens which were names are pushed to stack - i += 1; - } - } else { - // rules are always pushed - i += 1; - } - } - #if !MICROPY_ENABLE_DOC_STRING // this code discards lonely statements, such as doc strings if (input_kind != MP_PARSE_SINGLE_INPUT && rule->rule_id == RULE_expr_stmt && peek_result(&parser, 0) == MP_PARSE_NODE_NULL) { @@ -871,35 +847,29 @@ mp_parse_tree_t mp_parse(mp_lexer_t *lex, mp_parse_input_kind_t input_kind) { } #endif - // always emit these rules, even if they have only 1 argument - if (rule->rule_id == RULE_expr_stmt || rule->rule_id == RULE_yield_stmt) { - emit_rule = true; - } - - // if a rule has the RULE_ACT_ALLOW_IDENT bit set then this - // rule should not be emitted if it has only 1 argument - if (rule->act & RULE_ACT_ALLOW_IDENT) { - emit_rule = false; - } - - // always emit these rules, and add an extra blank node at the end (to be used by the compiler to store data) - if (ADD_BLANK_NODE(rule)) { - emit_rule = true; - push_result_node(&parser, MP_PARSE_NODE_NULL); - i += 1; - } - + // count number of arguments for the parse node + i = 0; size_t num_not_nil = 0; - for (size_t x = 0; x < i; ++x) { - if (peek_result(&parser, x) != MP_PARSE_NODE_NULL) { - num_not_nil += 1; + for (size_t x = n; x > 0;) { + --x; + if ((rule->arg[x] & RULE_ARG_KIND_MASK) == RULE_ARG_TOK) { + mp_token_kind_t tok_kind = rule->arg[x] & RULE_ARG_ARG_MASK; + if (tok_kind == MP_TOKEN_NAME) { + // only tokens which were names are pushed to stack + i += 1; + num_not_nil += 1; + } + } else { + // rules are always pushed + if (peek_result(&parser, i) != MP_PARSE_NODE_NULL) { + num_not_nil += 1; + } + i += 1; } } - if (emit_rule || num_not_nil != 1) { - // need to add rule when num_not_nil==0 for, eg, atom_paren, testlist_comp_3b - push_result_rule(&parser, rule_src_line, rule, i); - } else { - // single result, leave it on stack + + if (num_not_nil == 1 && (rule->act & RULE_ACT_ALLOW_IDENT)) { + // this rule has only 1 argument and should not be emitted mp_parse_node_t pn = MP_PARSE_NODE_NULL; for (size_t x = 0; x < i; ++x) { mp_parse_node_t pn2 = pop_result(&parser); @@ -908,6 +878,16 @@ mp_parse_tree_t mp_parse(mp_lexer_t *lex, mp_parse_input_kind_t input_kind) { } } push_result_node(&parser, pn); + } else { + // this rule must be emitted + + if (rule->act & RULE_ACT_ADD_BLANK) { + // and add an extra blank node at the end (used by the compiler to store data) + push_result_node(&parser, MP_PARSE_NODE_NULL); + i += 1; + } + + push_result_rule(&parser, rule_src_line, rule, i); } break; } @@ -7,6 +7,12 @@ HEADER_BUILD = $(BUILD)/genhdr # file containing qstr defs for the core Python bit PY_QSTR_DEFS = $(PY_SRC)/qstrdefs.h +# If qstr autogeneration is not disabled we specify the output header +# for all collected qstrings. +ifneq ($(QSTR_AUTOGEN_DISABLE),1) +QSTR_DEFS_COLLECTED = $(HEADER_BUILD)/qstrdefs.collected.h +endif + # some code is performance bottleneck and compiled with other optimization options CSUPEROPT = -O3 @@ -94,6 +100,7 @@ PY_O_BASENAME = \ parsenum.o \ emitglue.o \ runtime.o \ + runtime_utils.o \ nativeglue.o \ stackctrl.o \ argcheck.o \ @@ -167,9 +174,12 @@ PY_O_BASENAME = \ ../extmod/moduhashlib.o \ ../extmod/modubinascii.o \ ../extmod/machine_mem.o \ + ../extmod/machine_i2c.o \ ../extmod/modussl.o \ ../extmod/modurandom.o \ ../extmod/modwebsocket.o \ + ../extmod/modwebrepl.o \ + ../extmod/modframebuf.o \ ../extmod/fsusermount.o \ ../extmod/vfs_fat.o \ ../extmod/vfs_fat_ffconf.o \ @@ -182,6 +192,10 @@ PY_O_BASENAME = \ # prepend the build destination prefix to the py object files PY_O = $(addprefix $(PY_BUILD)/, $(PY_O_BASENAME)) +# Sources that may contain qstrings +SRC_QSTR_IGNORE = nlr% emitnx% emitnthumb% emitnarm% +SRC_QSTR = $(SRC_MOD) $(addprefix py/,$(filter-out $(SRC_QSTR_IGNORE),$(PY_O_BASENAME:.o=.c)) emitnative.c) + # Anything that depends on FORCE will be considered out-of-date FORCE: .PHONY: FORCE @@ -194,14 +208,13 @@ $(HEADER_BUILD)/mpversion.h: FORCE | $(HEADER_BUILD) MPCONFIGPORT_MK = $(wildcard mpconfigport.mk) # qstr data - # Adding an order only dependency on $(HEADER_BUILD) causes $(HEADER_BUILD) to get # created before we run the script to generate the .h # Note: we need to protect the qstr names from the preprocessor, so we wrap # the lines in "" and then unwrap after the preprocessor is finished. -$(HEADER_BUILD)/qstrdefs.generated.h: $(PY_QSTR_DEFS) $(QSTR_DEFS) $(PY_SRC)/makeqstrdata.py mpconfigport.h $(MPCONFIGPORT_MK) $(PY_SRC)/mpconfig.h | $(HEADER_BUILD) +$(HEADER_BUILD)/qstrdefs.generated.h: $(PY_QSTR_DEFS) $(QSTR_DEFS) $(QSTR_DEFS_COLLECTED) $(PY_SRC)/makeqstrdata.py mpconfigport.h $(MPCONFIGPORT_MK) $(PY_SRC)/mpconfig.h | $(HEADER_BUILD) $(ECHO) "GEN $@" - $(Q)cat $(PY_QSTR_DEFS) $(QSTR_DEFS) | $(SED) 's/^Q(.*)/"&"/' | $(CPP) $(CFLAGS) - | sed 's/^"\(Q(.*)\)"/\1/' > $(HEADER_BUILD)/qstrdefs.preprocessed.h + $(Q)cat $(PY_QSTR_DEFS) $(QSTR_DEFS) $(QSTR_DEFS_COLLECTED) | $(SED) 's/^Q(.*)/"&"/' | $(CPP) $(CFLAGS) - | sed 's/^"\(Q(.*)\)"/\1/' > $(HEADER_BUILD)/qstrdefs.preprocessed.h $(Q)$(PYTHON) $(PY_SRC)/makeqstrdata.py $(HEADER_BUILD)/qstrdefs.preprocessed.h > $@ # emitters @@ -87,20 +87,29 @@ mp_uint_t qstr_compute_hash(const byte *data, size_t len) { return hash; } -STATIC const qstr_pool_t const_pool = { +const qstr_pool_t mp_qstr_const_pool = { NULL, // no previous pool 0, // no previous pool 10, // set so that the first dynamically allocated pool is twice this size; must be <= the len (just below) - MP_QSTR_number_of, // corresponds to number of strings in array just below + MP_QSTRnumber_of, // corresponds to number of strings in array just below { +#ifndef __QSTR_EXTRACT #define QDEF(id, str) str, #include "genhdr/qstrdefs.generated.h" #undef QDEF +#endif }, }; +#ifdef MICROPY_QSTR_EXTRA_POOL +extern const qstr_pool_t MICROPY_QSTR_EXTRA_POOL; +#define CONST_POOL MICROPY_QSTR_EXTRA_POOL +#else +#define CONST_POOL mp_qstr_const_pool +#endif + void qstr_init(void) { - MP_STATE_VM(last_pool) = (qstr_pool_t*)&const_pool; // we won't modify the const_pool since it has no allocated room left + MP_STATE_VM(last_pool) = (qstr_pool_t*)&CONST_POOL; // we won't modify the const_pool since it has no allocated room left MP_STATE_VM(qstr_last_chunk) = NULL; } @@ -258,7 +267,7 @@ void qstr_pool_info(size_t *n_pool, size_t *n_qstr, size_t *n_str_data_bytes, si *n_qstr = 0; *n_str_data_bytes = 0; *n_total_bytes = 0; - for (qstr_pool_t *pool = MP_STATE_VM(last_pool); pool != NULL && pool != &const_pool; pool = pool->prev) { + for (qstr_pool_t *pool = MP_STATE_VM(last_pool); pool != NULL && pool != &CONST_POOL; pool = pool->prev) { *n_pool += 1; *n_qstr += pool->len; for (const byte **q = pool->qstrs, **q_top = pool->qstrs + pool->len; q < q_top; q++) { @@ -275,7 +284,7 @@ void qstr_pool_info(size_t *n_pool, size_t *n_qstr, size_t *n_str_data_bytes, si #if MICROPY_PY_MICROPYTHON_MEM_INFO void qstr_dump_data(void) { - for (qstr_pool_t *pool = MP_STATE_VM(last_pool); pool != NULL && pool != &const_pool; pool = pool->prev) { + for (qstr_pool_t *pool = MP_STATE_VM(last_pool); pool != NULL && pool != &CONST_POOL; pool = pool->prev) { for (const byte **q = pool->qstrs, **q_top = pool->qstrs + pool->len; q < q_top; q++) { mp_printf(&mp_plat_print, "Q(%s)\n", Q_GET_DATA(*q)); } @@ -37,10 +37,12 @@ // first entry in enum will be MP_QSTR_NULL=0, which indicates invalid/no qstr enum { +#ifndef __QSTR_EXTRACT #define QDEF(id, str) id, #include "genhdr/qstrdefs.generated.h" #undef QDEF - MP_QSTR_number_of, +#endif + MP_QSTRnumber_of, // no underscore so it can't clash with any of the above }; typedef size_t qstr; diff --git a/py/qstrdefs.h b/py/qstrdefs.h index f5e06f1a52..c98a253a69 100644 --- a/py/qstrdefs.h +++ b/py/qstrdefs.h @@ -36,461 +36,11 @@ QCFG(BYTES_IN_HASH, MICROPY_QSTR_BYTES_IN_HASH) Q() Q(*) Q(_) -Q(__build_class__) -Q(__class__) -Q(__doc__) -Q(__import__) -Q(__init__) -Q(__new__) -Q(__locals__) -Q(__main__) -Q(__module__) -Q(__name__) -Q(__dict__) -Q(__hash__) -Q(__next__) -Q(__qualname__) -Q(__path__) -Q(__repl_print__) -#if MICROPY_PY___FILE__ -Q(__file__) -#endif - -Q(__bool__) -Q(__contains__) -Q(__enter__) -Q(__exit__) -Q(__len__) -Q(__iter__) -Q(__getitem__) -Q(__setitem__) -Q(__delitem__) -Q(__add__) -Q(__sub__) -Q(__repr__) -Q(__str__) -#if MICROPY_PY_DESCRIPTORS -Q(__get__) -Q(__set__) -Q(__delete__) -#endif -Q(__getattr__) -Q(__del__) -Q(__call__) -Q(__lt__) -Q(__gt__) -Q(__eq__) -Q(__le__) -Q(__ge__) -Q(__reversed__) -#if MICROPY_PY_ALL_SPECIAL_METHODS -Q(__mul__) -Q(__truediv__) -Q(__floordiv__) -Q(__iadd__) -Q(__isub__) -Q(__invert__) -Q(__neg__) -Q(__pos__) -#endif - -Q(micropython) -Q(bytecode) -Q(const) - -#if MICROPY_EMIT_NATIVE -Q(native) -Q(viper) -Q(uint) -Q(ptr) -Q(ptr8) -Q(ptr16) -Q(ptr32) -#endif - -#if MICROPY_EMIT_INLINE_THUMB -Q(asm_thumb) -Q(label) -Q(align) -Q(data) -Q(uint) -Q(nop) -Q(mov) -Q(and_) -Q(cmp) -Q(add) -Q(sub) -Q(lsl) -Q(lsr) -Q(asr) -Q(ldr) -Q(ldrb) -Q(ldrh) -Q(str) -Q(strb) -Q(strh) -Q(b) -Q(bl) -Q(bx) -Q(push) -Q(pop) -Q(cpsid) -Q(cpsie) -Q(wfi) -Q(clz) -Q(rbit) -Q(movw) -Q(movt) -Q(movwt) -Q(mrs) -Q(sdiv) -Q(udiv) -Q(ldrex) -Q(strex) -#if MICROPY_EMIT_INLINE_THUMB_FLOAT -Q(vcmp) -Q(vneg) -Q(vcvt_f32_s32) -Q(vcvt_s32_f32) -Q(vsqrt) -Q(vmov) -Q(vmrs) -Q(vldr) -Q(vstr) -#endif -#endif - -Q(builtins) - -Q(Ellipsis) -Q(StopIteration) -#if MICROPY_PY_BUILTINS_NOTIMPLEMENTED -Q(NotImplemented) -#endif - -Q(BaseException) -Q(ArithmeticError) -Q(AssertionError) -Q(AttributeError) -Q(BufferError) -Q(EOFError) -Q(Exception) -Q(FileExistsError) -Q(FileNotFoundError) -Q(FloatingPointError) -Q(GeneratorExit) -Q(ImportError) -Q(IndentationError) -Q(IndexError) -Q(KeyboardInterrupt) -Q(KeyError) -Q(LookupError) -Q(MemoryError) -Q(NameError) -Q(NotImplementedError) -Q(OSError) -#if MICROPY_PY_BUILTINS_TIMEOUTERROR -Q(TimeoutError) -#endif -Q(OverflowError) -Q(RuntimeError) -Q(SyntaxError) -Q(SystemExit) -Q(TypeError) -Q(UnboundLocalError) -Q(ValueError) -#if MICROPY_EMIT_NATIVE -Q(ViperTypeError) -#endif -Q(ZeroDivisionError) -#if MICROPY_PY_BUILTINS_STR_UNICODE -Q(UnicodeError) -#endif - -Q(None) -Q(False) -Q(True) -Q(object) - -Q(NoneType) - -#if MICROPY_PY_COLLECTIONS_ORDEREDDICT -Q(OrderedDict) -#endif - -Q(abs) -Q(all) -Q(any) -Q(args) -#if MICROPY_PY_ARRAY -Q(array) -#endif -Q(bin) -Q({:#b}) -Q(bool) -#if MICROPY_PY_BUILTINS_BYTEARRAY -Q(bytearray) -#endif -#if MICROPY_PY_BUILTINS_MEMORYVIEW -Q(memoryview) -#endif -Q(bytes) -Q(callable) -Q(chr) -Q(classmethod) -Q(_collections) -#if MICROPY_PY_BUILTINS_COMPLEX -Q(complex) -Q(real) -Q(imag) -#endif -Q(dict) -Q(dir) -Q(divmod) -#if MICROPY_PY_BUILTINS_ENUMERATE -Q(enumerate) -#endif -Q(eval) -Q(exec) -#if MICROPY_PY_BUILTINS_EXECFILE -Q(execfile) -#endif -#if MICROPY_PY_BUILTINS_FILTER -Q(filter) -#endif -#if MICROPY_PY_BUILTINS_FLOAT -Q(float) -#endif -Q(from_bytes) -Q(getattr) -Q(setattr) -Q(globals) -Q(hasattr) -Q(hash) -Q(hex) -Q(%#x) -Q(id) -Q(int) -Q(isinstance) -Q(issubclass) -Q(iter) -Q(len) -Q(list) -Q(locals) -Q(map) -#if MICROPY_PY_BUILTINS_MIN_MAX -Q(max) -Q(min) -Q(default) -#endif -Q(namedtuple) -Q(next) -Q(oct) Q(%#o) -Q(open) -Q(ord) -Q(path) -Q(pow) -Q(print) -Q(range) -Q(read) -Q(repr) -Q(reversed) -Q(round) -Q(sorted) -Q(staticmethod) -Q(sum) -Q(super) -Q(str) -Q(sys) -Q(to_bytes) -Q(tuple) -Q(type) -Q(value) -Q(write) -Q(zip) - -#if MICROPY_PY_BUILTINS_COMPILE -Q(compile) -Q(code) -Q(single) -#endif - -Q(sep) -Q(end) - -#if MICROPY_PY_BUILTINS_RANGE_ATTRS -Q(step) -Q(stop) -#endif - -Q(clear) -Q(copy) -Q(fromkeys) -Q(get) -Q(items) -Q(keys) -Q(pop) -Q(popitem) -Q(setdefault) -Q(update) -Q(values) -Q(append) -Q(close) -Q(send) -Q(throw) -Q(count) -Q(extend) -Q(index) -Q(remove) -Q(insert) -Q(pop) -Q(sort) -Q(join) -Q(strip) -Q(lstrip) -Q(rstrip) -Q(format) -Q(key) -Q(reverse) -Q(add) -Q(clear) -Q(copy) -Q(pop) -Q(remove) -Q(find) -Q(rfind) -Q(rindex) -Q(split) -#if MICROPY_PY_BUILTINS_STR_SPLITLINES -Q(splitlines) -Q(keepends) +Q(%#x) +Q({:#b}) Q(\n) -#endif -Q(rsplit) -Q(startswith) -Q(endswith) -Q(replace) -Q(partition) -Q(rpartition) -Q(lower) -Q(upper) -Q(isspace) -Q(isalpha) -Q(isdigit) -Q(isupper) -Q(islower) -Q(iterable) -Q(start) - -Q(bound_method) -Q(closure) -Q(dict_view) -Q(function) -Q(generator) -Q(iterator) -Q(module) -Q(slice) - -#if MICROPY_PY_BUILTINS_SET -Q(discard) -Q(difference) -Q(difference_update) -Q(intersection) -Q(intersection_update) -Q(isdisjoint) -Q(issubset) -Q(issuperset) -Q(set) -Q(symmetric_difference) -Q(symmetric_difference_update) -Q(union) -Q(update) -#endif - -#if MICROPY_PY_BUILTINS_FROZENSET -Q(frozenset) -#endif - -#if MICROPY_PY_MATH || MICROPY_PY_CMATH -Q(math) -Q(e) -Q(pi) -Q(sqrt) -Q(pow) -Q(exp) -#if MICROPY_PY_MATH_SPECIAL_FUNCTIONS -Q(expm1) -#endif -Q(log) -#if MICROPY_PY_MATH_SPECIAL_FUNCTIONS -Q(log2) -Q(log10) -Q(cosh) -Q(sinh) -Q(tanh) -Q(acosh) -Q(asinh) -Q(atanh) -#endif -Q(cos) -Q(sin) -Q(tan) -Q(acos) -Q(asin) -Q(atan) -Q(atan2) -Q(ceil) -Q(copysign) -Q(fabs) -Q(fmod) -Q(floor) -Q(isfinite) -Q(isinf) -Q(isnan) -Q(trunc) -Q(modf) -Q(frexp) -Q(ldexp) -Q(degrees) -Q(radians) -#if MICROPY_PY_MATH_SPECIAL_FUNCTIONS -Q(erf) -Q(erfc) -Q(gamma) -Q(lgamma) -#endif -#endif - -#if MICROPY_PY_CMATH -Q(cmath) -Q(phase) -Q(polar) -Q(rect) -#endif - -#if MICROPY_PY_MICROPYTHON_MEM_INFO -#if MICROPY_MEM_STATS -Q(mem_total) -Q(mem_current) -Q(mem_peak) -#endif -Q(mem_info) -Q(qstr_info) -#if MICROPY_STACK_CHECK -Q(stack_use) -#endif -#endif -#if MICROPY_ENABLE_GC -Q(heap_lock) -Q(heap_unlock) -#endif - -#if MICROPY_ENABLE_EMERGENCY_EXCEPTION_BUF && (MICROPY_EMERGENCY_EXCEPTION_BUF_SIZE == 0) -Q(alloc_emergency_exception_buf) -#endif Q(maximum recursion depth exceeded) - Q(<module>) Q(<lambda>) Q(<listcomp>) @@ -499,265 +49,12 @@ Q(<setcomp>) Q(<genexpr>) Q(<string>) Q(<stdin>) - -#if MICROPY_CPYTHON_COMPAT -Q(encode) -Q(decode) Q(utf-8) -#endif - -#if MICROPY_PY_SYS -Q(argv) -Q(byteorder) -Q(big) -Q(exit) -Q(little) -#ifdef MICROPY_PY_SYS_PLATFORM -Q(platform) -#endif -Q(stdin) -Q(stdout) -Q(stderr) -#if MICROPY_PY_SYS_STDIO_BUFFER -Q(buffer) -#endif -Q(version) -Q(version_info) -#if MICROPY_PY_ATTRTUPLE -Q(name) -#endif -Q(implementation) -#if MICROPY_PY_SYS_MAXSIZE -Q(maxsize) -#endif -#if MICROPY_PY_SYS_MODULES -Q(modules) -#endif -#if MICROPY_PY_SYS_EXC_INFO -Q(exc_info) -#endif -Q(print_exception) -#endif - -#if MICROPY_PY_STRUCT -Q(struct) -Q(ustruct) -Q(pack) -Q(pack_into) -Q(unpack) -Q(unpack_from) -Q(calcsize) -#endif - -#if MICROPY_PY_UCTYPES -Q(uctypes) -Q(struct) -Q(sizeof) -Q(addressof) -Q(bytes_at) -Q(bytearray_at) - -Q(NATIVE) -Q(LITTLE_ENDIAN) -Q(BIG_ENDIAN) - -Q(VOID) - -Q(UINT8) -Q(INT8) -Q(UINT16) -Q(INT16) -Q(UINT32) -Q(INT32) -Q(UINT64) -Q(INT64) - -Q(BFUINT8) -Q(BFINT8) -Q(BFUINT16) -Q(BFINT16) -Q(BFUINT32) -Q(BFINT32) - -Q(FLOAT32) -Q(FLOAT64) - -Q(ARRAY) -Q(PTR) -//Q(BITFIELD) - -Q(BF_POS) -Q(BF_LEN) -#endif - -#if MICROPY_PY_IO -Q(_io) -Q(readall) -Q(readinto) -Q(readline) -Q(readlines) -Q(seek) -Q(tell) -Q(FileIO) -Q(TextIOWrapper) -Q(StringIO) -Q(BytesIO) -Q(getvalue) -Q(file) -Q(mode) -Q(r) -Q(encoding) -#if MICROPY_PY_IO_BUFFEREDWRITER -Q(BufferedWriter) -#endif -#endif - -#if MICROPY_PY_GC -Q(gc) -Q(collect) -Q(disable) -Q(enable) -Q(isenabled) -Q(mem_free) -Q(mem_alloc) -#endif - -#if MICROPY_PY_BUILTINS_PROPERTY -Q(property) -Q(getter) -Q(setter) -Q(deleter) -Q(doc) -#endif - -#if MICROPY_PY_UZLIB -Q(uzlib) -Q(decompress) -#endif - -#if MICROPY_PY_UJSON -Q(ujson) -Q(dumps) -Q(loads) -#endif - -#if MICROPY_PY_URE -Q(ure) -Q(compile) -Q(match) -Q(search) -Q(group) -Q(DEBUG) -#endif -#if MICROPY_PY_UHEAPQ -Q(uheapq) -Q(heappush) -Q(heappop) -Q(heapify) -#endif - -#if MICROPY_PY_UHASHLIB -Q(uhashlib) -Q(update) -Q(digest) -Q(sha256) -Q(sha1) -#endif - -#if MICROPY_PY_UBINASCII -Q(ubinascii) -Q(hexlify) -Q(unhexlify) -Q(a2b_base64) -Q(b2a_base64) -#endif - -#if MICROPY_PY_MACHINE -Q(umachine) -Q(mem) -Q(mem8) -Q(mem16) -Q(mem32) -#endif - -#if MICROPY_PY_USSL -Q(ussl) -Q(wrap_socket) -#endif - -#if MICROPY_PY_LWIP -// for lwip module -Q(lwip) -Q(reset) -Q(callback) -Q(socket) -Q(AF_INET) -Q(AF_INET6) -Q(SOCK_STREAM) -Q(SOCK_DGRAM) -Q(SOCK_RAW) -Q(SOL_SOCKET) -Q(SO_REUSEADDR) -// for lwip.socket -Q(close) -Q(bind) -Q(listen) -Q(accept) -Q(connect) -Q(send) -Q(recv) -Q(sendto) -Q(recvfrom) -Q(settimeout) -Q(setsockopt) -Q(makefile) -#if MICROPY_PY_LWIP_SLIP -// for lwip.slip -Q(slip) -Q(status) -#endif -#endif - -#if MICROPY_FSUSERMOUNT -// for user-mountable block devices -Q(mount) -Q(umount) -Q(readonly) -Q(mkfs) -Q(listdir) -Q(mkdir) -Q(remove) -Q(rename) -Q(readblocks) -Q(writeblocks) -Q(ioctl) -Q(sync) -Q(count) -#endif - -#if MICROPY_PY_OS_DUPTERM -Q(dupterm) -#endif - -#if MICROPY_PY_URANDOM -Q(urandom) -Q(getrandbits) -Q(seed) -#if MICROPY_PY_URANDOM_EXTRA_FUNCS -Q(randrange) -Q(randint) -Q(choice) -Q(random) -Q(uniform) -#endif -#endif - -#if MICROPY_VFS_FAT -Q(VfsFat) -Q(flush) -#endif - -#if MICROPY_PY_WEBSOCKET -Q(websocket) -#endif +// The following qstrings not referenced from anywhere in the sources +Q(__locals__) +Q(BufferError) +Q(FileExistsError) +Q(FileNotFoundError) +Q(FloatingPointError) +Q(UnboundLocalError) @@ -57,6 +57,9 @@ bool mp_repl_continue_with_input(const char *input) { || str_startswith_word(input, "with") || str_startswith_word(input, "def") || str_startswith_word(input, "class") + #if MICROPY_PY_ASYNC_AWAIT + || str_startswith_word(input, "async") + #endif ; // check for unmatched open bracket, quote or escape quote diff --git a/py/runtime.c b/py/runtime.c index adbab579f0..67534c4b5e 100644 --- a/py/runtime.c +++ b/py/runtime.c @@ -704,7 +704,12 @@ void mp_call_prepare_args_n_kw_var(bool have_self, mp_uint_t n_args_n_kw, const assert(args2_len + 2 * map->used <= args2_alloc); // should have enough, since kw_dict_len is in this case hinted correctly above for (mp_uint_t i = 0; i < map->alloc; i++) { if (MP_MAP_SLOT_IS_FILLED(map, i)) { - args2[args2_len++] = map->table[i].key; + // the key must be a qstr, so intern it if it's a string + mp_obj_t key = map->table[i].key; + if (MP_OBJ_IS_TYPE(key, &mp_type_str)) { + key = mp_obj_str_intern(key); + } + args2[args2_len++] = key; args2[args2_len++] = map->table[i].value; } } @@ -726,7 +731,12 @@ void mp_call_prepare_args_n_kw_var(bool have_self, mp_uint_t n_args_n_kw, const } mp_obj_t *items; mp_obj_get_array_fixed_n(item, 2, &items); - args2[args2_len++] = items[0]; + // the key must be a qstr, so intern it if it's a string + mp_obj_t key = items[0]; + if (MP_OBJ_IS_TYPE(key, &mp_type_str)) { + key = mp_obj_str_intern(key); + } + args2[args2_len++] = key; args2[args2_len++] = items[1]; } } diff --git a/py/runtime.h b/py/runtime.h index 1b58f4728f..3e325a31b3 100644 --- a/py/runtime.h +++ b/py/runtime.h @@ -95,6 +95,9 @@ mp_obj_t mp_call_function_2(mp_obj_t fun, mp_obj_t arg1, mp_obj_t arg2); mp_obj_t mp_call_function_n_kw(mp_obj_t fun, mp_uint_t n_args, mp_uint_t n_kw, const mp_obj_t *args); mp_obj_t mp_call_method_n_kw(mp_uint_t n_args, mp_uint_t n_kw, const mp_obj_t *args); mp_obj_t mp_call_method_n_kw_var(bool have_self, mp_uint_t n_args_n_kw, const mp_obj_t *args); +// Call function and catch/dump exception - for Python callbacks from C code +void mp_call_function_1_protected(mp_obj_t fun, mp_obj_t arg); +void mp_call_function_2_protected(mp_obj_t fun, mp_obj_t arg1, mp_obj_t arg2); typedef struct _mp_call_args_t { mp_obj_t fun; diff --git a/py/runtime_utils.c b/py/runtime_utils.c new file mode 100644 index 0000000000..e0495495aa --- /dev/null +++ b/py/runtime_utils.c @@ -0,0 +1,50 @@ +/* + * This file is part of the Micro Python project, http://micropython.org/ + * + * The MIT License (MIT) + * + * Copyright (c) 2015 Josef Gajdusek + * Copyright (c) 2015 Paul Sokolovsky + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + * THE SOFTWARE. + */ + +#include "py/runtime.h" +#include "py/obj.h" +#include "py/nlr.h" + +void mp_call_function_1_protected(mp_obj_t fun, mp_obj_t arg) { + nlr_buf_t nlr; + if (nlr_push(&nlr) == 0) { + mp_call_function_1(fun, arg); + nlr_pop(); + } else { + mp_obj_print_exception(&mp_plat_print, MP_OBJ_FROM_PTR(nlr.ret_val)); + } +} + +void mp_call_function_2_protected(mp_obj_t fun, mp_obj_t arg1, mp_obj_t arg2) { + nlr_buf_t nlr; + if (nlr_push(&nlr) == 0) { + mp_call_function_2(fun, arg1, arg2); + nlr_pop(); + } else { + mp_obj_print_exception(&mp_plat_print, MP_OBJ_FROM_PTR(nlr.ret_val)); + } +} @@ -1144,7 +1144,8 @@ yield: if (ret_kind == MP_VM_RETURN_NORMAL) { // Pop exhausted gen sp--; - if (ret_value == MP_OBJ_NULL) { + // TODO: When ret_value can be MP_OBJ_NULL here?? + if (ret_value == MP_OBJ_NULL || ret_value == MP_OBJ_STOP_ITERATION) { // Optimize StopIteration // TODO: get StopIteration's value PUSH(mp_const_none); |