Merge tag 'v1.8.6' into parse-bytecode

ESP8266 port uses SDK 2.0, has more heap, has support for 512k devices

This release brings some code size reductions to the core as well as
more tests and improved coverage which is now at 94.3%.

The time.ticks_diff(a, b) function has changed: the order of the arguments
has been swapped so that it behaves like "a - b", and it can now return a
negative number if "a" came before "b" (modulo the period of the ticks
functions).

For the ESP8266 port the Espressif SDK has been updated to 2.0.0, the
heap has been increased from 28k to 36k, and there is support for 512k
devices via "make 512k".  upip is included by default as frozen bytecode.
The network module now allows access-point reconnection without WiFi
credentials, and exposes configuration for the station DHCP hostname.  The
DS18B20 driver now handles negative temperatures, and NeoPixel and APA102
drivers handle 4 bytes-per-pixel LEDs.

For the CC3200 port there is now support for loading of precompiled .mpy
files and threading now works properly with interrupts.

A detailed list of changes follows.

py core:
- py.mk: automatically add frozen.c to source list if FROZEN_DIR is defined
- be more specific with MP_DECLARE_CONST_FUN_OBJ macros
- specialise builtin funcs to use separate type for fixed arg count
- {modbuiltins,obj}: use MP_PYTHON_PRINTER where possible
- modbuiltins: add builtin "slice", pointing to existing slice type
- add "delattr" builtin, conditional on MICROPY_CPYTHON_COMPAT
- sequence: fix reverse slicing of lists
- fix null pointer dereference in mpz.c, fix missing va_end in warning.c
- remove asserts that are always true in emitbc.c
- fix wrong assumption that m_renew will not move if shrinking
- change config default so m_malloc0 uses memset if GC not enabled
- add MICROPY_FLOAT_CONST macro for defining float constants
- move frozen bytecode Makefile rules from ports to common mk files
- strip leading dirs from frozen mpy files, so any path can be used

extmod:
- vfs_fat_file: check fatfs f_sync() and f_close() returns for errors
- vfs_fat_file: make file.close() a no-op if file already closed
- utime_mphal: ticks_diff(): switch arg order, return signed value
- utime_mphal: add MP_THREAD_GIL_EXIT/ENTER warppers for sleep functions
- utime_mphal: implement ticks_add(), add to all maintained ports
- utime_mphal: allow ticks functions period be configurable by a port

lib:
- utils/pyhelp.c: use mp_printf() instead of printf()
- utils/pyexec: add mp_hal_set_interrupt_char() prototype
- libm: move Thumb-specific sqrtf function to separate file

drivers:
- add "from micropython import const" when const is used

tools:
- upgrade upip to 1.1.4: fix error on unix when installing to non-existing
  absolute path
- pip-micropython: remove deprecated wrapper tool
- check_code_size.sh: code size validation script for CI
- replace upip tarball with just source file, to make its inclusion as
  frozen modules in multiple ports less magic

tests:
- extmod/vfs_fat: improve VFS test coverage
- basics/builtin_slice: add test for "slice" builtin name
- basics: add test for builtin "delattr"
- extmod/vfs_fat_fsusermount: improve fsusermount test coverage
- extmod/vfs_fat_oldproto: test old block device protocol
- basics/gc1: garbage collector threshold() coverage
- extmod/uhashlib_sha1: coverage for SHA1 algorithm
- extmod/uhashlib_sha256: rename sha256.py test
- btree1: fix out of memory error running on esp8266
- extmod/ticks_diff: test for new semantics of ticks_diff()
- extmod/framebuf1: test framebuffer pixel clear, and text function

minimal port:
- Makefile: split rule for firmware.bin generation

unix port:
- Makefile: remove references to deprecated pip-micropython
- modtime: use ticks_diff() implementation from extmod/utime_mphal.c
- mphalport.h: add warning of mp_hal_delay_ms() implementation
- modtime: switch ticks/sleep_ms/us() to utime_mphal
- fix symbol references for x86 Mac
- replace upip tarball with just source file

windows port:
- enable utime_mphal following unix, define mp_hal_ticks_*
- fix utime_mphal compilation for msvc
- implement mp_hal_ticks_cpu in terms of QueryPerformanceCounter

qemu-arm port:
- exclude ticks_diff test for qemu-arm port
- exclude extmod/vfs_fat_fileio.py test
- exclude new vfs_fat tests
- enable software floating point support, and float tests

stmhal port:
- modutime: refactor to use extmod's version of ticks_cpu
- refactor pin usage to use mp_hal_pin API
- led: refactor LED to use mp_hal_pin_output() init function
- Makefile: use standard rules for frozen module generation
- modutime: consistently convert to MP_ROM_QSTR/MP_ROM_PTR
- enable SD power save (disable CLK on idle)

cc3200 port:
- use mp_raise_XXX helper functions to reduce code size
- mods/pybspi: allow "write" arg of read/readinto to be positional
- enable loading of precompiled .mpy files
- fix thread mutex's so threading works with interrupts

teensy port:
- update to provide new mp_hal_pin_XXX functions following stmhal

esp8266 port:
- Makefile: use latest esptool.py flash size auto-detection
- esp_init_data: auto-initialize system params with vendor SDK 2.0.0
- esp8266.ld: move help.o to iROM
- esp8266.ld: move modmachine.o to iROM
- esp8266.ld: move main.o to iROM
- add MP_FASTCODE modifier to put a function to iRAM
- main: mark nlr_jump_fail() as MP_FASTCODE
- modules/webrepl: enforce only one concurrent WebREPL connection
- etshal.h: add few more ESP8266 vendor lib prototypes
- modesp: add flash_user_start() function
- add support for building firmware version for 512K modules
- scripts: make neopixel/apa102 handle 4bpp LEDs with common code
- modutime: consistently convert to MP_ROM_QSTR/MP_ROM_PTR
- modnetwork: config(): fix copy-paste error in setting "mac"
- scripts/port_diag: add descriptions for esf_buf types
- modnetwork.c: allows AP reconnection without WiFi credentials
- main: bump heap size to 36K
- etshal.h: add prototypes for SPIRead/SPIWrite/SPIEraseSector
- etshal.h: adjust size of MD5_CTX structure
- modules: fix negative temperature in ds18x20 driver
- rename "machine" module implementation to use contemporary naming
- rework webrepl_setup to run over wired REPL
- espneopixel.c: solve glitching LED issues with cpu at 80MHz
- include upip as a standard frozen bytecode module
- update docs for esptool 1.2.1/SDK 2.0 (--flash_size=detect)
- modnetwork.c: expose configuration for station DHCP hostname

zephyr port:
- implement utime module
- use board/SoC values for startup banner based on Zephyr config
- initial implementation of machine.Pin
- zephyr_getchar: update for recent Zephyr refactor of console hooks
- support time -> utime module "weaklink"
- README: update for the current featureset, add more info
- mpconfigport.h: move less important params to the bottom
- Makefile: allow to adjust heap size from make command line
- Makefile: update comments to the current state of affairs
- Makefile: allow to override Zephyr config from make command line
- Makefile: add minimal port
- Makefile: add -fomit-frame-pointer to reduce code size
- mphalport.h: update for new "unified" kernal API (sleep functions)

docs:
- machine.SPI: bring up to date with Hardware API, make vendor-neutral
- machine.SPI: improve descriptions of xfer methods
- library/builtins: add docs for delattr and slice
- library/network: reword intro paragraph
- library/network: typo fixes, consistent acronym capitalization
- library/index: update TOCs so builtins sorted before modules
- utime: document ticks_cpu() in more detail
- utime: describe new semantics of ticks_diff() (signed ring arithmetics)
- utime: add docs for ticks_add(), improvements for other ticks_*()
- esp8266: update for new WebREPL setup procedure
- */quickref.rst: use new semantics of ticks_diff()
- library/machine.Pin: update Pin docs to align with new HW API

travis:
- integrate tools/check_code_size.sh
- minimal: Use CROSS=1, for binary size check

examples:
- http_server_simplistic: add "not suitable for real use" note
- hwapi: example showing best practices for HW API usage in apps
- hwapi: add hwconfig for DragonBoard 410c

10 files changed, 60 insertions, 31 deletions

diff --git a/extmod/fsusermount.h b/extmod/fsusermount.h
index e1f26f2ce8..af6867d23c 100644
--- a/extmod/fsusermount.h
+++ b/extmod/fsusermount.h
@@ -57,6 +57,6 @@ typedef struct _fs_user_mount_t {
 fs_user_mount_t *fatfs_mount_mkfs(mp_uint_t n_args, const mp_obj_t *pos_args, mp_map_t *kw_args, bool mkfs);
 mp_obj_t fatfs_umount(mp_obj_t bdev_or_path_in);
 
-MP_DECLARE_CONST_FUN_OBJ(fsuser_mount_obj);
-MP_DECLARE_CONST_FUN_OBJ(fsuser_umount_obj);
-MP_DECLARE_CONST_FUN_OBJ(fsuser_mkfs_obj);
+MP_DECLARE_CONST_FUN_OBJ_KW(fsuser_mount_obj);
+MP_DECLARE_CONST_FUN_OBJ_1(fsuser_umount_obj);
+MP_DECLARE_CONST_FUN_OBJ_KW(fsuser_mkfs_obj);
diff --git a/extmod/machine_pulse.h b/extmod/machine_pulse.h
index 3d5d81c073..cc1c4eda5f 100644
--- a/extmod/machine_pulse.h
+++ b/extmod/machine_pulse.h
@@ -32,6 +32,6 @@
 
 mp_uint_t machine_time_pulse_us(mp_hal_pin_obj_t pin, int pulse_level, mp_uint_t timeout_us);
 
-MP_DECLARE_CONST_FUN_OBJ(machine_time_pulse_us_obj);
+MP_DECLARE_CONST_FUN_OBJ_VAR_BETWEEN(machine_time_pulse_us_obj);
 
 #endif // __MICROPY_INCLUDED_EXTMOD_MACHINE_PULSE_H__
diff --git a/extmod/machine_spi.h b/extmod/machine_spi.h
index 316d06646e..62f42a8851 100644
--- a/extmod/machine_spi.h
+++ b/extmod/machine_spi.h
@@ -47,9 +47,9 @@ typedef struct _mp_machine_soft_spi_obj_t {
 
 void mp_machine_soft_spi_transfer(mp_obj_base_t *self, size_t len, const uint8_t *src, uint8_t *dest);
 
-MP_DECLARE_CONST_FUN_OBJ(mp_machine_spi_read_obj);
-MP_DECLARE_CONST_FUN_OBJ(mp_machine_spi_readinto_obj);
-MP_DECLARE_CONST_FUN_OBJ(mp_machine_spi_write_obj);
-MP_DECLARE_CONST_FUN_OBJ(mp_machine_spi_write_readinto_obj);
+MP_DECLARE_CONST_FUN_OBJ_VAR_BETWEEN(mp_machine_spi_read_obj);
+MP_DECLARE_CONST_FUN_OBJ_VAR_BETWEEN(mp_machine_spi_readinto_obj);
+MP_DECLARE_CONST_FUN_OBJ_2(mp_machine_spi_write_obj);
+MP_DECLARE_CONST_FUN_OBJ_3(mp_machine_spi_write_readinto_obj);
 
 #endif // MICROPY_INCLUDED_EXTMOD_MACHINE_SPI_H
diff --git a/extmod/misc.h b/extmod/misc.h
index 634ea924d0..d7ead06549 100644
--- a/extmod/misc.h
+++ b/extmod/misc.h
@@ -30,7 +30,7 @@
 #include <stddef.h>
 #include "py/runtime.h"
 
-MP_DECLARE_CONST_FUN_OBJ(mp_uos_dupterm_obj);
+MP_DECLARE_CONST_FUN_OBJ_VAR_BETWEEN(mp_uos_dupterm_obj);
 
 #if MICROPY_PY_OS_DUPTERM
 void mp_uos_dupterm_tx_strn(const char *str, size_t len);
diff --git a/extmod/modubinascii.h b/extmod/modubinascii.h
index 71dd8a6936..33d0f1cbd8 100644
--- a/extmod/modubinascii.h
+++ b/extmod/modubinascii.h
@@ -33,10 +33,10 @@ extern mp_obj_t mod_binascii_a2b_base64(mp_obj_t data);
 extern mp_obj_t mod_binascii_b2a_base64(mp_obj_t data);
 extern mp_obj_t mod_binascii_crc32(size_t n_args, const mp_obj_t *args);
 
-MP_DECLARE_CONST_FUN_OBJ(mod_binascii_hexlify_obj);
-MP_DECLARE_CONST_FUN_OBJ(mod_binascii_unhexlify_obj);
-MP_DECLARE_CONST_FUN_OBJ(mod_binascii_a2b_base64_obj);
-MP_DECLARE_CONST_FUN_OBJ(mod_binascii_b2a_base64_obj);
-MP_DECLARE_CONST_FUN_OBJ(mod_binascii_crc32_obj);
+MP_DECLARE_CONST_FUN_OBJ_VAR_BETWEEN(mod_binascii_hexlify_obj);
+MP_DECLARE_CONST_FUN_OBJ_1(mod_binascii_unhexlify_obj);
+MP_DECLARE_CONST_FUN_OBJ_1(mod_binascii_a2b_base64_obj);
+MP_DECLARE_CONST_FUN_OBJ_1(mod_binascii_b2a_base64_obj);
+MP_DECLARE_CONST_FUN_OBJ_VAR_BETWEEN(mod_binascii_crc32_obj);
 
 #endif /* MICROPY_EXTMOD_MODUBINASCII */
diff --git a/extmod/moduos_dupterm.c b/extmod/uos_dupterm.c
index d888099dff..d888099dff 100644
--- a/extmod/moduos_dupterm.c
+++ b/extmod/uos_dupterm.c
diff --git a/extmod/utime_mphal.c b/extmod/utime_mphal.c
index 3ecdc94469..f447b3a686 100644
--- a/extmod/utime_mphal.c
+++ b/extmod/utime_mphal.c
@@ -33,14 +33,17 @@
 #include "py/obj.h"
 #include "py/mphal.h"
 #include "py/smallint.h"
+#include "py/runtime.h"
 #include "extmod/utime_mphal.h"
 
 STATIC mp_obj_t time_sleep(mp_obj_t seconds_o) {
+    MP_THREAD_GIL_EXIT();
     #if MICROPY_PY_BUILTINS_FLOAT
     mp_hal_delay_ms(1000 * mp_obj_get_float(seconds_o));
     #else
     mp_hal_delay_ms(1000 * mp_obj_get_int(seconds_o));
     #endif
+    MP_THREAD_GIL_ENTER();
     return mp_const_none;
 }
 MP_DEFINE_CONST_FUN_OBJ_1(mp_utime_sleep_obj, time_sleep);
@@ -48,7 +51,9 @@ MP_DEFINE_CONST_FUN_OBJ_1(mp_utime_sleep_obj, time_sleep);
 STATIC mp_obj_t time_sleep_ms(mp_obj_t arg) {
     mp_int_t ms = mp_obj_get_int(arg);
     if (ms > 0) {
+        MP_THREAD_GIL_EXIT();
         mp_hal_delay_ms(ms);
+        MP_THREAD_GIL_ENTER();
     }
     return mp_const_none;
 }
@@ -57,33 +62,47 @@ MP_DEFINE_CONST_FUN_OBJ_1(mp_utime_sleep_ms_obj, time_sleep_ms);
 STATIC mp_obj_t time_sleep_us(mp_obj_t arg) {
     mp_int_t us = mp_obj_get_int(arg);
     if (us > 0) {
+        MP_THREAD_GIL_EXIT();
         mp_hal_delay_us(us);
+        MP_THREAD_GIL_ENTER();
     }
     return mp_const_none;
 }
 MP_DEFINE_CONST_FUN_OBJ_1(mp_utime_sleep_us_obj, time_sleep_us);
 
 STATIC mp_obj_t time_ticks_ms(void) {
-    return MP_OBJ_NEW_SMALL_INT(mp_hal_ticks_ms() & MP_SMALL_INT_POSITIVE_MASK);
+    return MP_OBJ_NEW_SMALL_INT(mp_hal_ticks_ms() & (MICROPY_PY_UTIME_TICKS_PERIOD - 1));
 }
 MP_DEFINE_CONST_FUN_OBJ_0(mp_utime_ticks_ms_obj, time_ticks_ms);
 
 STATIC mp_obj_t time_ticks_us(void) {
-    return MP_OBJ_NEW_SMALL_INT(mp_hal_ticks_us() & MP_SMALL_INT_POSITIVE_MASK);
+    return MP_OBJ_NEW_SMALL_INT(mp_hal_ticks_us() & (MICROPY_PY_UTIME_TICKS_PERIOD - 1));
 }
 MP_DEFINE_CONST_FUN_OBJ_0(mp_utime_ticks_us_obj, time_ticks_us);
 
 STATIC mp_obj_t time_ticks_cpu(void) {
-    return MP_OBJ_NEW_SMALL_INT(mp_hal_ticks_cpu() & MP_SMALL_INT_POSITIVE_MASK);
+    return MP_OBJ_NEW_SMALL_INT(mp_hal_ticks_cpu() & (MICROPY_PY_UTIME_TICKS_PERIOD - 1));
 }
 MP_DEFINE_CONST_FUN_OBJ_0(mp_utime_ticks_cpu_obj, time_ticks_cpu);
 
-STATIC mp_obj_t time_ticks_diff(mp_obj_t start_in, mp_obj_t end_in) {
+STATIC mp_obj_t time_ticks_diff(mp_obj_t end_in, mp_obj_t start_in) {
     // we assume that the arguments come from ticks_xx so are small ints
-    uint32_t start = MP_OBJ_SMALL_INT_VALUE(start_in);
-    uint32_t end = MP_OBJ_SMALL_INT_VALUE(end_in);
-    return MP_OBJ_NEW_SMALL_INT((end - start) & MP_SMALL_INT_POSITIVE_MASK);
+    mp_uint_t start = MP_OBJ_SMALL_INT_VALUE(start_in);
+    mp_uint_t end = MP_OBJ_SMALL_INT_VALUE(end_in);
+    // Optimized formula avoiding if conditions. We adjust difference "forward",
+    // wrap it around and adjust back.
+    mp_int_t diff = ((end - start + MICROPY_PY_UTIME_TICKS_PERIOD / 2) & (MICROPY_PY_UTIME_TICKS_PERIOD - 1))
+                   - MICROPY_PY_UTIME_TICKS_PERIOD / 2;
+    return MP_OBJ_NEW_SMALL_INT(diff);
 }
 MP_DEFINE_CONST_FUN_OBJ_2(mp_utime_ticks_diff_obj, time_ticks_diff);
 
+STATIC mp_obj_t time_ticks_add(mp_obj_t ticks_in, mp_obj_t delta_in) {
+    // we assume that first argument come from ticks_xx so is small int
+    mp_uint_t ticks = MP_OBJ_SMALL_INT_VALUE(ticks_in);
+    mp_uint_t delta = mp_obj_get_int(delta_in);
+    return MP_OBJ_NEW_SMALL_INT((ticks + delta) & (MICROPY_PY_UTIME_TICKS_PERIOD - 1));
+}
+MP_DEFINE_CONST_FUN_OBJ_2(mp_utime_ticks_add_obj, time_ticks_add);
+
 #endif // MICROPY_PY_UTIME_MP_HAL
diff --git a/extmod/utime_mphal.h b/extmod/utime_mphal.h
index 4f2395a090..644387b676 100644
--- a/extmod/utime_mphal.h
+++ b/extmod/utime_mphal.h
@@ -27,10 +27,11 @@
 
 #include "py/obj.h"
 
-MP_DECLARE_CONST_FUN_OBJ(mp_utime_sleep_obj);
-MP_DECLARE_CONST_FUN_OBJ(mp_utime_sleep_ms_obj);
-MP_DECLARE_CONST_FUN_OBJ(mp_utime_sleep_us_obj);
-MP_DECLARE_CONST_FUN_OBJ(mp_utime_ticks_ms_obj);
-MP_DECLARE_CONST_FUN_OBJ(mp_utime_ticks_us_obj);
-MP_DECLARE_CONST_FUN_OBJ(mp_utime_ticks_cpu_obj);
-MP_DECLARE_CONST_FUN_OBJ(mp_utime_ticks_diff_obj);
+MP_DECLARE_CONST_FUN_OBJ_1(mp_utime_sleep_obj);
+MP_DECLARE_CONST_FUN_OBJ_1(mp_utime_sleep_ms_obj);
+MP_DECLARE_CONST_FUN_OBJ_1(mp_utime_sleep_us_obj);
+MP_DECLARE_CONST_FUN_OBJ_0(mp_utime_ticks_ms_obj);
+MP_DECLARE_CONST_FUN_OBJ_0(mp_utime_ticks_us_obj);
+MP_DECLARE_CONST_FUN_OBJ_0(mp_utime_ticks_cpu_obj);
+MP_DECLARE_CONST_FUN_OBJ_2(mp_utime_ticks_diff_obj);
+MP_DECLARE_CONST_FUN_OBJ_2(mp_utime_ticks_add_obj);
diff --git a/extmod/vfs_fat_file.c b/extmod/vfs_fat_file.c
index ecc9ed70f9..76ac23685a 100644
--- a/extmod/vfs_fat_file.c
+++ b/extmod/vfs_fat_file.c
@@ -110,14 +110,23 @@ STATIC mp_uint_t file_obj_write(mp_obj_t self_in, const void *buf, mp_uint_t siz
 
 STATIC mp_obj_t file_obj_flush(mp_obj_t self_in) {
     pyb_file_obj_t *self = MP_OBJ_TO_PTR(self_in);
-    f_sync(&self->fp);
+    FRESULT res = f_sync(&self->fp);
+    if (res != FR_OK) {
+        mp_raise_OSError(fresult_to_errno_table[res]);
+    }
     return mp_const_none;
 }
 STATIC MP_DEFINE_CONST_FUN_OBJ_1(file_obj_flush_obj, file_obj_flush);
 
 STATIC mp_obj_t file_obj_close(mp_obj_t self_in) {
     pyb_file_obj_t *self = MP_OBJ_TO_PTR(self_in);
-    f_close(&self->fp);
+    // if fs==NULL then the file is closed and in that case this method is a no-op
+    if (self->fp.fs != NULL) {
+        FRESULT res = f_close(&self->fp);
+        if (res != FR_OK) {
+            mp_raise_OSError(fresult_to_errno_table[res]);
+        }
+    }
     return mp_const_none;
 }
 STATIC MP_DEFINE_CONST_FUN_OBJ_1(file_obj_close_obj, file_obj_close);
diff --git a/extmod/vfs_fat_file.h b/extmod/vfs_fat_file.h
index 6d68889550..5c271b6ee6 100644
--- a/extmod/vfs_fat_file.h
+++ b/extmod/vfs_fat_file.h
@@ -27,6 +27,6 @@
 extern const byte fresult_to_errno_table[20];
 
 mp_obj_t fatfs_builtin_open(mp_uint_t n_args, const mp_obj_t *args, mp_map_t *kwargs);
-MP_DECLARE_CONST_FUN_OBJ(mp_builtin_open_obj);
+MP_DECLARE_CONST_FUN_OBJ_KW(mp_builtin_open_obj);
 
 mp_obj_t fat_vfs_listdir(const char *path, bool is_str_type);