Merge tag 'v1.8.3' into parse-bytecode

Many small improvements and additions, with btree support in ESP8266 port

This release brings various improvements and additions to the core,
extended modules and the ESP8266 port, as well as enhancements to the docs.
There is now a "threshold" function in the gc module for the user to
configure the garbage collector to run earlier than usual, in order to help
reduce fragmentation of the heap.  The btree module is now available in the
ESP8266 port, and there is improved WebREPL file transfer: get file is now
non-blocking (this change requires an update of the client software).

py core:
- nlrx64.S: prefix mp_thread_get_state with an underscore on Mac
- stream: implement 2- and 3-arg write() method as an extension to CPython
- gc: implement GC running by allocation threshold, user configurable
- fix nlrthumb.c when DEBUG=1 is defined
- obj: issue a warning when str and bytes objects are compared
- mpconfig.h: fix description for MICROPY_PY_STR_BYTES_CMP_WARN
- objstrunicode: str_index_to_ptr: should handle bytes too
- objstrunicode: str_index_to_ptr: implement positive indexing properly
- stream: implement generic flush() method, in terms of C-level ioctl
- objstringio: add MP_STREAM_FLUSH ioctl and flush() method
- objstringio: implement MP_STREAM_SEEK ioctl and add seek() method
- stream: add adapter methods with POSIX-compatible signatures
- mpconfig.h: add MICROPY_STREAMS_POSIX_API setting
- py.mk: extra switches to build "embedded" BerkeleyDB BTree lib
- mkrules.mk: allow to add more items for "clean" target using CLEAN_EXTRA
- objstr: make .partition()/.rpartition() methods configurable
- mpconfig.h: define MP_ALWAYSINLINE for reuse

extmod:
- modussl: rename to modussl_axtls.c, to allow impl using other SSL libs
- modussl_axtls: further changes to allow alternative SSL modules
- modussl_axtls: add dummy setblocking() method
- vfs_fat: implement rmdir() method (aliases to unlink)
- modussl_axtls: use mp_stream_close() method
- modwebrepl: use mp_stream_close() method
- modbtree: check __bt_open() return value for error
- modbtree: switch to accepting stream object instead of filename
- modbtree: implement __contains__ operation
- modwebrepl: factor out "GET" iteration to write_file_chunk()
- modwebrepl: make GET_FILE operation non-blocking
- modbtree: open(): add option kwargs
- modwebsocket: add readline method
- modwebsocket: make compatible with non-default object models
- modwebsocket: use mp_rom_map_elem_t and friends
- modwebrepl: add GET_VER operation to query MicroPython version

lib:
- axtls: upgrade to axTLS 1.5.4 + MicroPython patchset
- axtls: update, fixes esp8266 build
- utils/stdout_helpers: move from minimal/uart_extra.c for reuse
- embed/abort_: implementation of abort_() function raising uPy exception
- berkeley-db-1.xx: switch to "embedded" branch
- berkeley-db-1.xx: update, sets default page size to 4096

tests:
- add test for extended arguments to stream .write() method
- extmod/machine_pinbase: skip if machine.PinBase is not available
- bytes_compare: rework test for bytes <-> str comparison
- extmod/btree1: close database at the end of test
- unicode_subscr.py: detailed test for subscripting unicode strings
- run-tests: make "regex'ed .exp" facility available to device tests
- io/bytesio_ext: test for .seek()/.flush() on BytesIO
- extmod/btree1: tests against in-memory DB (using io.BytesIO)
- machine_mem.py: too non-portable, rework as an example for unix port
- machine1: revamp to work with unix port (which has "umachine")
- basics: bytes/str.partition/rpartition are now optional

mpy-cross:
- fix Makefile to handle gc-sections linker flags on OS X
- use binary file translation mode for creating mpy files on windows
- fix mingw and msys2 compilation

minimal port:
- disable MICROPY_GC_ALLOC_THRESHOLD

unix port:
- mpthreadport: adjust minimum thread stack, and stack limit check
- cache libaxtls.a in local build dir
- disable MICROPY_GC_ALLOC_THRESHOLD for minimal build
- enable MICROPY_PY_STR_BYTES_CMP_WARN
- mpconfigport.h: include stdio.h by default
- file: implement MP_STREAM_FLUSH ioctl
- file: ioctl(): check that file is open before operations
- file: fdfile_ioctl(): fix argument to check_fd_is_open()
- file: use generic stream flush() method
- enable websocket module
- moduselect: allow poll.register(), etc. accept fd-like objects

stmhal port:
- fix malloc when used with external libraries
- make SPI NSS pin definition optional
- fix I2C mappings for STM32F429DISC board

esp8266 port:
- select axTLS for SSL implementation, following recent refactor
- moduos: add rmdir() function
- cache Xtensa-built libaxtls.a in local build dir
- enable MICROPY_PY_STR_BYTES_CMP_WARN
- dupterm_task_init() should be called before running _boot.py, etc
- _boot.py: set GC alloc threshold to half of heap size
- _boot.py: decrease GC alloc threshold to quarter of heap size
- modpybuart: fix UART parity setting
- axtls_helpers: remove abort_(), now in lib/embed/
- mpconfigport.h: include sys/types.h for POSIX types definitions
- esp_mphal: implement libc's errno
- enable btree module
- eagle.rom.addr.v6.ld: add Enable_QMode symbol from SDK 2.0.0
- make APA102 driver inclusion configurable
- makeimg.py: store firmware size as last 4 bytes of padding area
- makeimg.py: append md5 hash to the generated binary
- modesp: add check_fw() function to check integrity of the firmware
- scripts/port_diag.py: include esp.check_fw() call
- flashbdev: reserve extra sysparam sector for SDK 2.0.0 compatibility
- scripts/inisetup: add commented-out call to esp.osdebug(None)
- modmachine: implement idle() function
- esp_mphal.h: add mp_hal_ticks_cpu() for reuse
- modutime: actually implement ticks_cpu()
- modmachine: implement dummy sleep() function
- tutorial/intro: reword para abou -fm dio switch
- modules/flashbdev: start filesystem at 0x90000
- esp8266.ld: increase firmware image size to 0x90000 (576K)

docs:
- add DHT to ESP8266 Quick Ref and Tutorial
- fix some spelling mistakes
- array: document array module
- library/index: include array module in ToC
- esp8266/intro: rename to "Getting started" from "Introduction"
- esp8266/intro: add troubleshooting section
- esp8266/quickref: link to installation instructions
- esp8266/tutorial/intro: add anchor for link from quickeref
- esp8266/intro: focus on hazards of unearthed power wrt electronics
- uio: mention seek()/flush() support for io.BytesIO

misc:
- logo/1bit-logo A black & white version of the logo

examples:
- http_server*: update for buffered-like streams (read line by line)
- embedding: example for embedding MicroPython in an app
- embedding: add README
- http_client*: be sure to close socket
- network: split recv- and read-based HTTP servers

qemu-arm:
- enable gcc LTO option for nlrthumb.c

1 files changed, 203 insertions, 0 deletions

diff --git a/extmod/modussl_axtls.c b/extmod/modussl_axtls.c
new file mode 100644
index 0000000000..ce86263c2c
--- /dev/null
+++ b/extmod/modussl_axtls.c
@@ -0,0 +1,203 @@
+/*
+ * This file is part of the Micro Python project, http://micropython.org/
+ *
+ * The MIT License (MIT)
+ *
+ * Copyright (c) 2015 Paul Sokolovsky
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <errno.h>
+
+#include "py/nlr.h"
+#include "py/runtime.h"
+#include "py/stream.h"
+
+#if MICROPY_PY_USSL && MICROPY_SSL_AXTLS
+
+#include "ssl.h"
+
+typedef struct _mp_obj_ssl_socket_t {
+    mp_obj_base_t base;
+    mp_obj_t sock;
+    SSL_CTX *ssl_ctx;
+    SSL *ssl_sock;
+    byte *buf;
+    uint32_t bytes_left;
+} mp_obj_ssl_socket_t;
+
+STATIC const mp_obj_type_t ussl_socket_type;
+
+STATIC mp_obj_ssl_socket_t *socket_new(mp_obj_t sock, bool server_side) {
+    mp_obj_ssl_socket_t *o = m_new_obj(mp_obj_ssl_socket_t);
+    o->base.type = &ussl_socket_type;
+    o->buf = NULL;
+    o->bytes_left = 0;
+    o->sock = sock;
+
+    uint32_t options = SSL_SERVER_VERIFY_LATER;
+    if ((o->ssl_ctx = ssl_ctx_new(options, SSL_DEFAULT_CLNT_SESS)) == NULL) {
+        nlr_raise(mp_obj_new_exception_arg1(&mp_type_OSError, MP_OBJ_NEW_SMALL_INT(EINVAL)));
+    }
+
+    if (server_side) {
+        o->ssl_sock = ssl_server_new(o->ssl_ctx, (long)sock);
+    } else {
+        o->ssl_sock = ssl_client_new(o->ssl_ctx, (long)sock, NULL, 0);
+
+        int res;
+        /* check the return status */
+        if ((res = ssl_handshake_status(o->ssl_sock)) != SSL_OK) {
+            printf("ssl_handshake_status: %d\n", res);
+            ssl_display_error(res);
+            nlr_raise(mp_obj_new_exception_arg1(&mp_type_OSError, MP_OBJ_NEW_SMALL_INT(EIO)));
+        }
+    }
+
+    return o;
+}
+
+STATIC void socket_print(const mp_print_t *print, mp_obj_t self_in, mp_print_kind_t kind) {
+    (void)kind;
+    mp_obj_ssl_socket_t *self = MP_OBJ_TO_PTR(self_in);
+    mp_printf(print, "<_SSLSocket %p>", self->ssl_sock);
+}
+
+STATIC mp_uint_t socket_read(mp_obj_t o_in, void *buf, mp_uint_t size, int *errcode) {
+    mp_obj_ssl_socket_t *o = MP_OBJ_TO_PTR(o_in);
+
+    while (o->bytes_left == 0) {
+        mp_int_t r = ssl_read(o->ssl_sock, &o->buf);
+        if (r == SSL_OK) {
+            // SSL_OK from ssl_read() means "everything is ok, but there's
+            // not user data yet. So, we just keep reading.
+            continue;
+        }
+        if (r < 0) {
+            if (r == SSL_CLOSE_NOTIFY || r == SSL_ERROR_CONN_LOST) {
+                // EOF
+                return 0;
+            }
+            *errcode = r;
+            return MP_STREAM_ERROR;
+        }
+        o->bytes_left = r;
+    }
+
+    if (size > o->bytes_left) {
+        size = o->bytes_left;
+    }
+    memcpy(buf, o->buf, size);
+    o->buf += size;
+    o->bytes_left -= size;
+    return size;
+}
+
+STATIC mp_uint_t socket_write(mp_obj_t o_in, const void *buf, mp_uint_t size, int *errcode) {
+    mp_obj_ssl_socket_t *o = MP_OBJ_TO_PTR(o_in);
+    mp_int_t r = ssl_write(o->ssl_sock, buf, size);
+    if (r < 0) {
+        *errcode = r;
+        return MP_STREAM_ERROR;
+    }
+    return r;
+}
+
+STATIC mp_obj_t socket_setblocking(mp_obj_t self_in, mp_obj_t flag_in) {
+    // Currently supports only blocking mode
+    (void)self_in;
+    if (!mp_obj_is_true(flag_in)) {
+        mp_not_implemented("");
+    }
+    return mp_const_none;
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_2(socket_setblocking_obj, socket_setblocking);
+
+STATIC mp_obj_t socket_close(mp_obj_t self_in) {
+    mp_obj_ssl_socket_t *self = MP_OBJ_TO_PTR(self_in);
+    ssl_free(self->ssl_sock);
+    ssl_ctx_free(self->ssl_ctx);
+    return mp_stream_close(self->sock);
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_1(socket_close_obj, socket_close);
+
+STATIC const mp_rom_map_elem_t ussl_socket_locals_dict_table[] = {
+    { MP_ROM_QSTR(MP_QSTR_read), MP_ROM_PTR(&mp_stream_read_obj) },
+    { MP_ROM_QSTR(MP_QSTR_readall), MP_ROM_PTR(&mp_stream_readall_obj) },
+    { MP_ROM_QSTR(MP_QSTR_readinto), MP_ROM_PTR(&mp_stream_readinto_obj) },
+    { MP_ROM_QSTR(MP_QSTR_readline), MP_ROM_PTR(&mp_stream_unbuffered_readline_obj) },
+    { MP_ROM_QSTR(MP_QSTR_write), MP_ROM_PTR(&mp_stream_write_obj) },
+    { MP_ROM_QSTR(MP_QSTR_setblocking), MP_ROM_PTR(&socket_setblocking_obj) },
+    { MP_ROM_QSTR(MP_QSTR_close), MP_ROM_PTR(&socket_close_obj) },
+};
+
+STATIC MP_DEFINE_CONST_DICT(ussl_socket_locals_dict, ussl_socket_locals_dict_table);
+
+STATIC const mp_stream_p_t ussl_socket_stream_p = {
+    .read = socket_read,
+    .write = socket_write,
+};
+
+STATIC const mp_obj_type_t ussl_socket_type = {
+    { &mp_type_type },
+    // Save on qstr's, reuse same as for module
+    .name = MP_QSTR_ussl,
+    .print = socket_print,
+    .getiter = NULL,
+    .iternext = NULL,
+    .protocol = &ussl_socket_stream_p,
+    .locals_dict = (void*)&ussl_socket_locals_dict,
+};
+
+STATIC mp_obj_t mod_ssl_wrap_socket(size_t n_args, const mp_obj_t *pos_args, mp_map_t *kw_args) {
+    // TODO: Implement more args
+    static const mp_arg_t allowed_args[] = {
+        { MP_QSTR_server_side, MP_ARG_KW_ONLY | MP_ARG_BOOL, {.u_bool = false} },
+    };
+
+    // TODO: Check that sock implements stream protocol
+    mp_obj_t sock = pos_args[0];
+
+    struct {
+        mp_arg_val_t server_side;
+    } args;
+    mp_arg_parse_all(n_args - 1, pos_args + 1, kw_args,
+        MP_ARRAY_SIZE(allowed_args), allowed_args, (mp_arg_val_t*)&args);
+
+    return MP_OBJ_FROM_PTR(socket_new(sock, args.server_side.u_bool));
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_KW(mod_ssl_wrap_socket_obj, 1, mod_ssl_wrap_socket);
+
+STATIC const mp_rom_map_elem_t mp_module_ssl_globals_table[] = {
+    { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_ussl) },
+    { MP_ROM_QSTR(MP_QSTR_wrap_socket), MP_ROM_PTR(&mod_ssl_wrap_socket_obj) },
+};
+
+STATIC MP_DEFINE_CONST_DICT(mp_module_ssl_globals, mp_module_ssl_globals_table);
+
+const mp_obj_module_t mp_module_ussl = {
+    .base = { &mp_type_module },
+    .name = MP_QSTR_ussl,
+    .globals = (mp_obj_dict_t*)&mp_module_ssl_globals,
+};
+
+#endif // MICROPY_PY_USSL