Merge tag 'v1.8.2' into parse-bytecode

Thread support, ESP8266 memory improvements, btree module, improved docs

This release brings multi-threading support in the form of the _thread
module, which closely matches the semantics of the corresponding CPython
module.  There is support for GIL and non-GIL builds; without the GIL
enabled one must protect concurrent access to mutable Python state at the
Python level using Lock objects.  Threading with the GIL is enabled in
the cc3200 port on the WiPy.  Threading without the GIL is enabled on the
unix port.  The ESP8266 port has support for frozen bytecode (for scripts
in the modules/ subdirectory), as well as optimisations for reduced
memory usage and decreased memory fragmentation.  The ESP8266 RTC also
resumes correctly after a deepsleep.  An initial "btree" module has been
implemented for database support (unix port only), and the documentation
has been further improved, with pre-built PDF versions of the docs now
available.

py core:
- parse: treat constants that start with underscore as private
- objdict: implemented OrderedDict equality check
- support to build berkeley db 1.85 and "btree" module
- mpconfig.h: MP_NOINLINE is universally useful, move from unix port
- makeqstrdefs.py: remove restriction that source path can't be absolute
- mkrules.mk: define "lib" outside conditional block
- rename __QSTR_EXTRACT flag to NO_QSTR
- objtype: instance: inherit protocol vtable from a base class
- mphal.h: if virtpin API is used, automagically include its header
- objtype: inherit protocol vtable from base class only if it exists
- add MP_STATE_THREAD to hold state specific to a given thread
- add basic _thread module, with ability to start a new thread
- modthread: properly cast concrete exception pointer to an object
- modthread: add stack_size() function
- modthread: add exit() function
- modthread: implement lock object, for creating a mutex
- modthread: add with-context capabilities to lock object
- gc: make memory manager and garbage collector thread safe
- modthread: satisfy unused-args warning
- gc: fix 2 cases of concurrent access to ATB and FTB
- modthread: be more careful with root pointers when creating a thread
- modthread: call mp_thread_start/mp_thread_finish around threads
- gc: fix GC+thread bug where ptr gets lost because it's not computed
- make interning of qstrs thread safe
- implement a simple global interpreter lock (GIL)
- don't use gc or qstr mutex when the GIL is enabled, there is no need
- modthread: make Lock objects work when GIL is enabled
- mpthread.h: move GIL macros outside MICROPY_PY_THREAD block
- modthread: allow to properly set the stack limit of a thread
- nlrthumb: convert NLR thumb funcs from asm to C with inline-asm
- nlrsetjmp: update to take into account new location of nlr_top
- mpthread: include mpstate.h when defining GIL macros
- gc: be sure to count last allocated block at heap end in stats
- gc: calculate (and report) maximum contiguous free block size
- builtinimport: disable "imported as namespace package" warning
- mpconfig.h: mention MICROPY_PY_BTREE config option
- objarray: split out header to allow direct access to object
- rename mp_obj_type_t::stream_p to "protocol"

extmod:
- modbtree: initial implementation of "btree" module based on BerkeleyDB
- modbtree: handle default value and error check
- modbtree: implement .items() iterator
- modbtree: actually implement end key support for .items()
- modbtree: items(): Implement "end key inclusive" flag
- modbtree: items(): Implement DESC flag
- modbtree: __getitem__() should raise KeyError for non-existing key
- modbtree: open(): Support "in-memory" database with filename=None
- machine_pinbase: implementation of PinBase class
- machine_pinbase: fix nanbox build
- modlwip: store a chain of incoming pbufs, instead of only one
- modbtree: implement keys(), values(), items() iterators
- modbtree: cleverly implement "for key in btree:" syntax
- modwebsocket: add readinto() method
- modwebrepl: add readinto() method
- modbtree: fix unused argument warning
- modbtree: fixes for nanbox build
- moduos_dupterm: reserve buffer bytearray object for dupterm
- moduos_dupterm: reuse dupterm_arr_obj for write operations

lib:
- berkeley-db-1.xx: add Berkeley DB 1.85 as a submodule
- berkeley-db-1.xx: update to upstream which builds for uPy
- fatfs/option/ccsbcs: follow uPy optional features model
- libm: format code to pass gcc v6.1.1 warning
- libm: remove unused definition of "one"

drivers:
- display/ssd1306: add width arg and support 64px wide displays

tests:
- recursive_iternext: clang/Linux is even more stack-frugal than MacOS
- bench: add testcase to compare bytes(N) vs b"\0" * N
- add testcase for OrderedDict equality
- add a testcase for machine.PinBase class
- extmod: add "btree" module test
- btree1: add testcase for iterating btree object directly
- add tests for _thread module
- add 3 more tests for _thread module
- thread: remove need to sleep to wait for completion in some tests
- thread: add tests for running GC within a thread, and heap stress
- thread: rename thread_stress_XXX.py to stress_XXX.py
- thread: add tests that mutate shared objects
- thread: add test for concurrent interning of strings
- thread: add test for concurrent mutating of user instance
- thread: make stack-size test run correctly and reliable on uPy
- thread: make sure that thread tests don't rely on floating point
- thread: allow thread_sleep1 to run without floating point
- thread: allow some tests to run on ports with not much heap
- meminfo: update for addition of "max free sz" output
- run-tests: If running thread tests on unix, don't run mutate ones

unix port:
- unbreak "minimal" target by disabling FatFs
- mpconfigport_minimal.h: clearly mark where user-define config ends
- deprecate support for GNU Readline (MICROPY_USE_READLINE=2)
- time to build with --gc-sections
- move "utime" module config to C level instead of make level
- Makefile: libffi: Build with -Os
- alloc: add option to use uPy's alloc-exec implementation even for libffi
- alloc: make coverage build and its overzealous warnings happy
- disable FatFs VFS for normal build, keep enabled for coverage
- modmachine: include PinBase class
- modmachine: enable time_pulse_us() function
- fix Makefile to handle gc-sections linker flags on Mac OS
- add basic thread support using pthreads
- file: if write syscall returns because of EINTR then try again
- implement garbage collection with threading
- mpthreadport: use SA_SIGINFO for GC signal handler
- gccollect: provide declaration of exported function
- mpthreadport: suppress compiler warning about unused arguments
- modtime: release the GIL when sleeping
- enable btree module
- Makefile: make "minimal" build be minimal again
- mpconfigport_minimal.h: allow to print a string within 1KB of heap
- main: error out on unknown value of suffix in -X heapsize= option
- main: improve help for -X options a bit
- main: when preparing sys.path, allocate exact strings on uPy heap
- disable the GIL to improve performance of non-thread code

windows port:
- follow unix port changes regarding "utime" module
- msvc: include machine_pinbase.c in build and enable umachine module

stmhal port:
- add board definition files for Olimex STM32-E407
- port of f4 hal commit 1d7fb82 to l4 hal (SD card modifications)
- correct DMA to allow SD card on L4 MCUs
- enable SD card on L4 MCUs

cc3200 port:
- update FreeRTOS to v9.0.0
- define our own FreeRTOS heap so it can go in a special segment
- add basic threading capabilities
- fix call to new exception to be _msg instead of _arg1
- use xTaskCreateStatic instead of osi_TaskCreate
- mpthreadport: make mutex statically allocated
- mpthreadport: properly initialise the main thread's data
- gccollect: use MP_STATE_THREAD(stack_top) to get top of stack
- mpthreadport: scan more root pointers from thread data
- mpthreadport: move mem alloc outside the thread_mutex lock
- enable the GIL
- in FreeRTOSConfig.h, comment on configSUPPORT_STATIC_ALLOCATION
- remove .boot section attribute from osi_TaskCreate function
- start the simplelink spawn task using the static task creator
- shrink the FreeRTOS heap and place TCB+stack in freed location
- allow to compile bootloader with threading enabled
- remove obsolete singleton heart-beat object

teensy port:
- update the README with OSX specific information and tips

esp8266 port:
- modnetwork: use struct bss_info::ssid_len for ESSID length
- let RTC work correctly after deepsleep
- switch floating-point arith routines to BootROM
- Makefile: enable --verify option for esptool.py write_flash
- use RTC to set date & time stamps for files
- main.c: clear the command line history when (re)booting
- enable frozen bytecode, with scripts in modules/ subdir
- README: describe how to build mpy-cross
- websocket_helper.py: avoid extra string allocations
- websocket_helper.py: fix typo in debug output
- modpybuart: allow setting baudrate and other params
- explicitly collect garbage in bootstrap scripts
- switch webrepl_setup to use frozen bytecode
- switch webrepl to use frozen bytecode
- README: promote from "highly experimental" to "experimental"
- main: init recently added dupterm_arr_obj port state var
- esp_mphal: call_dupterm_read: Use readinto() method
- esp_mphal: properly handle dupterm EOF after switching to readinto()

qemu-arm port:
- disable gcc LTO option for nlrthumb.c

docs:
- esp8266/tutorial/network_basics: minor typo fix, ap becomes ap_if
- add link to PDF version of docs in sidebar
- library/network: wipy: fix literal block indentation
- esp.socket: remove unused file
- ure: typo fix
- usocket: there must be empty line after .only::
- library/machine.Pin: mention GPIO and cross-link .value() function
- machine.*: add "currentmodule:: machine" directive
- pyb.*: add "currentmodule:: pyb" directive
- machine.Pin: remove explicit target
- machine.UART: cleanup pyboard section
- machine.*: use proper class case in method headers
- machine.Pin: add class designator to all constants
- pyb.CAN: mark CAN.initfilterbanks() as classmethod explicitly
- pyb.Pin: af_list() is a normal method, not a class method
- pyb.ExtInt,pyb.Pin: mark up class methods as such
- pyb.*: use proper class case in method headers
- pyb.Pin: sort .af() and .af_list() methods together
- builtins: enumerate all builtin functions implemented
- library/index: add builtins.rst
- machine.Pin: disambiguate object call method
- machine*: remove explicit targets and "machine." prefixes on classes
- uctypes: improve documentation
- sys: detailed description of print_exception() diff from traceback module
- add sphinx_selective_exclude extension suite
- conf.py: active sphinx_selective_exclude extensions
- rebuild docs from scratch, as required for proper only:: handling
- select: add an article
- sys: print_exception: Fixes/clarifications
- conf.py: exclude cmath from modindex for wipy
- library: fix typo in docs for usocket.listen()

examples:
- unix/ffi_example.py: update for current "ffi" module API
- unix/ffi_example.py: remove TODO, make output more clear

11 files changed, 537 insertions, 33 deletions

diff --git a/extmod/machine_pinbase.c b/extmod/machine_pinbase.c
new file mode 100644
index 0000000000..3c1f094834
--- /dev/null
+++ b/extmod/machine_pinbase.c
@@ -0,0 +1,88 @@
+/*
+ * This file is part of the MicroPython project, http://micropython.org/
+ *
+ * The MIT License (MIT)
+ *
+ * Copyright (c) 2016 Paul Sokolovsky
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "py/mpconfig.h"
+#if MICROPY_PY_MACHINE
+
+#include "py/obj.h"
+#include "py/runtime.h"
+#include "extmod/virtpin.h"
+
+// PinBase class
+
+// As this is abstract class, its instance is null.
+// But there should be an instance, as the rest of instance code
+// expects that there will be concrete object for inheritance.
+typedef struct _mp_pinbase_t {
+    mp_obj_base_t base;
+} mp_pinbase_t;
+
+STATIC const mp_obj_type_t pinbase_type;
+
+STATIC mp_pinbase_t pinbase_singleton = {
+    .base = { &pinbase_type },
+};
+
+STATIC mp_obj_t pinbase_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args) {
+    (void)type;
+    (void)n_args;
+    (void)n_kw;
+    (void)args;
+    return MP_OBJ_FROM_PTR(&pinbase_singleton);
+}
+
+mp_uint_t pinbase_ioctl(mp_obj_t obj, mp_uint_t request, uintptr_t arg, int *errcode);
+mp_uint_t pinbase_ioctl(mp_obj_t obj, mp_uint_t request, uintptr_t arg, int *errcode) {
+    (void)errcode;
+    switch (request) {
+        case MP_PIN_READ: {
+            mp_obj_t dest[2];
+            mp_load_method(obj, MP_QSTR_value, dest);
+            return mp_obj_get_int(mp_call_method_n_kw(0, 0, dest));
+        }
+        case MP_PIN_WRITE: {
+            mp_obj_t dest[3];
+            mp_load_method(obj, MP_QSTR_value, dest);
+            dest[2] = (arg == 0 ? mp_const_false : mp_const_true);
+            mp_call_method_n_kw(1, 0, dest);
+            return 0;
+        }
+    }
+    return -1;
+}
+
+STATIC const mp_pin_p_t pinbase_pin_p = {
+    .ioctl = pinbase_ioctl,
+};
+
+const mp_obj_type_t machine_pinbase_type = {
+    { &mp_type_type },
+    .name = MP_QSTR_PinBase,
+    .make_new = pinbase_make_new,
+    .protocol = &pinbase_pin_p,
+};
+
+#endif // MICROPY_PY_MACHINE
diff --git a/extmod/machine_pinbase.h b/extmod/machine_pinbase.h
new file mode 100644
index 0000000000..ece3384cc6
--- /dev/null
+++ b/extmod/machine_pinbase.h
@@ -0,0 +1,35 @@
+/*
+ * This file is part of the MicroPython project, http://micropython.org/
+ *
+ * The MIT License (MIT)
+ *
+ * Copyright (c) 2016 Paul Sokolovsky
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+
+#ifndef __MICROPY_INCLUDED_EXTMOD_MACHINE_PINBASE_H__
+#define __MICROPY_INCLUDED_EXTMOD_MACHINE_PINBASE_H__
+
+#include "py/obj.h"
+
+extern const mp_obj_type_t machine_pinbase_type;
+
+#endif // __MICROPY_INCLUDED_EXTMOD_MACHINE_PINBASE_H__
diff --git a/extmod/modbtree.c b/extmod/modbtree.c
new file mode 100644
index 0000000000..0a0c5271c0
--- /dev/null
+++ b/extmod/modbtree.c
@@ -0,0 +1,353 @@
+/*
+ * This file is part of the MicroPython project, http://micropython.org/
+ *
+ * The MIT License (MIT)
+ *
+ * Copyright (c) 2016 Paul Sokolovsky
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <errno.h>
+#include <fcntl.h>
+
+#include "py/nlr.h"
+#include "py/runtime.h"
+
+#if MICROPY_PY_BTREE
+
+#include <db.h>
+#include <../../btree/btree.h>
+
+typedef struct _mp_obj_btree_t {
+    mp_obj_base_t base;
+    DB *db;
+    mp_obj_t start_key;
+    mp_obj_t end_key;
+    #define FLAG_END_KEY_INCL 1
+    #define FLAG_DESC 2
+    #define FLAG_ITER_TYPE_MASK 0xc0
+    #define FLAG_ITER_KEYS   0x40
+    #define FLAG_ITER_VALUES 0x80
+    #define FLAG_ITER_ITEMS  0xc0
+    byte flags;
+    byte next_flags;
+} mp_obj_btree_t;
+
+STATIC const mp_obj_type_t btree_type;
+
+#define CHECK_ERROR(res) \
+        if (res == RET_ERROR) { \
+            nlr_raise(mp_obj_new_exception_arg1(&mp_type_OSError, MP_OBJ_NEW_SMALL_INT(errno))); \
+        }
+
+void __dbpanic(DB *db) {
+    printf("__dbpanic(%p)\n", db);
+}
+
+STATIC mp_obj_btree_t *btree_new(DB *db) {
+    mp_obj_btree_t *o = m_new_obj(mp_obj_btree_t);
+    o->base.type = &btree_type;
+    o->db = db;
+    o->start_key = mp_const_none;
+    o->end_key = mp_const_none;
+    o->next_flags = 0;
+    return o;
+}
+
+STATIC void btree_print(const mp_print_t *print, mp_obj_t self_in, mp_print_kind_t kind) {
+    (void)kind;
+    mp_obj_btree_t *self = MP_OBJ_TO_PTR(self_in);
+    mp_printf(print, "<btree %p>", self->db);
+}
+
+STATIC mp_obj_t btree_close(mp_obj_t self_in) {
+    mp_obj_btree_t *self = MP_OBJ_TO_PTR(self_in);
+    return MP_OBJ_NEW_SMALL_INT(__bt_close(self->db));
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_1(btree_close_obj, btree_close);
+
+STATIC mp_obj_t btree_put(size_t n_args, const mp_obj_t *args) {
+    (void)n_args;
+    mp_obj_btree_t *self = MP_OBJ_TO_PTR(args[0]);
+    DBT key, val;
+    // Different ports may have different type sizes
+    mp_uint_t v;
+    key.data = (void*)mp_obj_str_get_data(args[1], &v);
+    key.size = v;
+    val.data = (void*)mp_obj_str_get_data(args[2], &v);
+    val.size = v;
+    return MP_OBJ_NEW_SMALL_INT(__bt_put(self->db, &key, &val, 0));
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(btree_put_obj, 3, 4, btree_put);
+
+STATIC mp_obj_t btree_get(size_t n_args, const mp_obj_t *args) {
+    mp_obj_btree_t *self = MP_OBJ_TO_PTR(args[0]);
+    DBT key, val;
+    // Different ports may have different type sizes
+    mp_uint_t v;
+    key.data = (void*)mp_obj_str_get_data(args[1], &v);
+    key.size = v;
+    int res = __bt_get(self->db, &key, &val, 0);
+    if (res == RET_SPECIAL) {
+        if (n_args > 2) {
+            return args[2];
+        } else {
+            return mp_const_none;
+        }
+    }
+    CHECK_ERROR(res);
+    return mp_obj_new_bytes(val.data, val.size);
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(btree_get_obj, 2, 3, btree_get);
+
+STATIC mp_obj_t btree_seq(size_t n_args, const mp_obj_t *args) {
+    mp_obj_btree_t *self = MP_OBJ_TO_PTR(args[0]);
+    int flags = MP_OBJ_SMALL_INT_VALUE(args[1]);
+    DBT key, val;
+    if (n_args > 2) {
+        // Different ports may have different type sizes
+        mp_uint_t v;
+        key.data = (void*)mp_obj_str_get_data(args[2], &v);
+        key.size = v;
+    }
+
+    int res = __bt_seq(self->db, &key, &val, flags);
+    if (res == RET_SPECIAL) {
+        return mp_const_none;
+    }
+
+    mp_obj_t pair_o = mp_obj_new_tuple(2, NULL);
+    mp_obj_tuple_t *pair = MP_OBJ_TO_PTR(pair_o);
+    pair->items[0] = mp_obj_new_bytes(key.data, key.size);
+    pair->items[1] = mp_obj_new_bytes(val.data, val.size);
+    return pair_o;
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(btree_seq_obj, 2, 4, btree_seq);
+
+STATIC mp_obj_t btree_init_iter(size_t n_args, const mp_obj_t *args, byte type) {
+    mp_obj_btree_t *self = MP_OBJ_TO_PTR(args[0]);
+    self->next_flags = type;
+    self->start_key = mp_const_none;
+    self->end_key = mp_const_none;
+    if (n_args > 1) {
+        self->start_key = args[1];
+        if (n_args > 2) {
+            self->end_key = args[2];
+            if (n_args > 3) {
+                self->next_flags = type | MP_OBJ_SMALL_INT_VALUE(args[3]);
+            }
+        }
+    }
+    return args[0];
+}
+
+STATIC mp_obj_t btree_keys(size_t n_args, const mp_obj_t *args) {
+    return btree_init_iter(n_args, args, FLAG_ITER_KEYS);
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(btree_keys_obj, 1, 4, btree_keys);
+
+STATIC mp_obj_t btree_values(size_t n_args, const mp_obj_t *args) {
+    return btree_init_iter(n_args, args, FLAG_ITER_VALUES);
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(btree_values_obj, 1, 4, btree_values);
+
+STATIC mp_obj_t btree_items(size_t n_args, const mp_obj_t *args) {
+    return btree_init_iter(n_args, args, FLAG_ITER_ITEMS);
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(btree_items_obj, 1, 4, btree_items);
+
+STATIC mp_obj_t btree_getiter(mp_obj_t self_in) {
+    mp_obj_btree_t *self = MP_OBJ_TO_PTR(self_in);
+    if (self->next_flags != 0) {
+        // If we're called immediately after keys(), values(), or items(),
+        // use their setup for iteration.
+        self->flags = self->next_flags;
+        self->next_flags = 0;
+    } else {
+        // Otherwise, iterate over all keys.
+        self->flags = FLAG_ITER_KEYS;
+        self->start_key = mp_const_none;
+        self->end_key = mp_const_none;
+    }
+
+    return self_in;
+}
+
+STATIC mp_obj_t btree_iternext(mp_obj_t self_in) {
+    mp_obj_btree_t *self = MP_OBJ_TO_PTR(self_in);
+    DBT key, val;
+    int res;
+    // Different ports may have different type sizes
+    mp_uint_t v;
+    bool desc = self->flags & FLAG_DESC;
+    if (self->start_key != MP_OBJ_NULL) {
+        int flags = R_FIRST;
+        if (self->start_key != mp_const_none) {
+            key.data = (void*)mp_obj_str_get_data(self->start_key, &v);
+            key.size = v;
+            flags = R_CURSOR;
+        } else if (desc) {
+            flags = R_LAST;
+        }
+        res = __bt_seq(self->db, &key, &val, flags);
+        self->start_key = MP_OBJ_NULL;
+    } else {
+        res = __bt_seq(self->db, &key, &val, desc ? R_PREV : R_NEXT);
+    }
+
+    if (res == RET_SPECIAL) {
+        return MP_OBJ_STOP_ITERATION;
+    }
+    CHECK_ERROR(res);
+
+    if (self->end_key != mp_const_none) {
+        DBT end_key;
+        end_key.data = (void*)mp_obj_str_get_data(self->end_key, &v);
+        end_key.size = v;
+        BTREE *t = self->db->internal;
+        int cmp = t->bt_cmp(&key, &end_key);
+        if (desc) {
+            cmp = -cmp;
+        }
+        if (self->flags & FLAG_END_KEY_INCL) {
+            cmp--;
+        }
+        if (cmp >= 0) {
+            self->end_key = MP_OBJ_NULL;
+            return MP_OBJ_STOP_ITERATION;
+        }
+    }
+
+    switch (self->flags & FLAG_ITER_TYPE_MASK) {
+        case FLAG_ITER_KEYS:
+            return mp_obj_new_bytes(key.data, key.size);
+        case FLAG_ITER_VALUES:
+            return mp_obj_new_bytes(val.data, val.size);
+        default: {
+            mp_obj_t pair_o = mp_obj_new_tuple(2, NULL);
+            mp_obj_tuple_t *pair = MP_OBJ_TO_PTR(pair_o);
+            pair->items[0] = mp_obj_new_bytes(key.data, key.size);
+            pair->items[1] = mp_obj_new_bytes(val.data, val.size);
+            return pair_o;
+        }
+    }
+}
+
+STATIC mp_obj_t btree_subscr(mp_obj_t self_in, mp_obj_t index, mp_obj_t value) {
+    mp_obj_btree_t *self = MP_OBJ_TO_PTR(self_in);
+    // Different ports may have different type sizes
+    mp_uint_t v;
+    if (value == MP_OBJ_NULL) {
+        // delete
+        DBT key;
+        key.data = (void*)mp_obj_str_get_data(index, &v);
+        key.size = v;
+        int res = __bt_delete(self->db, &key, 0);
+        if (res == RET_SPECIAL) {
+            nlr_raise(mp_obj_new_exception(&mp_type_KeyError));
+        }
+        CHECK_ERROR(res);
+        return mp_const_none;
+    } else if (value == MP_OBJ_SENTINEL) {
+        // load
+        DBT key, val;
+        key.data = (void*)mp_obj_str_get_data(index, &v);
+        key.size = v;
+        int res = __bt_get(self->db, &key, &val, 0);
+        if (res == RET_SPECIAL) {
+            nlr_raise(mp_obj_new_exception(&mp_type_KeyError));
+        }
+        CHECK_ERROR(res);
+        return mp_obj_new_bytes(val.data, val.size);
+    } else {
+        // store
+        DBT key, val;
+        key.data = (void*)mp_obj_str_get_data(index, &v);
+        key.size = v;
+        val.data = (void*)mp_obj_str_get_data(value, &v);
+        val.size = v;
+        int res = __bt_put(self->db, &key, &val, 0);
+        CHECK_ERROR(res);
+        return mp_const_none;
+    }
+}
+
+STATIC const mp_rom_map_elem_t btree_locals_dict_table[] = {
+    { MP_ROM_QSTR(MP_QSTR_close), MP_ROM_PTR(&btree_close_obj) },
+    { MP_ROM_QSTR(MP_QSTR_get), MP_ROM_PTR(&btree_get_obj) },
+    { MP_ROM_QSTR(MP_QSTR_put), MP_ROM_PTR(&btree_put_obj) },
+    { MP_ROM_QSTR(MP_QSTR_seq), MP_ROM_PTR(&btree_seq_obj) },
+    { MP_ROM_QSTR(MP_QSTR_keys), MP_ROM_PTR(&btree_keys_obj) },
+    { MP_ROM_QSTR(MP_QSTR_values), MP_ROM_PTR(&btree_values_obj) },
+    { MP_ROM_QSTR(MP_QSTR_items), MP_ROM_PTR(&btree_items_obj) },
+};
+
+STATIC MP_DEFINE_CONST_DICT(btree_locals_dict, btree_locals_dict_table);
+
+STATIC const mp_obj_type_t btree_type = {
+    { &mp_type_type },
+    // Save on qstr's, reuse same as for module
+    .name = MP_QSTR_btree,
+    .print = btree_print,
+    .getiter = btree_getiter,
+    .iternext = btree_iternext,
+    .subscr = btree_subscr,
+    .locals_dict = (void*)&btree_locals_dict,
+};
+
+STATIC mp_obj_t mod_btree_open(size_t n_args, const mp_obj_t *pos_args, mp_map_t *kw_args) {
+    static const mp_arg_t allowed_args[] = {
+        { MP_QSTR_server_side, MP_ARG_KW_ONLY | MP_ARG_BOOL, {.u_bool = false} },
+    };
+
+    const char *fname = NULL;
+    if (pos_args[0] != mp_const_none) {
+        fname = mp_obj_str_get_str(pos_args[0]);
+    }
+
+    struct {
+        mp_arg_val_t server_side;
+    } args;
+    mp_arg_parse_all(n_args - 1, pos_args + 1, kw_args,
+        MP_ARRAY_SIZE(allowed_args), allowed_args, (mp_arg_val_t*)&args);
+
+    DB *db = __bt_open(fname, /*flags*/O_CREAT | O_RDWR, /*mode*/0770, /*openinfo*/NULL, /*dflags*/0);
+    return MP_OBJ_FROM_PTR(btree_new(db));
+}
+STATIC MP_DEFINE_CONST_FUN_OBJ_KW(mod_btree_open_obj, 1, mod_btree_open);
+
+STATIC const mp_rom_map_elem_t mp_module_btree_globals_table[] = {
+    { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_btree) },
+    { MP_ROM_QSTR(MP_QSTR_open), MP_ROM_PTR(&mod_btree_open_obj) },
+    { MP_ROM_QSTR(MP_QSTR_INCL), MP_ROM_INT(FLAG_END_KEY_INCL) },
+    { MP_ROM_QSTR(MP_QSTR_DESC), MP_ROM_INT(FLAG_DESC) },
+};
+
+STATIC MP_DEFINE_CONST_DICT(mp_module_btree_globals, mp_module_btree_globals_table);
+
+const mp_obj_module_t mp_module_btree = {
+    .base = { &mp_type_module },
+    .name = MP_QSTR_btree,
+    .globals = (mp_obj_dict_t*)&mp_module_btree_globals,
+};
+
+#endif // MICROPY_PY_BTREE
diff --git a/extmod/modlwip.c b/extmod/modlwip.c
index cb7cc6596f..80df662647 100644
--- a/extmod/modlwip.c
+++ b/extmod/modlwip.c
@@ -239,7 +239,7 @@ typedef struct _lwip_socket_obj_t {
     byte peer[4];
     mp_uint_t peer_port;
     mp_uint_t timeout;
-    uint16_t leftover_count;
+    uint16_t recv_offset;
 
     uint8_t domain;
     uint8_t type;
@@ -354,11 +354,17 @@ STATIC err_t _lwip_tcp_recv(void *arg, struct tcp_pcb *tcpb, struct pbuf *p, err
         socket->state = STATE_PEER_CLOSED;
         exec_user_callback(socket);
         return ERR_OK;
-    } else if (socket->incoming.pbuf != NULL) {
-        // No room in the inn, let LWIP know it's still responsible for delivery later
+    }
+
+    if (socket->incoming.pbuf == NULL) {
+        socket->incoming.pbuf = p;
+    } else {
+        #ifdef SOCKET_SINGLE_PBUF
         return ERR_BUF;
+        #else
+        pbuf_cat(socket->incoming.pbuf, p);
+        #endif
     }
-    socket->incoming.pbuf = p;
 
     exec_user_callback(socket);
 
@@ -536,22 +542,28 @@ STATIC mp_uint_t lwip_tcp_receive(lwip_socket_obj_t *socket, byte *buf, mp_uint_
 
     struct pbuf *p = socket->incoming.pbuf;
 
-    if (socket->leftover_count == 0) {
-        socket->leftover_count = p->tot_len;
+    mp_uint_t remaining = p->len - socket->recv_offset;
+    if (len > remaining) {
+        len = remaining;
     }
 
-    u16_t result = pbuf_copy_partial(p, buf, ((socket->leftover_count >= len) ? len : socket->leftover_count), (p->tot_len - socket->leftover_count));
-    if (socket->leftover_count > len) {
-        // More left over...
-        socket->leftover_count -= len;
-    } else {
+    memcpy(buf, (byte*)p->payload + socket->recv_offset, len);
+
+    remaining -= len;
+    if (remaining == 0) {
+        socket->incoming.pbuf = p->next;
+        // If we don't ref here, free() will free the entire chain,
+        // if we ref, it does what we need: frees 1st buf, and decrements
+        // next buf's refcount back to 1.
+        pbuf_ref(p->next);
         pbuf_free(p);
-        socket->incoming.pbuf = NULL;
-        socket->leftover_count = 0;
+        socket->recv_offset = 0;
+    } else {
+        socket->recv_offset += len;
     }
+    tcp_recved(socket->pcb.tcp, len);
 
-    tcp_recved(socket->pcb.tcp, result);
-    return (mp_uint_t) result;
+    return len;
 }
 
 /*******************************************************************************/
@@ -561,8 +573,8 @@ STATIC const mp_obj_type_t lwip_socket_type;
 
 STATIC void lwip_socket_print(const mp_print_t *print, mp_obj_t self_in, mp_print_kind_t kind) {
     lwip_socket_obj_t *self = self_in;
-    mp_printf(print, "<socket state=%d timeout=%d incoming=%p remaining=%d>", self->state, self->timeout,
-        self->incoming.pbuf, self->leftover_count);
+    mp_printf(print, "<socket state=%d timeout=%d incoming=%p off=%d>", self->state, self->timeout,
+        self->incoming.pbuf, self->recv_offset);
 }
 
 // FIXME: Only supports two arguments at present
@@ -612,7 +624,7 @@ STATIC mp_obj_t lwip_socket_make_new(const mp_obj_type_t *type, mp_uint_t n_args
     socket->incoming.pbuf = NULL;
     socket->timeout = -1;
     socket->state = STATE_NEW;
-    socket->leftover_count = 0;
+    socket->recv_offset = 0;
     return socket;
 }
 
@@ -749,7 +761,7 @@ STATIC mp_obj_t lwip_socket_accept(mp_obj_t self_in) {
     socket2->incoming.pbuf = NULL;
     socket2->timeout = socket->timeout;
     socket2->state = STATE_CONNECTED;
-    socket2->leftover_count = 0;
+    socket2->recv_offset = 0;
     socket2->callback = MP_OBJ_NULL;
     tcp_arg(socket2->pcb.tcp, (void*)socket2);
     tcp_err(socket2->pcb.tcp, _lwip_tcp_error);
@@ -1148,7 +1160,7 @@ STATIC const mp_obj_type_t lwip_socket_type = {
     .name = MP_QSTR_socket,
     .print = lwip_socket_print,
     .make_new = lwip_socket_make_new,
-    .stream_p = &lwip_socket_stream_p,
+    .protocol = &lwip_socket_stream_p,
     .locals_dict = (mp_obj_t)&lwip_socket_locals_dict,
 };
 
diff --git a/extmod/moduos_dupterm.c b/extmod/moduos_dupterm.c
index 4c9f9e4940..d888099dff 100644
--- a/extmod/moduos_dupterm.c
+++ b/extmod/moduos_dupterm.c
@@ -31,6 +31,7 @@
 #include "py/nlr.h"
 #include "py/runtime.h"
 #include "py/objtuple.h"
+#include "py/objarray.h"
 #include "py/stream.h"
 
 #if MICROPY_PY_OS_DUPTERM
@@ -51,8 +52,16 @@ void mp_uos_dupterm_tx_strn(const char *str, size_t len) {
         if (nlr_push(&nlr) == 0) {
             mp_obj_t write_m[3];
             mp_load_method(MP_STATE_PORT(term_obj), MP_QSTR_write, write_m);
-            write_m[2] = mp_obj_new_bytearray_by_ref(len, (char*)str);
+
+            mp_obj_array_t *arr = MP_OBJ_TO_PTR(MP_STATE_PORT(dupterm_arr_obj));
+            void *org_items = arr->items;
+            arr->items = (void*)str;
+            arr->len = len;
+            write_m[2] = MP_STATE_PORT(dupterm_arr_obj);
             mp_call_method_n_kw(1, 0, write_m);
+            arr = MP_OBJ_TO_PTR(MP_STATE_PORT(dupterm_arr_obj));
+            arr->items = org_items;
+            arr->len = 1;
             nlr_pop();
         } else {
             mp_uos_deactivate("dupterm: Exception in write() method, deactivating: ", nlr.ret_val);
@@ -69,9 +78,12 @@ STATIC mp_obj_t mp_uos_dupterm(mp_uint_t n_args, const mp_obj_t *args) {
         }
     } else {
         if (args[0] == mp_const_none) {
-            MP_STATE_PORT(term_obj) = NULL;
+            MP_STATE_PORT(term_obj) = MP_OBJ_NULL;
         } else {
             MP_STATE_PORT(term_obj) = args[0];
+            if (MP_STATE_PORT(dupterm_arr_obj) == MP_OBJ_NULL) {
+                MP_STATE_PORT(dupterm_arr_obj) = mp_obj_new_bytearray(1, "");
+            }
         }
         return mp_const_none;
     }
diff --git a/extmod/modussl.c b/extmod/modussl.c
index 567033cf48..4a3e297e1e 100644
--- a/extmod/modussl.c
+++ b/extmod/modussl.c
@@ -156,7 +156,7 @@ STATIC const mp_obj_type_t ussl_socket_type = {
     .print = socket_print,
     .getiter = NULL,
     .iternext = NULL,
-    .stream_p = &ussl_socket_stream_p,
+    .protocol = &ussl_socket_stream_p,
     .locals_dict = (void*)&ussl_socket_locals_dict,
 };
 
@@ -202,7 +202,8 @@ int mp_stream_errno;
 
 ssize_t mp_stream_posix_write(void *sock_obj, const void *buf, size_t len) {
     struct _mp_obj_base_t *o = (struct _mp_obj_base_t *)sock_obj;
-    mp_uint_t out_sz = o->type->stream_p->write(o, buf, len, &mp_stream_errno);
+    const mp_stream_p_t *stream_p = o->type->protocol;
+    mp_uint_t out_sz = stream_p->write(o, buf, len, &mp_stream_errno);
     if (out_sz == MP_STREAM_ERROR) {
         return -1;
     } else {
@@ -212,7 +213,8 @@ ssize_t mp_stream_posix_write(void *sock_obj, const void *buf, size_t len) {
 
 ssize_t mp_stream_posix_read(void *sock_obj, void *buf, size_t len) {
     struct _mp_obj_base_t *o = (struct _mp_obj_base_t *)sock_obj;
-    mp_uint_t out_sz = o->type->stream_p->read(o, buf, len, &mp_stream_errno);
+    const mp_stream_p_t *stream_p = o->type->protocol;
+    mp_uint_t out_sz = stream_p->read(o, buf, len, &mp_stream_errno);
     if (out_sz == MP_STREAM_ERROR) {
         return -1;
     } else {
diff --git a/extmod/modwebrepl.c b/extmod/modwebrepl.c
index 8ecab922aa..4f7b924e07 100644
--- a/extmod/modwebrepl.c
+++ b/extmod/modwebrepl.c
@@ -303,6 +303,7 @@ STATIC MP_DEFINE_CONST_FUN_OBJ_1(webrepl_set_password_obj, webrepl_set_password)
 
 STATIC const mp_map_elem_t webrepl_locals_dict_table[] = {
     { MP_OBJ_NEW_QSTR(MP_QSTR_read), (mp_obj_t)&mp_stream_read_obj },
+    { MP_OBJ_NEW_QSTR(MP_QSTR_readinto), (mp_obj_t)&mp_stream_readinto_obj },
     { MP_OBJ_NEW_QSTR(MP_QSTR_write), (mp_obj_t)&mp_stream_write_obj },
     { MP_OBJ_NEW_QSTR(MP_QSTR_close), (mp_obj_t)&webrepl_close_obj },
 };
@@ -317,7 +318,7 @@ STATIC const mp_obj_type_t webrepl_type = {
     { &mp_type_type },
     .name = MP_QSTR__webrepl,
     .make_new = webrepl_make_new,
-    .stream_p = &webrepl_stream_p,
+    .protocol = &webrepl_stream_p,
     .locals_dict = (mp_obj_t)&webrepl_locals_dict,
 };
 
diff --git a/extmod/modwebsocket.c b/extmod/modwebsocket.c
index fc5e29a05e..6354633af4 100644
--- a/extmod/modwebsocket.c
+++ b/extmod/modwebsocket.c
@@ -282,6 +282,7 @@ STATIC MP_DEFINE_CONST_FUN_OBJ_1(websocket_close_obj, websocket_close);
 
 STATIC const mp_map_elem_t websocket_locals_dict_table[] = {
     { MP_OBJ_NEW_QSTR(MP_QSTR_read), (mp_obj_t)&mp_stream_read_obj },
+    { MP_OBJ_NEW_QSTR(MP_QSTR_readinto), (mp_obj_t)&mp_stream_readinto_obj },
     { MP_OBJ_NEW_QSTR(MP_QSTR_write), (mp_obj_t)&mp_stream_write_obj },
     { MP_OBJ_NEW_QSTR(MP_QSTR_ioctl), (mp_obj_t)&mp_stream_ioctl_obj },
     { MP_OBJ_NEW_QSTR(MP_QSTR_close), (mp_obj_t)&websocket_close_obj },
@@ -298,7 +299,7 @@ STATIC const mp_obj_type_t websocket_type = {
     { &mp_type_type },
     .name = MP_QSTR_websocket,
     .make_new = websocket_make_new,
-    .stream_p = &websocket_stream_p,
+    .protocol = &websocket_stream_p,
     .locals_dict = (mp_obj_t)&websocket_locals_dict,
 };
 
diff --git a/extmod/vfs_fat.c b/extmod/vfs_fat.c
index a4a81370f9..df5cbb0d62 100644
--- a/extmod/vfs_fat.c
+++ b/extmod/vfs_fat.c
@@ -186,7 +186,7 @@ STATIC mp_obj_t fat_vfs_stat(mp_obj_t vfs_in, mp_obj_t path_in) {
     if (path_equal(path, "/")) {
         // stat root directory
         fno.fsize = 0;
-        fno.fdate = 0;
+        fno.fdate = 0x2821; // Jan 1, 2000
         fno.ftime = 0;
         fno.fattrib = AM_DIR;
     } else {
@@ -196,7 +196,7 @@ STATIC mp_obj_t fat_vfs_stat(mp_obj_t vfs_in, mp_obj_t path_in) {
             if (vfs != NULL && path_equal(path, vfs->str)) {
                 // stat mounted device directory
                 fno.fsize = 0;
-                fno.fdate = 0;
+                fno.fdate = 0x2821; // Jan 1, 2000
                 fno.ftime = 0;
                 fno.fattrib = AM_DIR;
                 res = FR_OK;
diff --git a/extmod/vfs_fat_file.c b/extmod/vfs_fat_file.c
index 857cd1f470..0cd61e4605 100644
--- a/extmod/vfs_fat_file.c
+++ b/extmod/vfs_fat_file.c
@@ -259,7 +259,7 @@ const mp_obj_type_t mp_type_fileio = {
     .make_new = file_obj_make_new,
     .getiter = mp_identity,
     .iternext = mp_stream_unbuffered_iter,
-    .stream_p = &fileio_stream_p,
+    .protocol = &fileio_stream_p,
     .locals_dict = (mp_obj_dict_t*)&rawfile_locals_dict,
 };
 #endif
@@ -278,7 +278,7 @@ const mp_obj_type_t mp_type_textio = {
     .make_new = file_obj_make_new,
     .getiter = mp_identity,
     .iternext = mp_stream_unbuffered_iter,
-    .stream_p = &textio_stream_p,
+    .protocol = &textio_stream_p,
     .locals_dict = (mp_obj_dict_t*)&rawfile_locals_dict,
 };
 
diff --git a/extmod/virtpin.c b/extmod/virtpin.c
index a5817ab4d0..dbfa21d669 100644
--- a/extmod/virtpin.c
+++ b/extmod/virtpin.c
@@ -28,12 +28,12 @@
 
 int mp_virtual_pin_read(mp_obj_t pin) {
     mp_obj_base_t* s = (mp_obj_base_t*)MP_OBJ_TO_PTR(pin);
-    mp_pin_p_t *pin_p = (mp_pin_p_t*)s->type->stream_p;
+    mp_pin_p_t *pin_p = (mp_pin_p_t*)s->type->protocol;
     return pin_p->ioctl(pin, MP_PIN_READ, 0, NULL);
 }
 
 void mp_virtual_pin_write(mp_obj_t pin, int value) {
     mp_obj_base_t* s = (mp_obj_base_t*)MP_OBJ_TO_PTR(pin);
-    mp_pin_p_t *pin_p = (mp_pin_p_t*)s->type->stream_p;
+    mp_pin_p_t *pin_p = (mp_pin_p_t*)s->type->protocol;
     pin_p->ioctl(pin, MP_PIN_WRITE, value, NULL);
 }