Merge tag 'v1.8.2' into parse-bytecode

Thread support, ESP8266 memory improvements, btree module, improved docs

This release brings multi-threading support in the form of the _thread
module, which closely matches the semantics of the corresponding CPython
module.  There is support for GIL and non-GIL builds; without the GIL
enabled one must protect concurrent access to mutable Python state at the
Python level using Lock objects.  Threading with the GIL is enabled in
the cc3200 port on the WiPy.  Threading without the GIL is enabled on the
unix port.  The ESP8266 port has support for frozen bytecode (for scripts
in the modules/ subdirectory), as well as optimisations for reduced
memory usage and decreased memory fragmentation.  The ESP8266 RTC also
resumes correctly after a deepsleep.  An initial "btree" module has been
implemented for database support (unix port only), and the documentation
has been further improved, with pre-built PDF versions of the docs now
available.

py core:
- parse: treat constants that start with underscore as private
- objdict: implemented OrderedDict equality check
- support to build berkeley db 1.85 and "btree" module
- mpconfig.h: MP_NOINLINE is universally useful, move from unix port
- makeqstrdefs.py: remove restriction that source path can't be absolute
- mkrules.mk: define "lib" outside conditional block
- rename __QSTR_EXTRACT flag to NO_QSTR
- objtype: instance: inherit protocol vtable from a base class
- mphal.h: if virtpin API is used, automagically include its header
- objtype: inherit protocol vtable from base class only if it exists
- add MP_STATE_THREAD to hold state specific to a given thread
- add basic _thread module, with ability to start a new thread
- modthread: properly cast concrete exception pointer to an object
- modthread: add stack_size() function
- modthread: add exit() function
- modthread: implement lock object, for creating a mutex
- modthread: add with-context capabilities to lock object
- gc: make memory manager and garbage collector thread safe
- modthread: satisfy unused-args warning
- gc: fix 2 cases of concurrent access to ATB and FTB
- modthread: be more careful with root pointers when creating a thread
- modthread: call mp_thread_start/mp_thread_finish around threads
- gc: fix GC+thread bug where ptr gets lost because it's not computed
- make interning of qstrs thread safe
- implement a simple global interpreter lock (GIL)
- don't use gc or qstr mutex when the GIL is enabled, there is no need
- modthread: make Lock objects work when GIL is enabled
- mpthread.h: move GIL macros outside MICROPY_PY_THREAD block
- modthread: allow to properly set the stack limit of a thread
- nlrthumb: convert NLR thumb funcs from asm to C with inline-asm
- nlrsetjmp: update to take into account new location of nlr_top
- mpthread: include mpstate.h when defining GIL macros
- gc: be sure to count last allocated block at heap end in stats
- gc: calculate (and report) maximum contiguous free block size
- builtinimport: disable "imported as namespace package" warning
- mpconfig.h: mention MICROPY_PY_BTREE config option
- objarray: split out header to allow direct access to object
- rename mp_obj_type_t::stream_p to "protocol"

extmod:
- modbtree: initial implementation of "btree" module based on BerkeleyDB
- modbtree: handle default value and error check
- modbtree: implement .items() iterator
- modbtree: actually implement end key support for .items()
- modbtree: items(): Implement "end key inclusive" flag
- modbtree: items(): Implement DESC flag
- modbtree: __getitem__() should raise KeyError for non-existing key
- modbtree: open(): Support "in-memory" database with filename=None
- machine_pinbase: implementation of PinBase class
- machine_pinbase: fix nanbox build
- modlwip: store a chain of incoming pbufs, instead of only one
- modbtree: implement keys(), values(), items() iterators
- modbtree: cleverly implement "for key in btree:" syntax
- modwebsocket: add readinto() method
- modwebrepl: add readinto() method
- modbtree: fix unused argument warning
- modbtree: fixes for nanbox build
- moduos_dupterm: reserve buffer bytearray object for dupterm
- moduos_dupterm: reuse dupterm_arr_obj for write operations

lib:
- berkeley-db-1.xx: add Berkeley DB 1.85 as a submodule
- berkeley-db-1.xx: update to upstream which builds for uPy
- fatfs/option/ccsbcs: follow uPy optional features model
- libm: format code to pass gcc v6.1.1 warning
- libm: remove unused definition of "one"

drivers:
- display/ssd1306: add width arg and support 64px wide displays

tests:
- recursive_iternext: clang/Linux is even more stack-frugal than MacOS
- bench: add testcase to compare bytes(N) vs b"\0" * N
- add testcase for OrderedDict equality
- add a testcase for machine.PinBase class
- extmod: add "btree" module test
- btree1: add testcase for iterating btree object directly
- add tests for _thread module
- add 3 more tests for _thread module
- thread: remove need to sleep to wait for completion in some tests
- thread: add tests for running GC within a thread, and heap stress
- thread: rename thread_stress_XXX.py to stress_XXX.py
- thread: add tests that mutate shared objects
- thread: add test for concurrent interning of strings
- thread: add test for concurrent mutating of user instance
- thread: make stack-size test run correctly and reliable on uPy
- thread: make sure that thread tests don't rely on floating point
- thread: allow thread_sleep1 to run without floating point
- thread: allow some tests to run on ports with not much heap
- meminfo: update for addition of "max free sz" output
- run-tests: If running thread tests on unix, don't run mutate ones

unix port:
- unbreak "minimal" target by disabling FatFs
- mpconfigport_minimal.h: clearly mark where user-define config ends
- deprecate support for GNU Readline (MICROPY_USE_READLINE=2)
- time to build with --gc-sections
- move "utime" module config to C level instead of make level
- Makefile: libffi: Build with -Os
- alloc: add option to use uPy's alloc-exec implementation even for libffi
- alloc: make coverage build and its overzealous warnings happy
- disable FatFs VFS for normal build, keep enabled for coverage
- modmachine: include PinBase class
- modmachine: enable time_pulse_us() function
- fix Makefile to handle gc-sections linker flags on Mac OS
- add basic thread support using pthreads
- file: if write syscall returns because of EINTR then try again
- implement garbage collection with threading
- mpthreadport: use SA_SIGINFO for GC signal handler
- gccollect: provide declaration of exported function
- mpthreadport: suppress compiler warning about unused arguments
- modtime: release the GIL when sleeping
- enable btree module
- Makefile: make "minimal" build be minimal again
- mpconfigport_minimal.h: allow to print a string within 1KB of heap
- main: error out on unknown value of suffix in -X heapsize= option
- main: improve help for -X options a bit
- main: when preparing sys.path, allocate exact strings on uPy heap
- disable the GIL to improve performance of non-thread code

windows port:
- follow unix port changes regarding "utime" module
- msvc: include machine_pinbase.c in build and enable umachine module

stmhal port:
- add board definition files for Olimex STM32-E407
- port of f4 hal commit 1d7fb82 to l4 hal (SD card modifications)
- correct DMA to allow SD card on L4 MCUs
- enable SD card on L4 MCUs

cc3200 port:
- update FreeRTOS to v9.0.0
- define our own FreeRTOS heap so it can go in a special segment
- add basic threading capabilities
- fix call to new exception to be _msg instead of _arg1
- use xTaskCreateStatic instead of osi_TaskCreate
- mpthreadport: make mutex statically allocated
- mpthreadport: properly initialise the main thread's data
- gccollect: use MP_STATE_THREAD(stack_top) to get top of stack
- mpthreadport: scan more root pointers from thread data
- mpthreadport: move mem alloc outside the thread_mutex lock
- enable the GIL
- in FreeRTOSConfig.h, comment on configSUPPORT_STATIC_ALLOCATION
- remove .boot section attribute from osi_TaskCreate function
- start the simplelink spawn task using the static task creator
- shrink the FreeRTOS heap and place TCB+stack in freed location
- allow to compile bootloader with threading enabled
- remove obsolete singleton heart-beat object

teensy port:
- update the README with OSX specific information and tips

esp8266 port:
- modnetwork: use struct bss_info::ssid_len for ESSID length
- let RTC work correctly after deepsleep
- switch floating-point arith routines to BootROM
- Makefile: enable --verify option for esptool.py write_flash
- use RTC to set date & time stamps for files
- main.c: clear the command line history when (re)booting
- enable frozen bytecode, with scripts in modules/ subdir
- README: describe how to build mpy-cross
- websocket_helper.py: avoid extra string allocations
- websocket_helper.py: fix typo in debug output
- modpybuart: allow setting baudrate and other params
- explicitly collect garbage in bootstrap scripts
- switch webrepl_setup to use frozen bytecode
- switch webrepl to use frozen bytecode
- README: promote from "highly experimental" to "experimental"
- main: init recently added dupterm_arr_obj port state var
- esp_mphal: call_dupterm_read: Use readinto() method
- esp_mphal: properly handle dupterm EOF after switching to readinto()

qemu-arm port:
- disable gcc LTO option for nlrthumb.c

docs:
- esp8266/tutorial/network_basics: minor typo fix, ap becomes ap_if
- add link to PDF version of docs in sidebar
- library/network: wipy: fix literal block indentation
- esp.socket: remove unused file
- ure: typo fix
- usocket: there must be empty line after .only::
- library/machine.Pin: mention GPIO and cross-link .value() function
- machine.*: add "currentmodule:: machine" directive
- pyb.*: add "currentmodule:: pyb" directive
- machine.Pin: remove explicit target
- machine.UART: cleanup pyboard section
- machine.*: use proper class case in method headers
- machine.Pin: add class designator to all constants
- pyb.CAN: mark CAN.initfilterbanks() as classmethod explicitly
- pyb.Pin: af_list() is a normal method, not a class method
- pyb.ExtInt,pyb.Pin: mark up class methods as such
- pyb.*: use proper class case in method headers
- pyb.Pin: sort .af() and .af_list() methods together
- builtins: enumerate all builtin functions implemented
- library/index: add builtins.rst
- machine.Pin: disambiguate object call method
- machine*: remove explicit targets and "machine." prefixes on classes
- uctypes: improve documentation
- sys: detailed description of print_exception() diff from traceback module
- add sphinx_selective_exclude extension suite
- conf.py: active sphinx_selective_exclude extensions
- rebuild docs from scratch, as required for proper only:: handling
- select: add an article
- sys: print_exception: Fixes/clarifications
- conf.py: exclude cmath from modindex for wipy
- library: fix typo in docs for usocket.listen()

examples:
- unix/ffi_example.py: update for current "ffi" module API
- unix/ffi_example.py: remove TODO, make output more clear

14 files changed, 410 insertions, 54 deletions

diff --git a/unix/Makefile b/unix/Makefile
index 90653e88e8..2e0cbfd35a 100644
--- a/unix/Makefile
+++ b/unix/Makefile
@@ -28,7 +28,7 @@ ifdef DEBUG
 CFLAGS += -g
 COPT = -O0
 else
-COPT = -Os #-DNDEBUG
+COPT = -Os -fdata-sections -ffunction-sections #-DNDEBUG
 # _FORTIFY_SOURCE is a feature in gcc/glibc which is intended to provide extra
 # security for detecting buffer overflows. Some distros (Ubuntu at the very least)
 # have it enabled by default.
@@ -63,10 +63,10 @@ endif
 ifeq ($(UNAME_S),Darwin)
 CC = clang
 # Use clang syntax for map file
-LDFLAGS_ARCH = -Wl,-map,$@.map
+LDFLAGS_ARCH = -Wl,-map,$@.map -Wl,-dead_strip
 else
 # Use gcc syntax for map file
-LDFLAGS_ARCH = -Wl,-Map=$@.map,--cref
+LDFLAGS_ARCH = -Wl,-Map=$@.map,--cref -Wl,--gc-sections
 endif
 LDFLAGS = $(LDFLAGS_MOD) $(LDFLAGS_ARCH) -lm $(LDFLAGS_EXTRA)
 
@@ -85,16 +85,6 @@ INC +=  -I../lib/mp-readline
 CFLAGS_MOD += -DMICROPY_USE_READLINE=1
 LIB_SRC_C_EXTRA += mp-readline/readline.c
 endif
-ifeq ($(MICROPY_USE_READLINE),2)
-CFLAGS_MOD += -DMICROPY_USE_READLINE=2
-LDFLAGS_MOD += -lreadline
-# the following is needed for BSD
-#LDFLAGS_MOD += -ltermcap
-endif
-ifeq ($(MICROPY_PY_TIME),1)
-CFLAGS_MOD += -DMICROPY_PY_TIME=1
-SRC_MOD += modtime.c
-endif
 ifeq ($(MICROPY_PY_TERMIOS),1)
 CFLAGS_MOD += -DMICROPY_PY_TERMIOS=1
 SRC_MOD += modtermios.c
@@ -103,6 +93,10 @@ ifeq ($(MICROPY_PY_SOCKET),1)
 CFLAGS_MOD += -DMICROPY_PY_SOCKET=1
 SRC_MOD += modsocket.c
 endif
+ifeq ($(MICROPY_PY_THREAD),1)
+CFLAGS_MOD += -DMICROPY_PY_THREAD=1 -DMICROPY_PY_THREAD_GIL=0
+LDFLAGS_MOD += -lpthread
+endif
 
 ifeq ($(MICROPY_PY_FFI),1)
 
@@ -138,10 +132,12 @@ SRC_C = \
 	main.c \
 	gccollect.c \
 	unix_mphal.c \
+	mpthreadport.c \
 	input.c \
 	file.c \
 	modmachine.c \
 	modos.c \
+	modtime.c \
 	moduselect.c \
 	alloc.c \
 	coverage.c \
@@ -162,9 +158,13 @@ endif
 LIB_SRC_C = $(addprefix lib/,\
 	$(LIB_SRC_C_EXTRA) \
 	utils/printf.c \
+	timeutils/timeutils.c \
+	)
+
+# FatFS VFS support
+LIB_SRC_C += $(addprefix lib/,\
 	fatfs/ff.c \
 	fatfs/option/ccsbcs.c \
-	timeutils/timeutils.c \
 	)
 
 OBJ = $(PY_O)
@@ -235,7 +235,11 @@ fast:
 
 # build a minimal interpreter
 minimal:
-	$(MAKE) COPT="-Os -DNDEBUG" CFLAGS_EXTRA='-DMP_CONFIGFILE="<mpconfigport_minimal.h>"' BUILD=build-minimal PROG=micropython_minimal MICROPY_PY_TIME=0 MICROPY_PY_TERMIOS=0 MICROPY_PY_SOCKET=0 MICROPY_PY_FFI=0 MICROPY_USE_READLINE=0
+	$(MAKE) COPT="-Os -DNDEBUG" CFLAGS_EXTRA='-DMP_CONFIGFILE="<mpconfigport_minimal.h>"' \
+	    BUILD=build-minimal PROG=micropython_minimal \
+	    MICROPY_PY_BTREE=0 MICROPY_PY_FFI=0 MICROPY_PY_SOCKET=0 MICROPY_PY_THREAD=0 \
+	    MICROPY_PY_TERMIOS=0 MICROPY_PY_USSL=0 \
+	    MICROPY_USE_READLINE=0 MICROPY_FATFS=0
 
 # build interpreter with nan-boxing as object model
 nanbox:
@@ -260,7 +264,7 @@ freedos:
 
 # build an interpreter for coverage testing and do the testing
 coverage:
-	$(MAKE) COPT="-O0" CFLAGS_EXTRA='-DMP_CONFIGFILE="<mpconfigport_coverage.h>" -fprofile-arcs -ftest-coverage -Wdouble-promotion -Wformat -Wmissing-declarations -Wmissing-prototypes -Wold-style-definition -Wpointer-arith -Wshadow -Wsign-compare -Wuninitialized -Wunused-parameter -DMICROPY_UNIX_COVERAGE' LDFLAGS_EXTRA='-fprofile-arcs -ftest-coverage' BUILD=build-coverage PROG=micropython_coverage
+	$(MAKE) COPT="-O0" MICROPY_PY_BTREE=0 CFLAGS_EXTRA='-DMP_CONFIGFILE="<mpconfigport_coverage.h>" -fprofile-arcs -ftest-coverage -Wdouble-promotion -Wformat -Wmissing-declarations -Wmissing-prototypes -Wold-style-definition -Wpointer-arith -Wshadow -Wsign-compare -Wuninitialized -Wunused-parameter -DMICROPY_UNIX_COVERAGE' LDFLAGS_EXTRA='-fprofile-arcs -ftest-coverage' BUILD=build-coverage PROG=micropython_coverage
 
 coverage_test: coverage
 	$(eval DIRNAME=$(notdir $(CURDIR)))
@@ -301,7 +305,7 @@ libffi:
 	cd ../lib/libffi; git clean -d -x -f
 	cd ../lib/libffi; ./autogen.sh
 	mkdir -p ../lib/libffi/build_dir; cd ../lib/libffi/build_dir; \
-	../configure $(CROSS_COMPILE_HOST) --prefix=$$PWD/out CC="$(CC)" CXX="$(CXX)" LD="$(LD)"; \
+	../configure $(CROSS_COMPILE_HOST) --prefix=$$PWD/out --disable-structs CC="$(CC)" CXX="$(CXX)" LD="$(LD)" CFLAGS="-Os -fomit-frame-pointer -fstrict-aliasing -ffast-math -fno-exceptions"; \
 	make install-exec-recursive; make -C include install-data-am
 
 axtls: ../lib/axtls/README
diff --git a/unix/alloc.c b/unix/alloc.c
index a0676a0aef..04c635ee97 100644
--- a/unix/alloc.c
+++ b/unix/alloc.c
@@ -33,7 +33,7 @@
 #include "py/mpstate.h"
 #include "py/gc.h"
 
-#if MICROPY_EMIT_NATIVE
+#if MICROPY_EMIT_NATIVE || (MICROPY_PY_FFI && MICROPY_FORCE_PLAT_ALLOC_EXEC)
 
 #if defined(__OpenBSD__) || defined(__MACH__)
 #define MAP_ANONYMOUS MAP_ANON
@@ -85,4 +85,23 @@ void mp_unix_mark_exec(void) {
     }
 }
 
-#endif // MICROPY_EMIT_NATIVE
+#if MICROPY_FORCE_PLAT_ALLOC_EXEC
+// Provide implementation of libffi ffi_closure_* functions in terms
+// of the functions above. On a normal Linux system, this save a lot
+// of code size.
+void *ffi_closure_alloc(size_t size, void **code);
+void ffi_closure_free(void *ptr);
+
+void *ffi_closure_alloc(size_t size, void **code) {
+    mp_uint_t dummy;
+    mp_unix_alloc_exec(size, code, &dummy);
+    return *code;
+}
+
+void ffi_closure_free(void *ptr) {
+    (void)ptr;
+    // TODO
+}
+#endif
+
+#endif // MICROPY_EMIT_NATIVE || (MICROPY_PY_FFI && MICROPY_FORCE_PLAT_ALLOC_EXEC)
diff --git a/unix/file.c b/unix/file.c
index 33acdccd06..a7620e079e 100644
--- a/unix/file.c
+++ b/unix/file.c
@@ -88,6 +88,14 @@ STATIC mp_uint_t fdfile_write(mp_obj_t o_in, const void *buf, mp_uint_t size, in
     }
     #endif
     mp_int_t r = write(o->fd, buf, size);
+    while (r == -1 && errno == EINTR) {
+        if (MP_STATE_VM(mp_pending_exception) != MP_OBJ_NULL) {
+            mp_obj_t obj = MP_STATE_VM(mp_pending_exception);
+            MP_STATE_VM(mp_pending_exception) = MP_OBJ_NULL;
+            nlr_raise(obj);
+        }
+        r = write(o->fd, buf, size);
+    }
     if (r == -1) {
         *errcode = errno;
         return MP_STREAM_ERROR;
@@ -242,7 +250,7 @@ const mp_obj_type_t mp_type_fileio = {
     .make_new = fdfile_make_new,
     .getiter = mp_identity,
     .iternext = mp_stream_unbuffered_iter,
-    .stream_p = &fileio_stream_p,
+    .protocol = &fileio_stream_p,
     .locals_dict = (mp_obj_dict_t*)&rawfile_locals_dict,
 };
 #endif
@@ -261,7 +269,7 @@ const mp_obj_type_t mp_type_textio = {
     .make_new = fdfile_make_new,
     .getiter = mp_identity,
     .iternext = mp_stream_unbuffered_iter,
-    .stream_p = &textio_stream_p,
+    .protocol = &textio_stream_p,
     .locals_dict = (mp_obj_dict_t*)&rawfile_locals_dict,
 };
 
diff --git a/unix/gccollect.c b/unix/gccollect.c
index 125c273a30..397c4ffe1c 100644
--- a/unix/gccollect.c
+++ b/unix/gccollect.c
@@ -136,15 +136,25 @@ STATIC void gc_helper_get_regs(regs_t arr) {
 
 #endif // MICROPY_GCREGS_SETJMP
 
-void gc_collect(void) {
-    //gc_dump_info();
+// this function is used by mpthreadport.c
+void gc_collect_regs_and_stack(void);
 
-    gc_collect_start();
+void gc_collect_regs_and_stack(void) {
     regs_t regs;
     gc_helper_get_regs(regs);
     // GC stack (and regs because we captured them)
     void **regs_ptr = (void**)(void*)&regs;
-    gc_collect_root(regs_ptr, ((uintptr_t)MP_STATE_VM(stack_top) - (uintptr_t)&regs) / sizeof(uintptr_t));
+    gc_collect_root(regs_ptr, ((uintptr_t)MP_STATE_THREAD(stack_top) - (uintptr_t)&regs) / sizeof(uintptr_t));
+}
+
+void gc_collect(void) {
+    //gc_dump_info();
+
+    gc_collect_start();
+    gc_collect_regs_and_stack();
+    #if MICROPY_PY_THREAD
+    mp_thread_gc_others();
+    #endif
     #if MICROPY_EMIT_NATIVE
     mp_unix_mark_exec();
     #endif
diff --git a/unix/main.c b/unix/main.c
index dedee28e7f..a1c057400f 100644
--- a/unix/main.c
+++ b/unix/main.c
@@ -45,6 +45,7 @@
 #include "py/gc.h"
 #include "py/stackctrl.h"
 #include "py/mphal.h"
+#include "py/mpthread.h"
 #include "extmod/misc.h"
 #include "genhdr/mpversion.h"
 #include "input.h"
@@ -292,7 +293,7 @@ STATIC int usage(char **argv) {
 "-v : verbose (trace various operations); can be multiple\n"
 "-O[N] : apply bytecode optimizations of level N\n"
 "\n"
-"Implementation specific options:\n", argv[0]
+"Implementation specific options (-X):\n", argv[0]
 );
     int impl_opts_cnt = 0;
     printf(
@@ -302,7 +303,7 @@ STATIC int usage(char **argv) {
     impl_opts_cnt++;
 #if MICROPY_ENABLE_GC
     printf(
-"  heapsize=<n> -- set the heap size for the GC (default %ld)\n"
+"  heapsize=<n>[w][K|M] -- set the heap size for the GC (default %ld)\n"
 , heap_size);
     impl_opts_cnt++;
 #endif
@@ -350,12 +351,20 @@ STATIC void pre_process_options(int argc, char **argv) {
                         heap_size *= 1024;
                     } else if ((*end | 0x20) == 'm') {
                         heap_size *= 1024 * 1024;
+                    } else {
+                        // Compensate for ++ below
+                        --end;
+                    }
+                    if (*++end != 0) {
+                        goto invalid_arg;
                     }
                     if (word_adjust) {
                         heap_size = heap_size * BYTES_PER_WORD / 4;
                     }
 #endif
                 } else {
+invalid_arg:
+                    printf("Invalid option\n");
                     exit(usage(argv));
                 }
                 a++;
@@ -379,6 +388,9 @@ STATIC void set_sys_argv(char *argv[], int argc, int start_arg) {
 MP_NOINLINE int main_(int argc, char **argv);
 
 int main(int argc, char **argv) {
+    #if MICROPY_PY_THREAD
+    mp_thread_init();
+    #endif
     // We should capture stack top ASAP after start, and it should be
     // captured guaranteedly before any other stack variables are allocated.
     // For this, actual main (renamed main_) should not be inlined into
@@ -432,10 +444,12 @@ MP_NOINLINE int main_(int argc, char **argv) {
         }
         if (p[0] == '~' && p[1] == '/' && home != NULL) {
             // Expand standalone ~ to $HOME
-            CHECKBUF(buf, PATH_MAX);
-            CHECKBUF_APPEND(buf, home, strlen(home));
-            CHECKBUF_APPEND(buf, p + 1, (size_t)(p1 - p - 1));
-            path_items[i] = MP_OBJ_NEW_QSTR(qstr_from_strn(buf, CHECKBUF_LEN(buf)));
+            int home_l = strlen(home);
+            vstr_t vstr;
+            vstr_init(&vstr, home_l + (p1 - p - 1) + 1);
+            vstr_add_strn(&vstr, home, home_l);
+            vstr_add_strn(&vstr, p + 1, p1 - p - 1);
+            path_items[i] = mp_obj_new_str_from_vstr(&mp_type_str, &vstr);
         } else {
             path_items[i] = MP_OBJ_NEW_QSTR(qstr_from_strn(p, p1 - p));
         }
diff --git a/unix/modmachine.c b/unix/modmachine.c
index 5c032c3466..166d47712d 100644
--- a/unix/modmachine.c
+++ b/unix/modmachine.c
@@ -31,6 +31,8 @@
 #include "py/obj.h"
 
 #include "extmod/machine_mem.h"
+#include "extmod/machine_pinbase.h"
+#include "extmod/machine_pulse.h"
 
 #if MICROPY_PLAT_DEV_MEM
 #include <errno.h>
@@ -78,6 +80,11 @@ STATIC const mp_rom_map_elem_t machine_module_globals_table[] = {
     { MP_ROM_QSTR(MP_QSTR_mem8), MP_ROM_PTR(&machine_mem8_obj) },
     { MP_ROM_QSTR(MP_QSTR_mem16), MP_ROM_PTR(&machine_mem16_obj) },
     { MP_ROM_QSTR(MP_QSTR_mem32), MP_ROM_PTR(&machine_mem32_obj) },
+
+    { MP_ROM_QSTR(MP_QSTR_PinBase), MP_ROM_PTR(&machine_pinbase_type) },
+    #if MICROPY_PY_MACHINE_PULSE
+    { MP_ROM_QSTR(MP_QSTR_time_pulse_us), MP_ROM_PTR(&machine_time_pulse_us_obj) },
+    #endif
 };
 
 STATIC MP_DEFINE_CONST_DICT(machine_module_globals, machine_module_globals_table);
diff --git a/unix/modsocket.c b/unix/modsocket.c
index 8e0d86fc79..cd68b20a45 100644
--- a/unix/modsocket.c
+++ b/unix/modsocket.c
@@ -381,7 +381,7 @@ STATIC const mp_obj_type_t usocket_type = {
     .make_new = socket_make_new,
     .getiter = NULL,
     .iternext = NULL,
-    .stream_p = &usocket_stream_p,
+    .protocol = &usocket_stream_p,
     .locals_dict = (mp_obj_dict_t*)&usocket_locals_dict,
 };
 
diff --git a/unix/modtime.c b/unix/modtime.c
index d3b780790c..6843238cf3 100644
--- a/unix/modtime.c
+++ b/unix/modtime.c
@@ -24,6 +24,9 @@
  * THE SOFTWARE.
  */
 
+#include "py/mpconfig.h"
+#if MICROPY_PY_UTIME
+
 #include <unistd.h>
 #include <errno.h>
 #include <string.h>
@@ -118,7 +121,9 @@ STATIC mp_obj_t mod_time_sleep(mp_obj_t arg) {
     tv.tv_sec = ipart;
     int res;
     while (1) {
+        MP_THREAD_GIL_EXIT();
         res = sleep_select(0, NULL, NULL, NULL, &tv);
+        MP_THREAD_GIL_ENTER();
         #if MICROPY_SELECT_REMAINING_TIME
         // TODO: This assumes Linux behavior of modifying tv to the remaining
         // time.
@@ -136,20 +141,26 @@ STATIC mp_obj_t mod_time_sleep(mp_obj_t arg) {
     RAISE_ERRNO(res, errno);
 #else
     // TODO: Handle EINTR
+    MP_THREAD_GIL_EXIT();
     sleep(mp_obj_get_int(arg));
+    MP_THREAD_GIL_ENTER();
 #endif
     return mp_const_none;
 }
 STATIC MP_DEFINE_CONST_FUN_OBJ_1(mod_time_sleep_obj, mod_time_sleep);
 
 STATIC mp_obj_t mod_time_sleep_ms(mp_obj_t arg) {
+    MP_THREAD_GIL_EXIT();
     usleep(mp_obj_get_int(arg) * 1000);
+    MP_THREAD_GIL_ENTER();
     return mp_const_none;
 }
 STATIC MP_DEFINE_CONST_FUN_OBJ_1(mod_time_sleep_ms_obj, mod_time_sleep_ms);
 
 STATIC mp_obj_t mod_time_sleep_us(mp_obj_t arg) {
+    MP_THREAD_GIL_EXIT();
     usleep(mp_obj_get_int(arg));
+    MP_THREAD_GIL_ENTER();
     return mp_const_none;
 }
 STATIC MP_DEFINE_CONST_FUN_OBJ_1(mod_time_sleep_us_obj, mod_time_sleep_us);
@@ -190,3 +201,5 @@ const mp_obj_module_t mp_module_time = {
     .name = MP_QSTR_utime,
     .globals = (mp_obj_dict_t*)&mp_module_time_globals,
 };
+
+#endif // MICROPY_PY_UTIME
diff --git a/unix/mpconfigport.h b/unix/mpconfigport.h
index 9601673a74..06c4edc1e2 100644
--- a/unix/mpconfigport.h
+++ b/unix/mpconfigport.h
@@ -103,6 +103,7 @@
 #define MICROPY_STACKLESS_STRICT    (0)
 
 #define MICROPY_PY_OS_STATVFS       (1)
+#define MICROPY_PY_UTIME            (1)
 #define MICROPY_PY_UERRNO           (1)
 #define MICROPY_PY_UCTYPES          (1)
 #define MICROPY_PY_UZLIB            (1)
@@ -119,6 +120,7 @@
 #define MICROPY_PY_USELECT          (1)
 #endif
 #define MICROPY_PY_MACHINE          (1)
+#define MICROPY_PY_MACHINE_PULSE    (1)
 #define MICROPY_MACHINE_MEM_GET_READ_ADDR   mod_machine_mem_get_addr
 #define MICROPY_MACHINE_MEM_GET_WRITE_ADDR  mod_machine_mem_get_addr
 
@@ -129,8 +131,8 @@
 #define MICROPY_FATFS_VOLUMES          (3)
 #define MICROPY_FATFS_MAX_SS           (4096)
 #define MICROPY_FATFS_LFN_CODE_PAGE    (437) /* 1=SFN/ANSI 437=LFN/U.S.(OEM) */
-#define MICROPY_FSUSERMOUNT            (1)
-#define MICROPY_VFS_FAT                (1)
+#define MICROPY_FSUSERMOUNT            (0)
+#define MICROPY_VFS_FAT                (0)
 
 // Define to MICROPY_ERROR_REPORTING_DETAILED to get function, etc.
 // names in exception messages (may require more RAM).
@@ -170,10 +172,10 @@ extern const struct _mp_obj_module_t mp_module_jni;
 #else
 #define MICROPY_PY_JNI_DEF
 #endif
-#if MICROPY_PY_TIME
-#define MICROPY_PY_TIME_DEF { MP_ROM_QSTR(MP_QSTR_utime), MP_ROM_PTR(&mp_module_time) },
+#if MICROPY_PY_UTIME
+#define MICROPY_PY_UTIME_DEF { MP_ROM_QSTR(MP_QSTR_utime), MP_ROM_PTR(&mp_module_time) },
 #else
-#define MICROPY_PY_TIME_DEF
+#define MICROPY_PY_UTIME_DEF
 #endif
 #if MICROPY_PY_TERMIOS
 #define MICROPY_PY_TERMIOS_DEF { MP_ROM_QSTR(MP_QSTR_termios), MP_ROM_PTR(&mp_module_termios) },
@@ -194,7 +196,7 @@ extern const struct _mp_obj_module_t mp_module_jni;
 #define MICROPY_PORT_BUILTIN_MODULES \
     MICROPY_PY_FFI_DEF \
     MICROPY_PY_JNI_DEF \
-    MICROPY_PY_TIME_DEF \
+    MICROPY_PY_UTIME_DEF \
     MICROPY_PY_SOCKET_DEF \
     { MP_ROM_QSTR(MP_QSTR_umachine), MP_ROM_PTR(&mp_module_machine) }, \
     { MP_ROM_QSTR(MP_QSTR_uos), MP_ROM_PTR(&mp_module_os) }, \
@@ -233,9 +235,10 @@ void mp_unix_free_exec(void *ptr, mp_uint_t size);
 void mp_unix_mark_exec(void);
 #define MP_PLAT_ALLOC_EXEC(min_size, ptr, size) mp_unix_alloc_exec(min_size, ptr, size)
 #define MP_PLAT_FREE_EXEC(ptr, size) mp_unix_free_exec(ptr, size)
-
-#ifndef MP_NOINLINE
-#define MP_NOINLINE __attribute__((noinline))
+#ifndef MICROPY_FORCE_PLAT_ALLOC_EXEC
+// Use MP_PLAT_ALLOC_EXEC for any executable memory allocation, including for FFI
+// (overriding libffi own implementation)
+#define MICROPY_FORCE_PLAT_ALLOC_EXEC (1)
 #endif
 
 #if MICROPY_PY_OS_DUPTERM
diff --git a/unix/mpconfigport.mk b/unix/mpconfigport.mk
index 88bd749a71..9f826661a9 100644
--- a/unix/mpconfigport.mk
+++ b/unix/mpconfigport.mk
@@ -6,11 +6,16 @@ MICROPY_FORCE_32BIT = 0
 # This variable can take the following values:
 #  0 - no readline, just simple input
 #  1 - use MicroPython version of readline
-#  2 - use GNU readline (causes binary to be licensed under GPL)
 MICROPY_USE_READLINE = 1
 
-# Subset of CPython time module
-MICROPY_PY_TIME = 1
+# Whether to enable FatFs VFS
+MICROPY_FATFS = 1
+
+# btree module using Berkeley DB 1.xx
+MICROPY_PY_BTREE = 1
+
+# _thread module using pthreads
+MICROPY_PY_THREAD = 1
 
 # Subset of CPython termios module
 MICROPY_PY_TERMIOS = 1
diff --git a/unix/mpconfigport_coverage.h b/unix/mpconfigport_coverage.h
index e5d5fd7a47..f9a6fbd9dd 100644
--- a/unix/mpconfigport_coverage.h
+++ b/unix/mpconfigport_coverage.h
@@ -31,3 +31,7 @@
 
 #define MICROPY_PY_URANDOM_EXTRA_FUNCS (1)
 #define MICROPY_PY_IO_BUFFEREDWRITER (1)
+#undef MICROPY_FSUSERMOUNT
+#undef MICROPY_VFS_FAT
+#define MICROPY_FSUSERMOUNT            (1)
+#define MICROPY_VFS_FAT                (1)
diff --git a/unix/mpconfigport_minimal.h b/unix/mpconfigport_minimal.h
index cffc9a6250..2ab95c67af 100644
--- a/unix/mpconfigport_minimal.h
+++ b/unix/mpconfigport_minimal.h
@@ -26,6 +26,12 @@
 
 // options to control how Micro Python is built
 
+#define MICROPY_ALLOC_QSTR_CHUNK_INIT (64)
+#define MICROPY_ALLOC_PARSE_RULE_INIT (8)
+#define MICROPY_ALLOC_PARSE_RULE_INC  (8)
+#define MICROPY_ALLOC_PARSE_RESULT_INIT (8)
+#define MICROPY_ALLOC_PARSE_RESULT_INC (8)
+#define MICROPY_ALLOC_PARSE_CHUNK_INIT (64)
 #define MICROPY_ALLOC_PATH_MAX      (PATH_MAX)
 #define MICROPY_ENABLE_GC           (1)
 #define MICROPY_ENABLE_FINALISER    (0)
@@ -38,6 +44,7 @@
 #define MICROPY_ENABLE_SOURCE_LINE  (0)
 #define MICROPY_ERROR_REPORTING     (MICROPY_ERROR_REPORTING_TERSE)
 #define MICROPY_WARNINGS            (0)
+#define MICROPY_ENABLE_EMERGENCY_EXCEPTION_BUF   (0)
 #define MICROPY_FLOAT_IMPL          (MICROPY_FLOAT_IMPL_NONE)
 #define MICROPY_LONGINT_IMPL        (MICROPY_LONGINT_IMPL_NONE)
 #define MICROPY_STREAMS_NON_BLOCK   (0)
@@ -76,6 +83,7 @@
 #define MICROPY_PY_SYS_STDFILES     (0)
 #define MICROPY_PY_CMATH            (0)
 #define MICROPY_PY_UCTYPES          (0)
+#define MICROPY_PY_UTIME            (0)
 #define MICROPY_PY_UZLIB            (0)
 #define MICROPY_PY_UJSON            (0)
 #define MICROPY_PY_URE              (0)
@@ -83,6 +91,18 @@
 #define MICROPY_PY_UHASHLIB         (0)
 #define MICROPY_PY_UBINASCII        (0)
 
+extern const struct _mp_obj_module_t mp_module_os;
+
+#define MICROPY_PORT_BUILTIN_MODULES \
+    { MP_OBJ_NEW_QSTR(MP_QSTR_uos), (mp_obj_t)&mp_module_os }, \
+
+#define MICROPY_PORT_ROOT_POINTERS \
+    mp_obj_t keyboard_interrupt_obj;
+
+//////////////////////////////////////////
+// Do not change anything beyond this line
+//////////////////////////////////////////
+
 // Define to 1 to use undertested inefficient GC helper implementation
 // (if more efficient arch-specific one is not available).
 #ifndef MICROPY_GCREGS_SETJMP
@@ -93,13 +113,6 @@
     #endif
 #endif
 
-#define MICROPY_ENABLE_EMERGENCY_EXCEPTION_BUF   (0)
-
-extern const struct _mp_obj_module_t mp_module_os;
-
-#define MICROPY_PORT_BUILTIN_MODULES \
-    { MP_OBJ_NEW_QSTR(MP_QSTR_uos), (mp_obj_t)&mp_module_os }, \
-
 // type definitions for the specific machine
 
 #ifdef __LP64__
@@ -124,9 +137,6 @@ typedef long mp_off_t;
 typedef void *machine_ptr_t; // must be of pointer size
 typedef const void *machine_const_ptr_t; // must be of pointer size
 
-#define MICROPY_PORT_ROOT_POINTERS \
-    mp_obj_t keyboard_interrupt_obj;
-
 // We need to provide a declaration/definition of alloca()
 #ifdef __FreeBSD__
 #include <stdlib.h>
diff --git a/unix/mpthreadport.c b/unix/mpthreadport.c
new file mode 100644
index 0000000000..e5cfe7a669
--- /dev/null
+++ b/unix/mpthreadport.c
@@ -0,0 +1,223 @@
+/*
+ * This file is part of the MicroPython project, http://micropython.org/
+ *
+ * The MIT License (MIT)
+ *
+ * Copyright (c) 2016 Damien P. George on behalf of Pycom Ltd
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <errno.h>
+
+#include "py/mpstate.h"
+#include "py/mpthread.h"
+#include "py/gc.h"
+
+#if MICROPY_PY_THREAD
+
+#include <signal.h>
+#include <sched.h>
+
+// this structure forms a linked list, one node per active thread
+typedef struct _thread_t {
+    pthread_t id;           // system id of thread
+    int ready;              // whether the thread is ready and running
+    void *arg;              // thread Python args, a GC root pointer
+    struct _thread_t *next;
+} thread_t;
+
+STATIC pthread_key_t tls_key;
+
+// the mutex controls access to the linked list
+STATIC pthread_mutex_t thread_mutex = PTHREAD_MUTEX_INITIALIZER;
+STATIC thread_t *thread;
+
+// this is used to synchronise the signal handler of the thread
+// it's needed because we can't use any pthread calls in a signal handler
+STATIC volatile int thread_signal_done;
+
+// this signal handler is used to scan the regs and stack of a thread
+STATIC void mp_thread_gc(int signo, siginfo_t *info, void *context) {
+    (void)info; // unused
+    (void)context; // unused
+    if (signo == SIGUSR1) {
+        void gc_collect_regs_and_stack(void);
+        gc_collect_regs_and_stack();
+        // We have access to the context (regs, stack) of the thread but it seems
+        // that we don't need the extra information, enough is captured by the
+        // gc_collect_regs_and_stack function above
+        //gc_collect_root((void**)context, sizeof(ucontext_t) / sizeof(uintptr_t));
+        thread_signal_done = 1;
+    }
+}
+
+void mp_thread_init(void) {
+    pthread_key_create(&tls_key, NULL);
+    pthread_setspecific(tls_key, &mp_state_ctx.thread);
+
+    // create first entry in linked list of all threads
+    thread = malloc(sizeof(thread_t));
+    thread->id = pthread_self();
+    thread->ready = 1;
+    thread->arg = NULL;
+    thread->next = NULL;
+
+    // enable signal handler for garbage collection
+    struct sigaction sa;
+    sa.sa_flags = SA_SIGINFO;
+    sa.sa_sigaction = mp_thread_gc;
+    sigemptyset(&sa.sa_mask);
+    sigaction(SIGUSR1, &sa, NULL);
+}
+
+// This function scans all pointers that are external to the current thread.
+// It does this by signalling all other threads and getting them to scan their
+// own registers and stack.  Note that there may still be some edge cases left
+// with race conditions and root-pointer scanning: a given thread may manipulate
+// the global root pointers (in mp_state_ctx) while another thread is doing a
+// garbage collection and tracing these pointers.
+void mp_thread_gc_others(void) {
+    pthread_mutex_lock(&thread_mutex);
+    for (thread_t *th = thread; th != NULL; th = th->next) {
+        gc_collect_root(&th->arg, 1);
+        if (th->id == pthread_self()) {
+            continue;
+        }
+        if (!th->ready) {
+            continue;
+        }
+        thread_signal_done = 0;
+        pthread_kill(th->id, SIGUSR1);
+        while (thread_signal_done == 0) {
+            sched_yield();
+        }
+    }
+    pthread_mutex_unlock(&thread_mutex);
+}
+
+mp_state_thread_t *mp_thread_get_state(void) {
+    return (mp_state_thread_t*)pthread_getspecific(tls_key);
+}
+
+void mp_thread_set_state(void *state) {
+    pthread_setspecific(tls_key, state);
+}
+
+void mp_thread_start(void) {
+    pthread_mutex_lock(&thread_mutex);
+    for (thread_t *th = thread; th != NULL; th = th->next) {
+        if (th->id == pthread_self()) {
+            th->ready = 1;
+            break;
+        }
+    }
+    pthread_mutex_unlock(&thread_mutex);
+}
+
+void mp_thread_create(void *(*entry)(void*), void *arg, size_t *stack_size) {
+    // default stack size is 8k machine-words, minimum is 2k
+    if (*stack_size == 0) {
+        *stack_size = 8192 * BYTES_PER_WORD;
+    } else if (*stack_size < 2048 * BYTES_PER_WORD) {
+        *stack_size = 2048 * BYTES_PER_WORD;
+    }
+
+    // set thread attributes
+    pthread_attr_t attr;
+    int ret = pthread_attr_init(&attr);
+    if (ret != 0) {
+        goto er;
+    }
+    ret = pthread_attr_setstacksize(&attr, *stack_size);
+    if (ret != 0) {
+        goto er;
+    }
+
+    pthread_mutex_lock(&thread_mutex);
+
+    // create thread
+    pthread_t id;
+    ret = pthread_create(&id, &attr, entry, arg);
+    if (ret != 0) {
+        pthread_mutex_unlock(&thread_mutex);
+        goto er;
+    }
+
+    // adjust stack_size to provide room to recover from hitting the limit
+    *stack_size -= 1024 * BYTES_PER_WORD;
+
+    // add thread to linked list of all threads
+    thread_t *th = malloc(sizeof(thread_t));
+    th->id = id;
+    th->ready = 0;
+    th->arg = arg;
+    th->next = thread;
+    thread = th;
+
+    pthread_mutex_unlock(&thread_mutex);
+
+    return;
+
+er:
+    nlr_raise(mp_obj_new_exception_arg1(&mp_type_OSError, MP_OBJ_NEW_SMALL_INT(ret)));
+}
+
+void mp_thread_finish(void) {
+    pthread_mutex_lock(&thread_mutex);
+    // TODO unlink from list
+    for (thread_t *th = thread; th != NULL; th = th->next) {
+        if (th->id == pthread_self()) {
+            th->ready = 0;
+            break;
+        }
+    }
+    pthread_mutex_unlock(&thread_mutex);
+}
+
+void mp_thread_mutex_init(mp_thread_mutex_t *mutex) {
+    pthread_mutex_init(mutex, NULL);
+}
+
+int mp_thread_mutex_lock(mp_thread_mutex_t *mutex, int wait) {
+    int ret;
+    if (wait) {
+        ret = pthread_mutex_lock(mutex);
+        if (ret == 0) {
+            return 1;
+        }
+    } else {
+        ret = pthread_mutex_trylock(mutex);
+        if (ret == 0) {
+            return 1;
+        } else if (ret == EBUSY) {
+            return 0;
+        }
+    }
+    return -ret;
+}
+
+void mp_thread_mutex_unlock(mp_thread_mutex_t *mutex) {
+    pthread_mutex_unlock(mutex);
+    // TODO check return value
+}
+
+#endif // MICROPY_PY_THREAD
diff --git a/unix/mpthreadport.h b/unix/mpthreadport.h
new file mode 100644
index 0000000000..51cf8d7860
--- /dev/null
+++ b/unix/mpthreadport.h
@@ -0,0 +1,36 @@
+/*
+ * This file is part of the MicroPython project, http://micropython.org/
+ *
+ * The MIT License (MIT)
+ *
+ * Copyright (c) 2016 Damien P. George on behalf of Pycom Ltd
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+#ifndef __MICROPY_INCLUDED_UNIX_MPTHREADPORT_H__
+#define __MICROPY_INCLUDED_UNIX_MPTHREADPORT_H__
+
+#include <pthread.h>
+
+typedef pthread_mutex_t mp_thread_mutex_t;
+
+void mp_thread_init(void);
+void mp_thread_gc_others(void);
+
+#endif // __MICROPY_INCLUDED_UNIX_MPTHREADPORT_H__