Merge tag 'v1.8.3' into parse-bytecode

Many small improvements and additions, with btree support in ESP8266 port

This release brings various improvements and additions to the core,
extended modules and the ESP8266 port, as well as enhancements to the docs.
There is now a "threshold" function in the gc module for the user to
configure the garbage collector to run earlier than usual, in order to help
reduce fragmentation of the heap.  The btree module is now available in the
ESP8266 port, and there is improved WebREPL file transfer: get file is now
non-blocking (this change requires an update of the client software).

py core:
- nlrx64.S: prefix mp_thread_get_state with an underscore on Mac
- stream: implement 2- and 3-arg write() method as an extension to CPython
- gc: implement GC running by allocation threshold, user configurable
- fix nlrthumb.c when DEBUG=1 is defined
- obj: issue a warning when str and bytes objects are compared
- mpconfig.h: fix description for MICROPY_PY_STR_BYTES_CMP_WARN
- objstrunicode: str_index_to_ptr: should handle bytes too
- objstrunicode: str_index_to_ptr: implement positive indexing properly
- stream: implement generic flush() method, in terms of C-level ioctl
- objstringio: add MP_STREAM_FLUSH ioctl and flush() method
- objstringio: implement MP_STREAM_SEEK ioctl and add seek() method
- stream: add adapter methods with POSIX-compatible signatures
- mpconfig.h: add MICROPY_STREAMS_POSIX_API setting
- py.mk: extra switches to build "embedded" BerkeleyDB BTree lib
- mkrules.mk: allow to add more items for "clean" target using CLEAN_EXTRA
- objstr: make .partition()/.rpartition() methods configurable
- mpconfig.h: define MP_ALWAYSINLINE for reuse

extmod:
- modussl: rename to modussl_axtls.c, to allow impl using other SSL libs
- modussl_axtls: further changes to allow alternative SSL modules
- modussl_axtls: add dummy setblocking() method
- vfs_fat: implement rmdir() method (aliases to unlink)
- modussl_axtls: use mp_stream_close() method
- modwebrepl: use mp_stream_close() method
- modbtree: check __bt_open() return value for error
- modbtree: switch to accepting stream object instead of filename
- modbtree: implement __contains__ operation
- modwebrepl: factor out "GET" iteration to write_file_chunk()
- modwebrepl: make GET_FILE operation non-blocking
- modbtree: open(): add option kwargs
- modwebsocket: add readline method
- modwebsocket: make compatible with non-default object models
- modwebsocket: use mp_rom_map_elem_t and friends
- modwebrepl: add GET_VER operation to query MicroPython version

lib:
- axtls: upgrade to axTLS 1.5.4 + MicroPython patchset
- axtls: update, fixes esp8266 build
- utils/stdout_helpers: move from minimal/uart_extra.c for reuse
- embed/abort_: implementation of abort_() function raising uPy exception
- berkeley-db-1.xx: switch to "embedded" branch
- berkeley-db-1.xx: update, sets default page size to 4096

tests:
- add test for extended arguments to stream .write() method
- extmod/machine_pinbase: skip if machine.PinBase is not available
- bytes_compare: rework test for bytes <-> str comparison
- extmod/btree1: close database at the end of test
- unicode_subscr.py: detailed test for subscripting unicode strings
- run-tests: make "regex'ed .exp" facility available to device tests
- io/bytesio_ext: test for .seek()/.flush() on BytesIO
- extmod/btree1: tests against in-memory DB (using io.BytesIO)
- machine_mem.py: too non-portable, rework as an example for unix port
- machine1: revamp to work with unix port (which has "umachine")
- basics: bytes/str.partition/rpartition are now optional

mpy-cross:
- fix Makefile to handle gc-sections linker flags on OS X
- use binary file translation mode for creating mpy files on windows
- fix mingw and msys2 compilation

minimal port:
- disable MICROPY_GC_ALLOC_THRESHOLD

unix port:
- mpthreadport: adjust minimum thread stack, and stack limit check
- cache libaxtls.a in local build dir
- disable MICROPY_GC_ALLOC_THRESHOLD for minimal build
- enable MICROPY_PY_STR_BYTES_CMP_WARN
- mpconfigport.h: include stdio.h by default
- file: implement MP_STREAM_FLUSH ioctl
- file: ioctl(): check that file is open before operations
- file: fdfile_ioctl(): fix argument to check_fd_is_open()
- file: use generic stream flush() method
- enable websocket module
- moduselect: allow poll.register(), etc. accept fd-like objects

stmhal port:
- fix malloc when used with external libraries
- make SPI NSS pin definition optional
- fix I2C mappings for STM32F429DISC board

esp8266 port:
- select axTLS for SSL implementation, following recent refactor
- moduos: add rmdir() function
- cache Xtensa-built libaxtls.a in local build dir
- enable MICROPY_PY_STR_BYTES_CMP_WARN
- dupterm_task_init() should be called before running _boot.py, etc
- _boot.py: set GC alloc threshold to half of heap size
- _boot.py: decrease GC alloc threshold to quarter of heap size
- modpybuart: fix UART parity setting
- axtls_helpers: remove abort_(), now in lib/embed/
- mpconfigport.h: include sys/types.h for POSIX types definitions
- esp_mphal: implement libc's errno
- enable btree module
- eagle.rom.addr.v6.ld: add Enable_QMode symbol from SDK 2.0.0
- make APA102 driver inclusion configurable
- makeimg.py: store firmware size as last 4 bytes of padding area
- makeimg.py: append md5 hash to the generated binary
- modesp: add check_fw() function to check integrity of the firmware
- scripts/port_diag.py: include esp.check_fw() call
- flashbdev: reserve extra sysparam sector for SDK 2.0.0 compatibility
- scripts/inisetup: add commented-out call to esp.osdebug(None)
- modmachine: implement idle() function
- esp_mphal.h: add mp_hal_ticks_cpu() for reuse
- modutime: actually implement ticks_cpu()
- modmachine: implement dummy sleep() function
- tutorial/intro: reword para abou -fm dio switch
- modules/flashbdev: start filesystem at 0x90000
- esp8266.ld: increase firmware image size to 0x90000 (576K)

docs:
- add DHT to ESP8266 Quick Ref and Tutorial
- fix some spelling mistakes
- array: document array module
- library/index: include array module in ToC
- esp8266/intro: rename to "Getting started" from "Introduction"
- esp8266/intro: add troubleshooting section
- esp8266/quickref: link to installation instructions
- esp8266/tutorial/intro: add anchor for link from quickeref
- esp8266/intro: focus on hazards of unearthed power wrt electronics
- uio: mention seek()/flush() support for io.BytesIO

misc:
- logo/1bit-logo A black & white version of the logo

examples:
- http_server*: update for buffered-like streams (read line by line)
- embedding: example for embedding MicroPython in an app
- embedding: add README
- http_client*: be sure to close socket
- network: split recv- and read-based HTTP servers

qemu-arm:
- enable gcc LTO option for nlrthumb.c

6 files changed, 153 insertions, 13 deletions

diff --git a/examples/network/http_client.py b/examples/network/http_client.py
index 3701e75e16..df66ace2a5 100644
--- a/examples/network/http_client.py
+++ b/examples/network/http_client.py
@@ -24,5 +24,7 @@ def main(use_stream=False):
         s.send(b"GET / HTTP/1.0\n\n")
         print(s.recv(4096))
 
+    s.close()
+
 
 main()
diff --git a/examples/network/http_client_ssl.py b/examples/network/http_client_ssl.py
index 53b1c732bc..46e039830f 100644
--- a/examples/network/http_client_ssl.py
+++ b/examples/network/http_client_ssl.py
@@ -32,5 +32,7 @@ def main(use_stream=True):
         s.send(b"GET / HTTP/1.0\n\n")
         print(s.recv(4096))
 
+    s.close()
+
 
 main()
diff --git a/examples/network/http_server.py b/examples/network/http_server.py
index 80dfc5db02..e3a66e8283 100644
--- a/examples/network/http_server.py
+++ b/examples/network/http_server.py
@@ -10,7 +10,7 @@ HTTP/1.0 200 OK
 Hello #%d from MicroPython!
 """
 
-def main(use_stream=False):
+def main(micropython_optimize=False):
     s = socket.socket()
 
     # Binding to all interfaces - server will be accessible to other hosts!
@@ -26,20 +26,37 @@ def main(use_stream=False):
     counter = 0
     while True:
         res = s.accept()
-        client_s = res[0]
+        client_sock = res[0]
         client_addr = res[1]
         print("Client address:", client_addr)
-        print("Client socket:", client_s)
-        print("Request:")
-        if use_stream:
-            # MicroPython socket objects support stream (aka file) interface
-            # directly.
-            print(client_s.read(4096))
-            client_s.write(CONTENT % counter)
+        print("Client socket:", client_sock)
+
+        if not micropython_optimize:
+            # To read line-oriented protocol (like HTTP) from a socket (and
+            # avoid short read problem), it must be wrapped in a stream (aka
+            # file-like) object. That's how you do it in CPython:
+            client_stream = client_sock.makefile("rwb")
         else:
-            print(client_s.recv(4096))
-            client_s.send(CONTENT % counter)
-        client_s.close()
+            # .. but MicroPython socket objects support stream interface
+            # directly, so calling .makefile() method is not required. If
+            # you develop application which will run only on MicroPython,
+            # especially on a resource-constrained embedded device, you
+            # may take this shortcut to save resources.
+            client_stream = client_sock
+
+        print("Request:")
+        req = client_stream.readline()
+        print(req)
+        while True:
+            h = client_stream.readline()
+            if h == b"" or h == b"\r\n":
+                break
+            print(h)
+        client_stream.write(CONTENT % counter)
+
+        client_stream.close()
+        if not micropython_optimize:
+            client_sock.close()
         counter += 1
         print()
 
diff --git a/examples/network/http_server_simplistic.py b/examples/network/http_server_simplistic.py
new file mode 100644
index 0000000000..f932e48f56
--- /dev/null
+++ b/examples/network/http_server_simplistic.py
@@ -0,0 +1,38 @@
+try:
+    import usocket as socket
+except:
+    import socket
+
+
+CONTENT = b"""\
+HTTP/1.0 200 OK
+
+Hello #%d from MicroPython!
+"""
+
+def main():
+    s = socket.socket()
+    ai = socket.getaddrinfo("0.0.0.0", 8080)
+    addr = ai[0][-1]
+
+    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+
+    s.bind(addr)
+    s.listen(5)
+    print("Listening, connect your browser to http://<this_host>:8080/")
+
+    counter = 0
+    while True:
+        res = s.accept()
+        client_s = res[0]
+        client_addr = res[1]
+        req = client_s.recv(4096)
+        print("Request:")
+        print(req)
+        client_s.send(CONTENT % counter)
+        client_s.close()
+        counter += 1
+        print()
+
+
+main()
diff --git a/examples/network/http_server_simplistic_commented.py b/examples/network/http_server_simplistic_commented.py
new file mode 100644
index 0000000000..b58e9eeb60
--- /dev/null
+++ b/examples/network/http_server_simplistic_commented.py
@@ -0,0 +1,76 @@
+#
+# MicroPython http_server_simplistic.py example
+#
+# This example shows how to write the smallest possible HTTP
+# server in MicroPython. With comments and convenience code
+# removed, this example can be compressed literally to ten
+# lines. There's a catch though - read comments below for
+# details, and use this code only for quick hacks, preferring
+# http_server.py for "real thing".
+#
+try:
+    import usocket as socket
+except:
+    import socket
+
+
+CONTENT = b"""\
+HTTP/1.0 200 OK
+
+Hello #%d from MicroPython!
+"""
+
+def main():
+    s = socket.socket()
+
+    # Bind to (allow to be connected on ) all interfaces. This means
+    # this server will be accessible to other hosts on your local
+    # network, and if your server has direct (non-firewalled) connection
+    # to the Internet, then to anyone on the Internet. We bind to all
+    # interfaces to let this example work easily on embedded MicroPython
+    # targets, which you will likely access from another machine on your
+    # local network. Take care when running this on an Internet-connected
+    # machine though! Replace "0.0.0.0" with "127.0.0.1" if in doubt, to
+    # make the server accessible only on the machine it runs on.
+    ai = socket.getaddrinfo("0.0.0.0", 8080)
+    print("Bind address info:", ai)
+    addr = ai[0][-1]
+
+    # A port on which a socket listened remains inactive during some time.
+    # This means that if you run this sample, terminate it, and run again
+    # you will likely get an error. To avoid this timeout, set SO_REUSEADDR
+    # socket option.
+    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+
+    s.bind(addr)
+    s.listen(5)
+    print("Listening, connect your browser to http://<this_host>:8080/")
+
+    counter = 0
+    while True:
+        res = s.accept()
+        client_s = res[0]
+        client_addr = res[1]
+        print("Client address:", client_addr)
+        print("Client socket:", client_s)
+        # We assume here that .recv() call will read entire HTTP request
+        # from client. This is usually true, at least on "big OS" systems
+        # like Linux/MacOS/Windows. But that doesn't have to be true in
+        # all cases, in particular on embedded systems, when there can
+        # easily be "short recv", where it returns much less than requested
+        # data size. That's why this example is called "simplistic" - it
+        # shows that writing a web server in Python that *usually works* is
+        # ten lines of code, and you can use this technique for quick hacks
+        # and experimentation. But don't do it like that in production
+        # applications - instead, parse HTTP request properly, as shown
+        # by http_server.py example.
+        req = client_s.recv(4096)
+        print("Request:")
+        print(req)
+        client_s.send(CONTENT % counter)
+        client_s.close()
+        counter += 1
+        print()
+
+
+main()
diff --git a/examples/network/http_server_ssl.py b/examples/network/http_server_ssl.py
index 04e0913448..9a69ca9d41 100644
--- a/examples/network/http_server_ssl.py
+++ b/examples/network/http_server_ssl.py
@@ -42,8 +42,13 @@ def main(use_stream=True):
             # next request they issue will likely be more well-behaving and
             # will succeed.
             try:
-                req = client_s.read(4096)
+                req = client_s.readline()
                 print(req)
+                while True:
+                    h = client_s.readline()
+                    if h == b"" or h == b"\r\n":
+                        break
+                    print(h)
                 if req:
                     client_s.write(CONTENT % counter)
             except Exception as e: