Merge tag 'v1.8' into parse-bytecode

First general release of ESP8266 port, and support for frozen bytecode

This release marks the first general release of official ESP8266 support
within the MicroPython code base.  The ESP8266 port has many improvements
and additions, including: websocket and webrepl modules, deep-sleep mode,
reading on UART, enhanced I2C support, enhanced network configuration,
full sequence of start-up scripts (built-in _boot.py, boot.py and
main.py), improved filesystem support with automatic flash-size detection
as well as documentation and a tutorial.

Known issues with ESP8266 port are:
- Basic SSL support is available but it works only with relatively short
  data lengths (a few kilobytes).
- WebREPL over SSL is not supported.
- Only 5 or so WebREPL sessions in succession can be started (including
  file transfer sessions), after that the board requires a hard reset.
- File transfer over WebREPL has issues with large files (above a few
  tens of kilobytes).
- By design, only one concurrent WebREPL connection is supported.
- Soft reset doesn't reset the network stack.

In addition to ESP8266 support, this release brings frozen bytecode
which allows to compile bytecode offline and link it into the firmware,
completely eliminating the need for compilation (and the associated
RAM usage) at runtime.  Basic async/await syntax is now supported, and
qstrs are now auto-generated in the build system.

A detailed list of changes is given below.

README:
- explicitly point to required dependencies section
- promote "docs" and "tests" to "major components"
- mention support "async" keyword from Python 3.5

ACKNOWLEDGEMENTS:
- add list of 842 backers from the ESP8266 campaign

py core:
- modbuiltins: add comment about setting "_" special var
- add async/await/async for/async with syntax
- fix constant folding and inline-asm to work with new async grammar
- emitglue: move typedef of mp_raw_code_t from .c to .h file
- emitglue: make mp_raw_code_t* arguments constant pointers
- makeqstrdata: factor out some code to functions that can be reused
- add ability to have frozen persistent bytecode from .mpy files
- makeqstrdata: add more names for escaped chars and esc non-printable
- simplify "and" action within parser by making ident-rules explicit
- makeqstrdata: fix rendering of qstrs that have non-printable ASCII
- makeqstrdata: add special case to handle \n qstr
- declare help, input, open builtins in core
- map: change hash-table allocation policy to be less aggressive
- makeqstrdefs: add script to automate extraction of qstr from sources
- frozenmod: pass the source name of the frozen module to the lexer
- rework QSTR extraction to work in simple and obvious way
- divide "split" and "cat" phases of qstr extraction for better efficiency
- fix bug passing a string as a keyword arg in a dict
- move call_function_*_protected() functions to py/ for reuse
- makeqstrdefs.py: windows compatibility
- obj.h: when constructing a small-int cast to mp_uint_t for bit-shift
- emitnative: use MP_OBJ_NEW_SMALL_INT instead of manual bit shifting
- vm: "yield from" didn't handle MP_OBJ_STOP_ITERATION optimization
- modio: rename module name to "uio" for consistency with other modules
- modcollections: rename module name have "u" prefix for consistency

extmod:
- add initial framebuf module
- add generic machine.I2C class, with bit-bang I2C
- machine_i2c: fix I2C reading by sending ack/nack at end of byte
- machine_i2c: implement I2C memory reading/writing
- moduos_dupterm: don't swallow exceptions in dupterm's read()/write()
- modlwip: lwip_tcp_receive(): Properly handle EOF for non-blocking sock
- modlwip: more debug messages for various edge conditions
- fsusermount: in mount/mkfs, deregister VFS object on error
- modlwip: lwip_tcp_send(): full error handling
- modlwip: lwip_tcp_receive(): full error handling
- modlwip: add ability to run callback on "recv" and "accept" events
- machine_i2c: allow mp_hal_pin_obj_t to be any type, not a ptr
- modlwip: protect recv/accept Python callback against exceptions
- modlwip: workaround esp8266 sendto issue where 1 is returned
- modlwip: add print_pcbs() debug function
- modwebsocket: handle CLOSE control frame
- modussl: throw Python exceptions in case of errors
- modussl: support server-side SSL sockets
- modussl: SSL_OK from ssl_read() means "no user data so far"
- modwebrepl: module to handle WebREPL protocol
- modwebrepl: initial implementation of "get file" operation
- modwebrepl: keep reading data when there's something to read
- modwebrepl: GET_FILE: Send length-prefix chunk with one write()
- modwebrepl: add rate-limiting workaround for broken network drivers
- modwebrepl: set debugging by default to off
- modwebrepl: add support for password
- modlwip: implement sendall() method for TCP sockets
- fix typo of macro that detects if float is enabled

lib:
- utils/printf: rework overriding printer of DEBUG_printf()
- utils/pyexec: condition-out GC calls from pyexec
- utils/pyhelp: extract implementation of help(obj) to a library function
- axtls: update to the latest upstream, fix reported MacOSX build issue

drivers:
- add SSD1306 OLED driver, with I2C and SPI interfaces

tools:
- mpy-tool.py: new tool to work with .mpy files; currently it can freeze them
- mpy-tool.py: add support for Python 2.7

tests:
- add 6 tests for async await/for/with
- add .exp files for async tests, so they can run with Python 3.4
- fix dict1.py so it doesn't rely on the order of dict elems
- extmod: move split-on-empty-match tests to a separate test file
- add testcase for yielding from a stopped generator
- run-bench-tests: process tests in alphabetical order
- update for _io/_collections module having been renamed

minimal port:
- add example of frozen persistent bytecode (.mpy file)
- disable async/await syntax

unix port:
- build with MICROPY_PY_UHASHLIB_SHA1 if already building with axTLS
- make sure build dir exists before accessing it for freezing upip
- Makefile: make install more compatible (BSD, etc.)

windows port:
- msvc: implement automatic qstr generation using makeqstrdefs

stmhal port:
- implement basic C-level pin HAL
- use new generic I2C object in machine module
- enable framebuf module
- properly handle RTS/CTS flow control for buf/unbuf transfers
- add Makefile option FROZEN_MPY_DIR to support frozen bytecode
- for frozen bytecode generation, add dependency of qstr file
- l4: add CMSIS files to support STM32L476
- l4: add basic STM32L4xx HAL files
- l4: adapt UART HAL to avoid 64-bit integer division
- update HALCOMMITS due to change to hal
- l4: add board definition files for STM32L476DISC
- l4: add line to Makefile for building L4 series
- l4: modify adc.c to add support for STM32L4 series
- l4: modify uart.c to support L4 MCU
- l4: modify usbd_conf.c to support L4 MCU
- l4: modify rtc.c to support L4 MCU
- l4: modify timer.c to support L4 MCU
- fix machine.unique_id() function to work for all MCUs
- l4: modify mphalport to support L4 MCU
- l4: modify flash.c and storage.c to support L4 MCU
- l4: add support for external interrupts/events
- accel: raise an exception if the accel couldn't be initialised
- use pyhelp_print_obj function
- change i2c.scan() method to scan addresses 0x08-0x77
- fix typo of macro that detects if float is enabled

cc3200 port:
- use pyhelp_print_obj function
- change i2c.scan() method to scan addresses 0x08-0x77
- disable async/await syntax

esp8266 port:
- enable websocket module
- modnetwork: .config(): Check interface whose config is requested
- modnetwork: .config(): Add "authmode" param
- modnetwork: .config(): Add "password" param (W/O)
- add Python modules for initial configuration
- scripts/inisetup.py: set WPA/WPA2 AP mode with a predefined password
- protect modpyb.h header file from multiple inclusions
- implement basic C-level pin HAL
- switch from using custom I2C driver to generic extmod one
- enable framebuf module
- in callback helpers, pop nlr_buf on successful call
- esp_mphal: don't swallow exceptions in dupterm's read()/write()
- esp_mphal: call_dupterm_read(): Fix order of deactivating on EOF
- remove pin_id field from C pin object
- add dummy entries for non-existing pins to simplify pin logic
- add hard IRQ callbacks for pin change on GPIO0-15
- separate 1-wire timing funcs from Python module to save iRAM
- switch integer arith routines to BootROM
- scripts/port_diag.py: module to collect diagnostic info
- enable input() builtin
- scripts/_boot: mount block device on "" instead of "/"
- moduos: add uos.mkdir function
- modmachine: add reset_cause() function
- adapt port to use new auto-qstr generation
- moduos: add dupterm_notify() function
- esp_mphal: protect dupterm_task_handler() from recursive exec
- README.md: update feature list for current state of affairs
- modesp: allow esp.deepsleep to take 2nd arg for RF wake opt
- scripts/flashbdev: use all available space in 1MB FlashROM for FS
- modesp: add flash_size() function
- scripts: don't try to create filesystem on 512KB devices or less
- modnetwork: .config(): Add "channel" param
- scripts/flashbdev: disable debug output/checks
- scripts/_boot: print notice when initial setup is executed
- scripts/flashbdev: correct bootloader flash size to match real size
- modnetwork: .config(): Add "hidden ESSID" param
- implement basic deep-sleep capabilities
- add uart_rx_wait and uart_rx_char functions
- implement UART.read functionality
- uart: remove obsolete UART rx buffering code
- esp_mphal: remove mp_hal_feed_watchdog
- convert mp_hal_pin_obj_t from pin ptr to simple integer
- allow GPIO16 to be used as a pin in the uPy pin HAL
- change software SPI driver to use general pin HAL
- scripts/websocket_helper: module encapsulating handshake sequences
- scripts/websocket_helper: disable debug output
- scripts/webrepl: webREPL based on C-level websocket object
- scripts/webrepl: convert to persistent daemon
- scripts/webrepl: don't start on import
- scripts/webrepl: allow to override port
- scripts/webrepl: print connection address
- scripts/webrepl: print client address for incoming connections
- scripts/flashbdev: use all available Flash for filesystem
- scripts/webrepl: add "ws://" to "daemon started at" message
- Makefile: add target to build axTLS
- esp8266.ld: put axTLS to FlashROM
- Makefile: override abort() when building axtls
- axtls_helpers: helper/wrapper functions for axTLS
- tests/onewire.py: don't run test on import
- Makefile: support linking with axTLS built from source
- Makefile: enable "ussl" module; axTLS should be built first using "make axtls"
- move pyb.info() function to esp module and remove pyb module
- move onewire.py, neopixel.py drivers from tests/ to scripts/
- scripts/onewire.py: simplify and improve 1-wire driver
- scripts/neopixel.py: remove test function from neopixel driver
- set suitable values for axtls's RT_MAX_PLAIN_LENGTH & RT_EXTRA
- README: add recently required step of 'make axtls'
- modnetwork: make WLAN.ifconfig() read/write
- help: implement help() builtin
- help: add cheatsheet for basic WiFi configuration
- enable WebREPL file transfer rate limiting
- enable webrepl module
- scripts: move all of initial setup to inisetup module
- scripts/inisetup: create default boot.py in filesystem
- scripts/webrepl: connection ack prompt is now printed by modwebrepl
- scripts/webrepl: switch to using _webrepl object wrapper
- scripts/webrepl: add "first connection" mode to setup password
- README: mention WebREPL
- Makefile: be sure to pass cross-compiling AR when building axtls
- scripts/webrepl_setup: reject too short passwords
- change platform name from ESP8266 to esp8266
- modnetwork: remove deprecated wifi_mode()
- esp_mphal: add ets_esf_free_bufs(), etc. functions
- modesp: add esf_free_bufs() debugging function
- modesp: add malloc() and free() functions
- scripts/inisetup: enable WebREPL auto-start on boot
- modnetwork: remove .mac() method, move to .config("mac")
- scripts/inisetup: update for nic.mac() method being gone
- scripts/neopixel.py: swap red and green in pixel accessor
- modpybpin: use enum+array instead of struct for parsing args
- modpybpin: use None instead of PULL_NONE for no-pull config
- modpybpin: make pin.irq() methods take keyword args
- tutorial: mention that esptool is available via pip
- scripts/ntptime: add simple NTP client

docs:
- esp: enumerate flash access functions
- machine: start to update for esp8266 port
- speed_python: clarify/generalize "Buffers" subsection
- speed_python: generalize "Floating point" subsection
- speed_python: add many more details on memoryviews
- machine: reset_cause() has been implemented for esp8266
- topindex.html: esp8266: Enable quickref/general on the main page
- esp8266/general: start "General information" for esp8266
- esp8266: add info about using deep-sleep mode to quickref
- esp8266/tutorial: add tutorial placeholder page
- ustruct: there's no complete "struct" module, only "ustruct" subset
- esp8266: include usocket module reference
- library/usocket: add link to CPython's socket module
- usocket: socket-specific exceptions are for WiPy only
- usocket: socket.IPPROTO_SEC is WiPy-specific
- usocket: describe address format once at the beginning
- network: esp8266: scan(): add note that bssid is bytes object
- library: group MicroPython-specific modules under separate heading
- library/index: move WiPy "micro-libraries" under corresponding heading
- library/index: esp8266 has the same set of stdlibs as pyboard/unix
- module "time" is actually "utime"
- library/index: make single section for "micro-ified" modules
- library/index: order sections from the most to least standard modules
- Makefile: default BUILDDIR based on MICROPY_PORT
- library: "os" module is actually "uos"
- fix uos and utime heading underlines to be the correct length
- library/utime: add more time functions for unix and esp8266 ports
- utime: describe time() peculiarities in MicroPython
- utime: describe sleep() peculiarities in MicroPython
- library: consistently use admonitions for CPython differences
- ubinascii: clean up grammar
- network: esp8266: Add wlan.ifconfig() method
- network: esp8266: Describe wlan.config() method
- make the short port names in the port/version sidebar lowercase
- esp8266: update quickref to reflect changes to 1-wire and NeoPixel
- library/utime: elaborate on epochs and calendar time maintenance
- esp8266/quickref: add info about WebREPL
- ustruct: document pack_into(), unpack_from()
- sys: document sys.modules
- ustruct: fix argument formatting
- sys: remove port-specific details from description of stdin/out/err
- sys: clarify description of sys.exit()
- sys: clean up print_exception() description
- sys: describe sys.implementation
- sys: describe sys.maxsize
- sys: describe sys.platform is port-neutral manner
- add _collections module reference
- add _io module reference
- _io and _collections were renamed to have standard "u" prefix
- library/machine.I2C: update to reflect ESP8266 implementation
- esp8266/general: webREPL is described in quickref for now
- esp8266/general: add more points to "Multitude of boards" section
- esp8266/general: fix list formatting
- esp8266/general: add techspec section
- esp8266/general: add "Boot process" section
- esp8266_contents: reference general and tutorial docs
- network: esp8266: MAC address is set via .config() method
- esp8266: add ESP8266 tutorial
- machine: generalize docs from just WiPy to other ports
- machine: move WiPy-specific hardware details to its general reference
- machine: more generic description of sleep's, WiPy details to its genref
- machine: idle() description generalization
- library/machine.Pin: update pin docs to reflect ESP8266 support
- esp8266/tutorial: update pins tutorial to reflect changes in API

examples:
- http_server_ssl.py: add HTTPS server example

1 files changed, 207 insertions, 0 deletions

diff --git a/docs/esp8266/tutorial/repl.rst b/docs/esp8266/tutorial/repl.rst
new file mode 100644
index 0000000000..078f31357c
--- /dev/null
+++ b/docs/esp8266/tutorial/repl.rst
@@ -0,0 +1,207 @@
+Getting a MicroPython REPL prompt
+=================================
+
+REPL stands for Read Evaluate Print Loop, and is the name given to the
+interactive MicroPython prompt that you can access on the ESP8266.  Using the
+REPL is by far the easiest way to test out your code and run commands.
+
+There are two ways to access the REPL: either via a wired connection through the
+UART serial port, or via WiFi.
+
+REPL over the serial port
+-------------------------
+
+The REPL is always available on the UART0 serial peripheral, which is connected
+to the pins GPIO1 for TX and GPIO3 for RX.  The baudrate of the REPL is 115200.
+If your board has a USB-serial convertor on it then you should be able to access
+the REPL directly from your PC.  Otherwise you will need to have a way of
+communicating with the UART.
+
+To access the prompt over USB-serial you need to use a terminal emulator program.
+On Windows TeraTerm is a good choice, on Mac you can use the built-in screen
+program, and Linux has picocom and minicom.  Of course, there are many other
+terminal programs that will work, so pick your favourite!
+
+For example, on Linux you can try running::
+
+    picocom /dev/ttyUSB0
+
+Once you have made the connection over the serial port you can test if it is
+working by hitting enter a few times.  You should see the Python REPL prompt,
+indicated by ``>>>``.
+
+WebREPL - a prompt over WiFi
+----------------------------
+
+WebREPL allows you to use the Python prompt over WiFi, connecting through a
+browser. The latest versions of Firefox and Chrome are supported.
+
+For your convinience, WebREPL client is hosted at
+`<http://micropython.org/webrepl>`__ . Alternatively, you can install it
+locally from the the GitHub repository
+`<https://github.com/micropython/webrepl>`__ .
+
+To use WebREPL connect your computer to the ESP8266's access point
+(MicroPython-xxxxxx, see the previous section about this).  If you have
+already reconfigured your ESP8266 to connect to a router then you can
+skip this part.
+
+Once you are on the same network as the ESP8266 you click the "Connect" button
+(if you are connecting via a router then you may need to change the IP address,
+by default the IP address is correct when connected to the ESP8266's access
+point).  If the connection succeeds then you should see a welcome message.
+
+On the first connection you need to set a password.  Make sure that the
+terminal widget is selected by clicking on it, and then follow prompts to
+type in your password twice (they should match each other).  Then ESP8266
+will then reboot with the password applied (the WiFi will go down but come
+back up again). Note that some modules may have troubles rebooting
+automatically and need reset button press or power cycle (do this if
+you don't see ESP8266 access point appearing in a minute or so).
+
+You should then click the "Connect" button again, and enter your password
+to connect.  If you type in the correct password you should get a prompt
+looking like ``>>>``.  You can now start typing Python commands!
+
+Using the REPL
+--------------
+
+Once you have a prompt you can start experimenting!  Anything you type at the
+prompt will be executed after you press the Enter key.  MicroPython will run
+the code that you enter and print the result (if there is one).  If there is an
+error with the text that you enter then an error message is printed.
+
+Try typing the following at the prompt::
+
+    >>> print('hello esp8266!')
+    hello esp8266!
+
+Note that you shouldn't type the ``>>>`` arrows, they are there to indicate that
+you should type the text after it at the prompt.  And then the line following is
+what the device should respond with.  In the end, once you have entered the text
+``print("hello esp8266!")`` and pressed the Enter key, the output on your screen
+should look exactly like it does above.
+
+If you already know some python you can now try some basic commands here.   For
+example::
+
+    >>> 1 + 2
+    3
+    >>> 1 / 2
+    0.5
+    >>> 12**34
+    4922235242952026704037113243122008064
+
+If your board has an LED attached to GPIO2 (the ESP-12 modules do) then you can
+turn it on and off using the following code::
+
+    >>> import machine
+    >>> pin = machine.Pin(2, machine.Pin.OUT)
+    >>> pin.high()
+    >>> pin.low()
+
+Note that ``high`` might turn the LED off and ``low`` might turn it on (or vice
+versa), depending on how the LED is wired on your board.
+
+Line editing
+~~~~~~~~~~~~
+
+You can edit the current line that you are entering using the left and right
+arrow keys to move the cursor, as well as the delete and backspace keys.  Also,
+pressing Home or ctrl-A moves the cursor to the start of the line, and pressing
+End or ctrl-E moves to the end of the line.
+
+Input history
+~~~~~~~~~~~~~
+
+The REPL remembers a certain number of previous lines of text that you entered
+(up to 8 on the ESP8266).  To recall previous lines use the up and down arrow
+keys.
+
+Tab completion
+~~~~~~~~~~~~~~
+
+Pressing the Tab key will do an auto-completion of the current word that you are
+entering.  This can be very useful to find out functions and methods that a
+module or object has.  Try it out by typing "ma" and then pressing Tab.  It
+should complete to "machine" (assuming you imported machine in the above
+example).  Then type "." and press Tab again to see a list of all the functions
+that the machine module has.
+
+Line continuation and auto-indent
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Certain things that you type will need "continuing", that is, will need more
+lines of text to make a proper Python statement.  In this case the prompt will
+change to ``...`` and the cursor will auto-indent the correct amount so you can
+start typing the next line straight away.  Try this by defining the following
+function::
+
+    >>> def toggle(p):
+    ...    p.value(not p.value())
+    ...    
+    ...    
+    ...    
+    >>>
+
+In the above, you needed to press the Enter key three times in a row to finish
+the compound statement (that's the three lines with just dots on them).  The
+other way to finish a compound statement is to press backspace to get to the
+start of the line, then press the Enter key.  (If you did something wrong and
+want to escape the continuation mode then press ctrl-C; all lines will be
+ignored.)
+
+The function you just defined allows you to toggle a pin.  The pin object you
+created earlier should still exist (recreate it if it doesn't) and you can
+toggle the LED using::
+
+    >>> toggle(pin)
+
+Let's now toggle the LED in a loop (if you don't have an LED then you can just
+print some text instead of calling toggle, to see the effect)::
+
+    >>> import time
+    >>> while True:
+    ...     toggle(pin)
+    ...     time.sleep_ms(500)
+    ...    
+    ...    
+    ...    
+    >>>
+
+This will toggle the LED at 1Hz (half a second on, half a second off).  To stop
+the toggling press ctrl-C, which will raise a KeyboardInterrupt exception and
+break out of the loop.
+
+The time module provides some useful functions for making delays and doing
+timing.  Use tab completion to find out what they are and play around with them!
+
+Paste mode
+~~~~~~~~~~
+
+Pressing ctrl-E will enter a special paste mode.  This allows you to copy and
+paste a chunk of text into the REPL.  If you press ctrl-E you will see the
+paste-mode prompt::
+
+    paste mode; Ctrl-C to cancel, Ctrl-D to finish
+    === 
+
+You can then paste (or type) your text in.  Note that none of the special keys
+or commands work in paste mode (eg Tab or backspace), they are just accepted
+as-is.  Press ctrl-D to finish entering the text and execute it.
+
+Other control commands
+~~~~~~~~~~~~~~~~~~~~~~
+
+There are four other control commands:
+
+* Ctrl-A on a blank line will enter raw REPL mode.  This is like a permanent
+  paste mode, except that characters are not echoed back.
+
+* Ctrl-B on a blank like goes to normal REPL mode.
+
+* Ctrl-C cancels any input, or interrupts the currently running code.
+
+* Ctrl-D on a blank line will do a soft reset.
+
+Note that ctrl-A and ctrl-D do not work with WebREPL.