docs: Add littlefs docs and a filesystem tutorial.

4 files changed, 322 insertions, 63 deletions

diff --git a/docs/library/esp32.rst b/docs/library/esp32.rst
index a593965ae2..68379624e6 100644
--- a/docs/library/esp32.rst
+++ b/docs/library/esp32.rst
@@ -56,10 +56,14 @@ This class gives access to the partitions in the device's flash memory.
     Returns a 6-tuple ``(type, subtype, addr, size, label, encrypted)``.
 
 .. method:: Partition.readblocks(block_num, buf)
+.. method:: Partition.readblocks(block_num, buf, offset)
 .. method:: Partition.writeblocks(block_num, buf)
+.. method:: Partition.writeblocks(block_num, buf, offset)
 .. method:: Partition.ioctl(cmd, arg)
 
-    These methods implement the block protocol defined by :class:`uos.AbstractBlockDev`.
+    These methods implement the simple and :ref:`extended
+    <block-device-interface>` block protocol defined by
+    :class:`uos.AbstractBlockDev`.
 
 .. method:: Partition.set_boot()
 
diff --git a/docs/library/uos.rst b/docs/library/uos.rst
index d8d8b7a972..84d341ac21 100644
--- a/docs/library/uos.rst
+++ b/docs/library/uos.rst
@@ -178,6 +178,35 @@ represented by VFS classes.
 
         Build a FAT filesystem on *block_dev*.
 
+.. class:: VfsLfs1(block_dev)
+
+    Create a filesystem object that uses the `littlefs v1 filesystem format`_.
+    Storage of the littlefs filesystem is provided by *block_dev*, which must
+    support the :ref:`extended interface <block-device-interface>`.
+    Objects created by this constructor can be mounted using :func:`mount`.
+
+    See :ref:`filesystem` for more information.
+
+    .. staticmethod:: mkfs(block_dev)
+
+        Build a Lfs1 filesystem on *block_dev*.
+
+.. class:: VfsLfs2(block_dev)
+
+    Create a filesystem object that uses the `littlefs v2 filesystem format`_.
+    Storage of the littlefs filesystem is provided by *block_dev*, which must
+    support the :ref:`extended interface <block-device-interface>`.
+    Objects created by this constructor can be mounted using :func:`mount`.
+
+    See :ref:`filesystem` for more information.
+
+    .. staticmethod:: mkfs(block_dev)
+
+        Build a Lfs2 filesystem on *block_dev*.
+
+.. _littlefs v1 filesystem format: https://github.com/ARMmbed/littlefs/tree/v1
+.. _littlefs v2 filesystem format: https://github.com/ARMmbed/littlefs
+
 Block devices
 -------------
 
@@ -187,9 +216,15 @@ implementation of this class will usually allow access to the memory-like
 functionality a piece of hardware (like flash memory).  A block device can be
 used by a particular filesystem driver to store the data for its filesystem.
 
+.. _block-device-interface:
+
+Simple and extended interface
+.............................
+
 There are two compatible signatures for the ``readblocks`` and ``writeblocks``
 methods (see below), in order to support a variety of use cases.  A given block
-device may implement one form or the other, or both at the same time.
+device may implement one form or the other, or both at the same time. The second
+form (with the offset parameter) is referred to as the "extended interface".
 
 .. class:: AbstractBlockDev(...)
 
@@ -247,64 +282,5 @@ device may implement one form or the other, or both at the same time.
             (*arg* is unused)
           - 6 -- erase a block, *arg* is the block number to erase
 
-By way of example, the following class will implement a block device that stores
-its data in RAM using a ``bytearray``::
-
-    class RAMBlockDev:
-        def __init__(self, block_size, num_blocks):
-            self.block_size = block_size
-            self.data = bytearray(block_size * num_blocks)
-
-        def readblocks(self, block_num, buf):
-            for i in range(len(buf)):
-                buf[i] = self.data[block_num * self.block_size + i]
-
-        def writeblocks(self, block_num, buf):
-            for i in range(len(buf)):
-                self.data[block_num * self.block_size + i] = buf[i]
-
-        def ioctl(self, op, arg):
-            if op == 4: # get number of blocks
-                return len(self.data) // self.block_size
-            if op == 5: # get block size
-                return self.block_size
-
-It can be used as follows::
-
-    import uos
-
-    bdev = RAMBlockDev(512, 50)
-    uos.VfsFat.mkfs(bdev)
-    vfs = uos.VfsFat(bdev)
-    uos.mount(vfs, '/ramdisk')
-
-An example of a block device that supports both signatures and behaviours of
-the :meth:`readblocks` and :meth:`writeblocks` methods is::
-
-    class RAMBlockDev:
-        def __init__(self, block_size, num_blocks):
-            self.block_size = block_size
-            self.data = bytearray(block_size * num_blocks)
-
-        def readblocks(self, block, buf, offset=0):
-            addr = block_num * self.block_size + offset
-            for i in range(len(buf)):
-                buf[i] = self.data[addr + i]
-
-        def writeblocks(self, block_num, buf, offset=None):
-            if offset is None:
-                # do erase, then write
-                for i in range(len(buf) // self.block_size):
-                    self.ioctl(6, block_num + i)
-                offset = 0
-            addr = block_num * self.block_size + offset
-            for i in range(len(buf)):
-                self.data[addr + i] = buf[i]
-
-        def ioctl(self, op, arg):
-            if op == 4: # block count
-                return len(self.data) // self.block_size
-            if op == 5: # block size
-                return self.block_size
-            if op == 6: # block erase
-                return 0
+See :ref:`filesystem` for example implementations of block devices using both
+protocols.
diff --git a/docs/reference/filesystem.rst b/docs/reference/filesystem.rst
new file mode 100644
index 0000000000..71d34e7954
--- /dev/null
+++ b/docs/reference/filesystem.rst
@@ -0,0 +1,278 @@
+.. _filesystem:
+
+Working with filesystems
+========================
+
+.. contents::
+
+This tutorial describes how MicroPython provides an on-device filesystem,
+allowing standard Python file I/O methods to be used with persistent storage.
+
+MicroPython automatically creates a default configuration and auto-detects the
+primary filesystem, so this tutorial will be mostly useful if you want to modify
+the partitioning, filesystem type, or use custom block devices.
+
+The filesystem is typically backed by internal flash memory on the device, but
+can also use external flash, RAM, or a custom block device.
+
+On some ports (e.g. STM32), the filesystem may also be available over USB MSC to
+a host PC. :ref:`pyboard_py` also provides a way for the host PC to access to
+the filesystem on all ports.
+
+Note: This is mainly for use on bare-metal ports like STM32 and ESP32. On ports
+with an operating system (e.g. the Unix port) the filesystem is provided by the
+host OS.
+
+VFS
+---
+
+MicroPython implements a Unix-like Virtual File System (VFS) layer. All mounted
+filesystems are combined into a single virtual filesystem, starting at the root
+``/``. Filesystems are mounted into directories in this structure, and at
+startup the working directory is changed to where the primary filesystem is
+mounted.
+
+On STM32 / Pyboard, the internal flash is mounted at ``/flash``, and optionally
+the SDCard at ``/sd``. On ESP8266/ESP32, the primary filesystem is mounted at
+``/``.
+
+Block devices
+-------------
+
+A block device is an instance of a class that implements the
+:class:`uos.AbstractBlockDev` protocol.
+
+Built-in block devices
+~~~~~~~~~~~~~~~~~~~~~~
+
+Ports provide built-in block devices to access their primary flash.
+
+On power-on, MicroPython will attempt to detect the filesystem on the default
+flash and configure and mount it automatically. If no filesystem is found,
+MicroPython will attempt to create a FAT filesystem spanning the entire flash.
+Ports can also provide a mechanism to "factory reset" the primary flash, usually
+by some combination of button presses at power on.
+
+STM32 / Pyboard
+...............
+
+The :ref:`pyb.Flash <pyb.Flash>` class provides access to the internal flash. On some
+boards which have larger external flash (e.g. Pyboard D), it will use that
+instead. The ``start`` kwarg should always be specified, i.e.
+``pyb.Flash(start=0)``.
+
+Note: For backwards compatibility, when constructed with no arguments (i.e.
+``pyb.Flash()``), it only implements the simple block interface and reflects the
+virtual device presented to USB MSC (i.e. it includes a virtual partition table
+at the start).
+
+ESP8266
+.......
+
+The internal flash is exposed as a block device object which is created in the
+``flashbdev`` module on start up. This object is by default added as a global
+variable so it can usually be accessed simply as ``bdev``. This implements the
+extended interface.
+
+ESP32
+.....
+
+The :class:`esp32.Partition` class implements a block device for partitions
+defined for the board. Like ESP8266, there is a global variable ``bdev`` which
+points to the default partition. This implements the extended interface.
+
+Custom block devices
+~~~~~~~~~~~~~~~~~~~~
+
+The following class implements a simple block device that stores its data in
+RAM using a ``bytearray``::
+
+    class RAMBlockDev:
+        def __init__(self, block_size, num_blocks):
+            self.block_size = block_size
+            self.data = bytearray(block_size * num_blocks)
+
+        def readblocks(self, block_num, buf):
+            for i in range(len(buf)):
+                buf[i] = self.data[block_num * self.block_size + i]
+
+        def writeblocks(self, block_num, buf):
+            for i in range(len(buf)):
+                self.data[block_num * self.block_size + i] = buf[i]
+
+        def ioctl(self, op, arg):
+            if op == 4: # get number of blocks
+                return len(self.data) // self.block_size
+            if op == 5: # get block size
+                return self.block_size
+
+It can be used as follows::
+
+    import os
+
+    bdev = RAMBlockDev(512, 50)
+    os.VfsFat.mkfs(bdev)
+    os.mount(bdev, '/ramdisk')
+
+An example of a block device that supports both the simple and extended
+interface (i.e. both signatures and behaviours of the
+:meth:`uos.AbstractBlockDev.readblocks` and
+:meth:`uos.AbstractBlockDev.writeblocks` methods) is::
+
+    class RAMBlockDev:
+        def __init__(self, block_size, num_blocks):
+            self.block_size = block_size
+            self.data = bytearray(block_size * num_blocks)
+
+        def readblocks(self, block, buf, offset=0):
+            addr = block_num * self.block_size + offset
+            for i in range(len(buf)):
+                buf[i] = self.data[addr + i]
+
+        def writeblocks(self, block_num, buf, offset=None):
+            if offset is None:
+                # do erase, then write
+                for i in range(len(buf) // self.block_size):
+                    self.ioctl(6, block_num + i)
+                offset = 0
+            addr = block_num * self.block_size + offset
+            for i in range(len(buf)):
+                self.data[addr + i] = buf[i]
+
+        def ioctl(self, op, arg):
+            if op == 4: # block count
+                return len(self.data) // self.block_size
+            if op == 5: # block size
+                return self.block_size
+            if op == 6: # block erase
+                return 0
+
+As it supports the extended interface, it can be used with :class:`littlefs
+<uos.VfsLfs2>`::
+
+    import os
+
+    bdev = RAMBlockDev(512, 50)
+    os.VfsLfs2.mkfs(bdev)
+    os.mount(bdev, '/ramdisk')
+
+Filesystems
+-----------
+
+MicroPython ports can provide implementations of :class:`FAT <uos.VfsFat>`,
+:class:`littlefs v1 <uos.VfsLfs1>` and :class:`littlefs v2 <uos.VfsLfs2>`.
+
+The following table shows which filesystems are included in the firmware by
+default for given port/board combinations, however they can be optionally
+enabled in a custom firmware build.
+
+====================  =====  ===========  ===========
+Board                 FAT    littlefs v1  littlefs v2
+====================  =====  ===========  ===========
+pyboard 1.0, 1.1, D   Yes    No           Yes
+Other STM32           Yes    No           No
+ESP8266               Yes    No           No
+ESP32                 Yes    No           Yes
+====================  =====  ===========  ===========
+
+FAT
+~~~
+
+The main advantage of the FAT filesystem is that it can be accessed over USB MSC
+on supported boards (e.g. STM32) without any additional drivers required on the
+host PC.
+
+However, FAT is not tolerant to power failure during writes and this can lead to
+filesystem corruption. For applications that do not require USB MSC, it is
+recommended to use littlefs instead.
+
+To format the entire flash using FAT::
+
+    # ESP8266 and ESP32
+    import os
+    os.umount('/')
+    os.VfsFat.mkfs(bdev)
+    os.mount(bdev, '/')
+
+    # STM32
+    import os, pyb
+    os.umount('/flash')
+    os.VfsFat.mkfs(pyb.Flash(start=0))
+    os.mount(pyb.Flash(start=0), '/flash')
+    os.chdir('/flash')
+
+Littlefs
+~~~~~~~~
+
+Littlefs_ is a filesystem designed for flash-based devices, and is much more
+resistant to filesystem corruption.
+
+Note: It can be still be accessed over USB MSC using the `littlefs FUSE
+driver`_. Note that you must use the ``-b=4096`` option to override the block
+size.
+
+.. _littlefs FUSE driver: https://github.com/ARMmbed/littlefs-fuse/tree/master/littlefs
+
+.. _Littlefs: https://github.com/ARMmbed/littlefs
+
+To format the entire flash using littlefs v2::
+
+    # ESP8266 and ESP32
+    import os
+    os.umount('/')
+    os.VfsLfs2.mkfs(bdev)
+    os.mount(bdev, '/')
+
+    # STM32
+    import os, pyb
+    os.umount('/flash')
+    os.VfsLfs2.mkfs(pyb.Flash(start=0))
+    os.mount(pyb.Flash(start=0), '/flash')
+    os.chdir('/flash')
+
+Hybrid (STM32)
+~~~~~~~~~~~~~~
+
+By using the ``start`` and ``len`` kwargs to :class:`pyb.Flash`, you can create
+block devices spanning a subset of the flash device.
+
+For example, to configure the first 256kiB as FAT (and available over USB MSC),
+and the remainder as littlefs::
+
+    import os, pyb
+    os.umount('/flash')
+    p1 = pyb.Flash(start=0, len=256*1024)
+    p2 = pyb.Flash(start=256*1024)
+    os.VfsFat.mkfs(p1)
+    os.VfsLfs2.mkfs(p2)
+    os.mount(p1, '/flash')
+    os.mount(p2, '/data')
+    os.chdir('/flash')
+
+This might be useful to make your Python files, configuration and other
+rarely-modified content available over USB MSC, but allowing for frequently
+changing application data to reside on littlefs with better resilience to power
+failure, etc.
+
+The partition at offset ``0`` will be mounted automatically (and the filesystem
+type automatically detected), but you can add::
+
+    import os, pyb
+    p2 = pyb.Flash(start=256*1024)
+    os.mount(p2, '/data')
+
+to ``boot.py`` to mount the data partition.
+
+Hybrid (ESP32)
+~~~~~~~~~~~~~~
+
+On ESP32, if you build custom firmware, you can modify ``partitions.csv`` to
+define an arbitrary partition layout.
+
+At boot, the partition named "vfs" will be mounted at ``/`` by default, but any
+additional partitions can be mounted in your ``boot.py`` using::
+
+    import esp32, os
+    p = esp32.Partition.find(esp32.Partition.TYPE_DATA, label='foo')
+    os.mount(p, '/foo')
+
diff --git a/docs/reference/index.rst b/docs/reference/index.rst
index 4dd52b9c83..1eaaa85c85 100644
--- a/docs/reference/index.rst
+++ b/docs/reference/index.rst
@@ -26,4 +26,5 @@ implementation and the best practices to use them.
    constrained.rst
    packages.rst
    asm_thumb2_index.rst
+   filesystem.rst
    pyboard.py.rst