py: Implement partial PEP-498 (f-string) support.

This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.

It is implemented in the lexer as a syntax translation to `str.format`:
  f"{a}" --> "{}".format(a)

It also supports:
  f"{a=}" --> "a={}".format(a)

This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.

There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
  `SyntaxError: raw f-strings are not supported`.

- literal concatenation of f-strings with adjacent strings will fail
    "{}" f"{a}" --> "{}{}".format(a)    (str.format will incorrectly use
                                         the braces from the non-f-string)
    f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)

- PEP-498 requires the full parser to understand the interpolated
  argument, however because this entirely runs in the lexer it cannot
  resolve nested braces in expressions like
    f"{'}'}"

- The !r, !s, and !a conversions are not supported.

Includes tests and cpydiffs.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>

1 files changed, 57 insertions, 0 deletions

diff --git a/tests/basics/string_fstring.py b/tests/basics/string_fstring.py
new file mode 100644
index 0000000000..efb7e5a8e1
--- /dev/null
+++ b/tests/basics/string_fstring.py
@@ -0,0 +1,57 @@
+def f():
+    return 4
+def g(_):
+    return 5
+def h():
+    return 6
+
+print(f'no interpolation')
+print(f"no interpolation")
+print(f"""no interpolation""")
+
+x, y = 1, 2
+print(f'{x}')
+print(f'{x:08x}')
+print(f'{x=}')
+print(f'{x=:08x}')
+print(f'a {x} b {y} c')
+print(f'a {x:08x} b {y} c')
+print(f'a {x=} b {y} c')
+print(f'a {x=:08x} b {y} c')
+
+print(f'a {"hello"} b')
+print(f'a {f() + g("foo") + h()} b')
+print(f'a {f() + g("foo") + h()=} b')
+print(f'a {f() + g("foo") + h()=:08x} b')
+
+def foo(a, b):
+    return f'{x}{y}{a}{b}'
+print(foo(7, 8))
+
+# PEP-0498 specifies that '\\' and '#' must be disallowed explicitly, whereas
+# MicroPython relies on the syntax error as a result of the substitution.
+
+print(f"\\")
+print(f'#')
+try:
+    eval("f'{\}'")
+except SyntaxError:
+    print('SyntaxError')
+try:
+    eval("f'{#}'")
+except SyntaxError:
+    print('SyntaxError')
+
+
+# PEP-0498 specifies that handling of double braces '{{' or '}}' should
+# behave like str.format.
+print(f'{{}}')
+print(f'{{{4*10}}}', '{40}')
+
+# A single closing brace, unlike str.format should raise a syntax error.
+# MicroPython instead raises ValueError at runtime from the substitution.
+try:
+    eval("f'{{}'")
+except (ValueError, SyntaxError):
+    # MicroPython incorrectly raises ValueError here.
+    print('SyntaxError')