py/lexer: Allow conversion specifiers in f-strings (e.g. !r).

PEP-498 allows for conversion specifiers like !r and !s to convert the
expression declared in braces to be passed through repr() and str()
respectively.

This updates the logic that detects the end of the expression to also stop
when it sees "![rs]" that is either at the end of the f-string or before
the ":" indicating the start of the format specifier. The "![rs]" is now
retained in the format string, whereas previously it stayed on the end
of the expression leading to a syntax error.

Previously: `f"{x!y:z}"` --> `"{:z}".format(x!y)`
Now: `f"{x!y:z}"` --> `"{!y:z}".format(x)`

Note that "!a" is not supported by `str.format` as MicroPython has no
`ascii()`, but now this will raise the correct error.

Updated cpydiff and added tests.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>

1 files changed, 19 insertions, 0 deletions

diff --git a/tests/basics/string_fstring.py b/tests/basics/string_fstring.py
index 8907a5c478..1a3960680e 100644
--- a/tests/basics/string_fstring.py
+++ b/tests/basics/string_fstring.py
@@ -61,3 +61,22 @@ except (ValueError, SyntaxError):
 print(f"a {1,} b")
 print(f"a {x,y,} b")
 print(f"a {x,1} b")
+
+# f-strings with conversion specifiers (only support !r and !s).
+a = "123"
+print(f"{a!r}")
+print(f"{a!s}")
+try:
+    eval('print(f"{a!x}")')
+except (ValueError, SyntaxError):
+    # CPython detects this at compile time, MicroPython fails with ValueError
+    # when the str.format is executed.
+    print("ValueError")
+
+# Mixing conversion specifiers with formatting.
+print(f"{a!r:8s}")
+print(f"{a!s:8s}")
+
+# Still allow ! in expressions.
+print(f"{'1' if a != '456' else '0'!r:8s}")
+print(f"{'1' if a != '456' else '0'!s:8s}")