From 7bd81a6a03a659ee46eb79533b6d37549dc9ecdd Mon Sep 17 00:00:00 2001 From: Matthew Woodcraft Date: Mon, 4 Mar 2024 21:32:01 +0000 Subject: [PATCH] tokens.md: say that lifetime-like tokens can't be immediately followed by ' Forms like 'ab'c are rejected, so we need some way to explain why they don't tokenise as two consecutive LIFETIME_OR_LABEL tokens. Address this by adding "not immediately followed by `'`" to each of the lexer rules for the lifetime-like tokens. This also means there can be no ambiguity between CHAR_LITERAL and these tokens (at present we don't say how such ambiguities are resolved). --- src/tokens.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/tokens.md b/src/tokens.md index cf76e72..eded55e 100644 --- a/src/tokens.md +++ b/src/tokens.md @@ -630,11 +630,14 @@ Examples of reserved forms: > **Lexer**\ > LIFETIME_TOKEN :\ ->       `'` [IDENTIFIER_OR_KEYWORD][identifier]\ +>       `'` [IDENTIFIER_OR_KEYWORD][identifier] +> _(not immediately followed by `'`)_\ >    | `'_` +> _(not immediately followed by `'`)_ > > LIFETIME_OR_LABEL :\ >       `'` [NON_KEYWORD_IDENTIFIER][identifier] +> _(not immediately followed by `'`)_ Lifetime parameters and [loop labels] use LIFETIME_OR_LABEL tokens. Any LIFETIME_TOKEN will be accepted by the lexer, and for example, can be used in