String
to a list of {@link Token}s.
*/
public class Lexer {
private static final MapLexer
that will split the given source code into {@link Token}s.
+ *
+ * @param source a String
containing the DSL source code to process.
+ * @param errorReporter a Consumer
used to report each DslError
that the
+ * Lexer
finds within the source string.
+ */
public Lexer(final String source, final Consumer super DslError> errorReporter) {
this.source = source;
this.errorReporter = errorReporter;
}
+ /**
+ * Runs the Lexer
.
+ *
+ * This method can only be called once per instance.
+ *
+ * @return the list of
+ * This method can only be called once per instance.
+ *
+ * @return the list of all valid
+ *
+ * Some types of token always have the same lexemes (for example,
+ * As a special case, tokens of type Token
s extracted from the source string.
+ * If any errors are reported, this list should not be considered to represent a valid set of DSL rules,
+ * but it can still be parsed to potentially find additional errors (which may allow the user to fix more
+ * errors before having to rerun the Lexer
).
+ */
public ListPatternRule
s.
+ * Converts a list of {@link Token}s to a list of {@link PatternRule}s.
*/
public class Parser {
private static final MapParser
that will produce a list of {@link PatternRule}s from the the given list of {@link Token}s.
+ *
+ * @param tokens the list of Token
s to process.
+ * @param errorReporter a Consumer
used to report each DslError
that the
+ * Parser
finds within the source string.
+ */
public Parser(final ListParser
.
+ * PatternRule
s constructed from the given tokens.
+ * If any errors are reported, this list should not be considered to represent a valid set of DSL rules,
+ * but each rule can still be analyzed to look for undefined sub-functions in replacement patterns and
+ * report them (which may allow the user to fix more errors before having to rerun the Lexer
+ * and Parser
).
+ */
public ListToken
s are produced by the {@link Lexer} and consumed by the {@link Parser}.
+ */
public class Token {
/** The type of the token. */
public final TokenType type;
- /** The source string which corresponds to the token. */
+ /**
+ * The part of the source code which corresponds to the token.
+ * PLUS
is always represented by
+ * "+"
), while others have variable lexemes (like IDENTIFIER
, which may correspond to any
+ * valid identifier).
+ * EOF
(which signal the end of the source code) have empty lexemes
+ * (""
). Such tokens only exist to simplify the parser code, by allowing the end of the input to be
+ * treated like any other token (which is especially useful for error handling, because an unexpected end of input
+ * just becomes an "unexpected token" error).
+ */
public final String lexeme;
/** The index at which the token starts in the source string. */
public final int position;
+ /**
+ * Constructs a Token
.
+ *
+ * @param type the type of the token.
+ * @param lexeme the part of the source string which corresponds to the token.
+ * @param position the index at which the token starts in the source string.
+ */
public Token(final TokenType type, final String lexeme, final int position) {
this.type = type;
this.lexeme = lexeme;
diff --git a/core/src/main/java/it/cavallium/warppi/math/rules/dsl/frontend/TokenType.java b/core/src/main/java/it/cavallium/warppi/math/rules/dsl/frontend/TokenType.java
index 55aee670..769eb45b 100644
--- a/core/src/main/java/it/cavallium/warppi/math/rules/dsl/frontend/TokenType.java
+++ b/core/src/main/java/it/cavallium/warppi/math/rules/dsl/frontend/TokenType.java
@@ -1,5 +1,8 @@
package it.cavallium.warppi.math.rules.dsl.frontend;
+/**
+ * Specifies the type of a Token
.
+ */
public enum TokenType {
EOF,
// Separators and grouping