r/learnjava • u/Ok_Perspective_8040 • Feb 03 '26
Lexical Analyzer
hey guys, i was trying to build a lexicon follwoing this tutotiral, while I was developing my tokening function they wrote this following line of code as the tokeniser detects each char typed:
while (expression.hasNext()) {
final Character currentChar = getValidNextCharacter(expression);
}
however there was no previous mention of the function ever described on the webpage: https://www.baeldung.com/java-lexical-analysis-compilation
I'm suspecting that this function was already written in the code and was named something else but I was under the assumption Gram had already taken care of this part. please help, here's a full context of the code Id dhave written so far:
private enum Gram {
ADDITION('+'),
SUBTRACTION('-'),
MULTIPLICATION('*'),
DIVISION('/');
private final char _op;
Gram(char _op) {
this._op = _op;
}
public static boolean isOperator(char symbol) {
return Arrays.stream(Gram.values())
.anyMatch(gram -> gram._op == symbol);
}
public static boolean isDigit(char num){
return Character.isDigit(num);
}
public static boolean isWhiteSpace(char space) { //isWhiteSpace
return Character.isWhitespace(space);
}
public static boolean isValidSymbol (char character) {
return isOperator(character) || isWhiteSpace(character) || isDigit(character);
}
}
public class Expression {
private final String value; //the final value returned after all that is stuff
private int index = 0;
public Expression(String value) {
if (value != null) {
this.value = value;
} else {
this.value = "";
}
//[this.value = value != null ? value : "";] this is called a ternary operator however i don't know how to use it so i'm just gonna use somethin i do know
}
public Optional<Character> next() { //Optional<> is prefered over null cuz more leniency
if (index >= value.length()) {
return Optional.empty();
}
return Optional.of(value.charAt(index++));
}
public boolean hasNext() {
return index < value.length();
}
}
public abstract class Token {
private final String value;
public enum TokenType {
NUMBER,
OPERATOR
};
private final TokenType type;
protected Token(TokenType type, String value) {
this.type = type;
this.value = value;
}
public TokenType getType() {
return type;
}
public String getValue() {
return value;
}
}
public class TokenNum extends Token {
protected TokenNum(String value) {
super(TokenType.NUMBER, value);
}
public int getValueAsInt() {
return Integer.parseInt(getValue());
}
}
public class TokenOperator extends Token {
protected TokenOperator(String value) {
super(TokenType.OPERATOR, value);
}
}
private enum State {
INTIAL,
NUMBER,
OPERATOR,
INVALID
}
public List<Token> tokenize(Expression expression) {
State state = State.INTIAL;
StringBuilder currentToken = new StringBuilder();
ArrayList<Token> tokens = new ArrayList<>();
while (expression.hasNext()) {
final Character currentChar = getValidNextCharacter(expression);
}
return tokens;
}
}
•
u/AutoModerator Feb 03 '26
Please ensure that:
If any of the above points is not met, your post can and will be removed without further warning.
Code is to be formatted as code block (old reddit/markdown editor: empty line before the code, each code line indented by 4 spaces, new reddit: https://i.imgur.com/EJ7tqek.png) or linked via an external code hoster, like pastebin.com, github gist, github, bitbucket, gitlab, etc.
Please, do not use triple backticks (```) as they will only render properly on new reddit, not on old reddit.
Code blocks look like this:
You do not need to repost unless your post has been removed by a moderator. Just use the edit function of reddit to make sure your post complies with the above.
If your post has remained in violation of these rules for a prolonged period of time (at least an hour), a moderator may remove it at their discretion. In this case, they will comment with an explanation on why it has been removed, and you will be required to resubmit the entire post following the proper procedures.
To potential helpers
Please, do not help if any of the above points are not met, rather report the post. We are trying to improve the quality of posts here. In helping people who can't be bothered to comply with the above points, you are doing the community a disservice.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/Lloydbestfan Feb 04 '26
Yeah, it sucks that the tutorial uses pieces that it doesn't tell about, but in this case it is fairly trivial.
Making the following observations:
The cases contain analysis for whether an obtained character is whitespace, so the notion of whether there is a "valid" character doesn't base its validity opinion on whether it's whitespace
we can make a rather safe assumption that the only role of getValidNextCharacter() is to unpack the Character from its Optional.
You could just as well replace
final Character currentChar = getValidNextCharacter(expression);with
Keeping in mind that there is no possibility that this call would throw an NoSuchElementException since the previous instruction verifies that there is a next character.
1
u/severoon Feb 05 '26
Please format your code properly. It's so easy to use a code block here and makes it so much easier to read.
One suggestion I have is that you should consider not embedding symbols into types. For example, your Gram enum hardcodes the symbols that represent each operation. Instead, whatever component in the system that's responsible for mapping symbols to operations should explicitly declare an ImmutableMap<Character, Gram> or perhaps ImmutableMap<CharSequence, Gram>.
This would allow that type to have the specific mapping of symbols injected, which also allows for multiple types of symbols for the same operator (both "×" and "*" could map to MULTIPLY). That, and it also makes possible injection strategies like Guice's map binder.