Tokenize The Code

Statement

Goal

Wikipedia provides an article about tokenizing (lexical analyzing): https://en.wikipedia.org/wiki/Lexical_analysis
Info (symbols enclosed in double quotes: "):
A comma is the symbol: ","
Space/whitespace is the empty character: " "
Decimal digits are: 0123456789, including a float (0.55 as an example)

Tokenize the numbers and symbols/operators using this reference:

[LIT:NUM, n], [OPR, o], [LPAREN | RPAREN]
n: a decimal
o: an operator, any of the following: + | - | / | * | ^ | %

note: LPAREN and RPAREN is equal to ( and )

Input

Variable string S: the string to tokenize
e.g. Input is

(5+3)*3

Output

String result: the result of the tokenizer
e.g. Output is

[LPAREN] [LIT:NUM, 5] [OPR, +] [LIT:NUM, 3] [RPAREN] [OPR, *] [LIT:NUM, 3]

-All tokens are made of square brackets- "[]" -that contains information: [token_type{s}, token_value]
-Token type(s) and value must be separated by a comma and a space (representation enclosed in double quotes): ", "
-Tokens are separated by a space: " "
- Return values are rounded to 2 decimal digits after the floating point if it's a decimal, rounded to no decimal digits if it's an integer. Examples:
5 -> [LIT:NUM, 5]
3.234 -> [LIT:NUM, 3.23]
7.9 -> [LIT:NUM, 7.90]

Constraints

Ignore whitespace

Example

Input

(5+3)*3

Output

[LPAREN] [LIT:NUM, 5] [OPR, +] [LIT:NUM, 3] [RPAREN] [OPR, *] [LIT:NUM, 3]

Comments

10 Comments

CodinBot Unnamed CodinGamer

about 3 years ago

Hey CodinGamer! Your contribution has unfortunately not been moderated on time and will temporarily be removed from the system... But no worries! You can republish it anytime to give it another chance! :)

k14 Unnamed CodinGamer

I'm unsure whether it's acceptable nor intended but I was able to pass all the tests only by regex-replacing parts of the input.

alze Unnamed CodinGamer

over 3 years ago

it is just a "sed" exercise. a "real" lexer should include at least one conflict case. Simple conflicts cases : a = b => assignment a == b => comparison, and not two assignment operators C++ templates are a good example of complex conflict cases (not necessary solved by lexer) cin >> value : >> is an operator vector<pair<int,int>> : >> is two ">"

mfnsvrtm Unnamed CodinGamer

That's true for "real" lexers, but this is clearly more of a toy. This exercise could serve as an introduction to lexing for someone. Why not?

cg123 Unnamed CodinGamer

In the 3th test you transform a 3.5 value into [LIT:NUM, 3.50] I don't think it's the purpose of a tokenizer to change the value's format. But at least, provide the expected format. Also, it's definitely not a hard puzzle.

SPDene Unnamed CodinGamer

I think you are assuming too much specific knowledge. For someone totally new to lexical analysis, I would include: - a definition/description of "tokenize", with examples and/or a link to further information - how to read the "reference". It's not too complicated if you know the meanings of the square brackets, commas and pipes, but could be quite offputting for someone unfamiliar with the syntax. --- Your first three "Constraints" should be in the "Output" section: -All tokens are made of square brackets- [] -that contains information: [token_type{s}, token_value] -Token type(s) and value must be separated by a comma and a space: ", " -Tokens are separated by a space --- '-Power operator is "^"' is irrelevant; you don't need to know the function of the ^ operator to change it to [OPR, ^] --- "All numbers in the output strings are floating point numbers" You should specify the required number of decimal places, and any rounding rules. Although your tests never have more than one decimal place in the input, your solution would give exactly one (rounded) decimal place, resulting in: 12 -> [LIT:NUM, 12.0] 1.23 -> [LIT:NUM, 1.2] 7.85 -> [LIT:NUM, 7.9] --- Your tests and validators are identical. They must be different to prevent hard-coded solutions. --- Your "reference" includes the operators < > <= >=, but these are never tested. If the input contains any of these, your solution results in "bad argument #2 to 'format' (number expected, got nil)" You should either remove these operators from the "reference", or add tests which contain them. --- Your stub generator is incorrect. The default code expects multiple lines of input, when there is only one [To fix this, remove the "loopline S number:string(4096)" line]. --- I think the difficult of this is "Easy" or "Medium". Once I was familiar with the task, the actual code took less than 10 minutes (admittedly, without dealing with < > <= >=, since I knew they aren't tested)

u_know_not Unnamed CodinGamer

I updated it and made integer inputs return integers as output to increase the difficulty. I removed <, >, <=, and >= because the letters after them (and them themselves) don't appear in the string. I removed the "loopline S number..." line. I changed the validator tests. Added info about the rounding rules.