Please read this handout carefully before submitting the project. Contact the TAs if you have any questions about the project. Also please read the newsgroup constantly for any updates in the project handout.
Introduction
Your task in this step is to generate a scanner (also referred as tokenizer) for the project's grammar provided to you. Scanner is a program that takes a sequence of characters (the source file of the program) and produces a sequence of tokens that will be used to feed the compiler's parser.
The figure below shows the general structure of a simple compiler similarly to the one presented in the Fischer & LeBlanc's book. The picture highlights the part of the compiler that will be developed at this step and conceptually relate it to other parts that will be developed in the future.
Scanner Generation Tools
The scanner's source code is normally generated by a scanner generation tool. You can use one of the tools available to generate your scanner in this step. These tools normally work by taking the token definitions expressed by regular expressions and generates the source code for the scanner automatically (for this project, the tokens are especified in the language's grammar). The programmer has to add the code to handle the scanner's output. For example, at this step you will print all the tokens in the standard output. In the step 2 of this project you will modify your scanner to feed the parser replacing the print routines by calls to the parserand passing the tokens as parameters. In order to learn how to merge the code of the scanner generator with the rest of your source code, please, read the user's manual of the tool you decide to use.
Some popular tools are:
- Antlr: The most popular option if you're developing the compiler in Java. (Recommended)
- Lex (or Flex) : Generates the scanner's source in C language.
- JFlex(Scanner generator): Lexical analyzer written in Java.
Step 1: The compiler's tokenizer (scanner)
At the end of this step, the program developed (the very first part of our compiler) should be able to open a program's source file written in the LITTLE LANGUAGE and recognize its tokens. At this step, the output of your program should be the prints of each token's type and value (see below). In the next step the same output will be used to feed the parser.
Your scanner program should be able to open and read LITTLE source file and print the all the valid tokens within the source file and their respective type in the standard output. You might want to redirect the output to a file and compare your results with the output files provided for the testcases.
Testcases:
Outputs:
See grammar file for token definitions (Keywords, Operator, Literals)
Output Format
The program is expected to print each token in a predefined format. Here is an example
Token Type: INTLITERALValue: 20
Please read the output samples for more details.
Running your scanner
This is how we will test your scanner (if you're using C/C++, we will use the appropriate equivalent):
java Micro fibonacci.micro > fibonacci.scanner
Note: This step will be graded automatically and the outputs generated by your code will be directly compared with the expected outputs using "diff -b -B" command. Please make sure the your outputs are identical to the sample outputs provided.
Project template
To help you start the project we are providing templates for step1. We have template for ANTLR available.
Readings
Several books introduce the theory necessary for a good understandingof lexical analysis and how a compiler tokenizer works, as well a gooddescription of how a simple compiler is structured. You may findit helpful to check some of the following references:
Alfred v. Aho, Ravi Sethi, Jeffrey D.Ullman, "Compilers: Principles, Techniques, and Tools", Addison-Wesley,1986.
A simple one-pass compiler -Chapter 2
Lexical analysis - Chapter 3
Andrew W. Appel, "Modern CompilerImplementation in Java", Cambridge Univ. Press, 1998.
Lexical Analysis - Chapter 2
Charles Fischer, Richard LeBlanc,"Crafting a Compiler with C", The Benjamin/Commings Publishing Company,Inc, 1991.
A simple compiler - Chapter 2
Scanning - Theory and Practice- Chapter 3
Q&A
- Q: What should I do if I encounter an illegal token?
A: Stop the lexer at that point. You are not required to do error handling.
FAQs
USMLE Step 1 Passing Score
Unlike the other USMLE exams, Step 1 is reported as pass/fail only, so you will not receive a numerical score. According to the USMLE program, examinees must correctly answer approximately 60 percent of items to pass each USMLE Step.
Does Step 1 try to trick you? ›
Virtually every sentence for Step 1 has a purpose. (You may not KNOW the purpose, but that's a different story). For Step 2, the NBME claims (and experience suggests) that there may be unnecessary information. However, they're not trying to mislead you intentionally.
Is doing questions enough to pass Step 1? ›
USMLE Step 1 is a question-based exam. Therefore, the best approach to prepare for it is to incorporate practice questions early and often. What is Spaced Repetition and Why it's SO Important for USMLE? One effective approach is to go through the questions once during the first two years of medical school.
How many questions can you miss to pass Step 1? ›
What percentage of questions must you answer correctly to pass the USMLE Step 1? In order to pass the USMLE Step 1, you must answer approximately 60% of questions correctly. In numerical terms, if there are 280 questions in total, you must answer at least 168 questions correctly.
What score is 50% on Step 1? ›
Until the scoring system for Step 1 changes to Pass/Fail, the scores on Step 1 can be interpreted based on historical percentiles. While 194 represents the 5th percentile, the average (50th percentile) falls between 230 and 235. Scores at or just above the mean are good scores!
How common is it to fail Step 1? ›
So, what happens if you fail USMLE Step 1? In 2022, 91% of students passed Step 1, leaving nearly 10%, 1489 students, to retake the test. While failing Step 1 certainly isn't ideal, it could happen, but it doesn't have to kill your hopes of becoming a doctor or matching into your ideal residency program.
Is Step 1 the hardest exam ever? ›
Traditionally, Step 1 has been thought of as both the most difficult and most important USMLE Step exam. There are several reasons for this. Firstly, this exam is the first in the series, and students taking it will have had less experience with the types of questions that the USMLE asks.
Is UWorld Step 1 harder than Step 1? ›
The frequently updated content ensures you learn and master the most relevant material that helps confront your individual USMLE Step 1 strengths and weaknesses. UWorld may be harder than Step 1, with some questions above the actual test's difficulty level.
Does Step 1 get curved? ›
If they fall short of this mark, they can still get a residency position, but it will be harder. Note that the USMLE examination is not scored “on a curve”.
Can I finish Step 1 in 3 months? ›
You can definitely study for the USMLE in 3 months. It may even take you less time if you have a good understanding of your foundational medical knowledge! But, it's better to be safe than sorry. With more study time, you can build your confidence and memory recall for the lengthy eight-hour exam day.
The sooner students begin this process, the better.
Although UWorld is valuable for learning and reviewing, it's not sufficient as the sole study tool for Step 1. We recommend utilizing UWorld in conjunction with First Aid and flashcards to get the most out of your dedicated.
Is 60% a pass on Step 1? ›
The percentages of correctly answered items required to pass varies by Step and from form to form within each Step. However, examinees typically must answer approximately 60% of items correctly to achieve a passing score.
What percent correct is a 196 on Step 1? ›
Assuming Step 1 is scored similarly to CBSSA 25-30, you would need somewhere between 63 to 65% correct to attain the passing score of 196.
What percentile is a 240 on Step 1? ›
Step 2 Percentiles: Side-by-Side Comparison with Step 1, Step 2 CK, and Step 3
| Score | Step 1 Percentile | Step 2 CK Percentile |
|---|
| 240 | 62 | 35 |
| 235 | 52 | 25 |
| 230 | 43 | 17 |
| 225 | 34 | 12 |
24 more rowsMar 12, 2024
You'll usually need to receive a letter grade between A and D to pass a class, often the numerical equivalent of 65 percent or higher. Receiving an F—which stands for “fail”—indicates that you did not pass the class. The cutoff to receive an F is usually 64 percent.
What percent right do you need to pass Step 1? ›
The percentages of correctly answered items required to pass varies by Step and from form to form within each Step. However, examinees typically must answer approximately 60% of items correctly to achieve a passing score.
What is a safe score to pass Step 1? ›
Let's consider various USMLE Step 1 score brackets. A score of minimum pass to 215 would be considered low, yet still a pass. The lowest passing score possible is 194, which equates to just north of 55% correct of the 350 questions.
What is the bare minimum score to pass Step 1? ›
The current minimum score to pass Step 1 is 196, so if you pass, you at least scored 196 or higher. Step 1 becoming pass/fail means that Step 2 will now be given more weight, as it will still be scored.
