Is white space tokenization enough?
In this assignment, you will use an online tokenization tool. Navigate to http://text-processing.com/demo/tokenize/  and try to following: 

  1. Enter several sample sentences (you can copy paste them from the web or write your own) into the textbox where it says “tokenize text”. Your sentences should include at least one contraction and at least one compound word (if you don’t know what a compound word is, see here).

  2. Observe how the different tokenizers handle your text. Look carefully at the whitespace tokenizer and answer the following question: Are spaces sufficient to tokenize English language text? Why or why not? Cite examples from your test to support your conclusion.

Try out a CALL tool

  1. Identify one CALL tool that you could try out. Spend at least 10 minutes trying out the tool. Possible examples include, but are not limited too, DuoLingo, Mango Languages, Babble, Rosetta Stone, etc.

  2. What type of feedback does the tool give? Is it individualized feedback?
  3. How does the tool handle clozes? Does it allow multiple possibly correct answers?
  4. Does the tool allow you to work on, around or through the language?

Scroll to Top