Msg: 902 *Conference*

11-11-91 23:51:03

From: TRACY ALLEN

To : RICHARD HANSON

Subj: REPLY TO MSG #776 (BASIC TOKENS)

Dear Rick,
  The question mark and apostrophe are not, strictly speaking, tokens.  The
tokens are the one byte values that represent the BASIC keywords in the M100's
memory.  When you use ? that is a shorthand that BASIC immediately converts to
the TOKEN for PRINT.  When you list your program, the ? is not there, it says
PRINT.  Neither the five letters PRINT nor the single letter ? is the actual
token.  The actual PRINT token looks like the symbol for the British Pound
Stirling!  (graph-8, ascii 163)  If you peek your way through a BASIC program
(FORI=32768to33768:printCHR$(PEEK(I));:NEXT)  You will that symbol before all
the stuff you would see a PRINT in the listing.                 When you enter
a apostrophe: ' the program converts it to a colon, a remark token, and a byte
value 255.  
  The best discussion of exactly how the process of tokenizing works is in
Chris Morgan's book.  An appendix lists all the tokens and key words, all 128
of them.  
  Ask a token question, receive a ......  ......
  -- Tracy