juno-compiler/src/JunoToken.i3


 Copyright (C) 1992, Digital Equipment Corporation                         
 All rights reserved.                                                      
 See the file COPYRIGHT for a full description.                            
                                                                           
 Last modified on Wed Jun  8 17:23:14 PDT 1994 by heydon                   
      modified on Fri Aug  7 21:53:59 PDT 1992 by myers                    

INTERFACE JunoToken;
A JunoToken.T represents one of the possible input tokens of the Juno language. Juno has 8 token classes: literal real numbers, literal texts, identifiers, operators, keywords, reserved identifiers, comments, and an end-of-stream marker. This procedure also provides the procedure ToText() for converting a token to a textual form.

IMPORT JunoValue;

TYPE
  Kind =
    { LitReal, LitText, Id,
      Semi, Dot, Comma, Colon, LPren, RPren, LBrace, RBrace, LBracket,
        RBracket, Assign, SuchThat, Else, Guard, Near, Equals, Differs, Less,
        Greater, AtMost, AtLeast, Plus, Minus, Times, Divide, Concat,
      Module, Private, End, Import, Const, Var, Pred, Func, Proc, UI, Is,
        Skip, Abort, If, Fi, Do, Od, Save, In, Nil, True, False, Or, And, Not,
        Exists, Cong, Para, Hor, Ver, Rel, Div, Mod,
      Real, Text, Pair, Int, Floor, Ceiling, Round, Abs, Sin, Cos, Ln,
        Exp, Car, Cdr, Max, Min, Atan,
      Comment, EndMarker, Unknown };
  Op = [Kind.Semi..Kind.Concat];	 (* operators *)
  KeyWd = [Kind.Module..Kind.Mod];	 (* keywords *)
  ResvdId = [Kind.Real..Kind.Atan];	 (* reserved identifiers *)

CONST
  AllOps = SET OF Kind{FIRST(Op)..LAST(Op)};
  AllKeyWds = SET OF Kind{FIRST(KeyWd)..LAST(KeyWd)};
  AllResvdIds = SET OF Kind{FIRST(ResvdId)..LAST(ResvdId)};

TYPE
  T = REF RECORD
    kind: Kind;				 (* kind of token *)
    text: TEXT;				 (* text of the token *)
    val: REFANY;			 (* TEXT or Atom.T *)
    num: JunoValue.Real;                 (* for real values *)
  END;
A token t has a kind and an optional value. The value t.val is meaningful if and only if t.kind is Kind.LitText, Kind.Id, or Kind.Comment. The value t.num is meaningful if and only if t.kind is Kind.LitReal. There are 8 token classes: literal reals, literal texts, identifiers, operators, keywords, reserved identifiers, comments, and end-of-stream. A token t is classified according to the following table:
      
         Class           Condition               Value Type
         literal real    t.kind = Kind.LitReal   t.num is a JunoValue.Real
         literal text    t.kind = Kind.LitText   t.val is a TEXT
         identifier      t.kind = Kind.Id        t.val is an Atom.T
         operator        t.kind IN AllOps
         keyword         t.kind IN AllKeyWds
         reserved id     t.kind IN AllResvdIds
         comment         t.kind = Kind.Comment   t.val is a TEXT
         end-of-stream   t.kind = Kind.EndMarker
      
   
The TEXT associated with a text literal does not include the start and end double-quote characters, and escape sequences in the input literal have been converted to ASCII characters in the result text. The TEXT associated with a comment *does* include the begin- and end-of-comment characters, as well as those of any nested comments.

PROCEDURE Copy(t: T): T;
Returns a new token with the same field values as t.

PROCEDURE ToName(t: T): TEXT;
Produces a human-readable representation of the token t as a TEXT. This representation has the form: kind(value), where kind is one of the strings Real, Text, Id, Op, KeyWd, ResvdId, or EndMarker, and value is the token value. When kind is Op, KeyWd, or ResvdId, the value printed is the implicit value of the operator, keyword, or reserved identifier, respectively, such as :=, IMPORT, or FLOOR. When kind is EndMarker, no value is printed.

PROCEDURE ToText(t: T): TEXT;
Produces a textual representation of the token t. If substituted for the text of the token actually found in the program, this representation will not change the program's meaning.

CONST
  KindName = ARRAY Kind OF TEXT {
    "numeric literal", "text literal", "identifier",
    "semicolon", "period", "comma", "colon", "left parenthesis",
      "right parenthesis", "left brace", "right brace",	"left bracket",
      "right bracket", ":=", "::", "|", "->", "~", "=", "#", "<", ">",
      "<=", ">=", "+", "-", "*", "/", "&",
    "MODULE", "PRIVATE", "END", "IMPORT", "CONST", "VAR",
      "PRED", "FUNC", "PROC", "UI", "IS", "SKIP", "ABORT", "IF", "FI",
      "DO", "OD", "SAVE", "IN", "NIL", "TRUE", "FALSE", "OR", "AND", "NOT",
      "EXISTS", "CONG", "PARA", "HOR", "VER", "REL", "DIV", "MOD",
    "REAL", "TEXT", "PAIR", "INT", "FLOOR", "CEILING", "ROUND", "ABS", "SIN",
      "COS", "LN", "EXP", "CAR", "CDR", "MAX", "MIN", "ATAN",
    "comment", "end of file", "unknown token" };

END JunoToken.

interface JunoValue is in: