Token Types
There are several standard token types available for use and styling.
In addition every type can have subtypes, for example we can have
single quoted and double quoted string - both are strings, but different.
So we can define string.single
and string.double
both are
string
s but string.single
is not string.double
.
Standard token types:
Token type | Description | Example |
---|---|---|
string |
String/Char literals | 'single quoted' "double quoted" |
number |
Number literals integers, doubles, floats etc. | 10 .15 10.5e10 |
symbol |
Code symbols - class names, function names etc. | Symbol |
symbol.class |
Class Name | SomeClass |
symbol.class.interface |
Interface Name | SomeInterface |
symbol.function |
Function definition (NOT CALL!) | FizzBuzz |
symbol.annotation |
Annotation | @annotation [annotation] |
constant |
Constants defined in language | true IS_DEBUG |
variable |
Variables, including sigils (like $ in PHP) |
$var |
variable.property |
Property of object | $var->property variable.property |
call |
Functions/subroutine/directives calls | call() |
format |
Formatting styles | |
format.italics |
Italic text | italics |
format.bold |
Bold text | bold |
format.strike |
Strike text | strike |
format.underline |
Underlined text | underline |
keyword |
Keywords | foreach |
operator |
Operators | += / % |
operator.punctuation |
Punctuation operators | ; , . |
delimiter |
Language delimiters | <?php <% |
language.name |
Embedded name language |
|
comment |
Comment | // some comment |
comment.docblock |
Documentation block comment | /// Documentation comment |
preprocessor |
Preprocessor definition | #preprocessor directive |
However every language can define new types of tokens, but I suggest to stick with standard and just extend them. Create new tokens only when it's really necessary and none from list above makes semantic sense.