//languages/com.aardarch.aardink.languages.internal.markdown/MarkdownTokenizer
MarkdownTokenizer
object MarkdownTokenizer : RegexTokenizer
Regex-driven Markdown tokenizer. The editor doesn’t ship a “Heading” token type, so token types are mapped onto visually-distinct existing categories:
- headings → TokenType.Annotation
- bold / italic emphasis markers → TokenType.Keyword
- inline code / fenced code → TokenType.StringLiteral
- links and images → TokenType.FunctionCall
- blockquote markers, list bullets → TokenType.Punctuation
Properties
| Name | Summary |
|---|---|
| commentSyntax | open override val commentSyntax: CommentSyntax |
Functions
| Name | Summary |
|---|---|
| canSpanLines | open override fun canSpanLines(lineIndex: Int, tokens: List<Token>): Boolean Always false: the incremental rescan in tokenizeLines tracks multi-line constructs itself. |
| keyboardToolbarChars | open override fun keyboardToolbarChars(): List<Char> |
| tokenizeFull | open override fun tokenizeFull(text: String): List<Token> |
| tokenizeFullCooperative | open suspend override fun tokenizeFullCooperative(text: String): List<Token> |
| tokenizeLines | open override fun tokenizeLines(text: String, dirtyRange: IntRange, previousTokens: List<Token>): List<Token> Re-tokenizes the lines around the change since the last scan and returns their tokens: at least dirtyRange, and further when the change altered how later text scans. |