FSharpLineTokenizer (F# Compiler Guide)

FSharpLineTokenizer Type

Namespace: FSharp.Compiler.Tokenization
Assembly: FSharp.Compiler.Service.dll
Base Type: obj

Object to tokenize a line of F# source code, starting with the given lexState. The lexState should be FSharpTokenizerLexState.Initial for the first line of text. Returns an array of ranges of the text and two enumerations categorizing the tokens and characters covered by that range, i.e. FSharpTokenColorKind and FSharpTokenCharKind. The enumerations are somewhat adhoc but useful enough to give good colorization options to the user in an IDE. A new lexState is also returned. An IDE-plugin should in general cache the lexState values for each line of the edited code.

Instance members

Instance member	Description
`this.ScanToken` Full Usage: `this.ScanToken` Parameters: lexState : `FSharpTokenizerLexState` Returns: `FSharpTokenInfo option * FSharpTokenizerLexState`	Scan one token from the line lexState : `FSharpTokenizerLexState` Returns: `FSharpTokenInfo option * FSharpTokenizerLexState`

Static members

Static member	Description
`FSharpLineTokenizer.ColorStateOfLexState(arg1)` Full Usage: `FSharpLineTokenizer.ColorStateOfLexState(arg1)` Parameters: arg0 : `FSharpTokenizerLexState` Returns: `FSharpTokenizerColorState`	Get the color state from the lexer state arg0 : `FSharpTokenizerLexState` Returns: `FSharpTokenizerColorState`
`FSharpLineTokenizer.LexStateOfColorState(arg1)` Full Usage: `FSharpLineTokenizer.LexStateOfColorState(arg1)` Parameters: arg0 : `FSharpTokenizerColorState` Returns: `FSharpTokenizerLexState`	Get a default lexer state for a color state. NOTE: This may result in an inaccurate lexer state not taking into account things such as the #if/#endif and string interpolation context within the file arg0 : `FSharpTokenizerColorState` Returns: `FSharpTokenizerLexState`