FSharpSourceTokenizer (F# Compiler Guide)

FSharpSourceTokenizer Type

Tokenizer for a source file. Holds some expensive-to-compute resources at the scope of the file.

Constructor	Description
`FSharpSourceTokenizer(conditionalDefines, fileName, langVersion)` Full Usage: `FSharpSourceTokenizer(conditionalDefines, fileName, langVersion)` Parameters: conditionalDefines : `string list` fileName : `string option` langVersion : `string option` Returns: `FSharpSourceTokenizer`	Create a tokenizer for a source file. conditionalDefines : `string list` fileName : `string option` langVersion : `string option` Returns: `FSharpSourceTokenizer`

Instance member	Description
`this.CreateBufferTokenizer` Full Usage: `this.CreateBufferTokenizer` Parameters: bufferFiller : `char[] * int * int -> int` Returns: `FSharpLineTokenizer`	Create a tokenizer for a line of this source file using a buffer filler bufferFiller : `char[] * int * int -> int` Returns: `FSharpLineTokenizer`
`this.CreateLineTokenizer` Full Usage: `this.CreateLineTokenizer` Parameters: lineText : `string` Returns: `FSharpLineTokenizer`	Create a tokenizer for a line of this source file lineText : `string` Returns: `FSharpLineTokenizer`