Contents
- TokenStream <TOKENS, TOKEN>
- IsTab (TOKEN token)
- IsNewline (TOKEN token)
- TokenToString (TOKEN token)
- TokenToChunk (in TOKEN token)
- TokensToChunk (in ReadOnlySpan<TOKEN> token)
- ChunkToTokens (in TOKENS tokens)
- ChunkLength (in TOKENS tokens)
- Take1 (in TOKENS stream, out TOKEN head, out TOKENS tail)
- Take (int amount, in TOKENS stream, out TOKENS head, out TOKENS tail)
- TakeWhile (Func<TOKEN, bool> predicate, in TOKENS stream, out TOKENS head, out TOKENS tail)
- TokenStream
- tokenToChunk <S, A> (in A token)
- tokensToChunk <S, A> (in ReadOnlySpan<A> token)
- chunkToTokens <S, A> (in S tokens)
- chunkLength <S, A> (in S tokens)
- chunkEmpty <S, A> (in S tokens)
- take1 <S, A> (in S stream, out A head, out S tail)
- take <S, A> (int amount, in S stream, out S head, out S tail)
- takeWhile <S, A> (Func<A, bool> predicate, in S stream, out S head, out S tail)
interface TokenStream <TOKENS, TOKEN> Source #
Low-level streaming trait. Used primarily by Megaparsec.
Parameters
| type | S | Token type |
Methods
method bool IsTab (TOKEN token) Source #
If the stream supports the concept of tabs, then this function should return true if the token is a tab.
Parameters
| param | token | Token to test |
| returns | True if a tab | |
method bool IsNewline (TOKEN token) Source #
If the stream supports the concept of newlines, then this function should return true if the token is a newline.
Parameters
| param | token | Token to test |
| returns | True if a newline | |
method ReadOnlySpan<char> TokenToString (TOKEN token) Source #
Create a textual respresentation of the token
Parameters
| param | token | Token |
| returns | Text | |
method TOKENS TokenToChunk (in TOKEN token) Source #
Lift a single token to chunk of the stream
method TOKENS TokensToChunk (in ReadOnlySpan<TOKEN> token) Source #
Lift many tokens to chunk of the stream
method ReadOnlySpan<TOKEN> ChunkToTokens (in TOKENS tokens) Source #
Turn a chunk into a sequence of tokens
method int ChunkLength (in TOKENS tokens) Source #
Get the length of a chunk
method bool Take1 (in TOKENS stream, out TOKEN head, out TOKENS tail) Source #
Take the first element of the stream if it exists.
Parameters
| param | stream | Stream |
| returns | Head element taken from the stream and a Tail of remaining stream items | |
method bool Take (int amount, in TOKENS stream, out TOKENS head, out TOKENS tail) Source #
Take should try to extract a chunk of length amount, or if the
stream is too short, the rest of the stream. Valid implementation
should follow the rules:
- If the requested length
amountis0(or less),Noneshould never be returned, insteadSome([], s)should be returned, where[]stands for the empty chunk, andsis the original stream (second argument). - If the requested length is greater than
0and the stream is empty,Noneshould be returned indicating end-of-input. - In other cases, take chunk of length
amount(or shorter if the stream is not long enough) from the input stream and return the chunk along with the rest of the stream.
Parameters
| param | amount | Number of elements to take |
| param | stream | Stream |
| returns | Head element taken from the stream and a Tail of remaining stream items | |
method void TakeWhile (Func<TOKEN, bool> predicate, in TOKENS stream, out TOKENS head, out TOKENS tail) Source #
Extract chunk of the stream taking tokens while the supplied predicate returns 'True'. Return the chunk and the rest of the stream.
For many types of streams, the method allows for significant performance improvements, although it is not strictly necessary from a conceptual point-of-view.
Parameters
| param | predicate | Token testing predicate |
| param | stream | Stream to read from |
| returns | ||
class TokenStream Source #
Methods
method S tokenToChunk <S, A> (in A token) Source #
Lift a single token to chunk of the stream
method S tokensToChunk <S, A> (in ReadOnlySpan<A> token) Source #
Lift many tokens to chunk of the stream
method ReadOnlySpan<A> chunkToTokens <S, A> (in S tokens) Source #
Turn a chunk into a sequence of tokens
method int chunkLength <S, A> (in S tokens) Source #
Get the length of a chunk
method bool chunkEmpty <S, A> (in S tokens) Source #
Is the chunk empty?
method bool take1 <S, A> (in S stream, out A head, out S tail) Source #
Take the first element of the stream if it exists.
Parameters
| param | stream | Stream |
| returns | Head element taken from the stream and a Tail of remaining stream items | |
method bool take <S, A> (int amount, in S stream, out S head, out S tail) Source #
Take should try to extract a chunk of length amount, or if the
stream is too short, the rest of the stream. Valid implementation
should follow the rules:
- If the requested length
amountis0(or less),Noneshould never be returned, insteadSome([], s)should be returned, where[]stands for the empty chunk, andsis the original stream (second argument). - If the requested length is greater than
0and the stream is empty,Noneshould be returned indicating end-of-input. - In other cases, take chunk of length
amount(or shorter if the stream is not long enough) from the input stream and return the chunk along with the rest of the stream.
Parameters
| param | amount | Number of elements to take |
| param | stream | Stream |
| returns | Head element taken from the stream and a Tail of remaining stream items | |
method void takeWhile <S, A> (Func<A, bool> predicate, in S stream, out S head, out S tail) Source #
Extract chunk of the stream taking tokens while the supplied predicate returns 'True'. Return the chunk and the rest of the stream.
For many types of streams, the method allows for significant performance improvements, although it is not strictly necessary from a conceptual point-of-view.
Parameters
| param | predicate | Token testing predicate |
| param | stream | Stream to read from |
| returns | ||