function Tokenizer::tokenize

Same name in this branch

11.1.x vendor/squizlabs/php_codesniffer/src/Tokenizers/Tokenizer.php \PHP_CodeSniffer\Tokenizers\Tokenizer::tokenize()

Tokenize selector source code.

File

vendor/symfony/css-selector/Parser/Tokenizer/Tokenizer.php, line 54

Class

Tokenizer: CSS selector tokenizer.

Namespace

Symfony\Component\CssSelector\Parser\Tokenizer

Code

public function tokenize(Reader $reader) : TokenStream {
    $stream = new TokenStream();
    while (!$reader->isEOF()) {
        foreach ($this->handlers as $handler) {
            if ($handler->handle($reader, $stream)) {
                continue 2;
            }
        }
        $stream->push(new Token(Token::TYPE_DELIMITER, $reader->getSubstring(1), $reader->getPosition()));
        $reader->moveForward(1);
    }
    return $stream->push(new Token(Token::TYPE_FILE_END, null, $reader->getPosition()))
        ->freeze();
}