class Tokenizer

Same name in this branch
  1. 11.1.x vendor/masterminds/html5/src/HTML5/Parser/Tokenizer.php \Masterminds\HTML5\Parser\Tokenizer
  2. 11.1.x vendor/theseer/tokenizer/src/Tokenizer.php \TheSeer\Tokenizer\Tokenizer
  3. 11.1.x vendor/squizlabs/php_codesniffer/src/Tokenizers/Tokenizer.php \PHP_CodeSniffer\Tokenizers\Tokenizer

CSS selector tokenizer.

This component is a port of the Python cssselect library, which is copyright Ian Bicking, @author Jean-François Simon <jeanfrancois.simon@sensiolabs.com>

@internal

Hierarchy

  • class \Symfony\Component\CssSelector\Parser\Tokenizer\Tokenizer

Expanded class hierarchy of Tokenizer

See also

https://github.com/SimonSapin/cssselect.

1 file declares its use of Tokenizer
Parser.php in vendor/symfony/css-selector/Parser/Parser.php
2 string references to 'Tokenizer'
Runner::checkRequirements in vendor/squizlabs/php_codesniffer/src/Runner.php
Exits if the minimum requirements of PHP_CodeSniffer are not met.
system_requirements in core/modules/system/system.install
Implements hook_requirements().

File

vendor/symfony/css-selector/Parser/Tokenizer/Tokenizer.php, line 29

Namespace

Symfony\Component\CssSelector\Parser\Tokenizer
View source 
class Tokenizer {
    
    /**
     * @var Handler\HandlerInterface[]
     */
    private array $handlers;
    public function __construct() {
        $patterns = new TokenizerPatterns();
        $escaping = new TokenizerEscaping($patterns);
        $this->handlers = [
            new Handler\WhitespaceHandler(),
            new Handler\IdentifierHandler($patterns, $escaping),
            new Handler\HashHandler($patterns, $escaping),
            new Handler\StringHandler($patterns, $escaping),
            new Handler\NumberHandler($patterns),
            new Handler\CommentHandler(),
        ];
    }
    
    /**
     * Tokenize selector source code.
     */
    public function tokenize(Reader $reader) : TokenStream {
        $stream = new TokenStream();
        while (!$reader->isEOF()) {
            foreach ($this->handlers as $handler) {
                if ($handler->handle($reader, $stream)) {
                    continue 2;
                }
            }
            $stream->push(new Token(Token::TYPE_DELIMITER, $reader->getSubstring(1), $reader->getPosition()));
            $reader->moveForward(1);
        }
        return $stream->push(new Token(Token::TYPE_FILE_END, null, $reader->getPosition()))
            ->freeze();
    }

}

Members

Title Modifiers Object type Summary
Tokenizer::$handlers private property
Tokenizer::tokenize public function Tokenize selector source code.
Tokenizer::__construct public function