function Tokenizer::tokenize

Same name in this branch

11.1.x vendor/symfony/css-selector/Parser/Tokenizer/Tokenizer.php \Symfony\Component\CssSelector\Parser\Tokenizer\Tokenizer::tokenize()

Creates an array of tokens when given some content.

Parameters

string $string The string to tokenize.:

Return value

array

1 call to Tokenizer::tokenize()

Tokenizer::__construct in vendor/squizlabs/php_codesniffer/src/Tokenizers/Tokenizer.php: Initialise and run the tokenizer.

2 methods override Tokenizer::tokenize()

JS::tokenize in vendor/squizlabs/php_codesniffer/src/Tokenizers/JS.php: Creates an array of tokens when given some JS code.
PHP::tokenize in vendor/squizlabs/php_codesniffer/src/Tokenizers/PHP.php: Creates an array of tokens when given some PHP code.

File

vendor/squizlabs/php_codesniffer/src/Tokenizers/Tokenizer.php, line 152

Class

Tokenizer

Namespace

PHP_CodeSniffer\Tokenizers

Code

protected abstract function tokenize($string);