function Lexer::tokenize

Same name in this branch

11.1.x vendor/phpstan/phpdoc-parser/src/Lexer/Lexer.php \PHPStan\PhpDocParser\Lexer\Lexer::tokenize()
11.1.x vendor/twig/twig/src/Lexer.php \Twig\Lexer::tokenize()

Tokenize the provided source code.

The token array is in the same format as provided by the PhpToken::tokenize() method in PHP 8.0. The tokens are instances of PhpParser\Token, to abstract over a polyfill implementation in earlier PHP version.

The token array is terminated by a sentinel token with token ID 0. The token array does not discard any tokens (i.e. whitespace and comments are included). The token position attributes are against this token array.

Parameters

string $code The source code to tokenize.:

ErrorHandler|null $errorHandler Error handler to use for lexing errors. Defaults to: ErrorHandler\Throwing.

Return value

Token[] Tokens

2 calls to Lexer::tokenize()

Emulative::tokenize in vendor/nikic/php-parser/lib/PhpParser/Lexer/Emulative.php: Tokenize the provided source code.
Emulative::tokenize in vendor/nikic/php-parser/lib/PhpParser/Lexer/Emulative.php: Tokenize the provided source code.

1 method overrides Lexer::tokenize()

Emulative::tokenize in vendor/nikic/php-parser/lib/PhpParser/Lexer/Emulative.php: Tokenize the provided source code.

File

vendor/nikic/php-parser/lib/PhpParser/Lexer.php, line 24

Class

Lexer

Namespace

PhpParser

Code

public function tokenize(string $code, ?ErrorHandler $errorHandler = null) : array {
    if (null === $errorHandler) {
        $errorHandler = new ErrorHandler\Throwing();
    }
    $scream = ini_set('xdebug.scream', '0');
    $tokens = @Token::tokenize($code);
    $this->postprocessTokens($tokens, $errorHandler);
    if (false !== $scream) {
        ini_set('xdebug.scream', $scream);
    }
    return $tokens;
}