Add support for PHP 7

[nikic]

Update the parser tables for PHP 7, add lexer support for new tokens. This includes support for PHP 7.1.

I have removed the AST node definitions and emptied the semantic actions of the parser. ACE does not use the AST and porting the semantic actions to JS would be way too much work.

For future reference, this is the rebuildParser.php I used:

<?php

const GRAMMAR_FILE = './php7.y';
const TMP_FILE     = './tmp_parser.jsy';
const RESULT_FILE  = './tmp_parser.js';
const TOKENS_FILE  = './tokens.y';

$tokens = file_get_contents(TOKENS_FILE);
$grammarCode = file_get_contents(GRAMMAR_FILE);
$grammarCode = str_replace('%tokens', $tokens, $grammarCode);

file_put_contents(TMP_FILE, $grammarCode);

echo 'Building parser. Output: "',
     trim(shell_exec('kmyacc -l -m kmyacc.js.parser ' . TMP_FILE . ' 2>&1')),
     '"', "\n";

preg_match_all('/T_\w+/', $tokens, $matches);
foreach ($matches[0] as $match) {
    echo '"' . $match . '",';
}

The skeleton file is https://github.com/niklasvh/php.js/blob/master/grammar/kmyacc.js.parser. The grammar and token files are from https://github.com/nikic/PHP-Parser/tree/master/grammar.

[SjonHortensius]

I noticed a discrepancy on https://3v4l.org/mo9V4 where PHP accepts a syntax and the worker doesn't:

$obj = (Factory::Generate('MyObj'))($someData);

throws Syntax error, unexpected ???

Same thing happens on https://3v4l.org/WThln

var_dump( DateTimeZone::listIdentifiers());

throws Syntax error, unexpected ???, expecting ',' or ')'

[adamjimenez]

I found some other code it doesn't like:

$cb = \Codebird\Codebird::getInstance();

[nikic]

@SjonHortensius Thanks for testing! Handling of :: should be fixed now.

The issue was that the lexer used the T_DOUBLE_COLON alias of T_PAAMAYIM_NEKUDOTAYIM, which was lost during the update. I've now dropped the alias entirely in the interest of simplifying future updates.

[SjonHortensius]

Thanks; that fixes all three indeed 👍

[adamjimenez]

Thanks, this is working great so far.

[adamjimenez]

Seeing as you're on a roll, here's something else that throws an error when it shouldn't:

print "$_SERVER[DOCUMENT_ROOT]";

Syntax error, unexpected T_ENCAPSED_AND_WHITESPACE, expecting T_STRING or T_VARIABLE or T_NUM_STRING

These are also still bust:
niklasvh/php.js#55
niklasvh/php.js#46

[SjonHortensius]

var_dump(`ls`);

throws Syntax error, unexpected $EOF.

Maybe these should be reported as separate issues btw; it is not caused by this PR. This PR fixes 99% of all php7 issues that were in the previous syntax-checker.

[nikic]

I started working on a lexer update that should fix these issues, but it will take a while. I'll send a separate PR.

[nikic]

Forgot to link, new lexer is at https://gist.github.com/nikic/91ee0bc7ac74659366a8ae90b8f2d15e.

[SjonHortensius]

Not sure if (or why not) you committed this on a branch somewhere but I manually inserted your lexer in the existing php.js and it seems to work fine. It does fix indeed all mentioned issues 👍

[SjonHortensius]

@nikic I think you should include the Lexer changes in this PR as well. Also; I guess you'll need to sign the CLA here and then this should be ready to be pulled and included in a release

Can you confirm @nightwing ?

[nikic]

Pulled in the lexer updates, this fixes the issues above as well as #2232, #2285. The lexer is modeled after PHP's own lexer and tested against token_get_all() for a wide range of inputs -- some unimportant distinctions (LNUMBER/DNUMBER, COMMENT/DOC_COMMENT) and certain error behavior notwithstanding.

[nikic]

Any news here?

[plugowski]

Will be nice if that pull request will be merged into ajaxorg/ace :)

[nightwing]

Thanks for awesome work! Merging.

[nikic]

Thanks! The issues #2908, #2285 and #2232 can now be closed as well.