http://pear.php.net/bugs/search.php?cmd=display&package_name%5B0%5D=PHP_LexerGenerator&PHPSESSID=6sh8hov6085gl1gd4s43b0k8h0 Search Results en-us pear-webmaster@lists.php.net pear-webmaster@lists.php.net hourly 1 2000-01-01T12:00+00:00 http://pear.php.net/gifs/pearsmall.gif http://pear.php.net/bugs http://pear.php.net/bugs/14731 PHP_LexerGenerator Bug Reported by lew21 2008-10-02T12:57:54+00:00 PHP: 5.2.6 OS: Linux Package Version: 0.3.4 Description: ------------ $ in regex is interpreted as an reference to variable Test script: --------------- label = #[$a-zA-Z_\x7f-\xff][$a-zA-Z0-9_\x7f-\xff]*# Expected result: ---------------- I expected it to work. Actual result: -------------- PHP Notice: Undefined variable: a in /*/lexer.php on line 122 PHP Notice: Undefined variable: a in /*/lexer.php on line 122]]> PHP_LexerGenerator Bug Reported by lew21 2008-10-02T12:57:54+00:00 PHP: 5.2.6 OS: Linux Package Version: 0.3.4 Description: ------------ $ in regex is interpreted as an reference to variable Test script: --------------- label = #[$a-zA-Z_\x7f-\xff][$a-zA-Z0-9_\x7f-\xff]*# Expected result: ---------------- I expected it to work. Actual result: -------------- PHP Notice: Undefined variable: a in /*/lexer.php on line 122 PHP Notice: Undefined variable: a in /*/lexer.php on line 122]]> 2008-10-09T09:57:52+00:00 lew21st &#x61;&#116; gmail &#x64;&#111;&#x74; com PHP_LexerGenerator Bug http://pear.php.net/bugs/13252 PHP_LexerGenerator Feature/Change Request Reported by clicky 2008-02-28T11:10:34+00:00 PHP: Irrelevant OS: Any Package Version: 0.3.4 Description: ------------ Currently, if you have something like this in your regexp declaration block: nmstart = /[_a-zA-Z]/ nmchar = /[_a-zA-Z0-9]/ And you want to define ident using something like this: ident = @{nmstart}{nmchar}*@ You're forced to copy/paste the whole declaration of both nmstart & nmchar in ident's declaration. I suggest to allow using {name} in a regexp pattern to reuse the rule called "name" in the rule being declared. You may use this syntax anywhere in the pattern, except in character classes (that is, /[0-9{name}]/ would really match any digit, '{', 'n', 'a', 'm', 'e' or '}'). This is not part of any PCRE specification though. Therefore, it would require adding some special handling. Also, the PCRE documentation doesn't mention any use of the {name} syntax anywhere. The closest one is r{N,M} which indicates r must be repeated between N & M times, but you probably already know this. Test script: --------------- /*!lex2php %input $this->data %counter $this->N %token $this->token %value $this->value %line $this->line %matchlongest 1 nmstart = /[_a-zA-Z]/ nmchar = /[_a-zA-Z0-9]/ ident = @{nmstart}{nmchar}*@ */ Expected result: ---------------- No error. "ident" should be able to match something like: "Some_ID123". It should not match something like "5ome_invalid_ID" because '5' does not match the pattern expressed by nmstart. Actual result: -------------- Popping SUBPATTERN Popping PATTERN Popping pattern_declarations Popping processing_instructions Popping COMMENTSTART Popping PHPCODE Popping $ Exception: Unexpected input at line29: { in /usr/share/php/PHP/LexerGenerator/Regex/Lexer.php on line 202 Call Stack: 0.0002 53992 1. {main}() /usr/share/php/PHP/LexerGenerator/cli.php:0 0.0287 1587664 2. PHP_LexerGenerator->__construct() /usr/share/php/PHP/LexerGenerator/cli.php:3 0.2520 1605640 3. PHP_LexerGenerator_Parser->doParse() /usr/share/php/PHP/LexerGenerator.php:283 0.2522 1605640 4. PHP_LexerGenerator_Parser->yy_reduce() /usr/share/php/PHP/LexerGenerator/Parser.php:1855 0.2522 1605640 5. PHP_LexerGenerator_Parser->yy_r29() /usr/share/php/PHP/LexerGenerator/Parser.php:1716 0.2522 1605640 6. PHP_LexerGenerator_Parser->_validatePattern() /usr/share/php/PHP/LexerGenerator/Parser.php:1643 0.2523 1605640 7. PHP_LexerGenerator_Regex_Lexer->yylex() /usr/share/php/PHP/LexerGenerator/Parser.php:503 0.2523 1605640 8. PHP_LexerGenerator_Regex_Lexer->yylex1() /usr/share/php/PHP/LexerGenerator/Regex/Lexer.php:59]]> PHP_LexerGenerator Feature/Change Request Reported by clicky 2008-02-28T11:10:34+00:00 PHP: Irrelevant OS: Any Package Version: 0.3.4 Description: ------------ Currently, if you have something like this in your regexp declaration block: nmstart = /[_a-zA-Z]/ nmchar = /[_a-zA-Z0-9]/ And you want to define ident using something like this: ident = @{nmstart}{nmchar}*@ You're forced to copy/paste the whole declaration of both nmstart & nmchar in ident's declaration. I suggest to allow using {name} in a regexp pattern to reuse the rule called "name" in the rule being declared. You may use this syntax anywhere in the pattern, except in character classes (that is, /[0-9{name}]/ would really match any digit, '{', 'n', 'a', 'm', 'e' or '}'). This is not part of any PCRE specification though. Therefore, it would require adding some special handling. Also, the PCRE documentation doesn't mention any use of the {name} syntax anywhere. The closest one is r{N,M} which indicates r must be repeated between N & M times, but you probably already know this. Test script: --------------- /*!lex2php %input $this->data %counter $this->N %token $this->token %value $this->value %line $this->line %matchlongest 1 nmstart = /[_a-zA-Z]/ nmchar = /[_a-zA-Z0-9]/ ident = @{nmstart}{nmchar}*@ */ Expected result: ---------------- No error. "ident" should be able to match something like: "Some_ID123". It should not match something like "5ome_invalid_ID" because '5' does not match the pattern expressed by nmstart. Actual result: -------------- Popping SUBPATTERN Popping PATTERN Popping pattern_declarations Popping processing_instructions Popping COMMENTSTART Popping PHPCODE Popping $ Exception: Unexpected input at line29: { in /usr/share/php/PHP/LexerGenerator/Regex/Lexer.php on line 202 Call Stack: 0.0002 53992 1. {main}() /usr/share/php/PHP/LexerGenerator/cli.php:0 0.0287 1587664 2. PHP_LexerGenerator->__construct() /usr/share/php/PHP/LexerGenerator/cli.php:3 0.2520 1605640 3. PHP_LexerGenerator_Parser->doParse() /usr/share/php/PHP/LexerGenerator.php:283 0.2522 1605640 4. PHP_LexerGenerator_Parser->yy_reduce() /usr/share/php/PHP/LexerGenerator/Parser.php:1855 0.2522 1605640 5. PHP_LexerGenerator_Parser->yy_r29() /usr/share/php/PHP/LexerGenerator/Parser.php:1716 0.2522 1605640 6. PHP_LexerGenerator_Parser->_validatePattern() /usr/share/php/PHP/LexerGenerator/Parser.php:1643 0.2523 1605640 7. PHP_LexerGenerator_Regex_Lexer->yylex() /usr/share/php/PHP/LexerGenerator/Parser.php:503 0.2523 1605640 8. PHP_LexerGenerator_Regex_Lexer->yylex1() /usr/share/php/PHP/LexerGenerator/Regex/Lexer.php:59]]> 2008-02-28T11:10:34+00:00 missingno &#x61;&#116; ifrance &#x64;&#111;&#x74; com PHP_LexerGenerator Feature/Change Request http://pear.php.net/bugs/13222 PHP_LexerGenerator Bug Reported by clicky 2008-02-26T17:36:54+00:00 PHP: Irrelevant OS: Linux (Ubuntu) Package Version: 0.3.4 Description: ------------ When using %matchlongest 1, the rules will be stored in an array ($rules), enclosed in single quotes. This causes problems when the pattern uses single quotes like /[a-zA-Z0-9']*/ Digging into the code, the "first match" algorithm uses double quotes to store the pattern, which doesn't exhibit this problem (as the code properly escapes double quotes). Changing line 219 of LexerGenerator/Parser.php from: \'/^' . $rule['pattern'] . '/\','); to: "/^' . $rule['pattern'] . '/",'); solves the problem. Test script: --------------- <?php class TestLexer { private $data; private $N; private $token; public $value; private $line; private $state = 1; public function __construct($data) { $this->data = $data; $this->N = 0; $this->line = 1; } /*!lex2php %input $this->data %counter $this->N %token $this->token %value $this->value %line $this->line %matchlongest 1 word = /[a-zA-Z0-9']+/ space = /[ \t\n]+/ */ /*!lex2php %statename START word { echo "Word\n"; var_dump($this->value); echo " word subpatterns: \n"; var_dump($yy_subpatterns); } space { return FALSE; } */ } $a = new TestLexer("They're gone forever"); $a->yylex(); var_dump('advance: ' . $a->value); $a->yylex(); var_dump('advance: ' . $a->value); $a->yylex(); var_dump('advance: ' . $a->value); Expected result: ---------------- looksup@looksup:~/plex$ php -f bug1.php Word string(7) "They're" word subpatterns: array(0) { } string(16) "advance: They're" Word string(4) "gone" word subpatterns: array(0) { } string(13) "advance: gone" Word string(7) "forever" word subpatterns: array(0) { } string(16) "advance: forever" Actual result: -------------- looksup@looksup:~/plex$ php -f bug1.php Parse error: syntax error, unexpected ']', expecting ')' in /home/looksup/plex/bug1.php on line 54]]> PHP_LexerGenerator Bug Reported by clicky 2008-02-26T17:36:54+00:00 PHP: Irrelevant OS: Linux (Ubuntu) Package Version: 0.3.4 Description: ------------ When using %matchlongest 1, the rules will be stored in an array ($rules), enclosed in single quotes. This causes problems when the pattern uses single quotes like /[a-zA-Z0-9']*/ Digging into the code, the "first match" algorithm uses double quotes to store the pattern, which doesn't exhibit this problem (as the code properly escapes double quotes). Changing line 219 of LexerGenerator/Parser.php from: \'/^' . $rule['pattern'] . '/\','); to: "/^' . $rule['pattern'] . '/",'); solves the problem. Test script: --------------- <?php class TestLexer { private $data; private $N; private $token; public $value; private $line; private $state = 1; public function __construct($data) { $this->data = $data; $this->N = 0; $this->line = 1; } /*!lex2php %input $this->data %counter $this->N %token $this->token %value $this->value %line $this->line %matchlongest 1 word = /[a-zA-Z0-9']+/ space = /[ \t\n]+/ */ /*!lex2php %statename START word { echo "Word\n"; var_dump($this->value); echo " word subpatterns: \n"; var_dump($yy_subpatterns); } space { return FALSE; } */ } $a = new TestLexer("They're gone forever"); $a->yylex(); var_dump('advance: ' . $a->value); $a->yylex(); var_dump('advance: ' . $a->value); $a->yylex(); var_dump('advance: ' . $a->value); Expected result: ---------------- looksup@looksup:~/plex$ php -f bug1.php Word string(7) "They're" word subpatterns: array(0) { } string(16) "advance: They're" Word string(4) "gone" word subpatterns: array(0) { } string(13) "advance: gone" Word string(7) "forever" word subpatterns: array(0) { } string(16) "advance: forever" Actual result: -------------- looksup@looksup:~/plex$ php -f bug1.php Parse error: syntax error, unexpected ']', expecting ')' in /home/looksup/plex/bug1.php on line 54]]> 2008-02-26T17:36:54+00:00 missingno &#x61;&#116; ifrance &#x64;&#111;&#x74; com PHP_LexerGenerator Bug http://pear.php.net/bugs/13221 PHP_LexerGenerator Feature/Change Request Reported by clicky 2008-02-26T17:07:49+00:00 PHP: Irrelevant OS: Irrelevant Package Version: 0.3.4 and CVS Description: ------------ If you want to use the hyphen character (-) in a character class, you need to escape it, eg.: use octal (\055) or hexadecimal (\x2D) notation. I think the regexp lexer/parser should be made a bit more permissive as, for example, [-a-zA-Z0-9] is a valid character class. I think this is more of a feature request than a real bug report though... Test script: --------------- <?php class TestLexer { // ... /*!lex2php %input $this->data %counter $this->N %token $this->token %value $this->value %line $this->line %matchlongest 1 space = /[ \t\n]+/ name = /[-a-zA-Z0-9]+/ */ // Please note that using name = /[\x2Da-zA-Z0-9]+/ // works as expected. /*!lex2php %statename START name { echo "Name\n"; var_dump($this->value); echo " name subpatterns: \n"; var_dump($yy_subpatterns); } space { return FALSE; } */ // ... } ?> Expected result: ---------------- Plex should accept the file without any exception being raised. Actual result: -------------- Reduce (29) [subpattern ::= SUBPATTERN]. Syntax Error on line 27: token '-' while parsing rule:End of Input OPENCHARCLASS Popping SUBPATTERN Popping PATTERN Popping processing_instructions Popping COMMENTSTART Popping PHPCODE Popping $ Exception: Unexpected HYPHEN(-), expected one of: NEGATE,TEXT,ESCAPEDBACKSLASH,BACKREFERENCE,COULDBEBACKREF in /usr/share/php/PHP/LexerGenerator/Regex/Parser.php on line 1779 Call Stack: 0.0002 54040 1. {main}() /usr/share/php/PHP/LexerGenerator/cli.php:0 0.0260 1586716 2. PHP_LexerGenerator->__construct() /usr/share/php/PHP/LexerGenerator/cli.php:3 0.0332 1597664 3. PHP_LexerGenerator_Parser->doParse() /usr/share/php/PHP/LexerGenerator.php:283 0.0334 1598224 4. PHP_LexerGenerator_Parser->yy_reduce() /usr/share/php/PHP/LexerGenerator/Parser.php:1855 0.0334 1598224 5. PHP_LexerGenerator_Parser->yy_r29() /usr/share/php/PHP/LexerGenerator/Parser.php:1716 0.0334 1598224 6. PHP_LexerGenerator_Parser->_validatePattern() /usr/share/php/PHP/LexerGenerator/Parser.php:1643 0.0340 1602532 7. PHP_LexerGenerator_Regex_Parser->doParse() /usr/share/php/PHP/LexerGenerator/Parser.php:505 0.0341 1602636 8. PHP_LexerGenerator_Regex_Parser->yy_syntax_error() /usr/share/php/PHP/LexerGenerator/Regex/Parser.php:1878]]> PHP_LexerGenerator Feature/Change Request Reported by clicky 2008-02-26T17:07:49+00:00 PHP: Irrelevant OS: Irrelevant Package Version: 0.3.4 and CVS Description: ------------ If you want to use the hyphen character (-) in a character class, you need to escape it, eg.: use octal (\055) or hexadecimal (\x2D) notation. I think the regexp lexer/parser should be made a bit more permissive as, for example, [-a-zA-Z0-9] is a valid character class. I think this is more of a feature request than a real bug report though... Test script: --------------- <?php class TestLexer { // ... /*!lex2php %input $this->data %counter $this->N %token $this->token %value $this->value %line $this->line %matchlongest 1 space = /[ \t\n]+/ name = /[-a-zA-Z0-9]+/ */ // Please note that using name = /[\x2Da-zA-Z0-9]+/ // works as expected. /*!lex2php %statename START name { echo "Name\n"; var_dump($this->value); echo " name subpatterns: \n"; var_dump($yy_subpatterns); } space { return FALSE; } */ // ... } ?> Expected result: ---------------- Plex should accept the file without any exception being raised. Actual result: -------------- Reduce (29) [subpattern ::= SUBPATTERN]. Syntax Error on line 27: token '-' while parsing rule:End of Input OPENCHARCLASS Popping SUBPATTERN Popping PATTERN Popping processing_instructions Popping COMMENTSTART Popping PHPCODE Popping $ Exception: Unexpected HYPHEN(-), expected one of: NEGATE,TEXT,ESCAPEDBACKSLASH,BACKREFERENCE,COULDBEBACKREF in /usr/share/php/PHP/LexerGenerator/Regex/Parser.php on line 1779 Call Stack: 0.0002 54040 1. {main}() /usr/share/php/PHP/LexerGenerator/cli.php:0 0.0260 1586716 2. PHP_LexerGenerator->__construct() /usr/share/php/PHP/LexerGenerator/cli.php:3 0.0332 1597664 3. PHP_LexerGenerator_Parser->doParse() /usr/share/php/PHP/LexerGenerator.php:283 0.0334 1598224 4. PHP_LexerGenerator_Parser->yy_reduce() /usr/share/php/PHP/LexerGenerator/Parser.php:1855 0.0334 1598224 5. PHP_LexerGenerator_Parser->yy_r29() /usr/share/php/PHP/LexerGenerator/Parser.php:1716 0.0334 1598224 6. PHP_LexerGenerator_Parser->_validatePattern() /usr/share/php/PHP/LexerGenerator/Parser.php:1643 0.0340 1602532 7. PHP_LexerGenerator_Regex_Parser->doParse() /usr/share/php/PHP/LexerGenerator/Parser.php:505 0.0341 1602636 8. PHP_LexerGenerator_Regex_Parser->yy_syntax_error() /usr/share/php/PHP/LexerGenerator/Regex/Parser.php:1878]]> 2008-02-28T12:46:55+00:00 missingno &#x61;&#116; ifrance &#x64;&#111;&#x74; com PHP_LexerGenerator Feature/Change Request http://pear.php.net/bugs/12564 PHP_LexerGenerator Bug Reported by instance 2007-12-01T23:45:56+00:00 PHP: 5.2.4 OS: Any Package Version: CVS Description: ------------ There are still some issues with processing Unicode, so although the parser code works, unit tests (i.e. LexerGeneratorTest::testLexerGeneratorUnicode()) fail.]]> PHP_LexerGenerator Bug Reported by instance 2007-12-01T23:45:56+00:00 PHP: 5.2.4 OS: Any Package Version: CVS Description: ------------ There are still some issues with processing Unicode, so although the parser code works, unit tests (i.e. LexerGeneratorTest::testLexerGeneratorUnicode()) fail.]]> 2008-01-08T22:05:02+00:00 jal &#x61;&#116; ambitonline &#x64;&#111;&#x74; com PHP_LexerGenerator Bug