Page 1 of 1

Tokenizing template engine

Posted: Sat Sep 24, 2005 11:56 pm
by php_wiz_kid
Hey everyone,
I'm creating a template engine and right now I have it setup using ereg_replace(). I've read around, and some sources say that using a tokenizer engine is faster. I was wondering if that is true, and if so where can I read up on tokenizers and how tokenizing works?

Posted: Sun Sep 25, 2005 12:59 am
by feyd
depending on how you are doing the tokenizing, yes, a tokenizing engine can process text faster. FYI, ereg_replace() is extremely slow compared to preg_replace()

Posted: Sun Sep 25, 2005 1:48 am
by php_wiz_kid
Are there any resources online where I can find out how tokenizing works?

Posted: Sun Sep 25, 2005 2:20 pm
by EvilWalrus
This code was originally for my AIM library, where i had to be able to reload PHP functions without having to shutdown and restart the script...

What the following function does ( Tokenizer::tokenize_module() ) is it tokenizes the requested file into 'function' and 'code'. This was custom created just for PHP's create_function....

Code: Select all

<?php

/**
 * Module Tokenizer Class
 */

class Tokenizer
{
    private $file;

    public function tokenize_module($file)
    {
        $this->file = file_get_contents($file);
        $tokens = token_get_all($this->file);

        $temp = '';
        $func = array();
        $code = '';
        $i = 0;
        $b = 0;

        if (is_array($tokens)) {
            foreach($tokens as $c) {
                if ($i == 0 && is_array($c) && token_name($c[0]) == 'T_FUNCTION') {
                    $i++;
                }

                if ($i == 1 && is_array($c) && token_name($c[0]) == 'T_STRING') {
                    $tmp = explode('_', $c[1], 2);
                    if (!empty($tmp[1])) {
                        $temp = $tmp[0];
                        $func[$temp]['callback'] = $tmp[1];
                        $i++;
                    } else {
                        $i = 0;
                    }
                }

                if ($i == 2 && $c == '{') {
                    $i++;
                }

                if ($i == 3) {
                    if (is_array($c)) {
                        $code .= $c[1];
                    } else {
                        $code .= $c;
                        if ($c == '{') $b++;
                        if ($c == '}') $b--;
                    }
                    if ($b == 0) {
                        $func[$temp]['code'] = substr_replace(substr_replace($code,'', -1), '',0,1);
                        // important to reset the variables, or you might
                        // end up debugging your code for 5 days *cough*
                        $i = 0; $b = 0; $code = '';
                    }
                }
            }
            return $func;
        }
        return false;
    }

}

?>