Tokenizing template engine

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
php_wiz_kid
Forum Contributor
Posts: 181
Joined: Tue Jun 24, 2003 7:33 pm

Tokenizing template engine

Post by php_wiz_kid »

Hey everyone,
I'm creating a template engine and right now I have it setup using ereg_replace(). I've read around, and some sources say that using a tokenizer engine is faster. I was wondering if that is true, and if so where can I read up on tokenizers and how tokenizing works?
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

depending on how you are doing the tokenizing, yes, a tokenizing engine can process text faster. FYI, ereg_replace() is extremely slow compared to preg_replace()
php_wiz_kid
Forum Contributor
Posts: 181
Joined: Tue Jun 24, 2003 7:33 pm

Post by php_wiz_kid »

Are there any resources online where I can find out how tokenizing works?
User avatar
EvilWalrus
Site Admin
Posts: 209
Joined: Thu Apr 18, 2002 3:21 pm
Location: Springmont, PA USA

Post by EvilWalrus »

This code was originally for my AIM library, where i had to be able to reload PHP functions without having to shutdown and restart the script...

What the following function does ( Tokenizer::tokenize_module() ) is it tokenizes the requested file into 'function' and 'code'. This was custom created just for PHP's create_function....

Code: Select all

<?php

/**
 * Module Tokenizer Class
 */

class Tokenizer
{
    private $file;

    public function tokenize_module($file)
    {
        $this->file = file_get_contents($file);
        $tokens = token_get_all($this->file);

        $temp = '';
        $func = array();
        $code = '';
        $i = 0;
        $b = 0;

        if (is_array($tokens)) {
            foreach($tokens as $c) {
                if ($i == 0 && is_array($c) && token_name($c[0]) == 'T_FUNCTION') {
                    $i++;
                }

                if ($i == 1 && is_array($c) && token_name($c[0]) == 'T_STRING') {
                    $tmp = explode('_', $c[1], 2);
                    if (!empty($tmp[1])) {
                        $temp = $tmp[0];
                        $func[$temp]['callback'] = $tmp[1];
                        $i++;
                    } else {
                        $i = 0;
                    }
                }

                if ($i == 2 && $c == '{') {
                    $i++;
                }

                if ($i == 3) {
                    if (is_array($c)) {
                        $code .= $c[1];
                    } else {
                        $code .= $c;
                        if ($c == '{') $b++;
                        if ($c == '}') $b--;
                    }
                    if ($b == 0) {
                        $func[$temp]['code'] = substr_replace(substr_replace($code,'', -1), '',0,1);
                        // important to reset the variables, or you might
                        // end up debugging your code for 5 days *cough*
                        $i = 0; $b = 0; $code = '';
                    }
                }
            }
            return $func;
        }
        return false;
    }

}

?>
Post Reply