Smarty Forum Index Smarty
WARNING: All discussion is moving to https://reddit.com/r/smarty, please go there! This forum will be closing soon.

Smarty without compile directory (patch included)

 
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    Smarty Forum Index -> Feature Requests
View previous topic :: View next topic  
Author Message
Zimzat
Smarty Rookie


Joined: 02 Apr 2004
Posts: 5
Location: The Abyss

PostPosted: Thu Apr 12, 2007 4:27 pm    Post subject: Smarty without compile directory (patch included) Reply with quote

I've been working with Smarty for a long time now. One of the biggest draw-backs for using Smarty in a distributed product is that you have to have a writable compile directory before you can use it. This makes scenarios like product installation or system errors unable to use Smarty.

A while back I started creating a framework for my personal use. Concerned with this problem and not wanting to duplicate code in the installation and core parts, I came up with a way to get Smarty to work without a compile directory. This involved using a custom stream wrapper and telling Smarty to use that. The only problem with this was that Smarty does a is_writable() on the directory, and stat() calls do not work on stream wrappers in PHP 4.

To this end I modified Smarty to have a flag to allow overriding this check. It's a fairly small patch, adding about a dozen lines (half of which are comments) and changing one line.

Patch: http://www.zimzat.com/code/override_compile.diff
Example Usage: http://tools.gallery2.org/pastebin/1480
A version of the var_stream class can be found on the PHP stream_wrapper_register() documentation page. I use a different version that supports other variable types (e.g. $_GLOBAL)

What are your thoughts on getting Smarty to run without a compile directory?
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger
boots
Administrator


Joined: 16 Apr 2003
Posts: 5611
Location: Toronto, Canada

PostPosted: Thu Apr 12, 2007 7:23 pm    Post subject: Reply with quote

Hi.

I haven't looked at the patch yet but I think that in-general, this is a good approach. I've been wanting to rewrite Smarty to use streams pervasively but stopped that effort because of the poor (and slow) streams support in PHP4. I definitely think that Smarty should rely on an abstracted view of the file system (ie: a streams approach) in the next major version. For the current series, I think it may not be a good idea to change it (also, streams require PHP 4.3+, IIR) but I will look at your patch.

Best Regards!
Back to top
View user's profile Send private message
Zimzat
Smarty Rookie


Joined: 02 Apr 2004
Posts: 5
Location: The Abyss

PostPosted: Thu Apr 12, 2007 8:19 pm    Post subject: Reply with quote

Stream support was added in 4.3; however, custom streams require 4.3.2.

All my patch does is enable the usage of streams in PHP 4 without causing an error in Smarty. The actual work of implementing and using a stream is done in an outside class. I've uploaded the version I'm using here: http://www.zimzat.com/code/stream-var.class.txt
(In truth, the version I use does a lot more than it really needs to)

I mentioned this idea to Gallery v2, and they suggested I see what you guys thought of getting Smarty to run without a compile directory. They won't actually be able to use my custom stream solution, though, because their minimum required PHP version is only 4.3.

I also agree with your assessment about the performance of using a stream like this. If it weren't that the uses of it were narrow and often single-use (installation), then it wouldn't really be worth it at all.
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger
Zimzat
Smarty Rookie


Joined: 02 Apr 2004
Posts: 5
Location: The Abyss

PostPosted: Thu Apr 12, 2007 8:21 pm    Post subject: Reply with quote

I was just thinking, that one alternate way of doing without a compile directory is to simply put the compiled template in a variable, then eval() that. It's been a while since I've looked at the guts of Smarty so I don't remember how feasible this solution is. If you think it would be an idea that's possible and something you might add, I'll look further into it and see about getting a patch back to you.
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger
boots
Administrator


Joined: 16 Apr 2003
Posts: 5611
Location: Toronto, Canada

PostPosted: Thu Apr 12, 2007 8:43 pm    Post subject: Reply with quote

Personally, I'm against the eval'ing option as a general solution. It comes very close to completely negating Smarty's compiling benefits, IMO.
Back to top
View user's profile Send private message
Zimzat
Smarty Rookie


Joined: 02 Apr 2004
Posts: 5
Location: The Abyss

PostPosted: Thu Apr 12, 2007 9:09 pm    Post subject: Reply with quote

This is true.

The situation I'm wondering about is for when there isn't a compile directory or really anywhere else to save the compiled version to in the first place (e.g. initial installation). We might be able to use the system's temp directory, but that could fail too. The variable stream I mentioned above doesn't save the compiled version anywhere, so on the next invocation it re-complies the entire thing anyway. In this scenario is there still a reason not to use eval()?
Back to top
View user's profile Send private message Send e-mail Visit poster's website AIM Address Yahoo Messenger MSN Messenger
boots
Administrator


Joined: 16 Apr 2003
Posts: 5611
Location: Toronto, Canada

PostPosted: Fri Apr 13, 2007 1:14 am    Post subject: Reply with quote

Hi.

Are user streams cacheable via APC/eaccelerator/etc ? I think so, which means you would lose that benefit if using eval instead of a normal include.

I can also imagine a slightly more complex stream wrapper which fully manages variable access and thereby enables timestamping. The variable store could then be cached (say in session). At this point, you aren't gauranteed to have recompiles.

Diverting the compile_dir via streams is one thing (which I do favor). Forgoing the caching nature of the compiled files is something else (which I do not favor); if the latter is the main goal, then I think that Smarty might not be the proper solution since it introduces a lot of overhead mechanics that are redundant to the needs of the problem.
Back to top
View user's profile Send private message
Chainfire
Smarty n00b


Joined: 19 Apr 2007
Posts: 4

PostPosted: Fri Apr 20, 2007 2:35 pm    Post subject: Reply with quote

Heya,

I am all for applying the path. Though I did it somewhat differently:

in core.write_file.php :
Code:

function smarty_core_write_file($params, &$smarty)
{
    $_dirname = dirname($params['filename']);

    // START INSERTED CODE
    if (strpos($params['filename'], '://') !== false) {
      $fd = fopen($params['filename'], 'wb');
      fwrite($fd, $params['contents']);
      fclose($fd);
      return true;
    }
    // END INSERTED CODE

    if ($params['create_dirs']) {


This is because of the temporary file being used first by Smarty which is later renamed. tempnam($_dirname) for most PHP versions will use the standard system temp folder instead of $_dirname you provide, so it will result in an error because you cannot rename files between disk and wrapper.

With those few lines added, I could use a stream for template_dir and compile_dir without problems (on PHP 5.2, stat calls work on stream wrappers). I wrote a stream wrapper to make use of XCache's variable cache (by replacing two functions can also be used with memcached instead) with a disk fallback. This totally eliminates smarty disk access for recurring requests (within a set TTL) and can improve performance by a fair bit under heavy load (as the templates are only checked and if necessary recompiled every TTL seconds, instead of on each request). It differs in 'output caching' as the templates are still 'executed'.

I can also confirm that XCache's opcode cache has no trouble with caching the compiled TPL PHP from the stream wrapper.

Combined with using XCache's (or memcached, or whatevers) variable cache for database stuff (and sessions), you can reduce disk access to absolute zero for recurring requests on some pages, keeping them fast under heavy load.
Back to top
View user's profile Send private message
boots
Administrator


Joined: 16 Apr 2003
Posts: 5611
Location: Toronto, Canada

PostPosted: Fri Apr 20, 2007 11:30 pm    Post subject: Reply with quote

Hmmmm, sounds intriguing.

Chainfire, does your patch supercede ZImzats or augments it? At first glance it seems that yours obviates the need for an additional config parameter (override_compile) which would be nice.

Can you also post your cache handler? Is it just a modification of the eaccelerator cache handler?

Thanks.
Back to top
View user's profile Send private message
Chainfire
Smarty n00b


Joined: 19 Apr 2007
Posts: 4

PostPosted: Sat Apr 21, 2007 11:46 am    Post subject: Reply with quote

boots wrote:
Hmmmm, sounds intriguing.

Chainfire, does your patch supercede ZImzats or augments it? At first glance it seems that yours obviates the need for an additional config parameter (override_compile) which would be nice.

Can you also post your cache handler? Is it just a modification of the eaccelerator cache handler?

Thanks.


My patch replaces Zimzats, and indeed it removes the need for the additional config parameter. My small patch there however does not implement the workaround Zimzats does for the is_writable/is_directory etc calls. These calls trigger the url_stat method of the stream wrapper for PHP 5.2, but IIRC not for all PHP versions that support stream wrappers. You can add that workaround tool by checking for '://' in the filename instead of the directive, though.

I do not use a special cache handler, just the custom stream wrapper, like so:

$smarty->template_dir = 'varcache://SmartyTemplates::/var/www/vhosts/mysite.com/templates/';
$smarty->compile_dir = 'varcache://SmartyCompiled::/var/www/vhosts/mysite.com/templates_c/';

if you would like to use output caching as well, I assume

$smarty->cache_dir = 'varcache://SmartyCache::/var/www/vhosts/mysite.com/cache/';

would do the trick. (I have not looked at eaccelerator cache handler so I cannot comment on it)
Back to top
View user's profile Send private message
Chainfire
Smarty n00b


Joined: 19 Apr 2007
Posts: 4

PostPosted: Thu Apr 26, 2007 4:29 pm    Post subject: Reply with quote

Some updates,

I have not done anything with the normal 'cache handler', as I don't use the smarty caching feature (yet), for now I like having my code actually processed, and with the opcode cache, variable cache and no disk access, i must say the speed is great.

Anyhoo, I had to patch some more in Smarty, as it seems the PHP devs, in their infinite wisdom, think it's a good idea (really they do, check bugtracker) that 'fread' works differently on stream wrappers than real files, just to keep things consistent I'm sure. The result is, that Smarty never reads more than 8192 bytes from the source template if read from a variable stream.

So aside from the changes to 'core.write_file.php' @ line 20:

Code:

function smarty_core_write_file($params, &$smarty)
{
    $_dirname = dirname($params['filename']);

    /* Hack for stream wrappers - START */
    /* No code is being replaced, this is inserted */
    if (strpos($params['filename'], '://') !== false) {
      $fd = fopen($params['filename'], 'wb');
      fwrite($fd, $params['contents']);
      fclose($fd);
      return true;
    }
    /* Hack for stream wrappers - END */

    if ($params['create_dirs']) {
        $_params = array('dir' => $_dirname);
        require_once(SMARTY_CORE_DIR . 'core.create_dir_structure.php');
        smarty_core_create_dir_structure($_params, $smarty);
    }

    // write to tmp file, then rename it to avoid
    // file locking race condition
    $_tmp_file = tempnam($_dirname, 'wrt');

    if (!($fd = @fopen($_tmp_file, 'wb'))) {
        $_tmp_file = $_dirname . DIRECTORY_SEPARATOR . uniqid('wrt');
        if (!($fd = @fopen($_tmp_file, 'wb'))) {
            $smarty->trigger_error("problem writing temporary file '$_tmp_file'");
            return false;
        }
    }

    fwrite($fd, $params['contents']);
    fclose($fd);

    // Delete the file if it allready exists (this is needed on Win,
    // because it cannot overwrite files with rename()
    if (file_exists($params['filename'])) {
        @unlink($params['filename']);
    }
    rename($_tmp_file, $params['filename']);
    @chmod($params['filename'], $smarty->_file_perms);

    return true;
}


'Smarty.class.php' also needs to be patched @ line 1708:

Code:

    function _read_file($filename)
    {
        if ( file_exists($filename) && ($fd = @fopen($filename, 'rb')) ) {
          /* Hack for stream wrappers - START */
          /* Replaced code 'commented' below */
             $size = filesize($filename);
             $contents = '';
           while (strlen($contents) != $size) {
         if (strlen($buffer = fread($fd, 8192)) == 0) { break; }
                  $contents .= $buffer;
             }
           return $contents;
           /* Hack for stream wrappers - REPLACED CODE
                    $contents = ($size = filesize($filename)) ? fread($fd, $size) : '';
                    fclose($fd);
                    return $contents;
            Hack for stream wrappers - END */            
        } else {
            return false;
        }
    }


Changed code is clearly marked. It's a bit of an odd construct, but it works for both real files and stream wrappers (though probably not socket stream wrappers). Sorry for the indentation being a bit off!

EDIT: The 8192 bytes issue is only seen on newer PHPs ( >= 5.0.5 ), and it doesn't seem to happen on my 5.2.1 windows version but it sure does for my 5.2.1 linux version, just FYI.
Back to top
View user's profile Send private message
Chainfire
Smarty n00b


Joined: 19 Apr 2007
Posts: 4

PostPosted: Thu Apr 26, 2007 10:04 pm    Post subject: Reply with quote

In case anyone is interested, here is some 'cleaned up' code ripped out of our framework to handle the streams for XCache; it works well for us, though YMMV. Ofcourse, you need to apply the two patches mentioned above. Tested only on PHP 5.2.1. As said, it is a bit cleaned up, ours uses different cache get/set function that can be replaced with memcached versions and such, and I stripped out the TTL stuff since it relied on other code, you can add the TTL param to the xcache_set calls manually or just use the default xcache variable cache TTL. We use this with xcache.stat = Off (whats the use if that is on Wink) but you need to clear the relevant caches each time you update your sites php/tpl files as a result.

Code:

<?php

/*

  (C) Copyright 2007 Jorrit Jongma ( jorrit@jongma.org )
  Public Domain or BSD license, whichever is legal where you are ;)

*/

class stream_smarty_xcache {
    private $_id = '';
    private $_file = '';
    private $_open = false;
    private $_pos = 0;
    private $_mode = 0; // 1 = read, 2 = write, 3 = both
    private $_data = null;
    private $_isdir = false;
    private $_time = 0;
    private $_size = 0;
    private $_path = '';
    private $_src = 'mem';
       
    private function exists_data($path, $store = false) {
      $tmp = substr($path, strpos($path, '://') + 3);
      $id = $tmp;
      $file = substr($tmp, strpos($tmp, '::') + 2);
       $data = xcache_get($id);
       if ($data == false) {
          if (file_exists($file)) {
             $data = array('time' => filemtime($file), 'size' => filesize($file));
             if ($store) {
                xcache_set($id, array('time' => $data['time'], 'size' => $data['size'], 'content' => file_get_contents($file)));
             }
             return $data;
          } else {
             return false;
          }
       } else {
          return array('time' => $data['time'], 'size' => $data['size']);
       }
    }

    private function load_data() {
       $data = xcache_get($this->_id);
       if ($data != false) {
          $this->_time = $data['time'];
          $this->_size = $data['size'];
          $this->_data = $data['content'];
          $this->_src = 'mem';
           return true;                  
       } else {
          if (file_exists($this->_file)) {
             $this->_time = filemtime($this->_file);
             $this->_size = filesize($this->_file);
             $this->_data = file_get_contents($this->_file);
             $this->_src = 'disk';
             return true;
          } else {
             $this->_data = '';
             $this->_time = time();
             $this->_size = 0;
             return false;
          }
       }
    }
   
    private function save_data() {
         if (($this->_mode & 2) || ($this->_src == 'disk')) {
         xcache_set($this->_id, array('time' => $this->_time, 'size' => $this->_size, 'content' => $this->_data));
         }
    }
   
    function stream_open($path, $mode, $options, &$opened_path)
    {       
        $tmp = substr($path, strpos($path, '://') + 3);
        $this->_id = $tmp;
        $this->_file = substr($tmp, strpos($tmp, '::') + 2);
        $this->_path = $path;
        $fmode = $this->_mode;
       
        $ok = false;
        $mode = strtolower($mode);
        switch ($mode{0}) {
            case    "r"   :   $ok = $this->load_data();
                                $fmode = $fmode | 1;
                               break;
            case    "w" :
            case    "a" : $this->load_data();
                                 $ok = true;
                               $fmode = $fmode | 2;
                               break;
            case    "x"   : $this->load_data();
                                 $ok = true;
                               $fmode = $fmode | 2;
                               break;
        }
       
          $opened_path = $this->_file;

          if (!$ok) { return false; }

        $this->_mode = $fmode;
        $this->_pos = 0;
        $this->_open = true;
          $this->_isdir = false;

        if ($mode{0} == 'a') {
            $this->stream_seek(0, SEEK_END);
        }

        if (strlen($mode) > 1 && $mode{1} == '+') {
            $this->_mode = $this->_mode | 1 | 2;
        }

        return true;
    }

    function stream_eof()
    {
        return ($this->_pos >= $this->_size);
    }

    function stream_tell()
    {
        return $this->_pos;
    }

    function stream_close()
    {
        $this->save_data();
        $this->_pos  = 0;
        $this->_open = false;
    }

    function stream_read($count)
    {
        if (!$this->_open) {
            return false;
        }

        if (!($this->_mode & 1)) {
            return false;
        }

        $data = substr($this->_data, $this->_pos, $count);
        $this->_pos = $this->_pos + strlen($data);
        return $data;
    }

    function stream_write($data)
    {
        if (!$this->_open) {
            return false;
        }
       
        if (!($this->_mode & 2)) {
            return false;
        }
       
        $datalen = strlen($data);
       
        $this->_data = substr($this->_data, 0, $this->_pos) . $data . substr($this->_data, $this->_pos+$datalen);
        $this->_pos = $this->_pos + $datalen;
        if ($this->_pos > $this->_size) {
           $this->_size = $this->_pos;
        }
        return $datalen;
    }

    function stream_seek($offset, $whence)
    {
        switch ($whence) {
            case SEEK_SET:
                if (($offset < $this->_size) && ($offset >= 0)) {
                     $this->_pos = $offset;
                     return true;
                } else {
                     return false;
                }
                break;
            case SEEK_CUR:
                if ($offset >= 0) {
                     $this->_pos += $offset;
                     return true;
                } else {
                     return false;
                }
                break;
            case SEEK_END:
                if ($this->_size + $offset >= 0) {
                     $this->_pos = $this->_size + $offset;
                     return true;
                } else {
                     return false;
                }
                break;
            default:
                return false;
        }
    }

    function stream_flush()
    {
        return true;
    }

    function stream_stat()
    {
        if ($this->_isdir) {
            $mode = 0040000;
            $size = 0;
                  $time = time();
            } else {
                $mode = 0100000;
                $size = $this->_size;
                  $time = $this->_time;
            }
        $mode = $mode | 0777;

        $keys = array(
            'dev'     => 0,
            'ino'     => 0,
            'mode'    => $mode,
            'nlink'   => 0,
            'uid'     => 0,
            'gid'     => 0,
            'rdev'    => 0,
            'size'    => $size,
            'atime'   => $time,
            'mtime'   => $time,
            'ctime'   => $time,
            'blksize' => 0,
            'blocks'  => 0
        );

        return array_merge(array_values($keys), $keys);
    }

      function url_stat($path, $flags) {
           if (strpos($path, '.tpl') === false) {
          $mode = 0040000;
          $size = 0;
          $time = time();
            } else {
             $mode = 0100000;
             $data = $this->exists_data($path, true);
             if ($data === false) { return false; }
             $time = $data['time'];
             $size = $data['size'];
            }
        $mode = $mode | 0777;

        $keys = array(
            'dev'     => 0,
            'ino'     => 0,
            'mode'    => $mode,
            'nlink'   => 0,
            'uid'     => 0,
            'gid'     => 0,
            'rdev'    => 0,
            'size'    => $size,
            'atime'   => $time,
            'mtime'   => $time,
            'ctime'   => $time,
            'blksize' => 0,
            'blocks'  => 0
        );
       
        return array_merge(array_values($keys), $keys);
    }
   
    function unlink($path) {
       return true;
    }
   
    function rename($from, $to) {
       return true;
    }
   
    function mkdir($path, $mode, $options) {
       return true;
    }
   
    function rmdir($path, $options) {
       return true;
    }

    function dir_opendir($path, $options)
    {
        $tmp = substr($path, strpos($tmp, '://') + 2);
        $this->_id = $tmp;
        $this->_file = substr($tmp, strpos($tmp, '::') + 1);
        $this->_path = $path;
        $this->_isdir = true;
       
        $this->_open = true;
        return true;
    }

    function dir_closedir()
    {
        $this->_open = false;
        return true;
    }

    function dir_rewinddir()
    {
        if (!$this->_open) {
            return false;
        }
        return true;
    }

    function dir_readdir()
    {
          return false;
    }
}

?>


Usage:

Code:

stream_wrapper_register('smartyxcache', 'stream_smarty_xcache');      

$smarty = new Smarty;
$smarty->template_dir = 'smartyxcache://SmartyTemplates::/path/to/smarty/templates/';
$smarty->compile_dir = 'smartyxcache://SmartyCompiled::/path/to/smarty/templates_c/';
...


Do note that that if you use this code from several PHP files in different directories, it will be cached multiple times, once for each different directory, due to a bug in xcache. Also note that the templates_c directory is not actually used if it does not exist.
Back to top
View user's profile Send private message
Display posts from previous:   
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    Smarty Forum Index -> Feature Requests All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group
Protected by Anti-Spam ACP