Smarty Forum Index Smarty
WARNING: All discussion is moving to https://reddit.com/r/smarty, please go there! This forum will be closing soon.

Inaccurate Lexer/Compiler - TRAILING NEWLINE Issue

 
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    Smarty Forum Index -> General
View previous topic :: View next topic  
Author Message
XPSystemServices
Smarty Rookie


Joined: 20 Apr 2017
Posts: 7

PostPosted: Thu Apr 20, 2017 11:46 am    Post subject: Inaccurate Lexer/Compiler - TRAILING NEWLINE Issue Reply with quote

Why Template Compiler removes trailing NEWLINE-s in some cases?
This will produce an inaccurate HTML-code.
Please look at the examples shown below:

Example 1
Code:
SOURCE TEMPLATE:
=====================
a
{* comment 1 *}
{* comment 2 *}
{* comment 3 *}
b

RESULT HTML (accurate, as expected):
=====================
a



b


Example 2
Code:
SOURCE TEMPLATE:
=====================
a
b{if ...}
d<span></span>
f{/if}
h

RESULT HTML (inaccurate):
=====================
a
bd<span></span>
fh

Expected result:
=====================
a
b
d<span></span>
f
h


Smarty should only remove Smarty-commands {...}, but shouldn't remove leading and trailing html-content (such as whitespaces and newlines). Please fix it, because this will produce undefined behavior (for example inside html <pre>-tags).

Please let developer consider how to deal with leading/trailing content (for example using smarty pre-filters).

Thank you!

Smarty version: 3.1.30


Last edited by XPSystemServices on Fri Apr 21, 2017 10:58 pm; edited 1 time in total
Back to top
View user's profile Send private message
AnrDaemon
Administrator


Joined: 03 Dec 2012
Posts: 1785

PostPosted: Thu Apr 20, 2017 11:34 pm    Post subject: Reply with quote

This is a PHP problem. PHP truncates EOL character immediately after "?>".
The only way you can get the code you want is to account for this PHP behavior.
Back to top
View user's profile Send private message
XPSystemServices
Smarty Rookie


Joined: 20 Apr 2017
Posts: 7

PostPosted: Fri Apr 21, 2017 7:20 am    Post subject: Reply with quote

Ok, this is documented feature in PHP, and this behavior will newer change.

What is the cause of the problem? And why you can't fix it?

Is it possible to add extra "newline" after "?>" when {/if} comes with trailing "newline". For example:

"{/if}\r\n" compile to "?>\r\n\r\n"

"{/if}\n" compile to "?>\n\n"

"{/if}" compile to "?>"

...
Back to top
View user's profile Send private message
AnrDaemon
Administrator


Joined: 03 Dec 2012
Posts: 1785

PostPosted: Fri Apr 21, 2017 4:25 pm    Post subject: Reply with quote

While it is rare to see Smarty used to construct binary data, it is not uncommon.
The problem is that you can't always say with certain that this or that EOL in template is significant for the resulting document. But as long as the compiled code matches template code closely, you can predict the behavior and account for it.
Back to top
View user's profile Send private message
XPSystemServices
Smarty Rookie


Joined: 20 Apr 2017
Posts: 7

PostPosted: Fri Apr 21, 2017 6:36 pm    Post subject: Reply with quote

AnrDaemon wrote:
While it is rare to see Smarty used to construct binary data, it is not uncommon.


We are not talking about binary data. You can't ignore, that output inside <pre></pre> is always broken.

AnrDaemon wrote:

The problem is that you can't always say with certain that this or that EOL in template is significant for the resulting document.


This is completely no problem. Developers can easily control output using pre/post filters. There are a lot useful solutions... for example this one:

https://gist.github.com/rodneyrehm/3013807

But at this time this is very hard and not intuitive because output content is inaccurate.

AnrDaemon wrote:

But as long as the compiled code matches template code closely, you can predict the behavior and account for it.


No and No! In the current version (3.1.30) this behavior is completely unpredictable, because Compiler sometimes merges several smarty-commands to the single PHP-block, and sometimes not. A NEWLINES between smarty-commands in some cases disappears, but kept in some other cases.

There is a simple (demo) solution to implement accurate output in the current version of Smarty using pre-filter. We can replace NEWLINES in the source template with some escape sequence, and then compile template, and finally replace NEWLINES back.

Code:

function smarty_prefilter_trailing_NL_control($string, Smarty_Internal_Template $template) {
  return preg_replace(
    '/' . preg_quote($template->smarty->right_delimiter) . '(\r?\n)' . '/msi',
    $template->smarty->right_delimiter . '{$smarty.const.PHP_EOL}',
    $string
  );
}


But this is really ugly solution. It currently does not handle {literal}-commands.

This behavior should be implemented in Smarty-core.

PS: Sorry for my english.
Back to top
View user's profile Send private message
AnrDaemon
Administrator


Joined: 03 Dec 2012
Posts: 1785

PostPosted: Fri Apr 21, 2017 7:53 pm    Post subject: Reply with quote

Please be more specific, than "several".
I can imagine certain sequence of blocks being merged due to compilation and caching reasons, but specific examples would help to better understand your issue.
Back to top
View user's profile Send private message
XPSystemServices
Smarty Rookie


Joined: 20 Apr 2017
Posts: 7

PostPosted: Fri Apr 21, 2017 10:34 pm    Post subject: Reply with quote

AnrDaemon wrote:
Please be more specific, than "several".
I can imagine certain sequence of blocks being merged due to compilation and caching reasons, but specific examples would help to better understand your issue.


Code:

These commands will be compiled to single PHP-block
=======
{/if}
{if}

These commands will be compiled to separate PHP-blocks
=======
{/if}

{if}


I mean this behavior is unpredictable, because of several problems:

1) Smarty margins some commands to a single PHP-block and omits NEWLINES between these commands.
2) PHP removes NEWLINES after PHP closing-tag (?>) but Smarty doesn't add it back.
3) Smarty removes NEWLINES between commands only for some specific sequence... For example NEWLINE for {if}\n{if} or {\if}\n{\if} will be removed, but for {* *}\n{* *} will be kept.

Below are shown absolutely equivalent examples... and absolutely identical outputs are expected for each example, but results are really unpredictable.

Code:
Source:
=====================
{if ...}
{foreach ...}
    <htmltag />
{/foreach}
{/if}
=====================

Result:
=====================
    <htmltag />

=====================

Code:
Source:
=====================
{if ...}
{if ...}
    <htmltag />
{/if}
{/if}
=====================

Result:
=====================
     <htmltag />
=====================

Code:
Source:
=====================
{* comment *}
{* comment *}
    <htmltag />
{* comment *}
{* comment *}
=====================

Result:
=====================


    <htmltag />


=====================
Back to top
View user's profile Send private message
U.Tews
Administrator


Joined: 22 Nov 2006
Posts: 5068
Location: Hamburg / Germany

PostPosted: Mon Nov 20, 2017 3:03 am    Post subject: Reply with quote

Indeed the behavior of the compiler did become unpredictable regarding the newline spacing between tags and text content.

The problem was caused by some efforts to reduce context switching between PHP code and text.

This will be fixed on github in the the master branch dev version 3.1.32-dev-36.

The behaviour will be indentical with Smarty2 for the time being.

However it's not identical with all results discussed here.

Example 1
Code:
SOURCE TEMPLATE:
=====================
a
{* comment 1 *}
{* comment 2 *}
{* comment 3 *}
b
=====================

RESULT HTML
=====================
a
b
=====================


Empty lines after removal of comments are removed. (Does make no sense to keep them.


Example 2
Code:

SOURCE TEMPLATE:
=====================
a
b{if ...}
d<span></span>
f{/if}
h
=====================

RESULT HTML
=====================
a
bd<span></span>
fh
=====================


This is the behaviour like Smarty 2. Multiline {if}{/if} will endup in many different opinions.

Example 3
Code:

SOURCE TEMPLATE:
=====================
{if ...}
{foreach ...}
    <htmltag />
{/foreach}
{/if}
=====================

RESULT HTML
=====================
    <htmltag />
=====================



Example 4
Code:

SOURCE TEMPLATE:
=====================
{if ...}
{if ...}
    <htmltag />
{/if}
{/if}
=====================

RESULT HTML
=====================
    <htmltag />
=====================


Example 5
Code:

SOURCE TEMPLATE:
=====================
{* comment *}
{* comment *}
    <htmltag />
{* comment *}
{* comment *}
=====================

RESULT HTML
=====================
    <htmltag />
=====================



I will make some tests how other template systems handle this and report ASAP.

I have added PHPUnit test for spacing at all tags to detect any side effects of code changes in future immedeately.
Back to top
View user's profile Send private message
U.Tews
Administrator


Joined: 22 Nov 2006
Posts: 5068
Location: Hamburg / Germany

PostPosted: Mon Nov 20, 2017 4:09 am    Post subject: Reply with quote

Update

Twig produce exacty the same results as my examples.
So we do now same thing.


Last edited by U.Tews on Mon Nov 20, 2017 12:56 pm; edited 1 time in total
Back to top
View user's profile Send private message
AnrDaemon
Administrator


Joined: 03 Dec 2012
Posts: 1785

PostPosted: Mon Nov 20, 2017 11:50 am    Post subject: Reply with quote

Remind me to re-check the email templates…
Back to top
View user's profile Send private message
XPSystemServices
Smarty Rookie


Joined: 20 Apr 2017
Posts: 7

PostPosted: Tue Dec 19, 2017 1:05 pm    Post subject: Reply with quote

Thanks for answer!

OK!

Can you please add these rules and behaviors as Smarty Standards and describe them in Smarty Documentation?

I think, these rules should be strict and standard!

We need to know how to Smarty removes their commands from templates and we need to know in what cases NEWLINES will be kept or removed.

We need that rules to deal with Smarty commands inside HTML<pre>-tags for example...
Back to top
View user's profile Send private message
Display posts from previous:   
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    Smarty Forum Index -> General All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group
Protected by Anti-Spam ACP