|
Smarty
WARNING: All discussion is moving to https://reddit.com/r/smarty, please go there! This forum will be closing soon. |
|
View previous topic :: View next topic |
Author |
Message |
boots Administrator
Joined: 16 Apr 2003 Posts: 5611 Location: Toronto, Canada
|
Posted: Mon Dec 13, 2004 9:30 pm Post subject: Banded Report Generator |
|
|
Source code (at the wiki).
Updates
v0.1.5 (June 21, 2005)
- general code cleanup (boots)
- headers now have access to grouping stats (boots)
- fixed grouping bug (sophistry)
General
Presented here are a set of plugins that form a banded report generator tool for recordsets. I am presently working on a more ambitous version of this code that will offer several enhanced features and optimizations; since I will be going on holidays shortly I have decided to release the current working version so as to hopefully elicit some comments and feedback. For those not familiar with banded report generators, they are essentially an advanced looping device that automate reporting functions by defining logical sections of reports as "bands". As the recordset is processed, each band is triggered as required. This can also be used to simplify other looping needs (eg: dynamic tabular columns). If you ever used Access or Crystal Reports, you know what a banded report generator is
As it is, there is some useful functionality already available but I do not plan full 100% BC with this code so if you plan on using this, I don't suggest you put this into a production system just yet.
When the plugins are completed I will provide better documentation as well as an article that describes how they were built. In the meantime, enjoy!
Special thanks go to messju for providing insight and help in untangling some of the looping details and showing the way on using the tag stack to store local data instead of using the traditional method of using a custom private member of the running Smarty instance. Thanks to sophistry for bug fixes and testing.
Notes
only text/items within the {report_*} blocks are emitted. All other output in the {report}{/report} block is ignored.
a recordset follows the standard definition and is an indexed array of records where each record is an associative array all of which have identical fields/typing.
you must specify grouping levels if you wish to use them
all {report_*} sub blocks are optional -- for example, you can produce a summary report by not including a {report_detail}{/report_detail} block.
Sample Usage
report.php
[php:1:9d7aef2b2b]<?php
$smarty = new Smarty();
if (!$smarty->is_cached('report.tpl')) {
$data = array();
foreach (array(2003, 2004, 2005) as $year) {
foreach (array(1, 2, 3, 4) as $quarter) {
foreach (array('ca', 'us') as $region) {
foreach (array('foo', 'bar', 'baz') as $item) {
$sales = rand(2000,20000);
$data[] = compact('year', 'quarter', 'region', 'item', 'sales');
}
}
}
}
$smarty->assign('data', $data);
}
$smarty->display('report.tpl');
?>[/php:1:9d7aef2b2b]
report.tpl
Code: | {report recordset=$data record=rec groups="region,year,quarter" resort=true}
{report_header}
START OF REPORT
<hr/>
<table width="400" border=1 cellspacing=1 cellpadding=1>
{/report_header}
{report_header group="region"}
<tr style="background:blue; color:white;">
<td>SUMMARY FOR REGION '{$rec.region}'</td>
<td>{$count.sales} items totalling {$sum.sales} for an average of {$avg.sales} per item</td>
</tr>
{/report_header}
{report_header group="year"}
<tr>
<td align="center" colspan=2>YEAR: {$rec.year}</td>
</tr>
{/report_header}
{report_header group="quarter"}
<tr>
<td align="center" colspan=2>Q{$rec.quarter}</td>
</tr>
{/report_header}
{report_detail}
<tr>
<td>{$rec.item}</td>
<td>{$rec.sales}</td>
</tr>
{/report_detail}
{report_footer group="region"}
<tr>
<td>TOTALS FOR REGION '{$rec.region}'</td>
<td>{$count.sales} items totalling {$sum.sales} for an average of {$avg.sales} per item</td>
</tr>
{/report_footer}
{report_footer}
<tr>
<td>GRAND TOTALS</td>
<td>{$count.sales} items totalling {$sum.sales} for an average of {$avg.sales} per item</td>
</tr>
</table>
<hr/>
END OF REPORT
{/report_footer}
{/report}
|
Last edited by boots on Wed Jun 22, 2005 3:43 am; edited 4 times in total |
|
Back to top |
|
jamilmalik Smarty n00b
Joined: 01 Jan 2005 Posts: 1 Location: Pakistan
|
Posted: Sat Jan 01, 2005 7:16 am Post subject: |
|
|
Dear,
After copying and placing the 3 files in the relavent directories. Your example produce the error as follows:
Fatal error: Smarty error: [in report.tpl line 3]: syntax error: unrecognized tag 'report_header' (Smarty_Compiler.class.php, line 565) in e:\php\includes\Smarty.class.php on line 1088
Please help me to resolve this problem.
Thanks & regards.
Jamil
|
|
Back to top |
|
messju Administrator
Joined: 16 Apr 2003 Posts: 3336 Location: Oldenburg, Germany
|
Posted: Sat Jan 01, 2005 6:46 pm Post subject: |
|
|
it seems block.header.php should be named block.report_header.php instead.
HTH
messju |
|
Back to top |
|
boots Administrator
Joined: 16 Apr 2003 Posts: 5611 Location: Toronto, Canada
|
Posted: Sun Jan 02, 2005 11:36 pm Post subject: |
|
|
messju wrote: | it seems block.header.php should be named block.report_header.php instead. |
Yes, silly me. block.report_header.php and block.report_footer.php are the correct file names. I corrected the original post. Thanks! |
|
Back to top |
|
sophistry Smarty Rookie
Joined: 31 Jan 2005 Posts: 33
|
Posted: Mon Jan 31, 2005 9:55 pm Post subject: requests: header sums and "average" optimization |
|
|
Hi boots!
Thanks for this nice piece. I have two observations:
1) it seems that sums can only be accessed in footers... am i correct? if this is the case what is your suggestion for implementing header sums (i call them leading sub-summaries)? is it possible to use {capture} block to do such a thing?
2) it looks like the code calculates the average everytime through. in most cases i would think this is not necessary unless you want a running average. perhaps "grand average" and "running average" can be split out in the name of optimization! This might even be a noticeable optimization on large datasets.
BTW, the code ran perfectly out-of-the-box. I'm going to try to break it now!
soph |
|
Back to top |
|
boots Administrator
Joined: 16 Apr 2003 Posts: 5611 Location: Toronto, Canada
|
Posted: Mon Jan 31, 2005 10:41 pm Post subject: Re: requests: header sums and "average" optimizati |
|
|
Hi Sophistry. Thanks for the kind words--I haven't touched this since I posted it but you've given me the impetous to revisit.
Your comments are very astute -- this is not at all optimized and there are some significant limitations. I have plans to rework the looping and calculations and if I ever get to it, will do a compiler version. For now, I'd be happy to come up with a slightly better overall design and some easy to implement optimizations. Indeed, the reason I posted this at all was to try to illicit some feedback on usage patterns. Now that you got the ball rolling, I'll throw out some ideas to see what you think.
I have several options. Presently, nearly all of the heavy work is done in the {report} tag and nearly nothing is done in any of the other tags. One thing I've considered is factoring out some of the grouping code into the other tags. For one thing, this would mean that some of the caclulation steps can be avoided and for another, it should make the overall logic easier to follow. In general this should save some cycles for simple reports but it will mean that you will have to specify things more directly.
Another idea is to create a new "grouping" tag which merely specifies the logic of the grouping condition. This is where all the internal calculations for a group would go and a group header or footer would then be triggered based on the groups current status. The conditions would be similar to what you might expect for a banded report and I would like to implement all of the following and perhaps with a few more Group On options.
Field/Expression (this is tricky...)
Sort Order (Ascending/Descending)
Group On (Each Value, Prefix Characters)
Group Interval (numeric)
One problem is that this would mean that headers/footers would have to be connected to a degined "group" in someway.
Another challenge is how to specify what metrics are to be calculated on every loop. At present, I avoid this issue by calculating everything always but if it is to be done on an as-needed-basis, then there has to be an easy way to specificy what metrics should be tallied. I don't have a good idea on this front yet.
As for allowing headers to contain totals and such, I have never been a fan of that. The present code could easily support it since the values are already computed but it doesn't assign them to the header group. It is certainly possible to add it in but I am hesitant. I'll definately consider it.
Another idea I had was to not put any of the grouping logic into the template at all and instead require that a secondary array be defined with a particular format that specified all of the grouping requirements. The designer would then focus on how the report would look and not on what the report would cover. I do think that is splitting hairs too fine but it is a cleaner separation in some ways. There is a benefit to this: if I do decide to create a "group" tag which allows the specification of the tags, then sorting becomes an issue since the sorting must be done in the {report} tag for it to work properly and I'm trying to avoid specifying anything more than once.
Again, thanks for the comments. If you have any other insights or thoughts I'd be happy to hear them. |
|
Back to top |
|
sophistry Smarty Rookie
Joined: 31 Jan 2005 Posts: 33
|
Posted: Mon Jan 31, 2005 11:36 pm Post subject: sort by sum or avg & general way of handling calculation |
|
|
I'd like to help think these things out, but you are light years ahead of me in smartyville!
I always try to drive design by features. I look for the features I want and then try to meet them with a "just right" design. Some features i have in my current (spaghetti) reporting system are:
1) sort on results of sum or avg - allows table rows to be sorted after all computations are done
2) ability to show "top performer" at any particular level - eg say "this quarter foo was the top seller" and this year "bar" was the top seller
3) general way to do math in the system - eg i often need a column in reports that displays percent change from one to another column (apropos of #1, also, sort on that percent change).
4) cross-tabs - calculate sums in columns with arbitrary date clumpings and data clumpings
right now, i think the banded report design would support creating these sorts of calculations and sorts but i can't immediately see a clean way to do it with the kind of groups tag you talk about. That's really just a function of my inexperience examining nice modular smarty plugin code rather than a comment on your ideas.
another thing i'm always aware of while building out reporting thingees is that i am just re-creating SQL. i don't know enough about SQL to actually apply it here though i know that one version of SQL has an operator called CUBE that does cross-tabs (#4 above).
anyhow, more tomorrow...
Cheers,
soph |
|
Back to top |
|
Edufa Smarty n00b
Joined: 05 Feb 2005 Posts: 2
|
Posted: Thu Feb 10, 2005 12:27 am Post subject: Great plugin |
|
|
Great plugin.
I download the code and testing to do sugestions.
Edufa |
|
Back to top |
|
iurisj Smarty n00b
Joined: 09 Mar 2005 Posts: 1
|
Posted: Wed Mar 09, 2005 4:39 pm Post subject: |
|
|
I can't understand how do you assign the {$rec} variable from inside the block function. The documentation says that the $content grabbed by the block function is not the source, but the output of the template already parsed. On your "report__open" function you do an assign before it finishes. How can this work? I mean, according to documentation this assign have to be done before it enters on the block function.
By the way, it is a very nice solution. Congratulations... |
|
Back to top |
|
boots Administrator
Joined: 16 Apr 2003 Posts: 5611 Location: Toronto, Canada
|
Posted: Wed Mar 09, 2005 6:35 pm Post subject: |
|
|
iurisj wrote: | I can't understand how do you assign the {$rec} variable from inside the block function. The documentation says that the $content grabbed by the block function is not the source, but the output of the template already parsed. On your "report__open" function you do an assign before it finishes. How can this work? I mean, according to documentation this assign have to be done before it enters on the block function.
By the way, it is a very nice solution. Congratulations... |
Hi iurisj -- first, thanks for the nice comments. I'm not sure I follow your query entirely, but I will answer as best I can. The details are not complicated but they are somewhat involved. I apologize in advance if I am not being clear.
I decided to implement output from these block plugins in a way that is somewhat different (perhaps heretical) to the way it is typically done. Here, the main {report} tag begins a container for other tags. I made the decision that only the ouput of {report_*} tags are relevant to the overall output. In other words, any items other than the tags associated with a {report} block are to be ignored (in terms of output). This includes both static elements and non-{report_*} tags.
Normally, a block tag returns (a possibly modified form of) the content it recieves from Smarty. As you look at smarty_block_report(), you notice that on the tag close (ie: $content != null) smarty_block_report__close() is called. Normally, this would be the time to return the content. smarty_block_report__close() instead returns the contents of a private buffer and completely ignores the $content it recieved. The buffer is first created during the {report} open tag in smarty_block_report__init() as a simple empty string.
How is the buffer filled with content? Each of the {report_*} plugins, instead of simply returning their $content (again, as a normal block would) to Smarty, instead appends its $content to the {report} tag's buffer. In other words, only the {report_*} tags are capable of adding content to the buffer.
This achieves my aims and additionally, allows further control if I so desired. For example, I could create a format for the buffer (perhaps an associative array?) so that despite what order the tags are provided in the template, it would be possible to resort the buffer into a rational order, say header, detail, footer. There are other processing details that would greatly complicate out-of-order buffering but that isn't relevant here.
The big trick in making all of this work requires a quick review of some Smarty internals. When processing a template, Smarty maintains an internal stack so that it can store passed parameters, track depth of block plugins (to ensure matched close tags) and the like. I mentioned that the {report} tag has a private buffer created in smarty_block_report__init(). We exploit the internal stack and store, along with other private data, this buffer directly with the {report} tag's parameters.
The handling of the internal stack has been abstracted into two private functions, smarty_get_current_plugin_params() and smarty_get_parent_plugin_params(). Each of these return a reference to the parameters for a specific plugin (tag) on the stack. The first gets a reference for parameters of the plugin currently being processed by Smarty while the second gets the parameters for a specified parent of the plugin currently being processed. In this way, it is possible for the {report_*} tags to access and modify the parameter--and therefore private--data of their parent {report} tag. Although not at all OO, if we can borrow a metaphor you might think of this as a subversion of the {report} tag's data from "private" to a quasi-"protected" form accessible to its children (ie. "friend"s).
That's all I will write for now. If I ever get time, I would like to write an article to go into more depth on topics like this. I think some of these techniques are interesting and they are not well discussed.
Again, thanks for your interest! |
|
Back to top |
|
sophistry Smarty Rookie
Joined: 31 Jan 2005 Posts: 33
|
Posted: Tue Jun 14, 2005 7:30 pm Post subject: Expanding the code? Any recent updates? |
|
|
Hi boots,
Do you have any updates to this code? I am finally getting around to giving it a run on a real project and I would like to adapt your plugin to take care of one thing I need it to do:
--access the calculated data in header sections that is currently only available to the footer sections (count, sum, avg)
Do you have any code that does this?
You suggested earlier that this was easy to do, could you describe to me how you would do it and I can get my feet wet with your structured way of coding these plugins?
Thanks! |
|
Back to top |
|
boots Administrator
Joined: 16 Apr 2003 Posts: 5611 Location: Toronto, Canada
|
Posted: Tue Jun 14, 2005 10:37 pm Post subject: |
|
|
Hi Sophistry. It is always nice to see your comments
Sadly, this little side-project is idling at a very low priority for me so I don't have much more to add at this time other than what I have already posted. I can say that if you are looking to get the header totals working, your best bet is to look at how the footer code works and duplicate it. I'll try to give you a few things to consider since you took the time to ask:
Basically, on every iteration the master plugin (dealt with in smarty_block_report__open) tracks running values and calculates statistics as required. If a group end is detected, a sentinal value is set; if a group start is detected, a separate sentinal is set.
This is what smarty_block_footer uses to determine if it has work to do. If it detects the "last" sentinal, it unpacks the statistics set in smarty_block_report__open and assigns them to the template. You can do the same in smarty_block_header but you may have to change some other logic there as well. Currently, smarty_block_header only wants to fire on the "first" sentinal but to get the finalized stats, you need to wait until "last" is fired. At that point, of course, it is not really a header anymore so I'll leave it to you to iron those things out.
As for the structured coding, this is a bit on an experiment of how to deal with related plugins that work together to form a cohesive system and share state. Smarty itself doesn't provide mechanisms to do such things so I developed this pattern to try to address that. The main thing here are the plugin helper functions which allow children plugins to ensure that a parent plugin is active and also to allow plugins to share state information. Because of this, plugins can remain focused on their portion of the output control; operations which affect multiple children can be delegated into a single spot in the parent plugin. The output control is the last thing to consider: because I developed this for text based reports, I wanted more control over the output so I implemented output via a buffer instead of simply returning it. This is against normal Smarty usage and complicates these plugins. You needn't do the same unless you want to exact that level of control.
I would like to improve the code and eventually write more about it but alas, time is precious. Although I intend to do something about it at some point in the future, if you can better it or add more detail, your contributions would be appreciated. |
|
Back to top |
|
sophistry Smarty Rookie
Joined: 31 Jan 2005 Posts: 33
|
Posted: Wed Jun 15, 2005 6:30 pm Post subject: smarty architecture of nested plugins |
|
|
boots wrote: | This is what smarty_block_footer uses to determine if it has work to do. If it detects the "last" sentinal, it unpacks the statistics set in smarty_block_report__open and assigns them to the template. You can do the same in smarty_block_header but you may have to change some other logic there as well. Currently, smarty_block_header only wants to fire on the "first" sentinal but to get the finalized stats, you need to wait until "last" is fired. At that point, of course, it is not really a header anymore so I'll leave it to you to iron those things out.
|
Ah yes. I did try hacking into the header code by just copying the footer code. It didn't work, but I was able to come to grips with the logic you describe. It seemed to screw up the ordering of the output text.
I also tried just putting the footer tag before the header tag in the template, but that didn't work either.
I am going to try two more things:
1. nest the report tags to see if i can get my leading summaries in the headers that way
2. try to code what i need in native smarty foreach or section tags and captured assigns with math.
I have just realized that you are really taking advantage of the Smarty plugin framework in a deep way. You rely on the fact that block plugins are essentially iterators. That's why there is no foreach loop in the main report block code. And, inside each of the plugins (at the parent level and the child / group level), by setting repeat to false, you are breaking out of the loop and passing control back to the calling block. Finally, the return statements return control to the parent block to decide what to do. It's pretty cool stuff - like a little puzzle for Smarty plugin newbies.
Another thing is that I got confused because you call the internal function smarty_block_report__open() but because of the implicit Smarty plugin architecture control structure it really is more like smarty_block_report__next_record().
So, if I want leading sub-summaries (headers that can display the calculated summary data) what I really need to set up is a state machine that will keep track of the row in which data in a sorted "column" changes so that counts, sums, and averages can be calculated and stored when the data in that column (and/or the sort level above it) changes. But, that means that I need to maintain a column set array as well as the record set array that is currently there.
This is how my current report code works, but I'm struggling with how best to integrate it to capitalize on the work you've done here.
I think that the best thing to do would be to take a step back and try to understand how to acheive the ad hoc ordering you mentioned earlier. That would be the general solution that would afford the most flexibility in the long run.
So, you say it needs an associative array that carries the various report parts that have been buffered? If I create this array in the report plugin it should be possible to fill it with the output from the child plugins at their normal exit/dump time.
Where you dump the buffer in the child plugins... is that where assoc array assigning should happen?
Thanks boots! |
|
Back to top |
|
sophistry Smarty Rookie
Joined: 31 Jan 2005 Posts: 33
|
Posted: Wed Jun 15, 2005 6:59 pm Post subject: bug? with nested sort levels... |
|
|
This section of code from the report__open function seems to have a bug when two or more groups are specified.
Code: | // process grouping levels
foreach ($params['groups'] as $_group) {
if (!is_null($record['curr'])) {
if ($record['prev'][$_group] != $record['curr'][$_group]) {
$group[$_group]['first'] = true;
foreach ($record['curr'] as $field=>$value) {
$group[$_group]['stats']['sum'][$field] = $value;
$group[$_group]['stats']['count'][$field] = 1;
$group[$_group]['stats']['avg'][$field] = $value;
}
} else {
$group[$_group]['first'] = false;
foreach ($record['curr'] as $field=>$value) {
if (is_numeric($value)) {
$group[$_group]['stats']['sum'][$field] += $value;
++$group[$_group]['stats']['count'][$field];
$group[$_group]['stats']['avg'][$field] = $group[$_group]['stats']['sum'][$field]/$group[$_group]['stats']['count'][$field];
}
}
}
if ($record['last']) {
$group[$_group]['last'] = true;
} else if ($record['curr'][$_group] != $record['next'][$_group]) {
$group[$_group]['last'] = true;
} else {
$group[$_group]['last'] = false;
}
} else {
$group[$_group]['first'] = is_null($record['prev']);
$group[$_group]['last'] = is_null($record['next']);
}
}
|
here is where I think there is a problem...
Code: | if ($record['last']) {
$group[$_group]['last'] = true;
} else if ($record['curr'][$_group] != $record['next'][$_group]) {
$group[$_group]['last'] = true;
} else {
$group[$_group]['last'] = false;
}
|
It looks like the code determines if it is the last record in the group by testing if the curr is equal to the prev record. However, if the group one level up changes and the current group doesn't, this code thinks that the group has finished.
So, I have two records that say:
[a]=>1
[b]=>something
[a]=>2
[b]=>something
if i send this data in with sort level a,b then the 'something' data tells this code that the group *hasn't* changed, but because the group above it *did* change, i need that to be recognized. it seems as if this code is breaking. I have some more tests to do, but thought I'd describe the problem to see if you had a quick answer. |
|
Back to top |
|
boots Administrator
Joined: 16 Apr 2003 Posts: 5611 Location: Toronto, Canada
|
Posted: Wed Jun 15, 2005 7:09 pm Post subject: Re: smarty architecture of nested plugins |
|
|
sophistry wrote: | So, if I want leading sub-summaries (headers that can display the calculated summary data) what I really need to set up is a state machine that will keep track of the row in which data in a sorted "column" changes so that counts, sums, and averages can be calculated and stored when the data in that column (and/or the sort level above it) changes. But, that means that I need to maintain a column set array as well as the record set array that is currently there.
[snip]
So, you say it needs an associative array that carries the various report parts that have been buffered? If I create this array in the report plugin it should be possible to fill it with the output from the child plugins at their normal exit/dump time.
Where you dump the buffer in the child plugins... is that where assoc array assigning should happen? |
Rather insightful!
The real problem I see, Sophistry, is that statistics are calculated as the recordset is processed. This implies that calculated stats are not available until it is too late. The real options I see are:
- precompute everything. This is a bit of a waste since it may not get used -- then again, the current code doesn't skip any of this work either so its not too much of a loss other than the fact that you reloop the recordset. [my update of this code began with the premise that it is possible to precompute on a lazy basis -- the work is incomplete, so don't ask ]
- exploit the buffering option that these plugins use. Since actual output is defered to the end of the report block, you can add additional logic to do replacements in the buffer itself before it is finally returned to Smarty. For example, you can have a new tag type (say {report_header_stats}) which is triggered using the same logic as {report_footer} (so it can have access to all of the stats that footer does) but writes to a separate buffer. You would still define a {report_header}, but that plugin would silently insert a placeholder or allow you to manually insert a placeholder (eg: to keep things simple, you might use something like ##HEADER_STATS## for now.) When the recordset is fully iterated and it came time to flush the output buffer (in smarty_block_report__close), you would first merge the output buffer with the special header stats buffer by replacing on the pre-inserted placeholders with the content collected in the header stat buffer. In this case, it would be easier to maintain the header stats buffer as an array while the normal output buffer would remain a string. None of the buffers would require associative arrays as far as I can see.
I think you are hinting at the second option but with this option you wouldn't need to write a state machine and the overall mechanism should be fairly easy to implement and visualize. Of course, my off-the-cuff description is lacking but I suspect you get what I am driving at.
Why is a new tag needed? Why can't the header simply write to a separate buffer directly? Because of the logic in the header: when Smarty processes a {report_header} tag, the requisite stats have not yet been computed; this is a necessary outcome of wanting to maintain proper sequencing while using a single-pass to effect processing (and barring any precomputation which implies multiple passes of the dataset). |
|
Back to top |
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
|