Page 2 of 3

Re: Statiscal lazy loading idea

Posted: Tue Dec 02, 2008 1:25 am
by alex.barylski
Syntax Error :drunk: :offtopic: :lol:

Re: Statiscal lazy loading idea

Posted: Tue Dec 02, 2008 2:42 pm
by josh
Good design leads to optimized code? What about platforms where structural constructs like functions and objects are actually very resource intensive to use, ie embedded applications, etc.. Optimization = indirection, good design = low semantic difference between code and problem domain. If you're going to stop mid project to optimize a specific interaction, what assurance do you have the optimization was not done in vein. What insurance do you have that that code will not need to be refactored or not needed in several weeks? I'd agree with arborint, you optimize when there is a problem, optimizations detract from design quality. The most elegant design would be the most direct. Every caching tier, added facet of indirection adds to the cognitive load of the developers, in my opinion essentially a worse design.

Re: Statiscal lazy loading idea

Posted: Tue Dec 02, 2008 3:04 pm
by Eran
Yes, the purpose of design is to manage complexity, while the purpose of optimization is to manage performance - and those two seldom coincide. A good design however will allow you to make the necessary optimizations when they are needed with as little added complexity as possible.

Re: Statiscal lazy loading idea

Posted: Tue Dec 02, 2008 9:09 pm
by josh
^^ - Nicely worded

Re: Statiscal lazy loading idea

Posted: Tue Dec 02, 2008 9:48 pm
by alex.barylski
Yes, the purpose of design is to manage complexity, while the purpose of optimization is to manage performance - and those two seldom coincide. A good design however will allow you to make the necessary optimizations when they are needed with as little added complexity as possible.
OK fine, then a *great* design will allow you to focus strictly on "money code" and never have to worry about optimizations in the first place. :P

Re: Statiscal lazy loading idea

Posted: Wed Dec 03, 2008 5:36 pm
by josh
This we can agree, that is why I said I think this is a teriffic idea, I just don't see it solving any immediate problems I currently have, and nor could I imagine that others had. It's a great idea, but first you need a working ORM. Seeing as there is none for PHP*, someone should get on that first. No bright ideas are needed to write an ORM either, there's enough existing stuff / books written about it to find all the answers you are looking for. Also I'd propose unless you have some AI breakthroughs (which you probably dont) its always going to be more cost effective to go in and write the SQL by hand as the performance pressure points start to hurt.. Realizing this if you're still willing to dedicate your time to create what you've described than my hat goes off to you

*My definition of working would be a working implementation of each ORM & Data Source Architectural pattern described in POEAA, allowing the user of the framework to make design trade offs according to generally accepted best practices in the community

Re: Statiscal lazy loading idea

Posted: Wed Dec 03, 2008 11:25 pm
by Kieran Huggins
No plans to write it really, just an idea I had kicking around.

If I were to develop something along these lines, it would probably be as a DataMapper plugin in Ruby, since a) I use it, b) DataMapper is a fairly mature, extensible ORM, and c) Ruby is a much easier language to author something like this in (IMO).

All great comments - I agree that it's easier to write a little SQL than to write an ORM / plugin, but as long as it was easily re-usable (i.e. a plugin) I can't help but suspect that the overall optimization would be "a good thing". It's sort of like lazy loading, in a sense... it's easier to just fetch the fields in SQL than to write lazy loading into an ORM, but if it's there, it's essentially free optimization.

Anyone else have any crazy ideas?

Re: Statiscal lazy loading idea

Posted: Wed Dec 03, 2008 11:29 pm
by allspiritseve
Kieran Huggins wrote:Anyone else have any crazy ideas?
Lots... you'll have to be more specific ;)

Re: Statiscal lazy loading idea

Posted: Thu Dec 04, 2008 12:10 am
by alex.barylski
Anyone else have any crazy ideas?
Sure...I'm always trying to discover new DSL's to further abstract my development.

I find most ideas are nothing but experiments, but its stilla great way to learn, from trial and error, personally I find it most effective in absorbing information and knowledge.

Re: Statiscal lazy loading idea

Posted: Thu Dec 04, 2008 12:53 am
by josh
I agree, the problem is sometimes lazy loading can be counter-productive, if you do a loop asking an object for a related object is the ORM going to issue 1 query per object or is it going to grab everything you need? How does it know what you need? Keep in mind it can't be common sense. Common sense is the hardest thing to state explicitly, SQL is tried and true. Nueral networks, etc.. by design are always going to be predictably wrong a % of the time, in the sense that computers can only generate psuedo random outputs, based on input seeds.

Although the arguments I'm giving were the arguments given when people conceived high level dynamically typed languages, etc... but I can't resist playing the role of devil's advocate, if someone questions someone else's logic, it forces that person to rethink their approach, perhaps leading to an even better idea. The idea in itself is great, but I don't know... do you really know anyone who has so much traffic they would need something like this? How much time would it take to create? How much would you have to charge for it to make it worth it? How much would it cost for that person to throw some more hardware at the problem, or manually optimize? Again, these are all cliche counter-arguments and may be invalid, like I said the idea is great, these are just alternative viewpoints to consider that I'm throwing out there.

If you want "crazy" ideas read Steven Pinker, or Chomsky
http://video.google.com/videosearch?q=S ... 4&ct=title#

Re: Statiscal lazy loading idea

Posted: Thu Dec 04, 2008 1:09 am
by allspiritseve
It's not as hard as you make it out to be...

Code: Select all

class AccessLogger  {
 
private $accessed = false;
private $object;
private $logger;
 
function __construct ($object, $logger) {
    $this->object = $object;
    $this->logger = $logger;
}
 
function __get ($key)   {
    return $object->$key;
}
 
function __set ($key, $value)   {
    $this->object->$key = $value;
}
 
function __call ($method, $params)  {
    if (!$this->accessed): $this->logger->log ('object accessed'); $this->accessed = true; endif;
    $this->logger->log ('method ' . $method . ' accessed');
    return call_user_func_array (array ($this->object, $method), $params); 
}
 
}
You could have another one for collections...

And then it would be simple enough to determine from logs access numbers for methods, objects, and collections, and determine what loading method to use. Stick that behind a loader interface and away you go.

In your mapper, something like this:

Code: Select all

$object->setProducts ($this->loader ($productMapper)->getById ($object_id);
Personally, I'd rather just know the stats, and decide myself what to use, but... its not that complicated to automate.

Re: Statiscal lazy loading idea

Posted: Thu Dec 04, 2008 1:14 am
by josh
allspiritseve wrote:Personally, I'd rather just know the stats, and decide myself what to use
I would too, and make no mistake that's all the code you posted would do, it would log it. Computers are exceedingly efficient at that task. Mysql already does this for you. Slow query log. ( and other vendor tools )
allspiritseve wrote: its not that complicated to automate.
The human mind????! Geeettt outttaaa heeeree :D The computer can't know what data is related to what other data, understanding it's semantic relationships, and will not be able to generate on it's own, concepts of the ramifications of design decisions that need to be weighed against eachother, and cannot understand the metrics in which those trade offs need to be considered. For instance the computer might say oh, if we don't select all this data, the query runs 1,000x faster... then when the program breaks ( because you can't expect to write a 100% solid test harness ) you get a call at 3am..

Re: Statiscal lazy loading idea

Posted: Thu Dec 04, 2008 1:21 am
by allspiritseve
jshpro2 wrote:
allspiritseve wrote:Personally, I'd rather just know the stats, and decide myself what to use
I would too, and make no mistake that's all the code you posted would do, it would log it. Computers are exceedingly efficient at that task. Mysql already does this for you. Slow query log. ( and other vendor tools )
allspiritseve wrote: its not that complicated to automate.
The human mind????! Geeettt outttaaa heeeree :D
Mysql logs can show you query usage... this shows you object/collection usage. What if you are eager loading, and the data is never accessed?

It's not automating the human mind... its automating picking between 4 options based on some simple numbers. If you don't like what it's doing, just replace the mapper call with direct access:

Code: Select all

$object->setProducts ($productMapper->getById ($object_id));

Re: Statiscal lazy loading idea

Posted: Thu Dec 04, 2008 3:01 am
by josh
I think on a small scale it would be simple to implement, but on a larger scale there would be too many cases where the device made paradoxical decisions in the choice of SQL lexicon, where only a human would provide enough sense to outweigh the different design decisions. I guess the idea wasn't to have it make schema based decisions though. Also we're purely speculating that the code to do this would be less resource intensive than just doing purely aggressive loading. If it simply logged usage that could be analyzed off-site, that could have a potential use.

Re: Statiscal lazy loading idea

Posted: Thu Dec 04, 2008 4:42 am
by alex.barylski
It's not as hard as you make it out to be...
That goes for any problem. Everything is simple once it is understood, the problem with problem solving, is our capacity as humans to think complex, we tend to forget simplicity is the best approach. Especially in software development where milestones and features are more important than code quality.

The paradox: It's easy to think about complex solutions for simple problems; And actually difficult to think of simple solutions for complex problems.

I would say 99% of the time, we as humans (developers included) solve problems with far more verbosity or details than actually needed, thus making simple solutions more complicated.

I recall reading a study/article years ago, titled: "what makes a super programmer" or something similar...the study was from 1978 in an old book I have so it's a little dated. The conclusion was that of all backgrounds tested (Physchologists, physicists, etc about a dozen in total) mathematicians made the best programmers. Assumingly because they are 'experts' at breaking complex problems into atomic problems.

The fact that 30 some years later, that practice is still equally important in software development, speaks volumes as a basic principle.

Cheers,
Alex