Page 1 of 1

Databases and Automation vs. Hand editing - Design Block

Posted: Mon Mar 21, 2005 9:03 pm
by Ambush Commander
I've hit a design block, primarily because of the way I percieve the data that's being handled. For instance, with a forum, you're working towards having it totally automated: you're not going to have people hand editing the database for posts. Yet I don't feel I want my application to be that way... which brings up my first fundamental question:

Should all web applications work towards being fully automated, and then build in a moderation system? Or should one try to get something running as soon as possible, making moderation the only choice, and then slowly change the parts as you move towards automation? The first choice won't get anything done in a while, but it's the much more direct path. The second way will ultimately result in more code being written and more steps in the wrong direction.

See, if I work towards that, it breaks one of the nice things about my site, which is intimacy through unpopularity. And fully automating things is going to be a monumental task for me to do... which is why I'm reluctant to take the try to target that first (I've got other things in the code I want to work on). So I'm trying to connect with what I have right now.

What do I have right now? A propriety data storage form: It works sorta like PHP's variable/array constructors, but that's all it can do (and that's all it's supposed to do). I've written code to parse it, and the code is kinda slow (it's okay if you call it only once, but many times and...) Plus the structure is ugly: you always seem to be missing things.

So, what I thought I would do was first clean up the data. Objects sounded like the way to go: they would let me blueprint and hold lots of data while letting me implement methods for easily hooking into the old interface. I finished the author class.

Of course, the author class was the easiest of them all: it didn't have anything too tricky, just a few values for the author. It would be the stories that would be difficult: there was complicated logic in determining the organization of a chapter story, logic I'm not quite sure I want to keep (although it may not be too bad after all...)

But then, looking at the code, I came up with a problem. Right now, this is how my data is managed:

Code: Select all

DATABASE ---PHP parser (slow)---> PROGRAM (in abstract form) ----> SERVED
When I found out about serialize, I intended to do this:

Code: Select all

OLD DATABASE ---parse---> PROGRAM (in abstract form) -----\/----> SERVED

---serialize----> SERIALIZED DATA (physical manifestation of abstraction) ---------> back to PROGRAM
But that meant I would have to write complicated subroutines when I meant to edit the object and it's associated code.

I had no idea why people had databases in the first place: why not just provide mechanisms for "storing" data structures, rather than make lots of queries to populate your structures?

What is the fundamental use for Databases: what makes them so good?

I'm beginning to suspect is that while data inside a data structure is hard to manipulate, the database itself is easy to edit.

Well, at least, easy for a program to edit.

Which comes back to the problem of automation: I have to extend my programming to have complete editing subroutines if I want the program to be able to do all the editing by itself. And that's going to take a lot of time.

When you want to move databases, you have to do "database dumps", which are essentially large files of lots of instructions on how to recreate a database. That's a nightmare for someone who just wants to make simple changes to something.

Ughh.... I can't think anymore. Can anyone answer the first two questions and I'll reveal more about my dilemma as I think some more... I think the main problem is I'm not thinking... I need to do some research too... and maybe I should just bite the bullet and go for the conceptually correct way.

Re: Databases and Automation vs. Hand editing - Design Block

Posted: Tue Mar 22, 2005 9:39 am
by timvw
Ambush Commander wrote:Should all web applications work towards being fully automated, and then build in a moderation system? Or should one try to get something running as soon as possible, making moderation the only choice, and then slowly change the parts as you move towards automation?
Usually people start with identifying the modules/classes/functionalities that a systeem needs and how they interact.

So you probably start with a system that allows people to change/read/update/delete a post. After that you plugin a security system that ensurs you only people with the right permissions can crud a post. Then you will probably add a mechanism to track which posts have been read and which not. Then you can also add subforums (groups of posts)....
Ambush Commander wrote: The first choice won't get anything done in a while, but it's the much more direct path. The second way will ultimately result in more code being written and more steps in the wrong direction.
That is a risk you always take. I still have to meet people that come up with a perfect design from the start. It always evolves to something that approaches perfectionism...
Ambush Commander wrote: I had no idea why people had databases in the first place: why not just provide mechanisms for "storing" data structures, rather than make lots of queries to populate your structures? What is the fundamental use for Databases: what makes them so good?
- databases offer you:
- access control
- consistency of data
- control of data redundancy
- backup & recovery
- ...


Ambush Commander wrote: I'm beginning to suspect is that while data inside a data structure is hard to manipulate, the database itself is easy to edit.
i'm beginning to suspect you might need a good introduction to dbms (read some codd / chen / date books)
Ambush Commander wrote: Well, at least, easy for a program to edit.
Use the technology that is available.

Ambush Commander wrote: Which comes back to the problem of automation: I have to extend my programming to have complete editing subroutines if I want the program to be able to do all the editing by itself. And that's going to take a lot of time.
You can reduce all operations to a set of crud actions. For example writing a class for crud actions on a sql table will offer you a very powerful and reusable component.

After that it's only a matter of writing some basic (list/create/read/update/delete) that use that sql-class on then it's only a matter of initialising the right sql-table class and dispatching control to that script.

Ambush Commander wrote: Ughh.... I can't think anymore. Can anyone answer the first two questions and I'll reveal more about my dilemma as I think some more... I think the main problem is I'm not thinking... I need to do some research too... and maybe I should just bite the bullet and go for the conceptually correct way.
Sometimes it can take a while to identify the patterns and situations that occurs time after time.. but in the end you will be able to identify them and are you able to program the perfect classes/routines...

Re: Databases and Automation vs. Hand editing - Design Block

Posted: Tue Mar 22, 2005 10:31 am
by CoderGoblin
Ambush Commander wrote:you're not going to have people hand editing the database for posts. Yet I don't feel I want my application to be that way...
Most people are not comfortable with dealing with a database. Even a lot programmers use GUI's to access there data rather than "complex" SQL. As such users of systems need an "interface",
Ambush Commander wrote:Should all web applications work towards being fully automated, and then build in a moderation system?
1) A web application is in my opinion, fully automated.
2) Looking at a web application you see the following:
User <--> application/moderation system <--> data
Given this "workflow" you need a good DESIGN for both the application and data. Both the application and the data are joined together. You cannot separate the two.

The hardest thing about data is effectively storing it and reducing redundancy. This is why databases are used. How easy it is to get and change data is all dependant on the DESIGN of the database tables.
Ambush Commander wrote:When you want to move databases, you have to do "database dumps", which are essentially large files of lots of instructions on how to recreate a database. That's a nightmare for someone who just wants to make simple changes to something.
Good design should allow flexibility but it depends on what you mean be simple changes to something. Normally you do not want to move databases only alter the data within the database.

The ideal workflow for an application is
1) Requirement Specification
2) Design Specification
3) Build Data Storage
4) Build Application
5) Test

The most important step is DESIGN

Posted: Tue Mar 22, 2005 9:02 pm
by Ambush Commander
Oh wow... that's a handful to bite off of. I've printed out your responses... I'm so thankful you replied! I'm going to read them over and over, until I think I've understood your response, and then I'll reply. I'll also look into some of the authors you suggested timvw, although a google search seemed to suggest that Codd's books were quite technical.

And this means more research for me! Yay! Thanks a lot. I'll be back with indepth responses to your responses. Thank you *so* much.