Trying to Decide on a Software Development Environment

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

Post Reply
User avatar
Ollie Saunders
DevNet Master
Posts: 3179
Joined: Tue May 24, 2005 6:01 pm
Location: UK

Trying to Decide on a Software Development Environment

Post by Ollie Saunders »

I have spent the last two hours scratching my head on this one. You know when you get so confused you completely lose the ability to think. I spent 20 minutes trying to decide if I should do something else or not. I still don't what I chose. I call it "headfuzz".

So anyway the problem is I'm struggling with setting up a decent web server platform for software development. In particular I'm struggling with the management of website specific files and libraries. So I would really appreciate it if people could describe what they have done in order to manage the following issues:
  • Local development of sites and libraries amongst teams and the concurrency issues concerned
  • Sites relying on different versions of the same libraries and then require()ing them in PHP.
  • Reverting mistakes
  • Deploying web sites from development environment to production environment
Subversion, symbolic linking, perhaps specialist PHP coded library loaders are all going to be possible solutions. I can give more specific information about what I'm trying to do if you like but regardless of that I'd like to know about other people's set ups. Perhaps some of these things aren't problems for you, what are the reasons for that?

Where am I again?
User avatar
ReverendDexter
Forum Contributor
Posts: 193
Joined: Tue May 29, 2007 1:26 pm
Location: Chico, CA

Post by ReverendDexter »

You'll probably want to wait for the experts to chime in, but we just recently implemented subversion, and the contact that I've had with it leads me to believe that it would certainly do everything you need it to. I've currently got a working copy of a site we're working on checked out into a subfolder under htdocs, you'd never know the difference hitting it via my webserver. With branching you should be able to have stable "releases" of your libraries; and it'll certainly give you the ability to roll-back mistakes.

I haven't done this yet, but I think it would be easy to check out a working copy to the webserver, then, you could update your local copy to the way you like it, check it back in, and update the version on the webserver...

Again, I'm no expert, I've only been using subversion for maybe a week, but I think it could fit the bill.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

Subversion is it.
User avatar
Ollie Saunders
DevNet Master
Posts: 3179
Joined: Tue May 24, 2005 6:01 pm
Location: UK

Post by Ollie Saunders »

I'm currently using subversion. But I'm interested in how people use it.
I've currently got a working copy of a site we're working on checked out into a subfolder under htdocs, you'd never know the difference hitting it via my webserver. With branching you should be able to have stable "releases" of your libraries; and it'll certainly give you the ability to roll-back mistakes.
Do you have a separate repo for each library and site? If you need to include in a branch of a library instead of the trunk do you have to change the includes in the source or do you have a way of checking out differently?
User avatar
seodevhead
Forum Regular
Posts: 705
Joined: Sat Oct 08, 2005 8:18 pm
Location: Windermere, FL

Post by seodevhead »

I remember some time ago I was recommended here to use subversion.. as hardly no one here could imagine developing without it. However, I spent a good 2 days reading the documentation trying to understand exactly how to set it up and work it, and I was unable to understand it enough to use it. Not having any more time to "get it", I just dropped it and left.

There is a thing called TortioseSVN, which I can't remember if I tried, but it is a windows GUI implentation of Subversion... which if I can remember correctly, still confused the hell out of me. Perhaps I am a lower echelon of computer geek. I sure wish I understood it all.
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

Local development of sites and libraries amongst teams and the concurrency issues concerned
http://svnbook.red-bean.com/en/1.2/svn. ... odels.html
Sites relying on different versions of the same libraries and then require()ing them in PHP.
If the same site requires different versions of the same library, you've got serious problems. If its different sites, you'll need to decentralize the library includes and then attempt to get them all up to date.
Reverting mistakes
svn update --revision for a bandaid fix, svn cat > file to retrieve older versions of the files and then svn commit to revert at the repository.
Deploying web sites from development environment to production environment
Post-commit hooks! Or, if you're not as wild, tar / PEAR / rsync and NFS all can work.
Do you have a separate repo for each library and site?
http://svnbook.red-bean.com/en/1.2/svn. ... jects.html
If you need to include in a branch of a library instead of the trunk do you have to change the includes in the source or do you have a way of checking out differently?
If the library is using SVN, svn:externals does the trick nicely. Otherwise, you'll need to version the library itself or create some "script" that will download the library and set it up.
However, I spent a good 2 days reading the documentation trying to understand exactly how to set it up and work it, and I was unable to understand it enough to use it. Not having any more time to "get it", I just dropped it and left.
That's a shame. Did you ever get a repository working?
User avatar
Ollie Saunders
DevNet Master
Posts: 3179
Joined: Tue May 24, 2005 6:01 pm
Location: UK

Post by Ollie Saunders »

Yeah I had everything set up and then I realised it wasn't workable. I dunno I think I might be going through a phase of coders block at the moment, completely unable to think clearly. Maybe it's something I'm eating, or not eating. Perhaps I'm not taking enough exercise.

Here's what I had

Code: Select all

/                           # root; obviously
    /web                    # one big single repo checkout here
        /repo               # the repo itself, yes, it's inside a checkout of itself, there's svn:ignore set for it
        /sites
            /foo
                /code
                    /foo    # site specific code
                    /libA   # svn:external included library /lib/libA
                    /libB   # svn:external included library from /lib/libB
                /www        # document root for this site (vhost)
                /log        # server specific and svn:ignored
            /bar
                ... much like foo ...
        /lib                # libraries
            /libA
                /trunk
                    .. library code
                /branches
            /libB
                ... much like libA ...
I was having to deal with "trunk" sub-folders in order to get access to libraries. But I knew I may not always be using the trunk. Also the constraint of an svn:external having to go in it's own folder was causing me problems.

Everything I've written was really heavily coupled to a very specific file system layout and it had all changed. I used to include all my stuff like this:

Code: Select all

set_include_path('../code'); // outside the doc root and into the code folder
require_once 'MyFramework/Controller/Front.php';
require_once 'simpletest/mock_objects.php';
require_once 'utf8/utf8.php';
require_once 'ThisWebsite/Controller/Action/Index.php';
In order to make my code run I was faced with:

Code: Select all

set_include_path('../code'); // outside the doc root and into the code folder
require_once 'MyFramework/trunk/MyFramework/Controller/Front.php'; // might change
require_once 'simpletest/trunk/mock_object.php'; // might change but probably won't
require_once 'utf8/trunk/utf8.php'; // might change but probably won't
require_once 'ThisWebsite/Controller/Action/Index.php'; // I can't do branching here x(
Also I have a dependency injection system that depends on being able to resolve Underscore_Separated_Class_Names to their/path/counterparts and so the whole thing was completed screwed up.

I've now discovered, this is not the kind of thing you want to solve with trial and error. For me not knowing where my files are is enough to keep me awake at night, this stuff really stressed me out. I've got bits in version control, bits out of version control, stuff on one server, stuff on another, stuff on my desktop machine. For a couple of days I was completely unable to actually run anything.

Here's my proposed solution that I have begun to implement:

Code: Select all

/
    /repos              # many repos not one
        /libs
            /libA       # a library repo
            /libB       # another library repo
        /sites
            /foo        # a site repo
            /bar        # another site repo
        /all            # a special repo with all the other site repos svn:externaled in
    /web
        /libs
            /libA       # checkout of /repos/libs/libA
                /trunk
                    .. library here ..
                /branches
            /libB
                .. much like libA ..
        /sites
            /foo
                /code
                    /foo
                    /libA   # svn:external included library /repos/libA/trunk (note: trunk!)
                    /libB   # svn:external included library from /lib/libB/trunk
                /www        # document root for this site (vhost)
                /log        # server specific and svn:ignored
In that last listing I've shown what is beneath /web but that I now more free to change because it's several checkouts of several different repos. Yes, you can checkout the a single repo at several different points in different places but there's a conceptual difference. For instance there is a greater incentive to ensure that library unit tests can be executed without having to be part of a site. I've also got the flexibility to change access control on a per repo basis and the revision number stays smaller.

In addition I'm beefing up the flexibility of inclusion in all my code with constants. I always thought these were silly because you can use set_include_path() to load anything from anywhere anyway but it turns out they give a lot greater certainly about what is going on and you can reference them in your code later instead of having to explode(PATH_SEPARATOR, get_include_path()) and work out which is which.

So now I can require like this:

Code: Select all

define('LIB_PATH_FRAMEWORK', '../code/MyFramework/');
define('LIB_PATH_SIMPLETEST', '../code/simpletest/');
define('LIB_PATH_PHPUTF8', ../code/'utf8/');
define('SITE_PATH', '../code/ThisWebsite/');
require_once LIB_PATH_FRAMEWORK . 'MyFramework/Controller/Front.php';
require_once LIB_PATH_SIMPLETEST . 'mock_object.php'
require_once LIB_PATH_PHPUTF8 . 'utf8.php'; // might change but probably won't
require_once SITE_PATH . 'Controller/Action/Index.php';
Now, comments:
AC wrote:If the same site requires different versions of the same library, you've got serious problems.
Uh yeah, that would be a problem. I don't have that though.
If its different sites, you'll need to decentralize the library includes and then attempt to get them all up to date.
I'm actually assuming svn:external is going to be good enough. One of the things that worried me is if I go to /sites and type "svn update" will it update all of the externals too? Some of those sites will depend on older revisions so executing "svn update" will break them. I think I'm slightly unique on this front because I don't want my framework to be constrained by backwards compatibility where most people seem to accept that as a necessary part of developing a framework, even a personal one.
svn update --revision for a bandaid fix, svn cat > file to retrieve older versions of the files and then svn commit to revert at the repository.
Thanks for that tip involving cat. I had been doing it a much more awkward way.
Post-commit hooks! Or, if you're not as wild, tar / PEAR / rsync and NFS all can work.
Well I was thinking of using subversion itself. There's the idea behind my "all" repo. I can check that out on as many production servers as I like and just svn update when I need to update them.
User avatar
ReverendDexter
Forum Contributor
Posts: 193
Joined: Tue May 29, 2007 1:26 pm
Location: Chico, CA

Post by ReverendDexter »

For those includes, why not check out a working copy of your library files to a "library" directory that your code knows where it is? The app wouldn't know that it was working with versioned files, and you only care about trunks and branches when you're checking it out, or updating the repository version (I don't like the term "checking it back in" for subversion; in my head, if I check something in, I don't have it anymore, and this isn't the case; when I "check it in", all I'm doing is uploading my changes to the server, but I still have my working copies).

Or am I totally misunderstanding what's going on?

EDIT: you're != your
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

A few general comments:

Why is the actual, file-system repository folder being version controlled? That, to me, makes no sense at all: a repository is nothing without its associated data, and there's not even any "real" configuration to version if you're using mod_dav_svn to access the repository (then it's the httpd.conf file you need to worry about).

The problems with includes all over the place seems to stem from the lack of a logical class name -> file name scheme. If you need Foo_Bar_Class, you should be able to include require_once 'Foo/Bar/Class.php'; and be done with it. With proper "fake" namespacing, it should be quite simple to inspect the include path and figure out where everything is coming from.

I've always found that a naive,

Code: Select all

project1/
project2/
project1-web/
other-stuff/
Actually works quite well in practice. This is because you're not publishing the Subversion directory structure as a way to get to the repositories: don't think of it as a glorified filesystem. You'll be referencing SVN URLs from other places, and they can enforce the necessary hierarchical system you want.
I'm actually assuming svn:external is going to be good enough. One of the things that worried me is if I go to /sites and type "svn update" will it update all of the externals too? Some of those sites will depend on older revisions so executing "svn update" will break them. I think I'm slightly unique on this front because I don't want my framework to be constrained by backwards compatibility where most people seem to accept that as a necessary part of developing a framework, even a personal one.
If they depend on a specific version, you shouldn't be checking out of the trunk. Instead, tag the revision you want and check that out. When you want to upgrade the library, update the svn:externals property. (externals also lets you specify a specific revision yourself, but if you can, tag them so they're more easily identifiable).
Well I was thinking of using subversion itself. There's the idea behind my "all" repo. I can check that out on as many production servers as I like and just svn update when I need to update them.
Right. The only problem is that `svn update` doesn't instantaneously update all the files, so there's a possibility for some funky stuff to happen in-between the update; I've never had a problem with it but it may show up on higher traffic sites.

post-commit hooks are nice because it means that the live checkout gets updated instantaneously.
User avatar
RobertGonzalez
Site Administrator
Posts: 14293
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Post by RobertGonzalez »

Our setup at work (for SVN) is:

Code: Select all

/custom
    /svn-repos
        /framework
        /general
        /intranet
        /secure
        /www
The /custom folder is named after the company, as a way to keep all of our things in a central location. All the repos contain their respective /branches, /tags and /trunk directories that contain the actual code for the repositories. We haven't implemented post commit hooks yet seeing as our dev server runs multiple virtual servers and our secure server runs one app on its own subnet that doesn't play well with the other side of the firewall.
User avatar
Ollie Saunders
DevNet Master
Posts: 3179
Joined: Tue May 24, 2005 6:01 pm
Location: UK

Post by Ollie Saunders »

Why is the actual, file-system repository folder being version controlled?
It was inside the checkout but wasn't under version control.
The problems with includes all over the place seems to stem from the lack of a logical class name -> file name scheme
Not all libraries are written by myself and the ones that are I already do that with.
If you need Foo_Bar_Class, you should be able to include require_once 'Foo/Bar/Class.php'; and be done with it.
I refer to it as PEAR-style class naming it powers my dependency injection as I mentioned earlier. Unless you can correct me on this I now believe that If set_include_path() trickery can be avoided it should. It is a hidden value that makes including less predictable, it can be changed during program flow with dire consequences, is less reflective and you run the risk, albeit slight, of file ambiguity. Also I can use default include paths now:

Code: Select all

if (!defined('LIB_PATH_SIMPLETEST')) {
    define('LIB_PATH_SIMPLETEST', 'simpletest/');
}
Which still allows me to selectively override things, in the bootstrap file, if I need to. The downside is the associated constant diarrhoea:

Code: Select all

require_once LIB_PATH_FOO . 'Foo.php';
require_once LIB_PATH_FOO . 'Bar.php';
require_once LIB_PATH_FOO . 'Zim.php';
and the fact that my dependency injection system has to be rewritten to handle multiple paths. Then again perhaps I should be considering __autoload() or dirname(__FILE__). __autoload() is look more appealing than ever.
This is because you're not publishing the Subversion directory structure as a way to get to the repositories: don't think of it as a glorified filesystem. You'll be referencing SVN URLs from other places, and they can enforce the necessary hierarchical system you want.
Could you explain this further.
If they depend on a specific version, you shouldn't be checking out of the trunk. Instead, tag the revision you want and check that out. When you want to upgrade the library, update the svn:externals property. (externals also lets you specify a specific revision yourself, but if you can, tag them so they're more easily identifiable).
Aahhh that's brilliant. Thanks. I've got to read up on tagging now.
Right. The only problem is that `svn update` doesn't instantaneously update all the files, so there's a possibility for some funky stuff to happen in-between the update; I've never had a problem with it but it may show up on higher traffic sites.
Are we talking race conditions here? No, wait isn't the repo locked whilst a commit is in progress?
post-commit hooks are nice because it means that the live checkout gets updated instantaneously.
Hmm yes, I guess I could write on that would look out for a certain string like, "Production Safe" in the commit message and then SSH over to my production server and perform the update. Is that the kind of thing you are suggesting?
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

It was inside the checkout but wasn't under version control.
Right. I believe this to be completely unnecessary.
Not all libraries are written by myself and the ones that are I already do that with.
So it's in your best interests to ensure the libraries are also in the proper format. You can edit them directly or make adapters that interface with the library. In this case, however, I'd agree that constants make sense, but don't hardwire them into the application code: they should be in "settings".
Which still allows me to selectively override things, in the bootstrap file, if I need to. The downside is the associated constant diarrhoea:
Continuing on the thought above, I would recommend using constants for external libraries with poor naming, and include path for internal libraries with reasonable naming. Hopefully, it's not necessary to include too many files from external files manually.
Could you explain this further.
The usual format is a directory full of projects, and each of those projects has trunk, branches, tags. In your format, you have projects in other categorization directories, in more directories, etc. Your SVN repository layout basically dictates the organization of your projects. My argument is that it's unnecessary.
Are we talking race conditions here? No, wait isn't the repo locked whilst a commit is in progress?
Operations on the repository are ACID-safe. Subversion is quite good about that. However, updating a production website with svn update is not: there will be a sizable chunk of time when the files are not all the same version. This is the race condition I'm talking about.
Hmm yes, I guess I could write on that would look out for a certain string like, "Production Safe" in the commit message and then SSH over to my production server and perform the update. Is that the kind of thing you are suggesting?
The more "Subversion" way to do it is to have a stable branch, and merge in changes whenever you want to make code live.
User avatar
Ollie Saunders
DevNet Master
Posts: 3179
Joined: Tue May 24, 2005 6:01 pm
Location: UK

Post by Ollie Saunders »

Right. I believe this to be completely unnecessary.
Bit of a funny choice of words but I'm not doing this any more anyway.
So it's in your best interests to ensure the libraries are also in the proper format. You can edit them directly or make adapters that interface with the library. In this case, however, I'd agree that constants make sense, but don't hardwire them into the application code: they should be in "settings".
I'm trying to be more flexible and efficient here. What you are suggesting is a great deal of work and what if the library is updated?
Continuing on the thought above, I would recommend using constants for external libraries with poor naming, and include path for internal libraries with reasonable naming. Hopefully, it's not necessary to include too many files from external files manually.
I've basically decided now I'm going to use __autoload(). It's going to save so much time and effort and it centralizes the problem.
The usual format is a directory full of projects, and each of those projects has trunk, branches, tags. In your format, you have projects in other categorization directories, in more directories, etc. Your SVN repository layout basically dictates the organization of your projects. My argument is that it's unnecessary.
I've only split them into libraries and sites to make things clearer. In development how much is strictly "necessary"?
However, updating a production website with svn update is not: there will be a sizeable chunk of time when the files are not all the same version. This is the race condition I'm talking about.
Ahh yes I see your point. I don't think that's likely to be an issue for me.
The more "Subversion" way to do it is to have a stable branch, and merge in changes whenever you want to make code live.
I'd still have to log in to the production server and run svn update though, surely?
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

I'm trying to be more flexible and efficient here. What you are suggesting is a great deal of work and what if the library is updated?
Yep. I was thinking that you ought to create an appropriately named adapter for the library, but the adapter still needs to know where the library is.
I've only split them into libraries and sites to make things clearer. In development how much is strictly "necessary"?
Not very much. Just a separation between projects, usually.
'd still have to log in to the production server and run svn update though, surely?
If the server hosting the repository can communicate to the production server, not even that's necessary.
User avatar
Ollie Saunders
DevNet Master
Posts: 3179
Joined: Tue May 24, 2005 6:01 pm
Location: UK

Post by Ollie Saunders »

Thanks for all your help on this. I'm in a much better place now, I know what I'm doing :)
Post Reply