Designing a "price comparison" script - Pitfalls?
Posted: Mon Mar 26, 2007 9:26 am
I want to write a script that does something similar to a price comparison script. I'm not actually going to be comparing prices, but my project idea does need to build up a set of price data from different sites. Some have feeds that I can utilise (Amazon for example), but most don't.
As it stands my method is to crawl the 'new releases' pages using a regexp to get ids, then crawl the individual product pages every few days using another regexp to get the price. Is there a better method than that? I imagine it'll be fragile as hell.
Has anyone here written some sort of price comparator? What did you find most tricky? What should I watch out for? How did you link similar products (I'm thinking of using the name with Levenshtein distances).
As it stands my method is to crawl the 'new releases' pages using a regexp to get ids, then crawl the individual product pages every few days using another regexp to get the price. Is there a better method than that? I imagine it'll be fragile as hell.
Has anyone here written some sort of price comparator? What did you find most tricky? What should I watch out for? How did you link similar products (I'm thinking of using the name with Levenshtein distances).