Page 1 of 1

Implementing a recommendation system for a video website

Posted: Sat Feb 05, 2011 7:02 am
by brownca
Hi,
I hope this is the right forum to post this in.

I'm making a website (mysql) that allows a user to view a video, and then recommends the next video to be viewed based on many variables (already stored in various tables). My initial thought of how to do this would be to make a 'score' for each video - the higher the score, the more relevant to the user, and thus the top 5 (say) recommended videos can be shown. However, this would mean storing a score for every video and for every user, as well as calculating new scores for every video every time a list of recommendations is requested. This sounds to me like it might be a bit of a problem, but as a newbie to php/mysql I'd really appreciate the input of those more experienced than myself. Is there a better way of doing it that I haven't thought of? Would this idea be feasible with heavy traffic?

If I haven't explained this very well please let me know and I'll try and go into a bit more detail. Thanks.

Re: Implementing a recommendation system for a video website

Posted: Sat Feb 05, 2011 2:47 pm
by josh
Bayesian or k nearest neighbor the most # of records you'll need is # of videos ^ 2. You don't need a row for each "hit", just a total count for each possible "path" thru the site (and there's likely existing implementations that have been enhanced to use way less storage.)

All you need to know is that after viewing video A that the user is x% likely to goto video C, etc.

More advanced 'related videos' engine would do image recognition and find visually similar things, and also weight in a high relevance to the 'SEO' of the video (title, description, tags). Heck you could do it without tracking views, just use mysql full text search to find videos that have tags that match the tags of the current video, for example. That already returns a 'score' you can go off. You could then use a function of mysql's score with the # of views of the video, and you're done.

Re: Implementing a recommendation system for a video website

Posted: Sat Feb 05, 2011 6:02 pm
by brownca
I see what you're saying, but I want to go a bit more in-depth with the recommendation. For example, I want to give more weight to the videos watched next by 'similar' users, or videos from a playlist that has been subscribed to; and less weight to a video watched recently by the user (they don't want to keep being recommended the same thing), etc. This is mostly all data that's being stored in tables anyway. The difficult part is getting all these different ideas into one score. I mean, calculating the score is not too much of an issue, my main focus is on where I should store it and when I should calculate it. Intuitively I feel I shouldn't be calculating and storing thousands of 'scores' in a database every time a user requests a suggestion, so that's probably the wrong way to go about it. Or is it?

Re: Implementing a recommendation system for a video website

Posted: Mon Feb 07, 2011 3:56 pm
by josh
brownca wrote:I see what you're saying, but I want to go a bit more in-depth with the recommendation. For example, I want to give more weight to the videos watched next by 'similar' users, or videos from a playlist that has been subscribed to;
The former is already possible with the correlation algorithms I suggested. K-nearest neighbor would literally find the users with the "closets" viewing history to the current user. It would then look at the top videos they viewed. Weighting in additional factors like subscription count is a combination of elementary multiplication, and trial & error.
Intuitively I feel I shouldn't be calculating and storing thousands of 'scores' in a database every time a user requests a suggestion, so that's probably the wrong way to go about it. Or is it?
You'd calculate user's viewing histories. You'd store that. A cron script would run whenever economically feasible. This scheduled task would update the recommendations.

There's a distinction between recommendation videos & related videos.

Recommendation - you need a way to collect feedback, don't go off views, instead have thumbs up & thumbs down like youtube. A view could indicate the user liked or disliked. You have to ask. Just like TIVO.
Related - you should not take the current user into account. Better suited to Bayesian as you're simply interested in "users who watched A are 10% likely to view B next"

Re: Implementing a recommendation system for a video website

Posted: Wed Feb 09, 2011 5:38 pm
by brownca
OK, thanks very much for the advice. I'll read into it :)