Removing duplicate strings

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
HiddenS3crets
Forum Contributor
Posts: 119
Joined: Fri Apr 22, 2005 12:23 pm
Location: USA

Removing duplicate strings

Post by HiddenS3crets »

I've got a 17 mb dictionary file that has some duplicate words in it. Is there a way to return everything unique?

I thought about loading the file into an array, then using array_unique() to return it. Should I use this way, or is there a more efficient way to?
User avatar
John Cartwright
Site Admin
Posts: 11470
Joined: Tue Dec 23, 2003 2:10 am
Location: Toronto
Contact:

Post by John Cartwright »

Not sure how to do this strictly in mysql, but with a php aproach you almost got it.
Simple gather your array of all your words, slap it with array_unique, delete your old table, input your fresh unique array.
Remember, this process only needs to be done once.
HiddenS3crets
Forum Contributor
Posts: 119
Joined: Fri Apr 22, 2005 12:23 pm
Location: USA

Post by HiddenS3crets »

Jcart wrote:Not sure how to do this strictly in mysql, but with a php aproach you almost got it.
Simple gather your array of all your words, slap it with array_unique, delete your old table, input your fresh unique array.
Remember, this process only needs to be done once.
I'm not using MySQL, my word file is uploaded as a text file on the server. I need to get the contents and store into an array... that should still work though, right?
timvw
DevNet Master
Posts: 4897
Joined: Mon Jan 19, 2004 11:11 pm
Location: Leuven, Belgium

Post by timvw »

Well, if you have a couple of GNU textutilities around:

Code: Select all

timvw@madoka:~$ sort words.txt | uniq > uniqwords.txt
HiddenS3crets
Forum Contributor
Posts: 119
Joined: Fri Apr 22, 2005 12:23 pm
Location: USA

Post by HiddenS3crets »

I'm not great working with files in PHP, how would I add each line from a file to an array?
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

file() ... should use array_map() to trim() all the elements so you can work from a uniform set of entries..
Post Reply