I've got a 17 mb dictionary file that has some duplicate words in it. Is there a way to return everything unique?
I thought about loading the file into an array, then using array_unique() to return it. Should I use this way, or is there a more efficient way to?
Removing duplicate strings
Moderator: General Moderators
-
HiddenS3crets
- Forum Contributor
- Posts: 119
- Joined: Fri Apr 22, 2005 12:23 pm
- Location: USA
- John Cartwright
- Site Admin
- Posts: 11470
- Joined: Tue Dec 23, 2003 2:10 am
- Location: Toronto
- Contact:
-
HiddenS3crets
- Forum Contributor
- Posts: 119
- Joined: Fri Apr 22, 2005 12:23 pm
- Location: USA
I'm not using MySQL, my word file is uploaded as a text file on the server. I need to get the contents and store into an array... that should still work though, right?Jcart wrote:Not sure how to do this strictly in mysql, but with a php aproach you almost got it.
Simple gather your array of all your words, slap it with array_unique, delete your old table, input your fresh unique array.
Remember, this process only needs to be done once.
Well, if you have a couple of GNU textutilities around:
Code: Select all
timvw@madoka:~$ sort words.txt | uniq > uniqwords.txt-
HiddenS3crets
- Forum Contributor
- Posts: 119
- Joined: Fri Apr 22, 2005 12:23 pm
- Location: USA