Of each filename listed, i need to get the content also and store on my table on file_contents field.
Can someone give me an expert advise on the most optimal, efficient, fast and effective way to do this?
by the way, the file below could contain large amount of data as much as 300k to 1M lines.
I already have my script but im getting random timeout and 'Unable to get the files' error
Code: Select all
Description: Master Index of EDGAR Dissemination Feed
Last Data Received: March 25, 2008
Comments: webmaster@sec.gov
Anonymous FTP: ftp://ftp.sec.gov/edgar/
CIK|Company Name|Form Type|Date Filed|Filename
--------------------------------------------------------------------------------
1000045|NICHOLAS FINANCIAL INC|10-Q|2008-02-11|edgar/data/1000045/0001193125-08-025292.txt
1000045|NICHOLAS FINANCIAL INC|4|2008-02-05|edgar/data/1000045/0001000045-08-000001.txt
1000045|NICHOLAS FINANCIAL INC|4|2008-02-07|edgar/data/1000045/0001000045-08-000002.txt
1000045|NICHOLAS FINANCIAL INC|4|2008-03-18|edgar/data/1000045/0001000045-08-000003.txt
1000045|NICHOLAS FINANCIAL INC|8-K|2008-02-05|edgar/data/1000045/0001193125-08-020320.txt
1000045|NICHOLAS FINANCIAL INC|SC 13G/A|2008-02-08|edgar/data/1000045/0000950135-08-000646.txt
1000045|NICHOLAS FINANCIAL INC|SC 13G/A|2008-02-14|edgar/data/1000045/0000315066-08-001913.txt
1000045|NICHOLAS FINANCIAL INC|SC 13G/A|2008-02-14|edgar/data/1000045/0001362310-08-000904.txt
1000045|NICHOLAS FINANCIAL INC|SC 13G|2008-02-11|edgar/data/1000045/0001362310-08-000627.txt
1000069|EMPIRIC FUNDS, INC|40-17G/A|2008-02-13|edgar/data/1000069/0000894189-08-000447.txt