How can I easily sort and display data from a large text file on my website?

Question

Hello!

I'm trying to sort and display data from a very large text file (about 128MB) on a website -- the text file has all its fields comma-separated and enclosed in double-quotes, like so:

"John","Adams","President","1796-1800"...

I'm trying to determine the best route to sort and display these fields so that by searching one parameter (ie, search for "Adams") it will sort through that list and return only the items with "Adams" in the last-name field.

I do have some interesting challenges:

1.) The size of the file - how much will that impact speed?

2.) The text file also gets updated regularly, so I need a way to keep the data current.

I have some basic programming skills, but I'm still a total noob. Anything you've got will make a world of difference for me. Thanks in advance!

Answer 1

With such a large text file -- 128MB is huge!!! -- it would be better to move the data into a (free) database such as MySQL. (Other databases will work fine but many of them are fairly pricey.)

If you're not using a database and insist on loading the text file each time you run a query, this could significantly impact performance. It would be best to read the file in only when it changes (once per day?) into a STATIC variable and run all your queries from that. That would minimize the impact on site performance.