Crawling the Web Again

A while back I started to work on the crawler and I am about to start working on it once again. This time I will be optimizing, storing the information and logging the speed of the database transaction to test how the cloud hosting service is handling this many connections.

Thank you for the unlimited bandwidth and transfer dreamhost because I will be making good use of your service tonight. I don’t plan on turning off this service and I would like to add in some form of multi tasking on the server. or at least increasing the connections made per second. So far it is indexing 10 sites a second but I think I could easily get that up to 50 with a little bit of work and a little lack of sleep.