Isn't it interesting what a name implies??
If you are reading:
According to:
Monday, June 13, 2011
The One True Translation
Isn't it interesting what a name implies??
If you are reading:
According to:
Saturday, June 11, 2011
Happenings
Have I been running lately? Nope. But this post is not about running.
About 4 months ago I had an idea. Now, I have ideas all the time - but as I told my boss awhile ago at work "it needs to be worth pursuing". I feel strongly enough about this idea that I've spent a considerable amount of time on it over the last several months. I've taken close to 2 weeks of leave from work, and lost countless hours of sleep because I refuse to take time from the family.
I can't show it off yet but I do have a proof of concept up an running. I can say that it involves crawling portions of the Internet. A web crawler is a robot that starts with a list of web pages, downloads them, finds the links in those pages and adds those newly found links to the download queue. My starting list of sites to crawl is ~35,000 domains long.
Now by comparison the big guys ( Google, Bing, Yahoo, Baidu ) are most likely crawling & indexing all 130 million registered domains.
Luckily for me my ISP does not have a bandwidth cap - though I am on DSL, so I can't ultimately use that much. Last month I was able to pull ~150 gigabytes in. I'm hoping to do double that this month.
When my index gets big enough to be useful I'll open it up and we'll see how useful people will find it.
About 4 months ago I had an idea. Now, I have ideas all the time - but as I told my boss awhile ago at work "it needs to be worth pursuing". I feel strongly enough about this idea that I've spent a considerable amount of time on it over the last several months. I've taken close to 2 weeks of leave from work, and lost countless hours of sleep because I refuse to take time from the family.
I can't show it off yet but I do have a proof of concept up an running. I can say that it involves crawling portions of the Internet. A web crawler is a robot that starts with a list of web pages, downloads them, finds the links in those pages and adds those newly found links to the download queue. My starting list of sites to crawl is ~35,000 domains long.
Now by comparison the big guys ( Google, Bing, Yahoo, Baidu ) are most likely crawling & indexing all 130 million registered domains.
Luckily for me my ISP does not have a bandwidth cap - though I am on DSL, so I can't ultimately use that much. Last month I was able to pull ~150 gigabytes in. I'm hoping to do double that this month.
When my index gets big enough to be useful I'll open it up and we'll see how useful people will find it.
Subscribe to:
Posts (Atom)