Have I been running lately? Nope. But this post is not about running.
About 4 months ago I had an idea. Now, I have ideas all the time - but as I told my boss awhile ago at work "it needs to be worth pursuing". I feel strongly enough about this idea that I've spent a considerable amount of time on it over the last several months. I've taken close to 2 weeks of leave from work, and lost countless hours of sleep because I refuse to take time from the family.
I can't show it off yet but I do have a proof of concept up an running. I can say that it involves crawling portions of the Internet. A web crawler is a robot that starts with a list of web pages, downloads them, finds the links in those pages and adds those newly found links to the download queue. My starting list of sites to crawl is ~35,000 domains long.
Now by comparison the big guys ( Google, Bing, Yahoo, Baidu ) are most likely crawling & indexing all 130 million registered domains.
Luckily for me my ISP does not have a bandwidth cap - though I am on DSL, so I can't ultimately use that much. Last month I was able to pull ~150 gigabytes in. I'm hoping to do double that this month.
When my index gets big enough to be useful I'll open it up and we'll see how useful people will find it.
Subscribe to:
Post Comments (Atom)
I think it is a great idea and I am proud of you for pursing it. I think it is awesome that you are so passionate about things. I love that you dream big, and then go for it!
ReplyDeleteHow awesome Zac! Joe and I were just talking about your skill set this morning.
ReplyDeleteTo God be the Glory!
PS-we're hoping to see you in a few weeks. Hoping we can connect!
keep moving forward man! you are really our inspiration :))
ReplyDelete-popoi-