Hosting For Search Engine With Web Crawler
Hey guys we’re having trouble finding good hosting for our search engine and web crawler. We spent over year and a lot of money developing the software that runs the crawler, search etc.. Our crawler obeys robots.txt and also has a niceness setting built in that makes sure it doesn’t crawl the same sites too fast or too often.
The problem we are having is when crawling the web, the server companies get sent these automated abuse notices and things like that because we are crawling millions of pages daily. Once again we are obeying all crawling rules but we still just get these never ending automated abuse messages that we are constantly having to reply to. It seems no matter what host we use they all have these automated messages we need to deal with.
Does anybody have any idea of a dedicated host that would allow our crawlers and understand that we are running a large scale web crawl that has absolutely no malicious activity going. During our testing phase we’ve used soyoustart, hetzner, nocix, reliablesite and a few others all of which we had this same problem with.