We’ve got an SEO consultant telling us that having a lot of bot traffic on our collector endpoint is a Bad Thing. (Googlebot hits it pretty regularly.)
- Is it possible to configure the Scala collector to respond to a request for robots.txt?
- Does it make any sense to do this?
Any advice appreciated.