Taptubot - Taptu's mobile-friendly website discovery tool
What is Taptubot?
Taptu Ltd are providers of a mobile search engine. We are working at actively discovering domains that provide versions of their website specifically tailored for the iPhone and other hand-held touch devices for inclusion in our index, to allow our users to find mobile-friendly content.
Because there is no standard URL pattern for a mobile version of a website, our website discovery tool, taptubot, tests common mobile variants (eg. m.domain.com, mobile.domain.com, domain.com/iphone, etc) to discover if a website has a mobile version.
The domains we perform our test on are taken from publicly available lists.
This isn't either a probe or vulnerability test, or an unrestricted web crawl. We're not hacking or trying to access restricted content.
Each test of a website sends between 10 and 20 web page requests. It's designed to not impair or disrupt a remote site and there are no subsequent requests made after we're done. It doesn't crawl a website - no links are followed. The load on a webserver should be about the same as a single user session.
If you have a mobile website you'd like to submit to us, please let us know
If you have any questions please email us at info@taptu.com
Why are you sending a fake user-agent header?
Our purpose is to discover "touch phone" friendly sites (eg. designed for the iPhone, T-Mobile G1, Nokia N5800, etc); unfortunately when many websites examine the user-agent header in order to determine which website version (desktop/iPhone) to serve to a browser, they often perform an exact comparison on the user-agent header.
In order to see for example iPhone sites as intended we must send a normal iPhone user-agent.
Why would I see repeated requests?
There's no standard URL pattern for a mobile website - so we test various likely common patterns.
If you host multiple domains, we may perform this test for each domain. Our URL list is randomised and sites are queried over many hours, so it's very unlikely that two or more domains that you own will be contacted at the same time.
How can I prevent Taptu from requesting my site or certain portions of it?
We follow the robots.txt robots exclusion standard. As such you can block taptubot from requesting specific paths, or from your entire website by specifying them in robots.txt.
We match against any user-agent rules which contain taptubot, as well as the catch-all *.
Or, if you'd prefer us to never contact your website again, please email abuse@taptu.com with your domain name(s) and we'll add you to our blacklist.
