Yahoo Mobile Crawler

Posted by joy

Today a crawler from 66.94.233.87, an IP which apparently belongs to test06.wap.search.scd.yahoo.com (WHOIS) started crawling my site. That in and of itself is not too unusual. What caught my eye, however, was that the crawler was attempting to find pages thathad something to do with presentation for mobile users. Which is fine, but the pages the crawler tried to get to are not industry standard, instead, it looks like some programmer at Yahoo just took a guess as to how, perhaps, a mobile version of a site may be presented.

I dunno, it seems like this crawler could have been written to be more efficient - aside from bandwidth issues, you’d think the crawler would try to look for a robots.txt first, which it failed to. A thread at WebmasterWorld comes to a similar conclusion.

More boring site logs after the jump…

/index.wml
Http Code: 404 Date: Dec 14 12:38:10 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/index.xhtml
Http Code: 404 Date: Dec 14 12:38:11 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

[tags]Yahoo, Yahoo Mobile, Robots, Bots, Crawlers, robots.txt, Mobile, Wireless WAP, WML [/tags]

/default.wml
Http Code: 404 Date: Dec 14 12:38:11 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/default.xhtml
Http Code: 404 Date: Dec 14 12:38:12 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/home.wml
Http Code: 404 Date: Dec 14 12:38:13 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/home.xhtml
Http Code: 404 Date: Dec 14 12:38:13 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mobile
Http Code: 404 Date: Dec 14 12:38:14 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mobile/index.wml
Http Code: 404 Date: Dec 14 12:38:14 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mobile/default.wml
Http Code: 404 Date: Dec 14 12:38:15 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mobile/default.xhtml
Http Code: 404 Date: Dec 14 12:38:16 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mobile/home.wml
Http Code: 404 Date: Dec 14 12:38:16 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mobile/home.xhtml
Http Code: 404 Date: Dec 14 12:38:17 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mob
Http Code: 404 Date: Dec 14 12:38:17 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mob/index.wml
Http Code: 404 Date: Dec 14 12:38:18 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mob/index.xhtml
Http Code: 404 Date: Dec 14 12:38:19 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mob/default.wml
Http Code: 404 Date: Dec 14 12:38:19 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/mob/home.xhtml
Http Code: 404 Date: Dec 14 12:38:20 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/wml/index.wml
Http Code: 404 Date: Dec 14 12:38:20 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/wml/default.wml
Http Code: 404 Date: Dec 14 12:38:21 Http Version: HTTP/1.1 Size inBytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/wml/home.wml
Http Code: 404 Date: Dec 14 12:38:21 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/xhtml/index.xhtml
Http Code: 404 Date: Dec 14 12:38:22 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/xhtml/default.xhtml
Http Code: 404 Date: Dec 14 12:38:23 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/xhtml/home.xhtml
Http Code: 404 Date: Dec 14 12:38:23 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/wap/index.wml
Http Code: 404 Date: Dec 14 12:38:24 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/wap/default.wml
Http Code: 404 Date: Dec 14 12:38:24 Http Version: HTTP/1.1 Size inBytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0
Profile/MIDP-2.0 Configuration/CLDC-1.0

/wap/default.xhtml
Http Code: 404 Date: Dec 14 12:38:25 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0
Profile/MIDP-2.0 Configuration/CLDC-1.0

/wap/home.wml
Http Code: 404 Date: Dec 14 12:38:26 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/wap/home.xhtml
Http Code: 404 Date: Dec 14 12:38:26 Http Version: HTTP/1.1 Size inBytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/wap
Http Code: 404 Date: Dec 14 12:38:27 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/wml
Http Code: 404 Date: Dec 14 12:38:27 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

/xhtml
Http Code: 404 Date: Dec 14 12:38:29 Http Version: HTTP/1.1 Size in Bytes: 29745
Referer: -
Agent: Nokia6600/1.0 (4.09.1) SymbianOS/7.0s Series60/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.0

Also, for those of you who are curious, the .WML file extension stands for Wireless Markup Language.

Share and Enjoy:These icons link to social bookmarking sites where readers can share and discover new web pages.
  • del.icio.us
  • digg
  • Fark
  • NewsVine
  • Netvouz
  • Reddit
  • YahooMyWeb

Leave a Reply