Yes, it seems as if the hot iPad social content app has a user agent that crawls the Web. Interestingly enough, the service is run on Amazon AWS according to the reverse IP lookup.

Host: 174.129.125.105
/2010/08/15/modern-email-marketing-two-odious-practices/
Http Code: 200 Date: Aug 15 12:27:35 Http Version: HTTP/1.1 Size in Bytes: -
Referer: -
Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (+http://flipboard.com/crawler)

Technorati Tags: Flipboard, iPad, Content, Social, Magazine, User Agent, Amazon AWS, iTunes

Being a Marketer who is focused on all things Internet related, I read a lot of email on a daily basis. I receive a lot of email, too. I can tell you quite honestly that I probably get more email than you in a day. I don’t really want more email that will take attention away from my primary concern, work. And having sent a lot of email (they don’t call it Email Deliverability for nothing) in a previous life, I’m pretty inured to Email Marketing practices - both good and meant with good intentions.

But I don’t know if it’s just me being the proverbial old grump with a full email inbox or what, but some Email Marketing practices of late have gotten pretty obnoxious.

Practice #1: Sending email to a catch all or general email address. We’ve heard the mantra from the opt-in evangelists about how you should not send email to a catch all address (i.e. sales@yourdomain.com or info@yourdomain.com or webmaster@yourdomain.com) because it weakens your deliverability - in terms of potential email bounces.

It isn’t just that though, as a marketer it shows you don’t know *jack* about the organization you’re marketing to. You’re basically proclaiming you’re too lazy to find out to find out who the decision makers really are in the organization. And that makes you a poor marketer.

Practice #2: Including a mailto: link to the recipients email address in the body of the email (mostly seen in the footer, near the unsubscribe link).

Whoa, wait, what? The first time I noticed this, I thought it was a newbie error on behalf of the sender. Now, I’m seeing the behavior from well known senders using well known Email Marketing services. So, I’m suspicious - because the thinking goes, if you add the recipient’s domain to the email, the email has less of a chance of being rejected by spam filtering software. Because of course you (the recipient) would not want to be using spam filtering software that would reject email with a link to your domain in it. Below is an example of the text:

This email was sent to: blog@cleverhack.com

You’ve received this message because you’ve registered to receive email or you’ve made a purchase from us.

If you no longer wish to receive email offers from us, unsubscribe here.

Sneaky, huh?

Technorati Tags: Email, Email Deliverability, Email Marketing, Email Addresses, Domains, Footer, Unsubscribe, Spam, Spam Filtering, B2B, B2C

Well, I guess when you have enough Twitter followers, you start seeing the phishing scams.

It looks pretty close - design wise - to an official Twitter email. However, the thing was a) sent to an address that isn’t used for Twitter and b) sent from a hotmail address, which means these guys were just hoping for a few clicks before getting shut down. The hover over shows the address of the phishing site.

Technorati Tags: Twitter, phishing, phish, Hotmail

Way back in late 2006, when the social Web was just starting and Twitter was but a mere messaging Web site, along MyBlogLog, which gave us the concept of a social Web profile and creating a community around your Web site.

The idea was a good one, and oh so exploitable by some unseemly social media marketers. And I think one of the better ideas to be popularized by MBL was the idea of the embeddable Web site widget featuring recent MBL visitors to your blog (see lower right hand side of cleverhack for the visitor widget).

Not that long after MBL arrived on the scene, they were acquired by Yahoo and their founders moved from New England to California. Unfortunately, Yahoo didn’t treat MBL all that well and never really improved the service or the MBL UI for that matter. The fact that MBL wasn’t really that integrated with Yahoo didn’t help matters much. And but three years later, we hear that Yahoo will kill MyBlogLog next month.

MyBlogLog - a great idea a little ahead of it’s time and never fully developed into a friendly usable product. I’ll be pouring out a 40 for one of the early pioneers of the social Web.

Technorati Tags: MyBlogLog, Yahoo, the Yahoo kiss of death, startup, hacking mybloglog, widget, visitor widget

Looks like someone with an Android handset visited cleverhack earlier today… Notice that Google has a special version of the search engine interface for Android (hint: click on the referrer). This seems to be the latest build of Android at 2.0.1, had no idea Google was using the AppleWebKit framework though. The screen size is also generous, too. Resolution : 854 x 480
Color Depth : 32 bits

Host: 75.209.219.99
*
/2007/12/10/hack-yahoo-fantasy-football/
Http Code: 200 Date: Dec 21 14:23:49 Http Version: HTTP/1.1 Size in Bytes: 13396
Referer: http://www.google.com/m?gl=us&source=android-launcher-search&q=yahoo++fantasy+foot
Agent: Mozilla/5.0 (Linux; U; Android 2.0.1; en-us; Droid Build/ESD56) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17

Technorati Tags: Google, Android, not the iPhone, AppleWebKit

Because of the hoopla around cuil today, I thought I’d take a peek at this newest search engine’s referrers.

Cuil crawler info. I know I’ve been seeing this bot for the past year or so. Cuil’s crawler is apparently called twiceler (is that a pun?) and the user agent string uses cuill.com which 302 redirects to the cuil.com domain. As of this writing, the
cuil Webmaster info URL has been updated from what is in the bot’s user agent string.

Host: 208.36.144.10

*

/

Http Code: 200 Date: Jul 28 15:02:12 Http Version: HTTP/1.0 Size in Bytes: 68965

Referer: -

Agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)

As for cuil visitor referrer info, here you go…

[Visitor’s IP Address]

*

/

Http Code: 200 Date: Jul 28 17:31:24 Http Version: HTTP/1.1 Size in Bytes: 17773

Referer: http://www.cuil.com/search?q=cleverhack

Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1

If you happen to see a “&sl=long” appended after the referrer i.e. (http://www.cuil.com/search?q=cleverhack&sl=long), it indicates that the visitor was using the two column layout. If cuil ever gets significant marketshare, you can bet there will be SEO’s stressing about how their sites show in the two column vs. three column layout.

Otherwise, a cuil visitor presents in your visitor logs pretty much as any other visitor from the big search engines. The IP address belongs to the user (not a proxy like ask.com) and so does the user agent.

As for my thoughts about cuil, I am not impressed with the image thumbnails with the search results, as nearly all I have seen so far have been wildly inappropriate for the results. As for information volume, I haven’t done a statistical survey, but google still presents a volume of results as opposed to cuil.

Technorati Tags: cuil, cuill, twiceler, referrer, crawler, bot, spider, referrer, user agent, former google employees, SERP

This past weekend there’s been a conversation about Shyftr a new RSS service that allows people to read and comment on full text stories on the Shyftr site, rather making the reader click through to the originating blog to comment. The thought is that folks who care about pageviews for advertising will lose out in such a scenario.

So, in the spirit of helping the wider, feathers in a ruffle, blogging community out, I’ve pasted the Shyftr RSS bot info below. The good news is that you can block the Shyftr IP address from accessing your blog (if you already have that capability through your blog hosting solution, etc.). As of present, the IP address is 66.234.234.34.

Unlike other annoying bots, I would not block the user agent in your .htaccess file as the RSS bot software the Shyftr folks are using is the generic MagpieRSS toolset, which is used by other RSS services. Hopefully, the people at Shyftr will rename the user agent to something more uniquely identifiable in the future so you can block via .htaccess.

(Note: Blocking a future unique Shyftr user agent via robots.txt probably won’t work as the crawler would need to fetch the robots.txt file first before fetching your feed and I didn’t see that behavior tonight.)

Host: 66.234.234.34
*
/feed
Http Code: 200 Date: Apr 12 19:48:28 Http Version: HTTP/1.0 Size in Bytes: 6244
Referer: -
Agent: MagpieRSS/0.72 (+http://magpierss.sf.net)
*
/favicon.ico
Http Code: 200 Date: Apr 12 19:48:28 Http Version: HTTP/1.0 Size in Bytes: 1406
Referer: -
Agent: -

Technorati Tags: Shyftr, RSS, RSS toolset, robots, crawlers, bots, lost pageviews, comments, blogger brouhaha

On the MSNBC developer blog, the question was posed How do you share?. Not in the grade school way, but in the newfangled Web 2.0 way.

Overall, the comments from MSNBC readers were pretty… negative. Aside from the “I’ll just paste the link I want to share in an email” or the “I’ll just add the page to my browser bookmarks” or the “they’re tracking your habits for nefarious purposes” comments, other commenters cited just one or two social bookmarking sites (the most popular seeming to be either del.icio.us or digg.com). And a few other commenters wondered, “Hey, MSNBC, don’t you own Newsvine?”

It appears that the zen habits of social bookmarking hasn’t been widely accepted by the at large Internet populace.

Technorati Tags: social bookmarking, bookmarking, Web 2.0, del.icio.us, digg, newsvine, putting it in my favorites

Had anyone else read this story of David Airey’s domains being stolen from him because of a Gmail exploit?

Both of David’s domains have been subsequently restored, thanks to the publicity he received this week.

Technorati Tags: Gmail, Gmail filters, exploit, cracked, domain stealing

Let’s all take a moment and remember the good old days of the Internet in the 1990s … the Netscape Web browser is being end of lifed as of Feb 2008.

If you didn’t catch Code Rush, a documentary on Netscape which was shown on PBS in 2000, I highly recommend you do so.

Technorati Tags: Netscape, Netscape Navigator, Netscape Communicator, Netscape Web Browser, AOL, Mosaic, Mozilla, Mozilla Foundation

This particular crawler is being deployed from the Semantic Web Search Engine (SWSE) project, which is attempting to crawl the nascent Semantic Web, including RSS and FOAF data.

This is yet another reason why deploying RSS is a good idea for any Web presence.

Here’s a link to the SWSE search demo.

Host: 140.203.154.196
/wp-rdf.php
Http Code: 304 Date: Dec 18 14:56:27 Http Version: HTTP/1.1 Size in Bytes: -
Referer: -
Agent: multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html)

Technorati Tags: Semantic Web, SWSE, RSS, FOAF, User Agents, Crawlers, Bots, Robots

Has anyone else seen some different activity coming from MSN? What I mean is that I’m seeing the following entries in my search logs, but it doesn’t appear like traditional MSNBot crawler behavior.

Why this activity is different:
1) The originating IP address is from the MSN netblock.
2) There is an alleged referrer that looks like it is from an MSN search http://search.live.com/results.aspx?q=keyword&mrt=en-us&FORM=LIVSOP
3) The user agent is showing as a browser.
4) This activity is showing very close to when I see MSNBot entries in my logs.

And no, the behavior does not appear to be a real life user.

Host: 65.55.165.38
*
/2006/06/17/live-blogging-from-the-philly-blogger-meeting/
Http Code: 200 Date: Dec 17 02:59:16 Http Version: HTTP/1.0 Size in Bytes: 40839
Referer: http://search.live.com/results.aspx?q=podcasts&mrt=en-us&FORM=LIVSOP
Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 1.1.4322)

Host: 65.55.165.42
*
/2006/06/20/how-e-commerce-will-be-affected-by-ie-7/
Http Code: 200 Date: Dec 17 03:13:02 Http Version: HTTP/1.0 Size in Bytes: 43238
Referer: http://search.live.com/results.aspx?q=podcasts&mrt=en-us&FORM=LIVSOP
Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 1.1.4322)

Technorati Tags: MSN, MSNBot, Live, Live search, User Agent, crawlers, bots