SEO Hardcore » spidering

.htaccess commands

John Sisler — Fri, 04 Jan 2008 16:39:46 +0000

1. Redirect to Files or Directories

Redirect permanent /oldfile.html http://www.domain.com/filename.html or
Redirect 301 /oldfile.html http://www.domain.com/filename.html

Redirect 301 /olddirectory/oldfile.html http://www.domain.com/newdirectory/newfile.html

Redirect 301 www.olddomain.com/

Also you can program your home page to always load the www.domain.com version of the URL regardless of your actualy homepage file or a user not typing www.

1a.To redirect the non-www version use:

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^domain.com [NC]
RewriteRule ^(.*)$ http://www.domain.com/$1 [L,R=301]

1b.To prevent

Search in 2007

John Sisler — Wed, 02 Jan 2008 22:47:54 +0000

The following was originally presented on SEW

Universal/Blended search

Google Universal search

Yahoo Panama

Yahoo Reorgs

Ask 3D

Microsoft Live Search/AdCenter

AOL Platform A

Social Media

Personalization & Customization of SERPs

Quality Score of Ads

SEM/SEO 2.0

Social Search

Local Search

Mobile Search

Paid Links

Search in the Mix/Integration

Click Fraud

It’s Time to See What’s In the PPC Sausage
Click Fraud Numbers Up, Content Networks Near 20%
Google Refutes Click Fraud Numbers, Once Again
Yahoo Steps Up Click Fraud Efforts
Google Shares More Click Fraud Numbers
Click Quality Council Outlines 8 Principles of Click Quality
Google, Yahoo Respond to Click Quality Council Guidelines
Fair Isaac Click Fraud Report Spreads False Alarm
Fair Isaac Pegs Billed Click Fraud at 10-15%

What can ACAP do for me?

John Sisler — Thu, 06 Dec 2007 00:28:33 +0000

Implement ACAP
What do I need to do to use ACAP on my website?
ACAP may be already be used in a variety of different business contexts, but its main initial purpose is for communicating access and usage permissions to web crawlers (also known as ‘spiders’ or ‘robots’) – the automated processes employed by search engine operators and others to “crawl” the web and harvest content for use in their services.

There are two stages to using ACAP on your website for communicating permissions to web crawlers. They are: implementation and verification.

1. Implementation
Implementing ACAP involves making some important changes to your website. These are of two kinds:

* changes to your ‘robots.txt’ file
* changes to content resources

The simplest way of communicating permissions information to web crawlers is through a ‘robots.txt’ file placed on your web server. If you already have a ‘robots.txt’ file, this will need to be modified to contain ACAP permissions in addition to the information that it already contains (remember that not all web crawlers will be able to interpret ACAP permissions, for a while at least, so you should retain the existing content of your ‘robots.txt’ file).

To save time and effort in modifying existing ‘robots.txt’ files, a tool for converting them to include the corresponding ACAP permissions is available here: >.

However, to gain the full benefit of using ACAP, you will need to make further modifications to ensure that the permissions say exactly what you need them to say. How to modify your ‘robots.txt’ file is explained in the ACAP Technical Framework documentation, which you will find here: >.

Another way of communicating permissions information to web crawlers is by embedding permissions information directly in the content resources themselves. ACAP permissions can be embedded in almost any kind of web resource, and how to do so is explained in the ACAP Technical Framework documentation, which you will find here: >.

Once you have implemented ACAP on your website, please add the “ACAP enabled” logo to your home page and other pages on your site. Go to: Add an

ACAP Launches, Robots.txt 2.0 For Blocking Search Engines?

John Sisler — Thu, 06 Dec 2007 00:20:41 +0000

After a year of discussions, ACAP — Automated Content Access Protocol — was released today as a sort of robots.txt 2.0 system for telling search engines what they can or can’t include in their listings. However, none of the major search engines support ACAP, and its future remains firmly one of “watch and see.” Below, more about the how and why of ACAP.
Let’s start with some history. ACAP got going in September 2006, backed by major European newspaper and publishing groups that in particular felt Google was using content without proper permissions and wanting a more flexible means to provide this than allowed by the long-standing robots.txt and meta robots standards.

These two standards are found at the robotstxt.org, and ACAP has been referring to them often at “Robots Exclusion Protocol” or REP, though within the SEO world, they’re generally known by their actual names.

Robots.txt was born in 1994 as a way to block content on a server-wide basis; meta robots emerged in 1996 as a system to block on a page-by-page basis (see Meta Robots Tag 101: Blocking Spiders, Cached Pages & More for more about it). Neither has been updated since those years ago, in terms of search engines coming together to agree on new universal standards. In short, REP has no “guardians” or group to take it forward.

Enter ACAP. If the search engines weren’t going to improve robots.txt, the aforementioned publishers decided they’d take on the challenge. Of course, creating a standard for search engine indexing is kind of a waste of time, if you don’t have the search engines themselves to actually support it. But ACAP didn’t let that be a deterrent. Over the past year, it has had a working group setting up a new system, with search engines Google and Ask.com, along with Exalead, taking part in the discussions. FYI, I’ve not been an active working member, but I’ve been included on the working group’s emails and chimed in from time to time with advice and thoughts.

The ACAP System

Now the new system has arrived, being unveiled at the ACAP conference in New York today. Before getting into support, let’s cover what’s in it. You’ll find an overview page for the specifications here, which leads to:

A robots.txt-to-ACAP conversion tool (don’t worry; this should make your robots.txt file still work as a regular one and double as an ACAP file)

Simple Silverlight SEO with ASP.Net and XSLT

John Sisler — Mon, 12 Nov 2007 22:33:54 +0000

This was originally published by Michael S. Scherotter

Introducing Microsoft Hyper-V

John Sisler — Mon, 12 Nov 2007 19:26:21 +0000

Today at Tech ED / IT FORUM in Barcelona we made a couple of important announcements:

Windows Server virtualization in Windows Server 2008 (formerly codenamed ‘Viridian’) will have an official product name of ‘Hyper-V’.

Get Safe Online Week II

John Sisler — Mon, 12 Nov 2007 19:26:18 +0000

We’re in “Get Safe Online Week”:

The initiative aims to help individuals and micro-businesses to protect themselves against Internet security risks by raising awareness of the need for Internet security and providing information, advice and updates through its website www.getsafeonline.org.

It’s entirely coincidental that I’ve just spent an hour at the IT Forum conference, listening to Vinny Guloto, General Manager for the Microsoft Malware Protection Centre. IT Forum is a gathering of over 5,000 IT Directors and service managers from all over Europe. It includes over 100 delegates from UK education establishments – mainly colleges and universities.

Anyway, back to Vinny. He presented today on the work that the centre is putting into the world of threat research, and both reactive and proactive response. Some of the numbers he used were astounding – around 1995, there were 20,000 viruses on the loose – now there are about a quarter of a million (with even more variants of core viruses). No wonder we need anti-virus software!

He mentioned some statistics in relation to Windows Defender (a piece of free software from MS which looks for things like adware and spyware, rather than specifically “malicious” software – it isn’t anti-virus software). Using Windows Defender as an example, he shared some facts about intercepted threats:

In the first six months of 2007, it detected 50 million pieces of potentially unwanted software
- 16 million items of Adware
- 7 million items of “potentially unwanted software”
- 6.5 million Trojans
- 3 millions items of “remote control” software

To read more about the work that Vinny’s team do, check out their Microsoft Malware Protection Center Portal

And if you want to help your staff and students avoid the 50 million bits of nasty software out there, let them know about Get Safe Online

Is the 'Ron Paul Effect' for Real?

John Sisler — Mon, 12 Nov 2007 18:56:49 +0000

Posted by great scott!

If you hang out online much (and if you’re reading this, chances are you do), you’ve no doubt noticed the seeming onslaught of fevered, near-rabid support for Republican presidential candidate, Ron Paul.

Passing Page Rank Across Servers

John Sisler — Mon, 12 Nov 2007 15:08:43 +0000

Q: How can I pass Page rank from my domain to directories of information that are hosted on other servers?

A: mod_proxy.

SMX London: A Sneak Preview

John Sisler — Fri, 09 Nov 2007 15:55:22 +0000

Last week’s inaugural SMX Sweden was a great success