There are two stages to using ACAP on your website for communicating permissions to web crawlers. They are: implementation and verification.

1. Implementation
Implementing ACAP involves making some important changes to your website. These are of two kinds:

* changes to your ‘robots.txt’ file
* changes to content resources

The simplest way of communicating permissions information to web crawlers is through a ‘robots.txt’ file placed on your web server. If you already have a ‘robots.txt’ file, this will need to be modified to contain ACAP permissions in addition to the information that it already contains (remember that not all web crawlers will be able to interpret ACAP permissions, for a while at least, so you should retain the existing content of your ‘robots.txt’ file).

To save time and effort in modifying existing ‘robots.txt’ files, a tool for converting them to include the corresponding ACAP permissions is available here: >.

However, to gain the full benefit of using ACAP, you will need to make further modifications to ensure that the permissions say exactly what you need them to say. How to modify your ‘robots.txt’ file is explained in the ACAP Technical Framework documentation, which you will find here: >.

Another way of communicating permissions information to web crawlers is by embedding permissions information directly in the content resources themselves. ACAP permissions can be embedded in almost any kind of web resource, and how to do so is explained in the ACAP Technical Framework documentation, which you will find here: >.

Once you have implemented ACAP on your website, please add the “ACAP enabled” logo to your home page and other pages on your site. Go to: Add an “ACAP-enabled” logo: > to find out how to add the logo to your website.

2. Verification
Once you have implemented ACAP on your website, you will probably wish to verify the quality of your implementation. We expect there to be a number of ACAP verification services that can tell you whether your implementation conforms to the ACAP Technical Framework or not, and details will shortly be available on the ACAP website. Once your ACAP implementation has been verified, you will be entitled to add the “ACAP compliant” logo to your website.

Looking for an answer

Try searching through our comprehensive FAQ section

Frequently Asked Questions

These two standards are found at the robotstxt.org, and ACAP has been referring to them often at “Robots Exclusion Protocol” or REP, though within the SEO world, they’re generally known by their actual names.

Robots.txt was born in 1994 as a way to block content on a server-wide basis; meta robots emerged in 1996 as a system to block on a page-by-page basis (see Meta Robots Tag 101: Blocking Spiders, Cached Pages & More for more about it). Neither has been updated since those years ago, in terms of search engines coming together to agree on new universal standards. In short, REP has no “guardians” or group to take it forward.

Enter ACAP. If the search engines weren’t going to improve robots.txt, the aforementioned publishers decided they’d take on the challenge. Of course, creating a standard for search engine indexing is kind of a waste of time, if you don’t have the search engines themselves to actually support it. But ACAP didn’t let that be a deterrent. Over the past year, it has had a working group setting up a new system, with search engines Google and Ask.com, along with Exalead, taking part in the discussions. FYI, I’ve not been an active working member, but I’ve been included on the working group’s emails and chimed in from time to time with advice and thoughts.

The ACAP System

Now the new system has arrived, being unveiled at the ACAP conference in New York today. Before getting into support, let’s cover what’s in it. You’ll find an overview page for the specifications here, which leads to:

A robots.txt-to-ACAP conversion tool (don’t worry; this should make your robots.txt file still work as a regular one and double as an ACAP file)
 
ACAP extensions to use with robots.txt (here, PDF file)
 
ACAP extensions to use with meta robots (here, PDF file)
 
ACAP logo for those that want to show they’re using ACAP (not required to make ACAP work, but expect publishers pushing ACAP to make use of it)
What does ACAP provide that robots.txt and meta robots does not? After going through the technical specs, which are pretty dense reading, I’d summarize it this way:

Emphasis on both granting permissions and blocking
 
Support for time-based inclusion or exclusion
That’s it. Discussions have covered concepts such as how password-protected content could be indexed, or whether you could issue permissions on a country-by-country basis, but some of these ideas haven’t made it into the first cut.

AP has a nice overview article about the ACAP launch, and I found the companion piece a nice summary if you’re looking for some faster specifics. A key part:

Some search engines have interpreted “disallow” to mean that the site cannot be added to the index but could be fetched for use in various algorithms employed to determine how high a site appears in search results….ACAP proposes to clarify that “disallow” refers to indexing.

A separate “crawl” command would be added to bar the indexing software or crawler entirely.

In addition, Web sites would be able to add qualifiers stipulating that the information expires from the search index on a specific date, in a given number of days or whenever the crawler returns to the site.

A “follow” command would permit or block the crawler from following links within a page.

“Preserve,” with similar time limits available for “index,” would stipulate whether a copy may be stored in a search engine’s cache.

“Present” would govern a search engine’s ability to display the copy, and a site may limit that further — for example, to a snippet or to a miniaturized version, or thumbnail.

As I said, there’s an emphasis on granting permission. By default, search engines assume everything is open to indexing. ACAP changes this assumption, asking those that create the files to explicitly indicate yes or no.

Should You Use It?

So now we have a new standard for expressing search engine permissions. Do site owners need to run out and immediately use it?

No. Not immediately. Not even long term.

Right now, none of the major search engines are supporting ACAP. If you were to use ACAP without ensuring that standard robots.txt or meta robots commands were also included, you’d fail to properly block search engines. Only Exalead, which is not a major multi-country service, would currently act upon your ACAP-only commands.

Even if ACAP were to magically get endorsed and supported by all the major search engines, robots.txt and meta robots support wouldn’t go away for many years. There are simply too many sites that use those systems, have used them for over a decade, and would fail to upgrade. Those two systems will continue to be supported in the same way Microsoft has had to support DOS programs despite the growth of Windows.

So why bother at all? Probably two reasons:

You want to personally test out how ACAP works, playing with the permissions and seeing what happens in Exalead
 
You want to support the ACAP system and hope that if enough people use it, perhaps the search engines will adopt it. FYI, ACAP is urging (PDF file) “universal adoption” by publishers by the end of next year.
Search Engine Support

What’s up with the major services? I emailed the big three, Google, Microsoft, and Yahoo, all of whom either took part in the working group or are at today’s conference. Google’s canned answer:

We are interested in all initiatives that allow web publishers and search engines to work more closely together. We have undertaken many efforts in this direction over the years including supporting file-extension and wildcard specifications in robots.txt, SiteMaps, our Webmaster Console, extending per-item indexing specification to non-html documents and specifying how long a url would be available. We will examine ACAP proposals when they become available. As a broad-based search engine, we need to keep in mind the needs of millions of web publishers worldwide.

As it happens, I was at Microsoft yesterday, and while I haven’t gotten a formal statement to post, the sentiment was the same as Google. Microsoft is interesting in supporting publishers, is continuing to grow its own tools and will also watch ACAP, wanting to support publishers in general

Yahoo’s not sent a statement back yet, but when it arrives, you can expect it will be pretty much the same as Google and Microsoft.

Why not just jump into ACAP? Between the lines time here — no one really wants to hand over control of the standard to the ACAP group, especially in my view when it has been born out of some anti-search engine hype.

So why not jump behind improving robots.txt and meta robots? Another issue here is that no one is officially in charge of those standards. The search engines are sort of the gatekeepers, because it’s what they decide to support that effectively becomes “law.” If they don’t support a particular exclusion command, it might as well not exist.

The various search engines tell me they have been talking more about making some collective improvements. Individually, they’ve already added to both robots.txt and meta robots over the years, extensions that may work with their particular search engines. Perhaps they will become more unified.

In particular, they’ve united around the sitemaps standard. That sort of picks up what ACAP does in terms of being a system to provide express permission of indexing, and it’s where I’d expect any search engine-driven, collective agreement about improved blocking tools to emerge.

Also be sure to read Up Close & Personal With Robots.txt, which summarizes the second robots.txt summit that I organized earlier this year. The article covers a lot of things that general site owners and SEOs have wished for, along with some search engine responses.

Conclusion

So has the entire ACAP project been a waste of time, or as Andy Beal’s great headline put it when ACAP was announced last year, Publishers to Spend Half Million Dollars on a Robots.txt File? That still makes me laugh.

No, I’d say not. I think it’s been very useful that some group has diligently and carefully tried to explore the issues, and having ACAP lurking at the very least gives the search engines themselves a kick in the butt to work on better standards. Plus, ACAP provides some groundwork they may want to use. Personally, I doubt ACAP will become Robots.txt 2.0 — but I suspect elements of ACAP will flow into that new version or a successor.

originally written by Danny Sullivan on Search Engine Land

My post last week on measuring visitor engagement was pretty long by the time I outlined the calculation, so I put off publishing examples of how the metric could be used until now. I’m excited to see that this topic has generated so much interest, both in terms of comments and emails sent to me directly.

My goal for this post is to provide a few examples and explanations to show how the metric can be used to supplement our otherwise already-rich set of web analytics data. Since so many folks have been willing to explore the engagement metric, I have embedded a bunch of questions in this post in italics that I’d love your feedback on.

Distribution of engagement scores and segmentation. Here is the distribution of engagement scores for about six months at Web Analytics Demystified by percent of visitors. As you can see, these scores are left-skewed and tail off as the score increases, showing that nearly half (47.6%) of visitors to my site are “poorly engaged”. When I look at this distribution it makes perfect sense to me — what do you think?

I have created segments to group visitors by their engagement score: “Well engaged” visitors have engagement scores over 30%, “moderately engaged” visitors are those between 10% and 30%, and”poorly engaged” visitors score less than 10%. These segments can then be used to explore how the behavior of visitors in each engagement group differs by looking at my page and referring source dimensions (page, content group, referring domain, campaign, search phrase, etc.)

Identify relationships that might otherwise not be found. At the top of this report you can see the pronounced difference in visitor engagement (and traditional metrics) for “branded” and unbranded searches (”None”) bringing visitors to my site. Now, because branded searches are a component of the calculation (Brand Index), you definitely expect to see a difference between the two engagement scores. What is interesting is that while other metrics (duration, sessions per visitor, page views per session) show a slight difference, visitor engagement and conversion are all three times higher for branded searches. I think this difference observed in all the metrics is further evidence that brand-driven searches are bringing more engaged visitors — what do you think?

In the middle table you can see search phrases bringing visitor to my site, showing visitor engagement, page views per session, and sessions per visitor. Here three phrases stand out to me:

1. “web analytics book” and “web analytics process”, neither of which are particularly distinguished from other search phrases based on page views per session or sessions per visitor but both of which have visitor engagement scores over double my site-wide average of 8.8%. This is important to me because these are un-branded search terms that are critically important to my business.
2. “vendor discovery tool” which would appear to be pretty important based on traditional metrics but only stands out slightly using the visitor engagement score (at 13.6%) I spend a lot of time trying to figure out how to drive folks using the vendor discovery tool to take other actions (buy books, inquire about consulting) and this data suggests that there is an unrealized opportunity.
3. “performance indicators” which shows that the visitor engagement metric is useful to identify terms that you’d think are important to the site but aren’t attracting the right audience (average engagement score for these visitors is only 5.6%)

I think this level of information is actually pretty helpful for identifying search marketing opportunities — what do you think?

Engagement-derivative metrics like “Percent Highly Engaged Visitors” are useful. Here you can see a select group of referring domains showing the percent of highly and percent moderately engaged visitors they’re sending my way (with conversion to show that engagement and conversion are in fact different!) Avinash Kaushik is sending me a few (0.2%) highly engaged visitors (thanks!) but Ian Thomas is sending me a bunch (70.4%) of moderately engaged visitors, many of whom are purchasing books (1.2% conversion rate.)

By looking at traffic from Avinash’s site over time (bar graph) I can see peaks and valleys in overall engagement from folks coming from his site, which would be useful to back into those peaks to try and determine what other blogger’s readers might be reacting to when they’re exhibiting highly-engaged behavior on my site (see late August and early September.) Given that Clint proved that conversion is a poor measure of success when trying to evaluate traffic from other bloggers, I think visitor engagement is useful for examining the non-revenue value of referring sources — what do you think?

Those of you who are looking for correlation between engagement and conversion, have a look at the data for Mr. Jim Sterne’s wonderful site emetrics.org —  5.6% of the folks coming from Jim’s site are highly engaged, 66.2% moderately engaged, and man-oh-man does Jim help sell some copies of Web Analytics Demystified.  You’re the man, Jim!

Visitor engagement is globally useful. At least in Visual Sciences Visual Site you can apply engagement metrics and segments to pretty much any dimension tracked. Here I’m looking at the percentage of “highly” engaged visitors (50% or more) in my “well engaged” segment broken down by country. Now, this is certainly more interesting in light of the total volume of traffic coming from each geographic location, and as I think about localizing my books and planning future trips around the world this information becomes very helpful.

There is more, including some of the more granular visitor-level stuff I talked about in the first series of posts on the subject, but I want to be sensitive to protecting the identity of individual users on my site. If you’re interested in helping me collect some “ground truth” regarding the engagement calculation, write me and I’ll explain how you can help.

So what do you think? Do the screen-shots help you understand the calculation better? Or do they still make it look super-complicated and scary? Is there something specific you’d like to see me demonstrate with the calculation? Or do you think you could come up with these same insights using more traditional metrics?

A number of you have commented that I have been oddly quiet on the subject of Omniture planning to acquire Visual Sciences and then the news that four senior-most managers at WebTrends were let go. It’s not that I don’t have an opinion — I can assure you that I do — but I wanted to take a little time to clarify my thoughts on these subjects before blogging about it.

On the Omniture/Visual Sciences deal, I sincerely do congratulate Josh James and the entire team at Omniture on building a company capable of completely taking out their biggest competitor. Over the years I have found myself having a somewhat topsy-turvy relationship with Mr. James and his organization: First I had to compete with them while at WebSideStory, winning some deals and losing others. Then I worked directly with them while at JupiterResearch, spending time both in their offices and also on their behalf through online seminars and client events. Finally I spent a little over a year competing with them again, this time at Visual Sciences, again winning some deals and losing others.

Regardless of where I worked, it was impossible to not develop a healthy respect for Omniture and their success. It pained me to watch deals like HP, AOL, CBS Sportsline, USAtoday, Overstock.com and others go their way, despite hard work from a talented group of individuals, and I absolutely hated going up against their particular version of salesmanship. But as an analyst it was encouraging to see Josh and John Pestana build a company that understood the underlying technology but also how that technology could make their customers more successful.

Their customers responded to this, and still do. It is not uncommon to meet Omniture customers who have “drank the kool aid” and for whom their customer status is very much a badge of honor. Hopefully Mr. James et al. will deliver the same Omniture experience for the 1,500-odd companies they’re purchasing from Visual Sciences/WebSideStory, because that is where I see the inherent risk in this deal.

All week last week people with a lot of money under their management asked me “what is the upside and what is the risk in this acquisition?” I’m not a financial analyst (disclosure: I don’t have any holdings in OMTR or VSCN) so all I could comment on was what I hoped the combined company would do and not do. And while nobody from Omniture has asked me — not that it would be particularly appropriate anyway — here are a few thoughts on what the new company needs to do to make this acquisition successful:

1. Suck it up and start the migration from HBX to SiteCatalyst immediately. I haven’t read all of the various transcripts on this deal, but nobody I am talking to expects HBX to survive the balance of 2008 if the deal is approved (which I sincerely believe it will be.) Omniture should smooth the transition path by splitting data collection at the gateways now and simultaneously loading whatever HBX-collected data into the SiteCatalyst data collectors, thusly giving HBX customers the easiest possible transition from one technology to the other. And while if I were on HBX I would be aggressively thinking about migrating to the SiteCatalyst code base, this transition is far from a slam-dunk at the customer-level. Splitting the data will give marketing something to show I.T. if they complain about needing to replace the JavaScript (again), and getting started on data collection now will potentially ease some of the pain associated with not being able to migrate years of HBX data that some customers might not want to lose (if that is the final assessment.)
2. Admit that Visual Workstation is the right interface for serious analysts. Again, I have not read the transcripts, but comments I have read are unclear about whether Discover 2 or Visual Workstation will live past the acquisition point. And while I have spent much time looking at Discover 2, I can assure you that of the two, Visual Workstation is the technology to keep (disclosure: I recently entered into a licensing agreement with Visual Sciences to use Visual Workstation at Web Analytics Demystified.) No disrespect to Omniture’s fine product team, but Visual Workstation is unparalleled for sheer analyst-class power, and I’m fairly sure that without modification Visual Workstaion can leverage whatever format Omniture stores visitor-level data to get up and running quickly. This may cause problems from a pure SaaS-perspective, and I could be wrong, but I suspect that most analysts wouldn’t actually mind having to run the software locally in exchange for having the robust data manipulation capabilities that Workstation provides.
3. As painful as it will be, resolve the internal stuff quickly. A huge potential pitfall in this deal is that it has tremendous potential to create confusion regarding who is managing what, when, where, and how all of these technologies are presented in sales and support situations. Not that this will be easy, any M&A transaction has the potential to be messy, but Josh and Jim MacIntyre won’t be doing anyone favors by sugarcoating what this deal is or being vague about who might be reassigned and who might be let go (keep in mind that these companies that were bitter enemies in the marketplace up until two weeks ago.) Any internal confusion about the transition will inevitably impact customers in the form of unclear deadlines, changing account managers, and other miscommunication that will only open the door for other vendors …

Which brings me to the other change in the web analytics market last week: WebTrends announcing that Greg Drew, Jason Palmer, Tore Steen, and Hamid Bahadori had all been asked to leave the company. I have to admit, I was more-or-less shocked by this announcement, specially given that I have been saying to folks since mid-July that I believe, at least from a software perspective, that WebTrends is finally getting back on track. I really do believe that WebTrends Score is one of the few true innovations we’ve seen in the web analytics marketplace recently, and learned WebTrends users far and wide have commented that they really like the stuff in the MarketingLab2 release.

I should also say that I personally really like Greg Drew and Jason Palmer. Now, I say that not having worked with or for them in any role other than that of an industry analyst, and anecdotally some of the recent flight from the company can be tied back to their leadership. But Greg has always struck me as one of the nicest guys in the entire industry and someone who was willing to do what it took to get the job done.

Regardless of Greg’s personal disposition, I again find myself nearly flabbergasted that the folks at Francisco Partners who are calling the shots would give up Greg and Jason’s experience in the field and knowledge about web analytics in general. I mean, it’s not like Eli Shapira is going to come back and run the company, or that it will be easy to find someone else to run the ship as experienced with web analytics as Josh James from Omniture, Joe Davis at Coremetrics, or Dennis Mortensen at IndexTools. Especially on the heels of the Omniture/Visual Sciences announcement, this whole thing sounds so fishy it’s almost unbelievable, but I have to believe that these four guys will be harder to replace than people think.

Case-in-point: when Jeff Lunsford showed up at WebSideStory sans web analytics experience, some of us were worried. But Jeff was a natural born-leader, and given time it was clear that Jeff had what it took to get the job done. Unfortunately, in retrospect, it is no longer clear exactly what that job was aside from making a small number of people a huge sum of money, and Jeff has moved on to even bigger deals. I liked working for Jeff tremendously, but I’m not 100 percent sure he left WebSideStory in better shape than he found it.

I’ll admit, I don’t have the experience that these guys have … I’ve been running a company of two people for seven months. But just as I felt like WebTrends was well positioned (along with Coremetrics) to be a strong solution with a great customer base, a good set of features, that was incidentally “not Omniture”, I now find myself questioning how strong the organization will really be when run by folks largely new to web analytics. No disrespect to Tim, John, Leo or Bruce, but web analytics is hard, the competition is big and about to get bigger, and sophisticated web analytics buyers will easily differentiate between passion and experience.

Trust me, I want to be wrong about this. I would like nothing more than to have someone clarify what happened at WebTrends and detail how the company is going to accelerate growth against Omniture given their recent momentum. I think despite Omniture’s strength and Google Analytics widespread deployment that the “web analytics wars” are far from over. Like others, I worry that a two horse race isn’t very exciting to watch, and despite believing that “it’s not the technology, it’s how you use it” that it’s nice to see innovation from time to time. For this to happen, I believe we need a strong WebTrends, a strong Coremetrics, and at least a small handful of smaller innovators out there in the world (Nedstat, IndexTools, Clicktracks, etc.) nipping at everyone else’s heels.

What do you think? Am I crazy? Am I just missing the most obvious thing? Am I too close to the situation, having worked with or for all of the companies involved in the past few weeks insanity? Or do you share some of the same concerns I do? Either way, I’d love to hear what you have to say.