Whether you’ve been naughty or nice plays a big role in where the search engines will list you. Good guys at the top, bad guys on page 37. So if Google thinks you’re employing unethical tactics to gain referrals from other sites (spam, purchased links, etc) it might be necessary to disassociate your site from those links.
Recently we began this exercise with a client. Actually, we are still engaged in a multi-step effort to strengthen their visibility and improve their content. But one issue that leapt out at us immediately was that they seemed to have taken turns for the worse that corresponded closely with Google’s Penguin algorithm updates. The primary areas of concern with Google are their two famous algorithms, Penguin and Panda, and their Manual Penalties. (Moz provided a very readable summary and comparison of these about a year ago in their blog).
Although there have been no manual penalties assessed to date (phew!) we identified a massive 25% drop in organic search traffic that seems to correspond with Penguin updates. And we can infer that Google doesn’t like, for instance, that link coming from a spammy web directory in the Ukraine to our client in Texas that serves only the US. Bad juju, indeed. So we set about to ferret out as many dubious backlinks as possible so that we could drop the big D on them: DISAVOW.
Last year when Google rolled out its tool for Disavowing Links, it set the web ablaze with conversation about how to use the tool, when and why to use the tool and whether the tool should be used at all. Matt Cutts’ video on the subject only fueled the debate because he explained the tool but stressed caution. Additionally, we found it somewhat amusing that there doesn’t seem to be a direct link to the Disavow tool within Webmaster tools – we had to Google it. So did this mean we should be scared to use the tool??
Our extensive research on the subject led to the following conclusions:
- Disavowing links is crucial if (a) there are many dubious links to consider and/or, more importantly, (b) the site has incurred Manual Penalties.
- If disavowing links in response to Manual Penalties, Google expects a concerted effort on the site owner’s part to try to have the links removed by the originating sites. A campaign of emails must be launched and documented prior to submitting a list of links to Google. In other words, they want you to do it yourself and use the disavow tool as a last resort, ensuring that documentation of clean-up efforts is provided. It is also imperative to contact Google and request reconsideration after uploading a list of links to disavow in response to Manual Penalties.
- If disavowing links due to a perceived loss of standing based on an algorithm change (as in our case with this client and Penguin), it is not necessary to try to clean up the links before disavowing. A note can be included with the list of links that indicates they are being submitted after an audit in response to a believed algorithm consequence instead.
So, we decided to first identify a good chunk of links we felt were damaging and then proceed to send them to Google. The latter portion of the task proved quick and easy, but the former was a lengthy and involved process. We went about it like so:
1. We obtained a list of ALL LINKS that pointed to all pages within our client’s site. There were thousands. Backlink information is available in Google Webmaster Tools but we also like to use Majestic SEO .
2. For this initial round of disavowing (and because we wanted to do it relatively quickly), it didn’t make sense to manually review ALL of those links. So we decided to somehow filter out the probable worst of them.
3. How better to sniff out the bad than to ask the public to do it for you? That’s right–we crowd sourced a filter for this gigantic list. We achieved this by writing a small Python program for categorizing sites. We used publicly available information for tagging. In doing so, we instantly reduced the list of 3,000+ domains to less than 300 questionable domains that we could then easily manually evaluate. If you need to categorize backlinks for your site or for a client’s site, send me a note and I’ll help you out – we may make this tool available for free on our site in the next few months!
4. Then we manually reviewed the 300 suspect domains to select for disavowing. For most of these it took mere seconds to determine the quality of the site in question but for some it took a little longer. To disavow or not to disavow? That was the question.
5. Once we had our “naughty list,” we created our disavow list per Google’s disavow link format. We chose to use domain level information vice specific links as we assume that any other links from different pages on the same site will also be bad. We entered these domains into a UTF-8 encoded text file, added a note that we believe an algorithm change may have been in play and then we uploaded it. Voila.
Here’s our process in a nutshell:
All in all, it took about 5 hours of manual effort and 10 hours of running a program to tag domains. Talk about an efficient way to disavow links!
We did not ask for Google reconsideration as the site hasn’t incurred any manual penalties. We will, however, keep an eye on any changes for about a month or so and if we see nothing that we can attribute to this Disavow list, we will consider sending the reconsideration request. More to follow!
