24th Jun 2017

How to do keyword research & analysis for SEO 2017

Lucy Kirkness
The goal of this article is to guide you through our keyword research process for SEO, the most important and valuable activity for any search marketing professional.

Keywords, or key phrases, (should) form the basis of any SEO campaign. They should provide a roadmap for everything from designing websites to content development, and are the key to understanding your customers. However, starting an SEO campaign based on the wrong research is a recipe for disaster. By disaster, I mean low traffic, low rankings, and potentially a Google penalty.

So, how can you make sure your keyword research is “right”? Traditional keyword research, where important optimisation decisions were made using a vague combination of search volume and competition, is simply no longer going to cut it. Consumer behaviour and the modern purchase journey have become much more complex, so our processes for analysing keywords must evolve as well. In short, we must also be analysing user intent, and placing these queries at the right point in the conversion funnel.

There are plenty of other excellent guides out there that explain how to do keyword research properly, however it is often useful to follow a real life example from start to finish. 

In this post, we’ll look at a cat product comparison site as an example, as this happens to be a project I am working on. The general framework I am going to go through for conducting keyword research (or search query analysis), can be applied to any industry.


Understand Your Niche
Generate Keyword Ideas
Evaluate Keyword Ideas

Understand your niche

Before you dive into the various keyword research tools out there, it’s crucial to get a better understanding of your customers, and how they are talking about your products or services online. There are plenty of ways you can start collecting data on your target user personas, whilst also identifying some potentially profitable keywords.


All seasoned SEO’s know that forums can be a gold mine when it comes to finding great keyword ideas. They are essentially live focus groups, and contain an incredibly accurate example of how people talk – and therefore, search.

The easiest way to search for forums related to your niche is to use simple search strings in Google. Here are a few examples I used:

"cat product" + "forum"
"best cat products"+ "forums"
"cat advice" + "forum"
"cat product" + "boards"

The petforums.co.uk site was frequently mentioned in my research, so I thought I’d take a peek.

The first thing to notice is how the forum is divided up into sections or categories. These sections are often niche topics, and can be very useful for when it comes to building your own category structures.


In a number of seconds, I’ve identified 2 (or 4) possible niche topics for my cat product comparison site:

  • Cat Health and Nutrition
  • Cat Training and Behaviour

Digging into the threads a little deeper, I find an A-Z of wet food for cats thread, a thread on the best food for neutered kittens, and insight into their love of cardboard boxes. Automatically I’ve got hundreds of keyword ideas, all related to one of the major niche topics on my site, and a better understanding as to the quirks of our feline friends.

Delving into each and every thread to better understand your user personas can get a little time consuming. What if we could automatically scrape relevant forums to find questions your target audience are asking, for free?

Welcome to my first tool suggestion: FAQ Fox. I came across this gem of a tool a few months ago, and have been using it to aid my keyword research ever since. While it’s not a typical keyword research tool, it’s great for understanding real user problems or queries. Let’s see it in action:


The tool lets you search by keyword, and then auto populates a bunch of forum sites for scraping, depending on the category chosen. I have also added in several cat forums that I found during my previous research phase. At a glance we can see a few more good threads for finding potential keyword opportunities (amongst a couple of totally unrelated threads). It is evident from this small amount of research that cat health and food are popular topics amongst the audience.

Generate keyword ideas

Now that we have a better understanding of our audience and the various niche topics, we can start generating some initial keyword ideas.

1. Use your brain

A good place to start is by rolling up your sleeves and trying to think of keyword ideas on your own. It’s typically easier to think of broader topics than long-tail keywords, but it’s good to begin with some brainstorming before digging into all of the research data. Try to think about other niche topics, product names, brand names etc and you should be able to come up with a good list.

I have managed to create a seed list of 70 keyword ideas.

2. Spy On Your Competitors

Doing some competitive analysis is the next best place to start. As well as learning about your audience, it is crucial to learn about your competitors. SEMRush is one of the most valuable search marketing tools for this, as it provides highly valuable organic and paid keyword and traffic data.

One of the most valuable tools to aid your keyword research in the SEMRush suite is the paid search keyword data. You can see exactly what keywords your competitors are bidding on, as well as the costs, giving you a good indication of the potential profitability of each keyphrase. Competitor research data doesn’t get much more sexy than that!

Here’s an example of the data I can get for a large price comparison website’s ‘cat product’ related organic ranking and search volume data.


SEMRush allows you to export these keyword lists. So, my next step is to run this tool for various competitors, export the keyword data, and add it straight into my keyword ideas spreadsheet. Due to the nature of a comparison site, researching retailers sites also generated some good keyword ideas.

This brings my seed list up to 1,372 keyword ideas. Note: there may be multiple duplicates at this point.

3. Use Ubersuggest and/or KeywordTool.io

We’ve already got quite a lot of keywords in our list, however if you want even more then Ubersuggest or KeywordTool.io are great for this purpose.

Both tools are free, and use Google’s Autocomplete to generate hundreds of relevant long-tail keyword ideas. Simply input your base keyword, and the tools will extract Google suggestions based on how often users have searched for a particular term, amongst other factors.

For the sake of this example, I used KeywordTool.io as it’s my personal preference out of the two options. I ran this tool several times using different head terms as my base keywords, for example:

  • Cat food – the tool generated 607 further keyword ideas based on real user queries
  • Cat litter – the tool generated 499 further keyword ideas based on real user queries
  • Cat toys – the tool generated 302 further keyword ideas based on real user queries


As well as keyword suggestions, KeywordTool.io also generates frequently asked ‘questions’. The result is a very useful set of data for long-tail content development ideas.

This brings my seed list up to 2,730 keyword ideas.


4. Use Keyword Snatcher

If you want to generate an even larger list of keywords, Keyword Snatcher is the answer to your prayers.

Simply open the tool, enter your base keyword, and click on ‘Get Suggestions’. You might need to wait for some time, but this tool will return thousands of keywords from your single seed term. If you don’t want to wait too long, you can stop the tool at any point, at which point you’ll already have generated a useful list of terms.

For this search, I’m going to focus on the ‘comparison’ feature of the site and use ‘compare cat’ as my seed term. After 5 minutes, the tool has already generated over 1,900 keywords. I’m going to stop it there, copy the list and paste it straight into my keyword ideas spreadsheet. Other keywords I can use for my seed term are, for example, ‘best cat’, ‘cheap cat’ etc.

This brings my seed list up to 4,625 keywords.


You may also be familiar with ScrapeBox, which you can use as an alternative to Keyword Snatcher.

5. Use Keyword Spy

Keyword Spy is a great tool for getting a quick lay of the land. With access to PPC ad data, related keywords, and competitor data, it’s almost a definite one for the keyword research toolbox!

I like to round up my keyword idea generation phase with this tool as it has a ‘misspell’ feature, which provides keyword suggestions based on common misspellings. Unfortunately in the case of my example, there weren’t too many useful misspelling suggestions.

This leaves my seed list at a total of 4,625 keywords.


6. Google Analytics & Search Console Data

If the site has been around for a while and you have historical data from your Analytics or Search Console accounts, then this would be one of your final stages before de-duplicating the list and running your keyword ideas through Google’s Keyword Planner to get monthly search volume and competitive data.

The more established your site is, the more data you will have to play with, and the more potential keywords you will find. In this example, the cat product comparison site is still in its development phase so we do not have access to this useful data.

7. Google Keyword Planner

Before I take my list and extract search volume data, I am going to run the final stage of my keyword idea generation phase through the Google Keyword Planner ‘search for new keywords’ option.


As you can see, Keyword Planner has suggested some very useful ad group ideas, which I can also use as ideas to build my site architecture and category structure. For the purpose of generating keyword ideas, I am simply going to click on the ‘Keyword Ideas’ tab and download the 801 results generated from my seed term ‘cat products’.

This brings my final seed list up to 5,426 keywords.

Don’t be overwhelmed. During the the next phase of our keyword research process we will remove thousands of keywords from our master list by using a few simple tactics.

Evaluate Keyword Ideas

Now that we have a list containing thousands of keyword ideas, it's time to clean the data and evaluate the best and most profitable keywords for our project.

1. Remove Duplicates

The first thing we want to do with our keyword ideas list is remove any duplicates we may have generated throughout the research phase. You can do this in Excel by simply selecting the data range, clicking on the ‘Data’ tab, and selecting ‘Remove Duplicates’ from the Data Tools group.

In this example, Excel has found and removed 418 duplicates, leaving a master list of 5,009 keywords.

Before moving onto the next step, we also want to remove any special characters as Keyword Planner doesn’t process results for these.

2. Collect Competitive Data

My go to tool for this is Term Explorer. Whilst not perfect (and I'll come to why in a minute), the tool allows us to collect some very important competitive data, including:

  • top 10 ranking URLs for each keyword
  • keyword use in page title, URL, domain, and in content
  • number of links to the domain
  • number of links to the ranking page
  • link strength scores
  • relevancy scores
  • difficulty scores

The first thing I do when I only have a small seed list is run a 'Bulk Keyword' job. You can generate up to 1,000 keyword ideas using the Free Account. Or, from 10,000 to 90,000 keyword ideas on Paid Plans. 

As we already have a seed list of 5,000 relevant keywords (having generated these ideas from other tools), I can jump straight into using the 'Keyword Analyser' tool. 


You will need a Paid Plan to use this tool. However, if you value good data, I would urge you to invest as this is where the truly valuable data comes in. 

Simply name your project, add your seed keywords to analyse (I have added 10 for the sake of this example), select a country and start your project. Once the Keyword Analyser report finishes, I am presented with a screen like this:

In order to see all of the competitive data that I want, I downloaded CSV, and then uploaded it to Google Drive so that I can use the data in Google Sheets.

One thing you may notice at this point is that the Average Monthly Searches data is wrong. However, the rest of the data is incredibly valuable. As you can see, the tool displays all of the URLs ranking on the first page for each keyword, along with all of their keyword and link data. 

Now I need to collect accurate search volume data from elsewhere to complete my data. In comes Google Sheets and SEMRush API.

3. Get Estimated Monthly Search Volume

For this step I deleted all of the data in the Average Monthly Searches column. I then added a column to the left (column C), which is used to input the SEMRush API import script. For those who need the syntax for importing this data, I have shared this below:

=transpose(importdata("<a href="http://api.semrush.com/?type=phrase_this&key=4bc6a69cfb25d50f5e349b9236c339f6&export_columns=Nq&phrase=%22&B2&%22&database=uk">http://api.semrush.com/?type=phrase_this&key=YourSEMRushAPIKey&export_columns=Nq&phrase="&C2&"&database=uk</a>"))

You will need to replace the section 'YourSEMRushAPIKEY' with your own API Key, and then you're ready to go - pasting this into cell C2 and dragging this down to the bottom of your list.

Once all of the rows have 'Loaded', the keyword data will have populated the Average Monthly Search column (D). In order to lock all of that data in, I simply copy and paste special values > values only.

Now that I have all the data I need, it's onto the next step.

4. Refine Keyword List

I instantly removed any keyword on the list with an estimated monthly search volume less than 100. I then browsed through the remaining terms to make sure there weren’t any irrelevant terms that might have been picked up during the keyword generation phases. If found, these were also removed. This included anything to do with ‘dogs’ or other animals which are not going to be the focus of a cat product site (duh!).

This brings my final list to 529, with a great selection of long-tail keywords.

5. Define User Intent

The next step, and arguably the most important, is to group the keywords into “buckets” according to their user intent. The 4 buckets I use are:

  • Navigational
  • Informational
  • Commercial Investigation
  • Transactional

This process can be done manually, where you go through each keyword and tag them appropriately. Or, you can automate the whole process by using an Excel macro. If you don’t know any Visual Basics, or don’t particularly want to learn, then you can buy Nick Eubanks ready-made search intent macro-enabled Excel template. Instructions on how to use it here.

With a few tweaks to the tagging criteria, you can auto populate your keyword research spreadsheet with the searchers inferred intent. Now we can start understanding the semantics behind our list of keywords, which will aid our analysis in choosing our target keywords and mapping these to different content targets in the purchase cycle.

You’ll end up with something that looks like this (which can then be combined with our keywords research and competitive data Google Sheet):


Or, you can use a tool called Sentinel, also built by Nick Eubanks and his team. One of the best things about this tool is that it's free! The tool allows you to automatically tag for intent based on your specific keyword vertical and the modifiers therein, and then automatically prioritise which keywords you should be going after based on rank potential.

You may find that some of your terms could apply to multiple categories. In this case, you need to fully understand how your product and service offering will affect your tagging, as well as the general semantic nature of the term, before assigning user intent.

Due to the nature of a comparison / review site, I am particularly interested in terms which have been tagged in the ‘Commercial Investigation’ intent group, and even the ‘Informational’ group. These happens to be the majority of my terms.

You can further segment your list into categories that represents the “funnel” stage the keyword belongs to. For example, brand terms may be attributed to “lower funnel” and research or “buy” related terms to “upper funnel”. For now, I am simply going to further segment my list into ‘micro-categories’, so that I can identify any potential holes in my keyword research from a product category perspective. As a product comparison site, this is going to make up the bulk of my site architecture.

5. Assign Priority Scores

Analysing your keywords for the highest growth potential and maximum impact is naturally a pretty manual process. It requires human input to review keywords according to certain criteria, and from here we can assign an aggregate priority score, effectively choosing our “Best” keywords. The criteria we are going to evaluate are:

  • Search Volume
  • Competition
  • Opportunity

Search Volume

For this criteria we are simply going to use the data collected from our SEMRush API call. As experienced SEO’s, we all know that this can be inaccurate to say the least (for example, Ahrefs will give you completely different data), however it gives you a good comparison of search volumes between different keywords.


While assessing competition, we need to estimate how difficult it will be to rank each page of our site for a given keyword. Fortunately, I have already collected some valuable competitor data, including difficulty scores, link strength scores, and more.

The best way to analyse this data is by using conditional formatting to help highlight potential opportunities. Green for low difficulty and link scores, and red for high. 

This will help to paint a very clear visual picture as to which keywords are worth going for based on organic competition, ready for you to start mapping keywords to site pages.


The process of evaluating keywords is both an art and a science. There are a lot of different ways you can evaluate your keywords for SEO opportunity, with most SEO’s having their own process that works for them. My keyword research process is one that I have refined over time, but also one which is tweaked depending on the nature of the business / topic I am analysing. The idea behind this process is to collect as much data as possible, in order to start making informed decisions about your keyword targeting and topical optimisation efforts.

I hope you have found the article useful. Please do share your thoughts in the comments, as well as any tips you might have for improving the process.

comments powered by Disqus