They provide a speedy, accurate and personal service. I am very impressed with the knowledge and the results achieved.
Mandi Cooper, FastBuy Properties
Results Focused
We've recently started using Big Surf for PPC and have been hugely impressed with the amount of research, knowledge, and time that's been invested in creating our campaign.
Caledonian Driver Training
Wealth of Knowledge
Calum at Big Surf Digital was available at short notice to deliver a comprehensive training session on SEO. We wouldn't hesitate in recommending his services.
H. Miller
Skilled and Professional
I needed some advice on marketing my website through google ads and Calum put me in the picture. He gave me loads of his time, even though he is obviously a very busy man. He is skilled and professional at what he does.
Stuart, The Academy of Professional Celebrants
Highly Recommend
Since Big Surf Digital have optimised my site, my rankings have increased considerably for my keywords and are continuing to do so. The impact on my business has been noticeable. I would have absolutely no hesitation in recommending them to prospective clients.
Rachael Roberts
How to get your website indexed by Google?
Indexing is a crucial step in getting your site ranked in Google. Getting a page indexed can be simple but there's also a lot of issues that come up once you
get further into the subject, especially for sites with lots of pages.
Indexing is the first crucial step for any new page when it comes to getting traffic from Google.
To understand indexing you need a basic understanding of how Google sources and ranks webpages. When a website publishes a page it doesn't automatically
show up in Google. First the page needs to be found and crawled by one of Google's spidering robots, or programs. This program called Googlebot is constantly scouring the web, following links to different pages, collecting these pages and then bringing them back to add to Google's vast index of the web.
Then when a user carries out a search Google's algorithm will search its index for suitable pages and then rank them in order of which is best, or most suitable.
So with that in mind, once a website publishes a page it's crucial that it gets indexed by Google if it's going to start ranking in the search results and bringing in traffic.
How to get your website indexed by Google?
Getting pages indexed is normally a simple task.
For established sites getting a new page indexed isn't even a task. Google crawls sites regularly and when they come across new pages they index them accordingly. The more clout a site has the quicker
it will get new pages indexed. What I mean by this is that all sites have a history with Google. If a site is well established with a long history and lots of links from other sites then it will have a good PageRank score with Google. If this is
the case then the site will be crawled regularly for new content. For newer sites Google's crawlers may not see the need to return to the site as frequently. Googlebot has a finite amount of resources so will spend more time
on the sites that have more content. So it may take a while longer to get pages indexed but as long as there's nothing holding the site back the pages should still be crawled and indexed.
How do you get a page indexed quickly?
Check your robots and submit for indexing.
There are times when pages need to be indexed as soon as possible. In these cases it's important to check that the robots on your site are set to allow Googlebot to access the page. This is covered in the next step.
Once you've checked that the page is accessible go into Google Search Console and follow these steps.
1. Paste the URL of the page into the search box at the top. 2. Click request indexing.
Robots.txt and Noindex Meta Tags
Granting access for Googlebot or blocking it.
There are a few different ways to block Google or allow access. In most cases when you publish a webpage you're going to want to get it indexed but there's all sorts of reasons
why you might not want to.
Possible reasons to block Googlebot from indexing your pages
New pages still in progress.
Test versions of landing pages for ads.
Internal documents for staff use.
Pages that serve no purpose for users.
Robots.txt file
A robots.txt file is used to tell Googlebot which folders it's not allowed to access. It is a suggestion rather than a hard rule and folders excluded from indexing can still end up
in Google's index if Googlebot finds the URL from another link. A robots file is added by creating a .TXT file and adding the following syntax to the root folder on the server:
User-Agent: *
Disallow:
You can use a robots file to disallow URLs or folders by including them like so:
User-Agent: Googlebot
Disallow:/images/
Disallow:/admin/
Disallow:/staff-files/
To create the file open up notepad and type in the above code. Then upload to the root folder of your server using an FTP program or through your hosting.
You can set the file to disallow all crawlers with * or specific crawlers like Googlebot or Bingbot.
Noindex Meta Tags
There is the option of setting the robots directive via a Meta tag in the source code of the page. When you set up a WordPress site you have the option of blocking the entire website from Google by adding a Noindex Meta tag.
There is no real benefit to using a tag over a file or vice versa other than one may be easier to implement than the other. A search engine's crawler will read and follow them accordingly.
Here is an example of a Noindex Meta tag <meta name="robots" content="noindex">
This should be placed in the head section of a websites code.
If you are only looking to block Google's crawlers you would add this: <meta name="googlebot" content="noindex">
Canonicals
Consolidate page strength using canonicals.
There are times online when one page of content can end up being displayed on multiple different URLs but on the same site. This happens for all sorts of reasons like pages being assigned
to multiple categories or different product options having their own URLs.
A product URL like this, https://www.shopping.com/black-shoes/
could end up looking like this. https://www.shopping.com/black-shoes/size-8 https://www.shopping.com/black-shoes/size-9 https://www.shopping.com/black-shoes/size-10
This is the same content, but split over these different URLs.
This isn't a problem for the websites users but it could be a problem for search engines. If you've been reading the rest of this site you'll know that links are very important for ranking in search engines.
The original black shoes URL will have more link strength than the other URLs as it will be linked from a menu or category page within the sites navigational structure.
If someone shares one of the other versions of the URL with the size added that URL could end up picking up links from other sites and gaining ranking strength. But we don't want the other pages receiving the strength. We want the
main page to receive it. So what do we do? We add a canonical tag to the original page which will show in the source code of the original page and the different size URL versions. This canonical tag will tell
search engines that any link strength that the other URLs have gained should be consolidated into the one original URL.
The Canonical Tag
Well aware of the issues caused by duplication the major search engines decided to do something about it. In 2009 Bing, Yahoo and Google collectively introduced the canonical tag. The canonical tag is a Meta
tag which allows a website to tell a search engine which version of a URL is the preferred one. It is also sometimes referred to as the Rel Canonical tag.
Let's say you have two of the URLs from before: https://www.shopping.com/black-shoes/ https://www.shopping.com/black-shoes/size-8 https://www.shopping.com/black-shoes/size-9
If you would like to tell search engines that the /black-shoes/ version is the preferred one you would add a Canonical tag to the code of that page. The canonical tag will also show up within the code of
the other versions of the URLs. When a search engine bot crawls all of these URLs it will be able to easily identify which one is the preferred version.
To make the canonical tag work you need to add the following to the head code of a page: rel="canonical" .
The full URL for the canonical tag of a page is : <link rel="canonical" href="your URL" />
A real world example of how canonicals can make or break your rankings
At one point this site didn't redirect and Google's algorithm decided that the http version of Big Surf Digital was the best version for it to list in its search results. The problem being that
the http doesn't have nearly the same amount of link strength as the https version. When Google started ranking the http version this sites rankings dropped dramatically. To fix it a canonical tag was added declaring to Google that the https version was the preferred option and the only
one it should rank. If there's a duplication issue or wrong version of a page ranking the canonical tag is a great tool to have.
Htaccess 301 redirects
301 server redirects are a hugely useful tool for handling changing URLs and site re-designs.
Do not underestimate the 301 redirect. It has saved many a websites traffic from disaster and its absence has caused ripples in many ranking and traffic reports.
To understand what makes redirects so important you need to understand how Google ranks pages. Google's algorithm works out which content is best based on numerous factors including content, keywords
and most importantly links. A page with lots of reputable links has a much higher chance of outranking a page with no links. It's also important to note that websites don't really rank in Google but
web pages do. When a page ranks well in search results the search engine has a lot of data about that page, all tied to the URL. A URL is the sign post for that page and its content. If you change that URL
you effectively create a brand new page in the eyes of search engines. Even if you take an existing page on a site and change one letter of the URL, it is now seen as a completely new page without any of the metrics or history that the original page had.
If you have a page that ranks in Google but then you change the URL Google will attempt to crawl the old URL but find nothing. Then that page will drop out of the search results like a stone.
Now consider if you were to change all of the URLs on your site like you would during a website redesign. This would lead to every page (other than the home page) dropping out of Google only to be replaced by all of the brand new URLs on your site, which have no ranking strength, links or authority.
This will make search engine traffic to the site nosedive. It's something that I have seen happen many times over the years and in fact I've picked up new clients who have hired me to try fix and the problem and reverse the traffic drop.
The solution to this problem is to add a server redirect that directs traffic from the old pages to the new. Doing this will transfer the ranking strength built up in each old URL to the new versions. Your rankings and traffic from Google will have a much better chance of being retained. There's still optimising all the new content to consider but that's
a different story.
Types of redirects:
There are a number of different types of redirects but the main ones you need to know for SEO are 301 redirects and 302 redirects.
301 permanent redirect
The 301 redirect is the standard for permanently redirecting URLs. It tells search engines to transfer all strength associated with the original URL to the new version.
It is added to a server .htaccess file on an apache server like so:
Redirect 301 /old-url/ https://www.bigsurfdigital.co.uk/new-url/
302 temporary redirect
A 302 is similar to a 301 but indicates that a page has been moved only temporarily. I have never had to use this in my career. In fact the only time I ever see it is when it has been used
incorrectly in place of a 301.
To add a 302 to an apache server add:
Redirect 302 /old-url/ https://www.bigsurfdigital.co.uk/test-url/
If your site is running WordPress there are a good few plugins that you can use to create redirects. A good one is Quick 301 Redirects.
Google Search Console
Submitting your URLs to Google
Google Search Console is a free service that Google provides to help website owners to manage and improve their sites presence in search results.
Search Console allows users to verify their site and gain insights into which search terms each page is ranking for in search results.
Google Search Console allows you to do the following:
Submit an XML sitemap for indexing
Submit individual URLs for indexing
View data on indexed URLs
View the search terms that each page is ranking for
View data on clicks, impressions and average ranking positions
Disavow links to your site
Remove URLs from search
Conduct crawl tests on URLs
View page enhancements showing in search
View data on Core Web Vitals, Page Experience and HTTPS
View Manual Actions and Security Issues
View External Links and Internal Links
An XML sitemap is a list of URLs on a website that can be submitted to a search engine for crawling. It's a good way of providing a map to all of your most important pages to
try and get as many of them indexed as possible. You don't need an XML sitemap to get pages indexed though. Search engine crawlers can also find your content through links in your navigation structure
or from external sites. With large sites especially XML sitemaps can be very useful.
There are lots of sites that can create a sitemap for you. Visit www.xml-sitemaps.com to generate a sitemap then download it. Then visit
Search Console and submit your sitemap.
Diagnosing indexing issues
How to find and fix indexing problems.
Indexing issues come up all the time when working in SEO. There are small indexing issues which are easily fixed and huge clangers that can cause major headaches. Usually the first
time I'm alerted to a page not being indexed is when I check its ranking. If it doesn't show up within the top 10 pages in Google then there might be a problem.
The site: command
The site command is really useful and I use it nearly every day. Type it into Google and it will tell you if a URL is indexed, or if it's the root of the domain you type in, will display a list
of all the URLs from the site that are currently indexed.
To use it go to Google.com or Bing.com and type in site:www.domainname.com
If the URL doesn't show up it's not indexed. Then the next step is to figure out why not?
The first thing to look at if a page isn't indexed is the source code. Have a look and see if there is a Meta Noindex tag which could stop a search engine from indexing the page.
After checking that have a look at the Robots.txt file on the server by going to www.domainname.com/robots.txt and see if the particular URL you're looking at is being blocked to crawlers.
The result from a site: command used to display a listing in Google with a link to the cache of the page. This has recently been removed which is a shame as it would show if the page
was indexed and on which date along with a copy of the indexed page.
Search Console
You can type a URL into the search bar at the top of Search Console and it will tell you if it has been indexed or not. If not it will give you possible reasons such as whether the URL
was found in a Sitemap, whether it has been indexed in the past and if it has a Canonical assigned to it.
Google selected canonical
If you paste in a URL into the top search box Search Console will tell you if it has found a Canonical assigned to your URL. It will also tell you if Google has selected a separate
Canonical for your page. If they differ you could have a problem. Google may have chosen a different URL as the canonical version because it has more links, it's been indexed longer or a number of different reasons.
Crawling tools
A web crawling tool is a great piece of software which will go through a website collecting data about each URL, much like a search engine crawler would. I use Screaming Frog SEO Spider.
There is both a free and a paid version. With this tool you can set it to crawl a domain where it will start at the root domain and follow every link it finds there. It will show which URLs have redirects and whether or not pages are crawlable and indexable
by search engines. It does a lot more than this but that's all that's needed for this issue.
Server codes
The crawl may tell you that the URL you're looking at is returning a 404, 301 or 302 code. This tells you a lot. 404 server code: This means that the page has not been found. So in this case you would need to look into why the page isn't displaying. 301 server code: This means that the page has been redirected permanently. 302 server code: This means that the page has been redirected temporarily.
The crawl might also show that there are a number of different URLS that are very similar to the original. The tool will show you if there is a Canonical Tag set for the URL and if it is set correctly.
The WayBack Machine
This is technically not a SEO tool but it is very useful. The WayBack Machine is an archive of the Internet. It has crawled popular sites for years and built
up a huge database of old and current websites. In some cases it's possible to use this site to see what a particular URL or URL structure looked like at a particular point in time, which
can be useful for diagnosing indexing issues.
Why isn't your URL indexed?
If you've checked a URL and it's not indexed what could the problem be? If you've only recently submitted it to Google you may just need to wait a little longer. It can take time. If it's been
over a week then check all the things that could be blocking a crawler. Check your Robots.txt, Meta tags and run a crawler check to view the server codes to check for any issues.
If the page still isn't indexed then perhaps there's another canonical being chosen by Google. If so Search Console will show you. If not the issue could be to do with the content of the page
or incoming links. Insufficient content could stop a page being indexed. As could paid links or links from spam sites
