Smart FAQ SEO

Description

Since Smart FAQ is entirely generated in Javascript, it will not be optimally referenced by search engines, such as Google in particular. For this reason, we use the most efficient method to date, namely that of Dynamic Rendering which consists in "serving" static HTML templates equivalent to the display made for the Internet user, which we will have previously generated , and which will be delivered exclusively to search engines for optimal referencing.

Implementation

Apache

Apache: Step 1

In the httpd.conf file you have to check that the modules are activated

• LoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so
• LoadModule rewrite_module lib/httpd/modules/mod_rewrite.so

Apache: Step 2

Integrate the following rules into APACHE:

### Activate the rewrite rule engine
RewriteEngine On

### If the request comes from a bot, set a flag to proxy the request to SEO4Ajax
RewriteCond %{ENV:PROXY} !true
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_URI} ^/DEFINE-URI
RewriteCond %{HTTP_USER_AGENT} (google|bot|spider|pinterest|crawler|archiver|flipboardproxy|mediapartners|facebookexternalhit|insights|quora|whatsapp|slurp) [NC,OR]
RewriteCond %{HTTP:from} .+
RewriteRule .* - [E=PROXY:true,E=REQUEST_PATH:%{REQUEST_URI}]

### Proxify the request to SEO4Ajax
RewriteCond %{ENV:PROXY} true
RewriteRule ^(.*)$ http://api.seo4ajax.com/DEFINE-KEY%{ENV:REQUEST_PATH} [P,QSA,L]

### If request sitemap file, set a flag to proxy the request to SEO4Ajax
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_URI} ^/sitemap_seo4ajax.xml
RewriteRule .* - [E=PROXY_SITEMAP:true]

### Proxify the request to SEO4Ajax
RewriteCond %{ENV:PROXY_SITEMAP} true
RewriteRule ^(.*)$ http://api.seo4ajax.com/DEFINE-KEY/sitemap.xml [P,QSA,L]

Apache: Step 3

Replace the terms DEFINE-KEY and DEFINE-URI with the elements below.

Variables	Description
DEFINE-KEY	This is data that Smart Tribune provides to you. It is the token allowing to authorize calls to SEO4AJAX.
DEFINE-URI	Customer site FAQ page URI.

Apache: Example

For the FAQ url http://example.com/en/faq, I have to change into the snippet above ^/DEFINE-URI to ^/en/faq

RewriteCond %{REQUEST_URI} ^/DEFINE-URI

will become :

RewriteCond %{REQUEST_URI} ^/en/faq

Nginx

Nginx: Step 1

Integrate the following rules under NGINX:

location /DEFINE-URI {
    ### If the files exists on the server, serve it, otherwise execute the @s4a_analyse location
    try_files $uri @s4a_analyse;
}

### This location determines if a request comes from bots
location @s4a_analyse {

    ### If the request comes from a bot, proxy the request through /s4a_proxy location
    if ($http_user_agent ~* (google|bot|spider|pinterest|crawler|archiver|flipboardproxy|mediapartners|facebookexternalhit|insights|quora|whatsapp|slurp)) {
        rewrite ^(.*)$ /s4a_proxy last;
    }

    if ($http_from ~* .+) {
        rewrite ^(.*)$ /s4a_proxy last;
    }
}

### This location proxy requests coming from bots to SEO4Ajax
### You can update the resolver directive with your own DNS provider if needed
location /s4a_proxy {
    set $s4a_domain 'https://api.seo4ajax.com/DEFINE-KEY';
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    resolver 8.8.8.8 8.8.4.4;
    proxy_pass $s4a_domain$request_uri;
}

### Following rules are used to provide FAQ dedicated sitemap
location /sitemap_seo4ajax.xml {
    rewrite ^(.*)$ /s4a_proxy_sitemap last;
}

location /s4a_proxy_sitemap {
    set $s4a_domain 'https://api.seo4ajax.com/DEFINE-KEY';
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    resolver 8.8.8.8 8.8.4.4;
    proxy_pass $s4a_domain/sitemap.xml;
}

Nginx: Step 2

• Replace the terms DEFINE-KEY and DEFINE-URI with the elements below.
• These rules need to be adapted according to your usage in order to serve the FAQ page of your application in case the request does not come from a bot.

Variables	Description
DEFINE-KEY	This is data that Smart Tribune provides to you. It is the token allowing to authorize calls to SEO4AJAX.
DEFINE-URI	Customer site FAQ page URI.

Nginx: Example

For the FAQ url http://example.com/en/faq, I have to change into the snippet above /DEFINE-URI to /en/faq

location /DEFINE-URI

will become :

location /en/faq

If you have multiple location for your faq :

location ~ (/en/faq|/us/faq)

info

IIS server: rules available for IIS Server on request

Validation

Method 1

In order to check the implementation of the rules and at the same time speed up the referencing of your new FAQ, you can use the Google Search Console (GSC) tool to check that the content of static HTML templates is correctly delivered to search engines and if necessary force the first indexing manually.

Method 2

To check the correct implementation, you need to check the behavior of the page as a search engine bot, so to simply summarize as GoogleBot.

Procedure A

1. Install a Chrome extension provided for this purpose

2. Click on Googlebot (Google's spider)

3. Go to the home page of the FAQ (or refresh it if you are already on it)

4. Right click > Show page source code

5. Search the tag <div id="st-faq"></div>

   • If you do not have content, then it is highly possible that the implementation of the rules is not good, you must check your server configuration (proxy_http for APACHE, blocking at the level of your CDN, robots.txt, firewall, ...) and repeat the procedure.
   • If you have content inside the tag, then the snapshots are there and the bots are redirected to the page in static HTML. The procedure ends here but we advise you to use procedure B (below) for further analysis.

Procedure B

1. Install CURL on your computer if this is not the case (How to install CURL)

2. Launch the following command from your CMD/Terminal:

   curl -L -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://example.com/en/faq

3. Search the tag <div id="st-faq"></div>

   • If you do not have content, then it is highly possible that the implementation of the rules is not good, you must check your server configuration (proxy_http for APACHE, blocking at the level of your CDN, robots.txt, firewall, ...) and repeat the procedure.
   • If you have content inside the tag, then the snapshots are there and the bots are redirected to the page in static HTML.

4. To confirm that you have a SEO4AJAX snapshot, launch the following command line:

   curl -I -A "Googlebot" https://example.com/en/faq

5. The result should return the page header and a line called Vary containing X-S4a-Debug

   • This is the case, the procedure ends here, everything is OK.
   • This is not the case and you have to check your configuration because it is possible that you have a CDN which takes care of this or a Server Side Rendering tool which is prioritized compared to SEO4AJAX. You need to investigate and start the procedure again.

Force indexing of content manually

Go to Google Search Console (GSC) then select your site (if it does not exist, please add it following the procedure described).

Go to the Exploration section then "Explore as google" (Fetch as Google) and enter the url of your FAQ then launch the exploration. If you observe the HTML content of the FAQ (categories, questions between the div tag having the identifier st-faq - <div id="st-faq"></div>), you can "Send for indexing" by clicking on the button provided for this purpose.

It is sometimes necessary to specify to Google how to behave with the parameters (query string) used by the FAQ, for that you just have to edit the settings in the section "Old tools and Reports" then "URL parameters" then for the following parameters:

• question
• category

Configure as follows:

• Does this parameter modify the content of the page displayed by the Internet user? => Yes
• How does this setting affect the content of the page? => Restriction
• Which of the URLs containing this parameter should be crawled by Googlebot? => All URLs