Home / User Guide / Configuration

Configuration

 

URL Checking

Open the Configuration -> URL Checking screen.

Max Redirects To Follow - the HEADMasterSEO redirect checker can follow up to 9 redirections per imported URL. The default value is 9.

The program will ignore this setting and use the max value of 9 when running URL mapping and http <-> https and www <-> non-www redirect tests.

Response Timeout - the amount of time in seconds that HEADMasterSEO waits before timing out an URL.

On Windows this timeout setting is applied separately to every step in the fetching of URL responses (connecting, sending a request, getting a response..). If you set it to 10 seconds, it would mean 10 seconds to send a request, then 10 seconds to connect etc. For domain name resolution on Windows, the program uses the default Windows timeout.

On Mac OS the option applies to all the steps in the URL checking process combined. If you set it to 10 seconds, it would mean that if a response is not received in 10 seconds, then the URL is timed out.

Retry Timed Out URLs - the number of times to retry URLs that time out, cannot DNS resolve or have a connection error.

Retry 5XX Server Errors - the number of times to retry URLs that throw server errors (status codes 500 to 599).

 

Rewrite URLs

HEADMasterSEO can rewrite every URL from your list before checking it.

Open the Configuration -> URL Rewriting screen and check the "Rewrite URLs Before Checking Them" checkbox.

Enter your URL rewriting rules in the big textbox below.

Enter one rule per line. Rules have a simple format: searchstring replacestring

If HEADMasterSEO finds searchstring in a link, it will replace the first occurrence of searchstring with replacestring.

If multiple rules apply to a single URL, only the first (by order) rule will get executed.

Example: to rewrite all non-www urls to www urls for mydomain.com, use the following rule (it will work for both http and https links):

mydomain.com www.mydomain.com

To rewrite mydomain.com urls to a staging environment that is in a subfolder on a different server:

mydomain.com/ stagingserver.com/folder/

To rewrite only http non-www urls to the https www variants:

http://mydomain https://www.mydomain

 

How to Change The Request User Agent

HEADMasterSEO can bulk check URL response headers with any User Agent you want to use.

To set the Request User Agent, go to the Configuration -> User Agents screen.

HEADMasterSEO comes with 20 predefined user agents (all major search engine bots and browsers).

The default User Agent is HEADMasterSEO.

 

HEAD or GET - Set Request Methods

HEADMasterSEO supports both HEAD and GET request methods to check URLs. The default mode is HEAD requests because when you want to examine only the http headers, GET requests are wasteful.

A HEAD request will only ask for the response headers. A GET request fetches both the response headers and the content. HEAD requests save bandwidth (server costs) and can check URLs faster.

Unfortunately, some servers don't allow HEAD requests and others are misconfigured and buggy and return incorrect headers when serving HEAD responses.

If the server does not allow HEAD requests, you will get a server status code 405 (Method Not Allowed) and you'll need to switch to GET requests.

Open the Configuration -> HEAD/GET Method Config screen.

You have 3 options:

• Always Use HEAD Requests: best setting if you are are checking links on servers that support HEAD requests.
• Always Use GET Requests: this is a conservative setting. Use it only if you are working on servers you know will not allow HEAD requests.
• Use GET Requests Only For the Following Domains: with this setting, you can add all domains that have problems with HEAD requests to a list and HEADMasterSEO will use GET requests for these problematic domains.

Enter one domain per line for the list of domains.

 

URL Checking Speed

HEADMasterSEO can use from 1 to 200 asynchronous threads to check URL server response headers. Each thread sends one asynchronous request and waits for the response before sending another request.

To increase speed for fast servers, increase the amount of threads. To limit URL checking speed (slow servers), lower the number of threads and use the "Limit URLs/s" option.

The optimal number of threads depends on your connection and the speed of the servers you are checking.

Going above the optimal number of threads will not make the URL checking faster. It may overload the servers and result in response timeouts.

The default setting is: 30 threads.

Check out the actual crawling speed in the status bar.

If the speed (responses per second) is significantly lower than the number of threads, decrease them next time you are checking the same URL list.

If the speed is about the same as the number of threads, don't change this setting.

If the speed is higher than the number of threads, increase this setting next time. We've seen speeds of about 1000 urls per second at the maximum 200 thread level (for very fast servers).

With very slow servers, use the "Limit URLs/s" option to cap the amount of request per second.

 

Custom HTTP Request Options

The "Allow Cookies" check makes it possible to accept and resend cookies with subsequent requests.

You can completely configure HTTP requests by adding custom headers of your choice and setting the "Add Custom Headers To Every Request" check and specifying your custom headers in the textbox below.

Enter one request header per line.

Header fields are in the format "Header: Value"

In the above screenshot the header is Accept-Language and the value is "da, en-gb;q=0.8, en;q=0.7"

Note: to configure the User Agent, use the User Agents Configuration Screen.

 

Proxy Settings

HEADMasterSEO can use proxies and supports password protected proxy servers.

Check the "Use Proxy Settings" and enter the proxy address and port in the Proxy configuration screen.

Leave the user and password fields empty, if the proxy is not password protected.