How to Send Bulk 410 Headers and 301 Redirects in NGINX
Google’s Webmaster Tools warns you when there are a lot of 404 Not Found pages found on your website. If you’ve just deleted a lot of spam from a site, Google might keep recrawling those URLs as long as they send 404 headers.
- A
404
header means Not Found. - A
410
header means Gone.
If you want to stop Google from adding those removed pages back to Google Webmaster Tools, you can try sending 410
headers for the removed URLs.
It took me a while to figure out how to send bulk 410
headers with NGINX, but I finally came up with this solution:
Create a new file called /etc/nginx/header-maps.conf
. In that file map each gone URL to a variable, in this case the number 1 (chosen arbitrarily).
map $request_uri $gone_var {
/path-one 1;
/path-two 1;
/path-three 1;
/path-four 1;
}
Then in your main conf file for your domain, like /etc/nginx/sites-available/example.com
(replacing example.com
with your domain):
include header-include.conf;
server {
# other things here...
if ($gone_var) {
return 410 $gone_var;
}
# other things here...
}
Bulk 301
redirects in NGINX
You can also set up bulk 301
redirects in the same file, for example:
# header-include.conf
# These are for the 410 headers.
map $request_uri $gone_var {
/path-one 1;
/path-two 1;
/path-three 1;
/path-four 1;
}
# These are for the 301 redirects.
map $request_uri $redirect_uri {
/path-five /path-five.html;
/path-six /path-six.html;
/path-seven /path-seven.html;
}
Here’s the NGINX conf file that tells NGINX which headers to send:
# Your domain's conf file
include header-include.conf;
server {
# other things here...
# if there is a mapped $gone_var set for the current path
if ($gone_var) {
return 410 $gone_var;
}
# if there is a $redirect_uri set for the current URI:
if ($redirect_uri) {
return 301 $redirect_uri;
}
# other things here...
}
map_hash_bucket_size
Errors
If you get an error about map_hash_bucket_size
being to small, open the /etc/nginx/nginx.conf
file (or wherever the http
block lives) and increase it something like this: map_hash_bucket_size 256
Read More
Here are some pages that might be useful for sending bulk status codes with NGINX:
- Module ngx_http_map_module
- Module ngx_http_rewrite_module
- NGINX redirects
- How to Use NGINX’s map Module on Ubuntu 16.04
I don’t know if it this technique will affect performance on a huge site, but it worked fine with hundreds of URLs for a site with thousands of visits per day on a relatively small server.
If it doesn’t work for you or if you have any suggestions for improving this idea, please comment below.