You are here: Home / NewsItems / Bernie phone dailer website using bad robots.txt

Bernie phone dailer website using bad robots.txt

by Acetolyne published Mar 22, 2016 09:06 PM, last modified Mar 22, 2016 09:06 PM
Bernie phone dailer website using bad robots.txt

Recently I saw on the 2600 group on facebook an article showing that Bernie Sanders made a website where you can get a free phone number for the day. It reads "Bernie has an open phone bank to allow people to help call them to get them to vote for them! " after investigating I decided to check the robots.txt file to see if it was being indexed by search engines and well it appears it was supposed to be blocked but someone forgot to uncomment the lines so their robot rules are being ignored! Here is the robots.txt file I am speaking of as they may fix it in the future.

From https://bernie-phonebank.herokuapp.com/robots.txt as of 3/22/2016

# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /