crawlercontrol.com

Get an email when its DNS, website or certificate changes.

crawlercontrol.com serves a website of its own over HTTPS. It answers on both IPv4 and IPv6, with a valid TLS certificate and MX records for mail.

4,330 other domains share an IP address with it.

The most recent crawl returned the page title “CrawlerControl — robots.txt generator and checker for AI bots” and 70,972 bytes of HTML.

Domain properties

Property	Value	Explanation
Top-level domain	.com	Statistics for all domains we know under this TLD.
Has website	yes	A website is hosted under the domain, which does not redirect to an external domain.
Has MX-record	yes	The domain has an MX record.
External redirect	no	The domain redirects to another domain.
Redirect to https	yes	The domain redirects to https.
Redirect to www.	no	The domain redirects to www.domain.
Only with www.	no	The domain can only be reached with the subdomain "www".
Only DNS, no website	no	There are no A or AAAA records in DNS.
IP-Address	Both	Shows whether the domain only has IPv4 or IPv6 or both.
SSL-Certificate	valid	Shows whether the domain has a valid SSL certificate.
Redirects	0	Number of domains that redirect to this domain.
Reverse IP	4,330	Number of domains with the same IP(s).

This data through the API

The data on this page is also available as JSON: /lookup for the key facts, /reverseip, /redirect and /similar for the lists, and /dnshistory with Basic and Premium. A free account comes with an API key and 100 calls a day.

curl -X POST https://api.webxtrend.com/v1.0/lookup \
  -H 'Content-Type: application/json' \
  -d '{"token": "YOUR_TOKEN", "query": "crawlercontrol.com"}'

Create a free account Read the API docs

Similar websites

Website	Similarity
sivd.dev AI Crawler Checker: Test GPTBot, ClaudeBot & robots.txt Free AI crawler checker. See whether GPTBot, ClaudeBot, PerplexityBot and other AI bots can access your site: robots.txt rules, Cloudflare managed robots.txt...	89 %
crawlcheck.dev AI Crawler Audit & robots.txt Generator — CrawlCheck Free robots.txt auditor for 60+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider…). See which AI engines can see your site, generate a clean policy...	89 %
aibotaccess.com AI Bot Access Checker — Is ChatGPT, Claude & Perplexity Reading Your Site? (Free) Free tool that checks if your robots.txt is blocking AI crawlers like GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Scan any domain in 5 seconds — no...	88 %
whataisee.com Is your website ready for AI? Free AI visibility checker See what ChatGPT, Claude and Perplexity can read on your site. Free AI crawler test: robots.txt, GPTBot, llms.txt, JSON-LD. Score and fixes in 10 seconds.	87 %
aigcrobots.com AI Robots.txt Generator \| Manage GPTBot, ClaudeBot, Google-Extended and AI Crawlers Draft and check robots.txt rules for GPTBot, ClaudeBot, Google-Extended, search crawlers, and private site paths.	87 %
aicrawlerchecker.com AI Crawler Checker - Detect GPTBot & ClaudeBot Access Detect AI crawler access blockers by checking robots.txt, metadata, sitemap, llms.txt, and fetcher context for GPTBot, OAI-SearchBot, OAI-AdsBot, ClaudeBot,...	86 %
aibotaccesschecker.com Misha Manko AI Bot Access Checker Free WordPress plugin that shows which AI bots can reach your site - ChatGPT, Claude, Perplexity, Google AI and more - and why the others are blocked.	86 %
aicrawlercheck.com Free AI Crawler Check: Test AI Crawlability and Bot Access Free AI crawler check: test AI crawlability and bot access for 257 AI and search bots, including Googlebot. Get a 0-100 score and copy-paste fixes. No sign-up.	85 %
robots2.org robots2.txt — AI Policy for the Open Web robots2.txt extends robots.txt with AI-specific directives. Tell AI agents what they can and cannot do with your content.	85 %
resulthack.com AI crawler directory: block or allow each in robots.txt 49 AI crawler robots.txt tokens from 19 operators: what each does, whether it obeys robots.txt, how to verify it and how many top sites block it.	84 %
scanaivisibility.com AI Visibility Checker — Free Instant Scan for ChatGPT, Claude & Perplexity Free AI visibility checker: scan any website in seconds to see if ChatGPT, Claude, Perplexity and other AI search engines can crawl and cite it. Checks robot...	84 %
aibotcheck.tech aibotcheck — Is your website invisible to AI crawlers? Evidence-first AI crawler access diagnostics. Probe robots.txt, bot headers, infrastructure feeds, and calculate your AI Visibility Index.	84 %
aicrawlability.com AI Crawlability Checker — Test If AI Bots Read Your Site A practical, source-grounded guide to making your website visible and citable to AI answer engines like ChatGPT, Perplexity, and Google AI Overviews.	84 %
robotstxtlab.net AI Crawler robots.txt Generator & Checker \| Robots.txt Lab Generate crawler-specific robots.txt rules for AI training, AI search, user-triggered agents, Googlebot, Bingbot, and more, then verify the live file.	84 %
robotscheck.tools Robots.txt Checker: See Which Bots You Really Block Fetch any domain's robots.txt and see which of 24 bots are allowed or blocked, with the exact rule behind each verdict. Includes a free generator.	84 %
amicrawlable.com Am I Crawlable Am I Crawlable — trusted ai visibility information and resources.	84 %
robotstxt.tools Robots.txt Generator – Free Online Robots.txt Tool Generate a valid robots.txt file for free. Add crawler rules, Allow and Disallow paths, sitemaps and presets, validate errors, then copy or download your rob...	83 %
aeoradar.com AEO Radar: free AEO tools and AI search data Answer engine optimization (AEO) is the practice of making your content easy for AI systems like ChatGPT, Perplexity and Google AI Overviews to read, trust a...	83 %
crawlreadiness.com Crawl Readiness — Check, Fix & Monitor Your AI Search Visibility (GEO) All-in-one AI GEO toolkit (generative engine optimization): check whether ChatGPT, Claude, Perplexity & 50+ AI crawlers can read your site, generate the fixe...	83 %
uawatch.com UAWatch — Know Which AI Bots Can Reach Your Website UAWatch monitors 124 AI crawlers, search engines and SEO bots daily. Get instant email alerts when Cloudflare, WAF rules or robots.txt changes block a bot. F...	82 %

Websites whose start page offers something similar, found by comparing the text of their start pages, across languages. Near-identical copies, parking pages and error pages are left out. The full list of up to 100 comes through the API: similar websites.

Last SSL certificate

Issuer	Google Trust Services
Valid from	9/9/2026
Valid to	12/8/2026
Time until expiry	47 days

Last successful crawl

Title	CrawlerControl — robots.txt generator and checker for AI bots
Description	Generate and audit your robots.txt with a clear line between AI search bots (ChatGPT, Claude, Perplexity) and AI training bots — every rule shows its real consequence in plain language.
Content-Length	70,972 bytes

Title and description as published by the page itself in its own <title> and <meta name="description"> tags.

DNS history

IPv4 address	Last updated
104.21.96.2	7/13/2026
172.67.171.235	7/13/2026
216.239.32.21	1/5/2024
216.239.34.21	1/5/2024
216.239.36.21	1/5/2024

IPv6 address	Last updated
2606:4700:3035:0:0:0:6815:6002	7/13/2026
2606:4700:3036:0:0:0:ac43:abeb	7/13/2026
2001:4860:4802:32:0:0:0:15	1/5/2024
2001:4860:4802:34:0:0:0:15	1/5/2024
2001:4860:4802:36:0:0:0:15	1/5/2024

Mail server	Priority	Last updated
route2.mx.cloudflare.net	22	7/13/2026
route1.mx.cloudflare.net	30	7/13/2026
route3.mx.cloudflare.net	32	7/13/2026

Name server	Last updated
jonah.ns.cloudflare.com	7/13/2026
sky.ns.cloudflare.com	7/13/2026
ns-cloud-d1.googledomains.com	12/10/2023
ns-cloud-d2.googledomains.com	12/10/2023
ns-cloud-d3.googledomains.com	12/10/2023

The five most recent states of each record type. The full history comes through the API: DNS history.

Most recent redirect domains

No domains found.

Reverse IP neighbours

2606:4700:3035::6815:6002 1,083 domains · here since 9/18/2026

Domain	On this IP since
zzrlsy.cn	10/14/2026
zzdmfs.com	10/5/2026
zo0b3.buzz	9/30/2026
zimoa.fr	9/16/2026
ziloempire.net	9/13/2026
zdrowieisila.live	9/22/2026
zbhaoyang.com	9/14/2026
zbam.ru	9/22/2026
zakriya.site	9/23/2026
zajazdvictoria.pl	9/12/2026
z21.co.uk	9/16/2026
z101.app	9/27/2026
and 1,070 more

2606:4700:3036::ac43:abeb 1,083 domains · here since 9/18/2026

Domain	On this IP since
zzrlsy.cn	10/14/2026
zzdmfs.com	10/5/2026
zo0b3.buzz	9/30/2026
zimoa.fr	9/16/2026
ziloempire.net	9/13/2026
zdrowieisila.live	9/22/2026
zbhaoyang.com	9/14/2026
zbam.ru	9/22/2026
zakriya.site	9/23/2026
zajazdvictoria.pl	9/12/2026
z21.co.uk	9/16/2026
z101.app	9/27/2026
and 1,070 more

104.21.96.2 1,082 domains · here since 9/18/2026

Domain	On this IP since
zzdmfs.com	10/5/2026
zo0b3.buzz	9/30/2026
zimoa.fr	9/16/2026
ziloempire.net	9/13/2026
zdrowieisila.live	9/22/2026
zbhaoyang.com	9/14/2026
zbam.ru	9/22/2026
zakriya.site	9/23/2026
zajazdvictoria.pl	9/12/2026
z21.co.uk	9/16/2026
z-hank.com	9/12/2026
yvura.com	9/22/2026
and 1,069 more

172.67.171.235 1,082 domains · here since 9/18/2026

Domain	On this IP since
zzdmfs.com	10/5/2026
zo0b3.buzz	9/30/2026
zimoa.fr	9/16/2026
ziloempire.net	9/13/2026
zdrowieisila.live	9/22/2026
zbhaoyang.com	9/14/2026
zbam.ru	9/22/2026
zakriya.site	9/23/2026
zajazdvictoria.pl	9/12/2026
z21.co.uk	9/16/2026
z-hank.com	9/12/2026
yvura.com	9/22/2026
and 1,069 more

Sharing an address does not always mean sharing a server: on a CDN or a large hosting platform, millions of unrelated domains answer on the same few addresses. We measured how far that goes in Half the web sits on 276 IP addresses. The complete list of neighbours, also for an IP address, comes through the API: reverse IP lookup.

Dive into our data, search now!