crawlcheck.dev

Get an email when its DNS, website or certificate changes.

crawlcheck.dev serves a website of its own over HTTPS. It answers on both IPv4 and IPv6, with a valid TLS certificate and MX records for mail.

3,968 other domains share an IP address with it.

The most recent crawl returned the page title “AI Crawler Audit & robots.txt Generator — CrawlCheck” and 13,855 bytes of HTML.

Domain properties

Property	Value	Explanation
Top-level domain	.dev	Statistics for all domains we know under this TLD.
Has website	yes	A website is hosted under the domain, which does not redirect to an external domain.
Has MX-record	yes	The domain has an MX record.
External redirect	no	The domain redirects to another domain.
Redirect to https	yes	The domain redirects to https.
Redirect to www.	no	The domain redirects to www.domain.
Only with www.	no	The domain can only be reached with the subdomain "www".
Only DNS, no website	no	There are no A or AAAA records in DNS.
IP-Address	Both	Shows whether the domain only has IPv4 or IPv6 or both.
SSL-Certificate	valid	Shows whether the domain has a valid SSL certificate.
Redirects	0	Number of domains that redirect to this domain.
Reverse IP	3,968	Number of domains with the same IP(s).

This data through the API

The data on this page is also available as JSON: /lookup for the key facts, /reverseip, /redirect and /similar for the lists, and /dnshistory with Basic and Premium. A free account comes with an API key and 100 calls a day.

curl -X POST https://api.webxtrend.com/v1.0/lookup \
  -H 'Content-Type: application/json' \
  -d '{"token": "YOUR_TOKEN", "query": "crawlcheck.dev"}'

Create a free account Read the API docs

Similar websites

Website	Similarity
crawlreadiness.com Crawl Readiness — Check, Fix & Monitor Your AI Search Visibility (GEO) All-in-one AI GEO toolkit (generative engine optimization): check whether ChatGPT, Claude, Perplexity & 50+ AI crawlers can read your site, generate the fixe...	91 %
aibotaccess.com AI Bot Access Checker — Is ChatGPT, Claude & Perplexity Reading Your Site? (Free) Free tool that checks if your robots.txt is blocking AI crawlers like GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Scan any domain in 5 seconds — no...	90 %
crawlercontrol.com CrawlerControl — robots.txt generator and checker for AI bots Generate and audit your robots.txt with a clear line between AI search bots (ChatGPT, Claude, Perplexity) and AI training bots — every rule shows its real co...	89 %
aigcrobots.com AI Robots.txt Generator \| Manage GPTBot, ClaudeBot, Google-Extended and AI Crawlers Draft and check robots.txt rules for GPTBot, ClaudeBot, Google-Extended, search crawlers, and private site paths.	88 %
aicrawlercheck.com Free AI Crawler Check: Test AI Crawlability and Bot Access Free AI crawler check: test AI crawlability and bot access for 257 AI and search bots, including Googlebot. Get a 0-100 score and copy-paste fixes. No sign-up.	88 %
aeosurge.com Answer Engine Optimization Platform \| AEO Surge AEO Surge is the answer engine optimization platform that gets your brand cited by ChatGPT, Perplexity, Gemini, and Copilot. Run a free AI-visibility audit.	87 %
scanaivisibility.com AI Visibility Checker — Free Instant Scan for ChatGPT, Claude & Perplexity Free AI visibility checker: scan any website in seconds to see if ChatGPT, Claude, Perplexity and other AI search engines can crawl and cite it. Checks robot...	87 %
sivd.dev AI Crawler Checker: Test GPTBot, ClaudeBot & robots.txt Free AI crawler checker. See whether GPTBot, ClaudeBot, PerplexityBot and other AI bots can access your site: robots.txt rules, Cloudflare managed robots.txt...	87 %
aicrawlerchecker.com AI Crawler Checker - Detect GPTBot & ClaudeBot Access Detect AI crawler access blockers by checking robots.txt, metadata, sitemap, llms.txt, and fetcher context for GPTBot, OAI-SearchBot, OAI-AdsBot, ClaudeBot,...	87 %
geokit.site GEOKit — Free GEO Tools for AI Search Optimization Free GEO toolkit to optimize your site for AI search: llms.txt generator, schema markup, robots.txt, readiness checker & more. No signup.	87 %
resulthack.com AI crawler directory: block or allow each in robots.txt 49 AI crawler robots.txt tokens from 19 operators: what each does, whether it obeys robots.txt, how to verify it and how many top sites block it.	86 %
whataisee.com Is your website ready for AI? Free AI visibility checker See what ChatGPT, Claude and Perplexity can read on your site. Free AI crawler test: robots.txt, GPTBot, llms.txt, JSON-LD. Score and fixes in 10 seconds.	86 %
crawlguard.dev CrawlGuard - Modern Crawl Guard & AI robots.txt Generator Generate modern, RFC-compliant robots.txt, .htaccess, and Cloudflare WAF rules to block aggressive AI scrapers (GPTBot, ClaudeBot, Bytespider) while preservi...	86 %
airobotstxt.com AI Robots.txt Validator & Generator \| Block ChatGPT, Claude & Gemini Generate and validate custom robots.txt files. Control search engines, social media preview bots, SEO scrapers, and 15+ AI/LLM crawlers like GPTBot, ClaudeBo...	85 %
aeoradar.com AEO Radar: free AEO tools and AI search data Answer engine optimization (AEO) is the practice of making your content easy for AI systems like ChatGPT, Perplexity and Google AI Overviews to read, trust a...	85 %
aicrawlability.com AI Crawlability Checker — Test If AI Bots Read Your Site A practical, source-grounded guide to making your website visible and citable to AI answer engines like ChatGPT, Perplexity, and Google AI Overviews.	85 %
robotstxtlab.net AI Crawler robots.txt Generator & Checker \| Robots.txt Lab Generate crawler-specific robots.txt rules for AI training, AI search, user-triggered agents, Googlebot, Bingbot, and more, then verify the live file.	85 %
amicrawlable.com Am I Crawlable Am I Crawlable — trusted ai visibility information and resources.	85 %
robotstxt.tools Robots.txt Generator – Free Online Robots.txt Tool Generate a valid robots.txt file for free. Add crawler rules, Allow and Disallow paths, sitemaps and presets, validate errors, then copy or download your rob...	84 %
crawlerview.com CrawlerView - AI Crawler Accessibility Testing Test how ChatGPT, Claude, Google, and Bing see your website. Validate SEO, structured data, and AI accessibility in seconds.	84 %

Websites whose start page offers something similar, found by comparing the text of their start pages, across languages. Near-identical copies, parking pages and error pages are left out. The full list of up to 100 comes through the API: similar websites.

Last SSL certificate

Issuer	Google Trust Services
Valid from	9/19/2026
Valid to	12/18/2026
Time until expiry	57 days

Last successful crawl

Title	AI Crawler Audit & robots.txt Generator — CrawlCheck
Description	Free robots.txt auditor for 60+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider…). See which AI engines can see your site, generate a clean policy in seconds. 100% client-side, no signup.
Content-Length	13,855 bytes

Title and description as published by the page itself in its own <title> and <meta name="description"> tags.

DNS history

IPv4 address	Last updated
104.21.16.253	9/26/2026
172.67.217.192	9/26/2026

IPv6 address	Last updated
2606:4700:3033:0:0:0:6815:10fd	9/26/2026
2606:4700:3033:0:0:0:ac43:d9c0	9/26/2026

Mail server	Priority	Last updated
route2.mx.cloudflare.net	18	9/26/2026
route3.mx.cloudflare.net	76	9/26/2026
route1.mx.cloudflare.net	96	9/26/2026

Name server	Last updated
adaline.ns.cloudflare.com	9/26/2026
lee.ns.cloudflare.com	9/26/2026

The five most recent states of each record type. The full history comes through the API: DNS history.

Most recent redirect domains

No domains found.

Reverse IP neighbours

104.21.16.253 996 domains · here since 9/20/2026

Domain	On this IP since
zsjhlt.com.cn	9/12/2026
zonastuntmedellin.com	9/15/2026
zhw158.my	10/12/2026
zh-zhs-pgsoftcn.com	9/15/2026
zernixqalo.online	9/18/2026
zdkiyz.com	9/18/2026
zd2so48l.cn	9/20/2026
za99.bio	9/16/2026
z16-168galaxy.com	9/17/2026
yyrr888.com	10/21/2026
yxnsbag.com	10/8/2026
ys1gp.sbs	9/18/2026
and 983 more

172.67.217.192 996 domains · here since 9/20/2026

Domain	On this IP since
zsjhlt.com.cn	9/12/2026
zonastuntmedellin.com	9/15/2026
zhw158.my	10/12/2026
zh-zhs-pgsoftcn.com	9/15/2026
zernixqalo.online	9/18/2026
zdkiyz.com	9/18/2026
zd2so48l.cn	9/20/2026
za99.bio	9/16/2026
z16-168galaxy.com	9/17/2026
yyrr888.com	10/21/2026
yxnsbag.com	10/8/2026
ys1gp.sbs	9/18/2026
and 983 more

2606:4700:3033::6815:10fd 988 domains · here since 9/20/2026

Domain	On this IP since
zsjhlt.com.cn	9/12/2026
zonastuntmedellin.com	9/15/2026
zhw158.my	10/12/2026
zh-zhs-pgsoftcn.com	9/15/2026
zernixqalo.online	9/18/2026
zdkiyz.com	9/18/2026
zd2so48l.cn	9/20/2026
za99.bio	9/16/2026
z16-168galaxy.com	9/17/2026
yyrr888.com	10/21/2026
yxnsbag.com	10/8/2026
ys1gp.sbs	9/18/2026
and 975 more

2606:4700:3033::ac43:d9c0 988 domains · here since 9/20/2026

Domain	On this IP since
zsjhlt.com.cn	9/12/2026
zonastuntmedellin.com	9/15/2026
zhw158.my	10/12/2026
zh-zhs-pgsoftcn.com	9/15/2026
zernixqalo.online	9/18/2026
zdkiyz.com	9/18/2026
zd2so48l.cn	9/20/2026
za99.bio	9/16/2026
z16-168galaxy.com	9/17/2026
yyrr888.com	10/21/2026
yxnsbag.com	10/8/2026
ys1gp.sbs	9/18/2026
and 975 more

Sharing an address does not always mean sharing a server: on a CDN or a large hosting platform, millions of unrelated domains answer on the same few addresses. We measured how far that goes in Half the web sits on 276 IP addresses. The complete list of neighbours, also for an IP address, comes through the API: reverse IP lookup.

Dive into our data, search now!