CrawlerCheck
Instantly check if Google bots and AI crawlers can access your website.
crawlercheck.com SourceMarketing / SEO Tools
- Monthly visits
- 24.5K
- Down29.3%MoMOct 2026
- Public revenue
- Undisclosed
- No public revenue evidence yet
- Entry price
- Free
- No monthly subscription pricing
- Launched
- 17mo ago
- 2025-07-08 · Est.
Traffic
Marketing moves
Launches, search & content, referrals, promotionsLaunched on Show HN (1 pts)Launch
CrawlerCheck v1.5 – Operators Directory and 25 new AI crawlers — Since the last Show HN 7 months ago, I've shipped some major updates: • Operators Directory – browse 160+ crawlers by company (Google, Ope…
Feb 9 · 9mo agoLaunch page
Launched on Show HN (1 pts · 1 comments)Launch
A directory of 150 AI bots and crawlers with verifying tools
Dec 10, 2025 · 11mo agoLaunch page
Launched on Product HuntLaunch
Identify, check, manage unwanted web crawlers and bots.
Dec 7, 2025 · 11mo agoLaunch page
Launched on Uneed (#4)Launch
CrawlerCheck — Free crawler checker and website crawlability test for Googlebot, and AI bots.
Oct 11, 2025 · 13mo agoLaunch page
Launched on BetaListLaunch
CrawlerCheck – This tool analyzes if you are blocking search engines or AI crawlers — This tool analyzes if you are blocking search engines or AI crawlers
Aug 9, 2025 · 15mo agoLaunch page
Third-party reviews & case dossiers
Acquisition breakdowns, engineer reviews & open-source metrics- Post excerpt:
- Hi HN, I'm a long-time developer and SEO consultant. Over the years, I've seen clients suffer from a simple, costly mistake: accidentally blocking Googlebot or other important crawlers with a misplaced rule in robots.txt or a noindex tag. Manually checking the robots.txt file, then the page's meta tags, then the X-Robots-Tag HTTP header is a tedious process.
- Post excerpt:
- Since the last Show HN 7 months ago, I've shipped some major updates: • Operators Directory – browse 160+ crawlers by company (Google, OpenAI, Anthropic, etc.). See all their bots, safety stats, and official docs in one place. • Enhanced crawler profiles with "Blocking Impact" analysis and common use cases • Added 25+ new AI crawlers including latest models
Open Source & Self-Hosting
- GitHub Stars
- Public repo
- Forks
- —
- License
- See repository
- Primary language
- —
Traffic sources
Traffic breakdown
Channels
- Direct39%
- Organic Search36%
- Referrals10%
- Social6%
- Other9%
Countries
- United States15%
- India13%
- Pakistan8%
- Indonesia7%
- Brazil7%
Search
Domain authority —Domain registered 2025-07-02
| Keyword | Monthly volume |
|---|---|
| how to check if your google adbot is being blocked | 1,170 |
| googlebot schekcer | 220 |
| status checherk googlebot | 220 |
| search engines that crawl the web list | 180 |
| google ads crawler test | 90 |
Traffic data: Vaneform
Pricing
Website does not list public plan prices.
Trial: None · Observed 2026-11-03 · Source page
Pages & changes
Key pages
Weekly snapshotsChanges
Design
Minimal Monochrome (Light canvas, balanced density) featuring centered headline + input layout and interactive hero demo / tool visuals.
First-screen preview · click to zoom
First-screen pitch & conversion flow
“Instantly check if Google bots and AI crawlers can access your website.”
Centered Headline + Input hero structure paired with direct input / instant try CTA and no explicit trust bar.
Conversion path: Centered Headline + Input + Interactive Hero Demo / Tool → Direct Input / Instant Try
- Outcome & Speed Promise
- Centered Prompt / Input
Palette · click to copy
- Typography
- 7.0
- Color & Contrast
- 6.0
- Layout & Spacing
- 7.0
- Craft & Polish
- 6.0
- Distinctiveness
- 5.0
Structure
- Fonts
- Inter
- Hero layout
- Centered Headline + Input
- Primary visual
- Interactive Hero Demo / Tool
- Conversion CTA
- Direct Input / Instant Try
- Social proof
- No Explicit Trust Bar
- Navigation
- Full-Width Top Bar
- Theme & density
- Light · Balanced
- Corners & depth
- Small Radius · Hairline borders
Preview full AI design brief (includes :root CSS variables & layout rules)Style tokens only; never copy brand names, logos, copy, or images
# Design Brief: Minimal Monochrome
Monochrome palette with a single accent color, hairline dividers, and generous whitespace.
## Design Tokens
```css
:root {
--background: #ffffff;
--surface: #f7f7f7;
--foreground: #1a1a1a;
--muted-foreground: #6b7280;
--accent: #6b8f71;
--radius: 4px;
--font-heading: "Inter", ui-sans-serif, system-ui, sans-serif;
--font-body: "Inter", ui-sans-serif, system-ui, sans-serif;
}
```
Color area ratio: Background #FFFFFF (~62%) · Surface #F7F7F7 (~22%) · Text #1A1A1A (~8%) · Muted Text #6B7280 (~5%) · Accent #6B8F71 (~3%). Background and surface dominate the canvas; reserve the accent color for primary buttons and active states.
## Layout & Conversion Structure
- Hero layout: Centered Headline + Input
- Navigation: Full-Width Top Bar
- Theme & density: Light · Balanced
- Border radius & elevation: Small Radius; no drop shadows—separate layers with hairline borders and whitespace
- Primary visual medium: Interactive Hero Demo / Tool
- Conversion CTA: Direct Input / Instant Try
- Headline hook: Outcome & Speed Promise
- Hero components: Centered Prompt / Input
- Spatial layout notes: Centered Headline + Input hero structure paired with direct input / instant try CTA and no explicit trust bar.
- Visual summary: Minimal Monochrome (Light canvas, balanced density) featuring centered headline + input layout and interactive hero demo / tool visuals.
## Avoid
- Multi-stop neon gradients
- Heavy drop shadows
- More than one competing accent color
Implement the UI following the design tokens and layout specifications above: use only these color variables, and reserve the accent color strictly for primary actions and active states. Adopt only this visual style—never copy any reference product’s brand name, logo, copy, or imagery.
Questions & evidence
- How much website traffic does CrawlerCheck get?
- Vaneform estimates 24.5K website visits in 2026-10. Visits can include repeat visits; this estimate measures web traffic separately from paying customers or revenue.Source
- What is CrawlerCheck and what platforms does it support?
- CrawlerCheck is catalogued under SEO Tools: Instantly check if Google bots and AI crawlers can access your website.. Supported platforms: Web.
- Is CrawlerCheck open source?
- CrawlerCheck has a public repository (https://github.com/waaab/crawlercheck) with its source repository published.Source
Related products in this category
These products share a catalog category. Compare customer tasks, features and deployment requirements before treating them as alternatives.
| Product | Entry price | Visits | Revenue |
|---|---|---|---|
| SEMrushGiantBe found and chosen everywhere search happens | $117.33/mo | 11.4M | $443.6M/yrClaimed |
| ahrefsGiantMake your business discoverable—in search, AI, and beyond | $50/mo | 7.8M | $100M/yrClaimed |
| smallseotoolsMatureYou need effective SEO tools which are 100% free? Check out our Search Engine Optimization tools for checking backlinks | $5.99/mo | 2.8M | Undisclosed |
| SE RankingMatureSearch data you can trust. Results you can explain. | $29/mo | 1.2M | Undisclosed |
| TubePilotMature100% Free YouTube Tools | — | 862.8K | Undisclosed |
| rankmathMatureSEO for WordPress Made Easy | $5.59/mo | 710K | Undisclosed |