Crawlee
Build reliable web scrapers. Fast.
crawlee.dev Source★ 3,714Apache-2.0 DiscordData & Analytics / Web Scraping & Extraction
- Monthly visits
- 38.3K
- Down28.6%MoMOct 2026
- Public revenue
- Undisclosed
- No public revenue evidence yet
- Entry price
- Free
- No monthly subscription pricing
- Launched
- 4y ago
- 2022-08-23 · Est.
Traffic
Marketing moves
Launches, search & content, referrals, promotionsLaunched on Product HuntLaunch
Crawlee for Python is now out of beta with new features!
Sep 30, 2025 · 14mo agoLaunch page
Launched on Show HN (254 pts · 57 comments)Launch
Crawlee for Python – a web scraping and browser automation library — Hey all, This is Jan, the founder of Apify ( ) — a full-stack web scraping platform. After the success of Crawlee for JavaScript ( ) and t…
Jul 9, 2024 · 2y agoLaunch page
Launched on Product HuntLaunch
Build reliable web scrapers and robots, fast!
Aug 23, 2022 · 4y agoLaunch page
Launched on Show HN (282 pts · 80 comments)Launch
Crawlee – Web scraping and browser automation library for Node.js — Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're laun…
Aug 23, 2022 · 4y agoLaunch page
Third-party reviews & case dossiers
Acquisition breakdowns, engineer reviews & open-source metrics- Post excerpt:
- Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see the short video [2] or read the announcemen
- What engineers like:
- Uses TypeScript and integrates with (abstracts) common headless browsers like Puppeteer & Playwright which makes it easier to keep scripts up to date. Manages queuing, storing results, scaling & proxies. Extensive docs and examples. Open source.
- Trade-offs & limitations:
- Primarily designed to run locally (or in Docker), but there is a commercial SaaS option for hosting and scaling in the cloud.
- Best suited for:
- Developers using TypeScript and headless browsers should consider Crawlee for web scraping automation
- Post excerpt:
- Hey all, This is Jan, the founder of Apify ( https://apify.com/ ) — a full-stack web scraping platform. After the success of Crawlee for JavaScript ( https://github.com/apify/crawlee/ ) and the demand from the Python community, we're launching Crawlee for Python today! The main features are: - A unified programming interface for both HTTP (HTTPX with Beautif
Open Source & Self-Hosting
- GitHub Stars
- ★ 3,714
- Forks
- 310
- License
- Apache-2.0
- Primary language
- JavaScript
Traffic sources
Traffic breakdown
Channels
- Organic Search37%
- Referrals30%
- Direct29%
- AI Assistants3%
Countries
- United States33%
- Brazil14%
- India14%
- Vietnam4%
- Indonesia4%
Search
Domain authority —Domain registered 2022-07-08
| Keyword | Monthly volume |
|---|---|
| apify | 962K |
| crawlee | 3,120 |
| sitemap crawler | 350 |
| cheeriocrawler | 300 |
| crawlee, scrapy | 220 |
Traffic data: Vaneform
Pricing
Website does not list public plan prices.
Trial: None · Observed 2026-11-03 · Source page
Pages & changes
Key pages
Weekly snapshots- Homecrawlee.dev/
- Featurescrawlee.dev/js Snapshot 2026-11-03
Changes
Design
Minimal Monochrome (Light canvas, balanced density) featuring centered headline + cta layout and code block / terminal / cli visuals.
First-screen preview · click to zoom
First-screen pitch & conversion flow
“Build reliable web scrapers. Fast.”
Centered Headline + CTA hero structure paired with download app / install extension CTA and no explicit trust bar.
Conversion path: Centered Headline + CTA + Code Block / Terminal / CLI → Download App / Install Extension
- Outcome & Speed Promise
- Code & Terminal
Palette · click to copy
- Typography
- 7.0
- Color & Contrast
- 7.0
- Layout & Spacing
- 7.0
- Craft & Polish
- 7.0
- Distinctiveness
- 6.0
Structure
- Fonts
- Inter / JetBrains Mono
- Hero layout
- Centered Headline + CTA
- Primary visual
- Code Block / Terminal / CLI
- Conversion CTA
- Download App / Install Extension
- Social proof
- No Explicit Trust Bar
- Navigation
- Full-Width Top Bar
- Theme & density
- Light · Balanced
- Corners & depth
- Medium Radius · Hairline borders
Preview full AI design brief (includes :root CSS variables & layout rules)Style tokens only; never copy brand names, logos, copy, or images
# Design Brief: Minimal Monochrome (with Precision Swiss Grid)
Monochrome palette with a single accent color, hairline dividers, and generous whitespace.
## Design Tokens
```css
:root {
--background: #ffffff;
--surface: #f5f6f8;
--foreground: #1b1d24;
--muted-foreground: #6e7480;
--accent: #f58220;
--radius: 8px;
--font-heading: "Inter", ui-sans-serif, system-ui, sans-serif;
--font-body: "Inter", ui-sans-serif, system-ui, sans-serif;
--font-mono: "JetBrains Mono", ui-monospace, monospace;
}
```
Color area ratio: Background #FFFFFF (~55%) · Surface #F5F6F8 (~20%) · Text #1B1D24 (~12%) · Muted Text #6E7480 (~8%) · Accent #F58220 (~5%). Background and surface dominate the canvas; reserve the accent color for primary buttons and active states.
## Layout & Conversion Structure
- Hero layout: Centered Headline + CTA
- Navigation: Full-Width Top Bar
- Theme & density: Light · Balanced
- Border radius & elevation: Medium Radius; no drop shadows—separate layers with hairline borders and whitespace
- Primary visual medium: Code Block / Terminal / CLI
- Conversion CTA: Download App / Install Extension
- Headline hook: Outcome & Speed Promise
- Hero components: Code & Terminal
- Spatial layout notes: Centered Headline + CTA hero structure paired with download app / install extension CTA and no explicit trust bar.
- Visual summary: Minimal Monochrome (Light canvas, balanced density) featuring centered headline + cta layout and code block / terminal / cli visuals.
## Avoid
- Multi-stop neon gradients
- Heavy drop shadows
- More than one competing accent color
Implement the UI following the design tokens and layout specifications above: use only these color variables, and reserve the accent color strictly for primary actions and active states. Adopt only this visual style—never copy any reference product’s brand name, logo, copy, or imagery.
Questions & evidence
- How much website traffic does Crawlee get?
- Vaneform estimates 38.3K website visits in 2026-10. Visits can include repeat visits; this estimate measures web traffic separately from paying customers or revenue.Source
- What is Crawlee and what platforms does it support?
- Crawlee is catalogued under Web Scraping & Extraction: Build reliable web scrapers. Fast.. Supported platforms: API.
- Is Crawlee open source?
- Crawlee has a public repository (https://github.com/apify/crawlee) under the Apache-2.0 license, recording 3,714 GitHub stars.Source
Related products in this category
These products share a catalog category. Compare customer tasks, features and deployment requirements before treating them as alternatives.
| Product | Entry price | Visits | Revenue |
|---|---|---|---|
| Apify118,333 tools for your AI | $19/mo | 4.2M | Undisclosed |
| IGExportIGExport Instagram Tools & Social Insights | — | 3.9M | Undisclosed |
| ExportTokTikTok Comment Exporter | $9.99/mo | 2.8M | Undisclosed |
| RosintRosint — Search deleted Reddit posts, removed comments, and private profiles | — | 2.7M | Undisclosed |
| Omar-Thing User FinderLook up public info for any TikTok account | — | 2.6M | Undisclosed |
| FirecrawlGive your AI agents web data and beyond | $19/mo | 1.8M | Undisclosed |