Navid MoazzezNavid Moazzez
AI consulting

Crawl4AI

Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.

Apply here for early access We’ll be onboarding in phases and working closely with early users. Limited slots.

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.

View Crawl4AI on GitHub
Navid MoazzezCurated by Navid Moazzez·Updated Oct 2, 2026·1 min read·
Crawl4AI

Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.

Apply here for early access We’ll be onboarding in phases and working closely with early users. Limited slots.

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.

Crawl4AI at a glance

Stars85k
Forks8.8k
LanguagePython
LicenseApache-2.0
Last update2026-09-25
Contributors95

How to install Crawl4AI

bash pip install crawl4ai[sync] 

Where Crawl4AI is listed

Navid Moazzez

AI business strategist & AI OS builder

Navid Moazzez helps creators and founders master AI and build their own AI Operating System (AI OS) to automate their business and life.

Navid.me is reader-supported. When you buy through links on this site, I may earn an affiliate commission. Read the affiliate disclaimer.

GitHub repos like this

View all GitHub repos

n8n

★ 206k · TypeScript · Active

Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.

AutomationDeveloper toolsSelf-hosted apps

Firecrawl

★ 188k · TypeScript · Active

The context API to search, scrape, and interact with the web at scale.

Automation

Core

★ 91k · Python · Active

house_with_garden: Open source home automation that puts local control and privacy first.

Automation

Agent Reach

★ 86k · Python · Active

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, one CLI, zero API fees.

AutomationDeveloper tools

Scrapling

★ 85k · Python · Active

An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

AutomationMCP servers

Deer Flow

★ 83k · Python · Active

An open-source long-horizon SuperAgent harness that researches, codes, and creates.

Agent frameworksAutomation

Awesome Claude Skills

★ 76k · Python · Active

A curated list of awesome Claude Skills, resources, and tools for customizing Claude AI workflows

Agent skillsAutomation

Career Ops

★ 73k · JavaScript · Active

Open-source AI job search: scan job portals, evaluate listings with a structured A-F rubric into a 1.0-5.

AutomationDeveloper tools

Awesome Claude Code

★ 55k · Python · Active

A hand-picked collection of the finest of resources for the most awesome of agents, Claude Code, the undisputed champion of coding companions, from…

Agent skillsAutomation

Shannon

★ 48k · TypeScript · Active

Shannon is an AI pentester for web applications and APIs.

Agent frameworksAutomation

Tooljet

★ 41k · JavaScript · Active

ToolJet is the open-source foundation of ToolJet AI - the enterprise app generation platform for building internal tools, dashboard, business…

Automation

Maigret

★ 38k · Python · Active

‍ Collect a dossier on a person by username from 3000+ sites

AutomationDeveloper tools

More repo topics

View all GitHub repos
githubSelf-hosted apps GitHub repos91 repos
githubAgent skills GitHub repos42 repos
githubAgent frameworks GitHub repos37 repos
githubDeveloper tools GitHub repos36 repos
githubMCP servers GitHub repos27 repos
githubLearning GitHub repos22 repos
githubMedia & audio GitHub repos22 repos
githubAI interfaces GitHub repos21 repos
githubRAG & memory GitHub repos19 repos
githubSecurity & infrastructure GitHub repos19 repos
githubCoding agents GitHub repos14 repos
githubLLM serving GitHub repos12 repos
githubPrompt engineering GitHub repos6 repos
githubAwesome lists GitHub repos6 repos
githubEvals & observability GitHub repos2 repos

More free tools

All free tools

llms.txt generator

Build your llms.txt from your live site, check every link and your AI crawler access, and add llms-full.txt.

AIAny website

DESIGN.md generator

Build a DESIGN.md from your CSS or Tailwind tokens, let your AI write the words, and check it against Google's format.

AIAny website

Image to 3D converter

Turn a photo or a video into a three.js model with your own AI, then download GLB, USDZ, STL, a GIF or website code.

AIAny website

Fake ChatGPT chat maker

Draw a ChatGPT conversation the way ChatGPT shows it today, in light or dark, with real markdown.

AI

AI tool finder

Find AI tools for your workflow, compare your options, and explore the tools creators use.

AIAny website

Skill generator

Build an Agent Skill for Claude Code, the Claude apps or Codex, with a graded description and test prompts.

AIAny website

AI tools library

The AI tools worth using, by category, pricing and platform, each with its own page.

AIAny website

AI prompt generator

Write the prompt that works the first time, for ChatGPT, Claude, Gemini, Midjourney, Seedance, Veo and more.

AIAny website

Prompt improver

Paste a prompt and get it back sharper, to the platform's own rules, with every variable kept.

AIAny website

ChatGPT prompt generator

Write ChatGPT prompts to OpenAI's own GPT-6 guidance, checked in code, ready to paste or save as custom instructions.

AIAny website

Claude prompt generator

Write Claude prompts to Anthropic's own best practices: XML sections, examples in tags, the reason behind each rule.

AIAny website

Gemini prompt generator

Write Gemini prompts to Google's own Gemini 3 strategies: the role first, examples, and the question after the material.

AIAny website

Related MCP servers & CLIs

All MCP servers & CLIs

Bluesky MCP server & CLI

Post, read and search Bluesky from Claude, Codex or Cursor, or from your terminal.

MCP servers & CLIsblueskyBluesky

Substack MCP server & CLI

Draft posts, publish Notes and check your stats on Substack from Claude, Codex or Cursor.

MCP servers & CLIssubstackSubstack

Threads MCP server & CLI

Post, reply and read your insights on Threads from Claude, Codex or Cursor.

MCP servers & CLIsthreadsThreads

Mastodon MCP server & CLI

Post, edit and read your timelines on any Mastodon instance from Claude, Codex or Cursor.

MCP servers & CLIsmastodonMastodon

TikTok MCP server & CLI

Rank your videos, read your stats and publish to TikTok from Claude, Codex or Cursor.

MCP servers & CLIstiktokTikTok

ThriveCart MCP server & CLI

Check revenue, customers, subscriptions and affiliates in ThriveCart from Claude, Codex or Cursor.

MCP servers & CLIs

WordPress MCP server & CLI

Edit posts, pages, Elementor and Rank Math SEO on WordPress from Claude, Codex or Cursor.

MCP servers & CLIs

Midjourney MCP server & CLI

Generate images, follow jobs and download the files from Midjourney with Claude, Codex or Cursor.

MCP servers & CLIs

Google Photos MCP server & CLI

Pick photos, build albums and upload to Google Photos from Claude, Codex or Cursor.

MCP servers & CLIs

Apple Photos MCP server & CLI

Search, look at and organize your Apple Photos library from Claude, Codex or Cursor, on your Mac.

MCP servers & CLIs

Apple Podcasts MCP server & CLI

Track chart ranks, read reviews and search your podcast library from Claude, Codex or Cursor.

MCP servers & CLIsapple-podcastsApple Podcasts

Podcast Index MCP server & CLI

Search 4 million podcasts and read episode transcripts from Claude, Codex or Cursor.

MCP servers & CLIs
Free AI newsletter

The most actionable AI newsletter for founders

Every week, get proven AI strategies, curated tools, and step-by-step systems to grow your audience, create better content, and build a profitable creator business.

No fluff, no filler, no BS. Just five minutes each week that might level up your online business and life.

P.S. Sign up now to get free access to my ultimate AI tools guide for creators.

Loved by 10,000+ readers
Loved by 10,000+ readers