From c2273fff203136d0320fb37af0ebea7a8dc10a8e Mon Sep 17 00:00:00 2001 From: Shubhamsaboo Date: Fri, 12 Jun 2026 19:57:42 -0700 Subject: [PATCH] docs: rewrite web_scraping_ai_agent README to match shipped code MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The folder only ships a local ScrapeGraphAI implementation (ai_scrapper.py, local_ai_scrapper.py), but the README documented an entire "Cloud SDK" version referencing a scrapegraph_ai_sdk/ folder and quickstart.py / smart_scraper_demo.py / scrapegraph_app.py β€” none of which exist anywhere in the repo. Remove all of that imaginary content and reframe as the local-only agent it actually is. Also fix the clone-path typo (web_scrapping -> web_scraping) and drop the now-inaccurate "(Local & Cloud SDK)" label in the root README. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 2 +- .../web_scraping_ai_agent/README.md | 161 +----------------- 2 files changed, 5 insertions(+), 158 deletions(-) diff --git a/README.md b/README.md index 5582bef3..9fe17917 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ streamlit run travel_agent.py * [πŸ”„ Mixture of Agents](starter_ai_agents/mixture_of_agents/) * [πŸ“Š xAI Finance Agent](starter_ai_agents/xai_finance_agent/) * [πŸ” OpenAI Research Agent](starter_ai_agents/openai_research_agent/) -* [πŸ•ΈοΈ Web Scraping AI Agent (Local & Cloud SDK)](starter_ai_agents/web_scraping_ai_agent/) +* [πŸ•ΈοΈ Web Scraping AI Agent](starter_ai_agents/web_scraping_ai_agent/) ### πŸš€ Advanced AI Agents *Production-style agents with tools, memory, and multi-step reasoning.* diff --git a/starter_ai_agents/web_scraping_ai_agent/README.md b/starter_ai_agents/web_scraping_ai_agent/README.md index 3e9b3ae7..5feb62a0 100644 --- a/starter_ai_agents/web_scraping_ai_agent/README.md +++ b/starter_ai_agents/web_scraping_ai_agent/README.md @@ -3,19 +3,15 @@ ### πŸŽ“ FREE Step-by-Step Tutorial **πŸ‘‰ [Click here to follow our complete step-by-step tutorial](https://www.theunwindai.com/p/build-a-web-scraping-ai-agent-with-llama-3-2-running-locally) and learn how to build this from scratch with detailed code walkthroughs, explanations, and best practices.** -AI-powered web scraping using **ScrapeGraph AI** - extract structured data from websites using natural language prompts. This folder contains two implementations: - -1. **🏠 Local Library** - Using `scrapegraphai` library (runs locally) -2. **☁️ Cloud SDK** - Using ScrapeGraph AI API (managed service) +AI-powered web scraping using **ScrapeGraphAI** - extract structured data from websites using natural language prompts. This agent runs locally with the open-source `scrapegraphai` library. --- ## πŸ“ What's Inside -### 🏠 Local Library Version **Files**: `ai_scrapper.py`, `local_ai_scrapper.py` -Use the open-source ScrapeGraph AI library that runs on your local machine. +Use the open-source ScrapeGraphAI library that runs on your local machine. **βœ… Pros:** - Free to use (no API costs) @@ -27,47 +23,14 @@ Use the open-source ScrapeGraph AI library that runs on your local machine. - Limited by your hardware - Need to manage updates -**Quick Start:** -```bash -pip install -r requirements.txt -streamlit run ai_scrapper.py -``` - ---- - -### ☁️ Cloud SDK Version -**Folder**: `scrapegraph_ai_sdk/` - -Use the managed ScrapeGraph AI API with advanced features and no setup required. - -**βœ… Pros:** -- No setup required (just API key) -- Scalable and fast -- Advanced features (SmartCrawler, SearchScraper, Markdownify) -- Always up-to-date - -**❌ Cons:** -- Pay-per-use (credit-based) -- Requires internet connection - -**Quick Start:** -```bash -cd scrapegraph_ai_sdk/ -pip install -r requirements.txt -export SGAI_API_KEY='your-api-key' -python quickstart.py -``` - --- ## πŸš€ Getting Started -### Local Library Version - 1. **Clone the repository** ```bash git clone https://github.com/Shubhamsaboo/awesome-llm-apps.git -cd awesome-llm-apps/starter_ai_agents/web_scrapping_ai_agent +cd awesome-llm-apps/starter_ai_agents/web_scraping_ai_agent ``` 2. **Install dependencies** @@ -86,57 +49,6 @@ streamlit run ai_scrapper.py streamlit run local_ai_scrapper.py ``` -### Cloud SDK Version - -1. **Navigate to SDK folder** -```bash -cd scrapegraph_ai_sdk/ -``` - -2. **Install dependencies** -```bash -pip install -r requirements.txt -``` - -3. **Get your ScrapeGraph AI API Key** -- Sign up at [scrapegraphai.com](https://scrapegraphai.com) -- Get your API key - -4. **Set API key** -```bash -export SGAI_API_KEY='your-api-key-here' -``` - -5. **Run demos** -```bash -# Quick test -python quickstart.py - -# SmartScraper demo -python smart_scraper_demo.py - -# Interactive app -streamlit run scrapegraph_app.py -``` - ---- - -## πŸ“Š Feature Comparison - -| Feature | Local Library | Cloud SDK | -|---------|--------------|-----------| -| **Setup** | Install dependencies | API key only | -| **Cost** | Free (+ LLM costs) | Pay-per-use | -| **Processing** | Your hardware | Cloud-based | -| **Speed** | Depends on hardware | Fast & optimized | -| **SmartScraper** | βœ… | βœ… | -| **SearchScraper** | ❌ | βœ… | -| **SmartCrawler** | ❌ | βœ… | -| **Markdownify** | ❌ | βœ… | -| **Scheduled Jobs** | ❌ | βœ… | -| **Scalability** | Limited | Unlimited | -| **Maintenance** | Self-managed | Fully managed | - --- ## πŸ’‘ Use Cases @@ -169,81 +81,16 @@ prompt = "Find company names, emails, and phone numbers" ## πŸ”§ How It Works -### Local Library 1. You provide your OpenAI API key 2. Select the model (GPT-4o, GPT-5, or local models) 3. Enter the URL and extraction prompt 4. The app uses ScrapeGraphAI to scrape and extract data locally 5. Results are displayed in the app -### Cloud SDK -1. You provide your ScrapeGraph AI API key -2. Choose the scraping method (SmartScraper, SearchScraper, etc.) -3. Define extraction prompt and optional output schema -4. API processes the request in the cloud -5. Structured results are returned - ---- - -## 🌟 Cloud SDK Features - -### πŸ€– SmartScraper -Extract structured data using natural language: -```python -response = client.smartscraper( - website_url="https://example.com", - user_prompt="Extract all products with prices" -) -``` - -### πŸ” SearchScraper -AI-powered web search with structured results: -```python -response = client.smartscraper( - user_prompt="Find top 5 AI news websites", - num_results=5 -) -``` - -### πŸ“ Markdownify -Convert webpages to clean markdown: -```python -response = client.markdownify( - website_url="https://example.com/article" -) -``` - -### πŸ•·οΈ SmartCrawler -Crawl multiple pages intelligently: -```python -request_id = client.smartcrawler( - url="https://docs.example.com", - user_prompt="Extract all API endpoints", - max_pages=50 -) -``` - --- ## πŸ“– Documentation -- **Local Library**: [ScrapeGraphAI GitHub](https://github.com/VinciGit00/Scrapegraph-ai) -- **API Docs**: https://docs.scrapegraphai.com +- **ScrapeGraphAI Library**: [ScrapeGraphAI GitHub](https://github.com/VinciGit00/Scrapegraph-ai) --- - -## 🀝 Which Version Should I Use? - -**Use Local Library if:** -- βœ… You want free, open-source solution -- βœ… You have good hardware -- βœ… You need full control -- βœ… Privacy is critical - -**Use Cloud SDK if:** -- βœ… You want quick setup -- βœ… You need advanced features -- βœ… You want scalability -- βœ… You prefer managed service - -**πŸ’‘ Pro Tip**: Start with the local version to learn, then switch to SDK for production!