mirror of
https://github.com/Shubhamsaboo/awesome-llm-apps.git
synced 2026-08-28 11:07:19 +08:00
docs: rewrite web_scraping_ai_agent README to match shipped code
The folder only ships a local ScrapeGraphAI implementation (ai_scrapper.py, local_ai_scrapper.py), but the README documented an entire "Cloud SDK" version referencing a scrapegraph_ai_sdk/ folder and quickstart.py / smart_scraper_demo.py / scrapegraph_app.py — none of which exist anywhere in the repo. Remove all of that imaginary content and reframe as the local-only agent it actually is. Also fix the clone-path typo (web_scrapping -> web_scraping) and drop the now-inaccurate "(Local & Cloud SDK)" label in the root README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Shubhamsaboo
@@ -137,7 +137,7 @@ streamlit run travel_agent.py
|
||||
* [🔄 Mixture of Agents](starter_ai_agents/mixture_of_agents/)
|
||||
* [📊 xAI Finance Agent](starter_ai_agents/xai_finance_agent/)
|
||||
* [🔍 OpenAI Research Agent](starter_ai_agents/openai_research_agent/)
|
||||
* [🕸️ Web Scraping AI Agent (Local & Cloud SDK)](starter_ai_agents/web_scraping_ai_agent/)
|
||||
* [🕸️ Web Scraping AI Agent](starter_ai_agents/web_scraping_ai_agent/)
|
||||
|
||||
### 🚀 Advanced AI Agents
|
||||
*Production-style agents with tools, memory, and multi-step reasoning.*
|
||||
|
||||
@@ -3,19 +3,15 @@
|
||||
### 🎓 FREE Step-by-Step Tutorial
|
||||
**👉 [Click here to follow our complete step-by-step tutorial](https://www.theunwindai.com/p/build-a-web-scraping-ai-agent-with-llama-3-2-running-locally) and learn how to build this from scratch with detailed code walkthroughs, explanations, and best practices.**
|
||||
|
||||
AI-powered web scraping using **ScrapeGraph AI** - extract structured data from websites using natural language prompts. This folder contains two implementations:
|
||||
|
||||
1. **🏠 Local Library** - Using `scrapegraphai` library (runs locally)
|
||||
2. **☁️ Cloud SDK** - Using ScrapeGraph AI API (managed service)
|
||||
AI-powered web scraping using **ScrapeGraphAI** - extract structured data from websites using natural language prompts. This agent runs locally with the open-source `scrapegraphai` library.
|
||||
|
||||
---
|
||||
|
||||
## 📁 What's Inside
|
||||
|
||||
### 🏠 Local Library Version
|
||||
**Files**: `ai_scrapper.py`, `local_ai_scrapper.py`
|
||||
|
||||
Use the open-source ScrapeGraph AI library that runs on your local machine.
|
||||
Use the open-source ScrapeGraphAI library that runs on your local machine.
|
||||
|
||||
**✅ Pros:**
|
||||
- Free to use (no API costs)
|
||||
@@ -27,47 +23,14 @@ Use the open-source ScrapeGraph AI library that runs on your local machine.
|
||||
- Limited by your hardware
|
||||
- Need to manage updates
|
||||
|
||||
**Quick Start:**
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
streamlit run ai_scrapper.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ☁️ Cloud SDK Version
|
||||
**Folder**: `scrapegraph_ai_sdk/`
|
||||
|
||||
Use the managed ScrapeGraph AI API with advanced features and no setup required.
|
||||
|
||||
**✅ Pros:**
|
||||
- No setup required (just API key)
|
||||
- Scalable and fast
|
||||
- Advanced features (SmartCrawler, SearchScraper, Markdownify)
|
||||
- Always up-to-date
|
||||
|
||||
**❌ Cons:**
|
||||
- Pay-per-use (credit-based)
|
||||
- Requires internet connection
|
||||
|
||||
**Quick Start:**
|
||||
```bash
|
||||
cd scrapegraph_ai_sdk/
|
||||
pip install -r requirements.txt
|
||||
export SGAI_API_KEY='your-api-key'
|
||||
python quickstart.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### Local Library Version
|
||||
|
||||
1. **Clone the repository**
|
||||
```bash
|
||||
git clone https://github.com/Shubhamsaboo/awesome-llm-apps.git
|
||||
cd awesome-llm-apps/starter_ai_agents/web_scrapping_ai_agent
|
||||
cd awesome-llm-apps/starter_ai_agents/web_scraping_ai_agent
|
||||
```
|
||||
|
||||
2. **Install dependencies**
|
||||
@@ -86,57 +49,6 @@ streamlit run ai_scrapper.py
|
||||
streamlit run local_ai_scrapper.py
|
||||
```
|
||||
|
||||
### Cloud SDK Version
|
||||
|
||||
1. **Navigate to SDK folder**
|
||||
```bash
|
||||
cd scrapegraph_ai_sdk/
|
||||
```
|
||||
|
||||
2. **Install dependencies**
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. **Get your ScrapeGraph AI API Key**
|
||||
- Sign up at [scrapegraphai.com](https://scrapegraphai.com)
|
||||
- Get your API key
|
||||
|
||||
4. **Set API key**
|
||||
```bash
|
||||
export SGAI_API_KEY='your-api-key-here'
|
||||
```
|
||||
|
||||
5. **Run demos**
|
||||
```bash
|
||||
# Quick test
|
||||
python quickstart.py
|
||||
|
||||
# SmartScraper demo
|
||||
python smart_scraper_demo.py
|
||||
|
||||
# Interactive app
|
||||
streamlit run scrapegraph_app.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Feature Comparison
|
||||
|
||||
| Feature | Local Library | Cloud SDK |
|
||||
|---------|--------------|-----------|
|
||||
| **Setup** | Install dependencies | API key only |
|
||||
| **Cost** | Free (+ LLM costs) | Pay-per-use |
|
||||
| **Processing** | Your hardware | Cloud-based |
|
||||
| **Speed** | Depends on hardware | Fast & optimized |
|
||||
| **SmartScraper** | ✅ | ✅ |
|
||||
| **SearchScraper** | ❌ | ✅ |
|
||||
| **SmartCrawler** | ❌ | ✅ |
|
||||
| **Markdownify** | ❌ | ✅ |
|
||||
| **Scheduled Jobs** | ❌ | ✅ |
|
||||
| **Scalability** | Limited | Unlimited |
|
||||
| **Maintenance** | Self-managed | Fully managed |
|
||||
|
||||
---
|
||||
|
||||
## 💡 Use Cases
|
||||
@@ -169,81 +81,16 @@ prompt = "Find company names, emails, and phone numbers"
|
||||
|
||||
## 🔧 How It Works
|
||||
|
||||
### Local Library
|
||||
1. You provide your OpenAI API key
|
||||
2. Select the model (GPT-4o, GPT-5, or local models)
|
||||
3. Enter the URL and extraction prompt
|
||||
4. The app uses ScrapeGraphAI to scrape and extract data locally
|
||||
5. Results are displayed in the app
|
||||
|
||||
### Cloud SDK
|
||||
1. You provide your ScrapeGraph AI API key
|
||||
2. Choose the scraping method (SmartScraper, SearchScraper, etc.)
|
||||
3. Define extraction prompt and optional output schema
|
||||
4. API processes the request in the cloud
|
||||
5. Structured results are returned
|
||||
|
||||
---
|
||||
|
||||
## 🌟 Cloud SDK Features
|
||||
|
||||
### 🤖 SmartScraper
|
||||
Extract structured data using natural language:
|
||||
```python
|
||||
response = client.smartscraper(
|
||||
website_url="https://example.com",
|
||||
user_prompt="Extract all products with prices"
|
||||
)
|
||||
```
|
||||
|
||||
### 🔍 SearchScraper
|
||||
AI-powered web search with structured results:
|
||||
```python
|
||||
response = client.smartscraper(
|
||||
user_prompt="Find top 5 AI news websites",
|
||||
num_results=5
|
||||
)
|
||||
```
|
||||
|
||||
### 📝 Markdownify
|
||||
Convert webpages to clean markdown:
|
||||
```python
|
||||
response = client.markdownify(
|
||||
website_url="https://example.com/article"
|
||||
)
|
||||
```
|
||||
|
||||
### 🕷️ SmartCrawler
|
||||
Crawl multiple pages intelligently:
|
||||
```python
|
||||
request_id = client.smartcrawler(
|
||||
url="https://docs.example.com",
|
||||
user_prompt="Extract all API endpoints",
|
||||
max_pages=50
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📖 Documentation
|
||||
|
||||
- **Local Library**: [ScrapeGraphAI GitHub](https://github.com/VinciGit00/Scrapegraph-ai)
|
||||
- **API Docs**: https://docs.scrapegraphai.com
|
||||
- **ScrapeGraphAI Library**: [ScrapeGraphAI GitHub](https://github.com/VinciGit00/Scrapegraph-ai)
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Which Version Should I Use?
|
||||
|
||||
**Use Local Library if:**
|
||||
- ✅ You want free, open-source solution
|
||||
- ✅ You have good hardware
|
||||
- ✅ You need full control
|
||||
- ✅ Privacy is critical
|
||||
|
||||
**Use Cloud SDK if:**
|
||||
- ✅ You want quick setup
|
||||
- ✅ You need advanced features
|
||||
- ✅ You want scalability
|
||||
- ✅ You prefer managed service
|
||||
|
||||
**💡 Pro Tip**: Start with the local version to learn, then switch to SDK for production!
|
||||
|
||||
Reference in New Issue
Block a user