opensource/FossFLOW
3
0
Fork 0
mirror of https://github.com/stan-smith/FossFLOW.git synced 2026-09-01 18:19:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

ci stuff

Browse Source
This commit is contained in:
stan-smith
2025-10-09 13:56:45 +01:00
parent a82ca72feb
commit b7b13297f5
9 changed files with 433 additions and 278 deletions
Download Patch File Download Diff File Expand all files Collapse all files
e2e-tests/.gitignore
+8 -14
View File
@@ -1,14 +1,8 @@
# Rust
/target/
Cargo.lock
# IDE
.idea/
.vscode/
*.swp
*.swo
*~
# Test artifacts
screenshots/
test-results/
__pycache__/
*.pyc
.pytest_cache/
htmlcov/
.coverage
*.log
venv/
env/
e2e-tests/Cargo.toml
-16
View File
@@ -1,16 +0,0 @@
[package]
name = "fossflow-e2e-tests"
version = "0.1.0"
edition = "2021"
[dependencies]
thirtyfour = "0.34.0"
tokio = { version = "1.47", features = ["full"] }
anyhow = "1.0"
[dev-dependencies]
assert_cmd = "2.0"
[[test]]
name = "basic_load"
path = "tests/basic_load.rs"
e2e-tests/README.md
+114 -57
View File
@@ -1,53 +1,57 @@
# FossFLOW E2E Tests
End-to-end tests for FossFLOW using Selenium WebDriver via the [thirtyfour](https://github.com/Vrtgs/thirtyfour) Rust library.
End-to-end tests for FossFLOW using Selenium WebDriver with Python and pytest.
## Prerequisites
1. **Rust** - Install from https://rustup.rs/
2. **Chrome/Chromium** browser
3. **ChromeDriver** or Selenium Server
1. **Python 3.11+** - Install from https://www.python.org/
2. **Docker** - For running Selenium Grid
3. **Chrome/Chromium** browser (provided by Selenium Docker image)
## Running Tests Locally
### Option 1: Using Selenium Standalone (Recommended)
### Quick Start (Recommended)
Use the provided test runner script:
```bash
cd e2e-tests
./run-tests.sh
```
The script will:
- Check for required dependencies (Docker, Python)
- Start Selenium container automatically
- Create a Python virtual environment
- Install test dependencies
- Prompt you to start the FossFLOW app if not running
- Run the tests
- Clean up Selenium container
### Manual Setup
1. Start Selenium server with Chrome:
```bash
docker run -d -p 4444:4444 -p 7900:7900 --shm-size="2g" selenium/standalone-chrome:latest
docker run -d --name fossflow-selenium -p 4444:4444 -p 7900:7900 --shm-size="2g" selenium/standalone-chrome:latest
```
2. Start the FossFLOW dev server:
```bash
cd .. # Go to project root
npm run dev
```
3. Run the tests:
3. Install Python dependencies:
```bash
cd e2e-tests
cargo test -- --test-threads=1
```
**Note**: Tests must run serially (`--test-threads=1`) because Selenium standalone only supports one session at a time.
### Option 2: Using ChromeDriver directly
1. Download ChromeDriver matching your Chrome version from https://chromedriver.chromium.org/
2. Start ChromeDriver:
```bash
chromedriver --port=4444
```
3. Start the FossFLOW dev server:
```bash
npm run dev
python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
```
4. Run the tests:
```bash
cd e2e-tests
cargo test -- --test-threads=1
pytest -v
```
## Environment Variables
@@ -57,7 +61,7 @@ End-to-end tests for FossFLOW using Selenium WebDriver via the [thirtyfour](http
Example:
```bash
FOSSFLOW_TEST_URL=http://localhost:8080 cargo test
FOSSFLOW_TEST_URL=http://localhost:8080 pytest -v
```
## Available Tests
@@ -74,50 +78,103 @@ Tests run automatically in GitHub Actions on:
The CI workflow:
1. Builds the app
2. Starts the app server
2. Starts the app server in background
3. Starts Selenium standalone Chrome
4. Runs all E2E tests
4. Installs Python dependencies
5. Runs all E2E tests with pytest
## Test Structure
```
e2e-tests/
├── tests/
│ └── test_basic_load.py # Main test suite
├── requirements.txt # Python dependencies
├── pytest.ini # Pytest configuration
├── run-tests.sh # Test runner script
└── README.md # This file
```
## Adding New Tests
1. Create a new test file in `tests/` directory
2. Add it to `Cargo.toml` under `[[test]]` sections
3. Use the thirtyfour API: https://docs.rs/thirtyfour/latest/thirtyfour/
1. Create a new test file in `tests/` directory (must start with `test_`)
2. Import required modules:
```python
import pytest
from selenium import webdriver
from selenium.webdriver.common.by import By
```
Example:
```rust
use anyhow::Result;
use thirtyfour::prelude::*;
3. Use the `driver` fixture:
```python
def test_my_feature(driver):
driver.get("http://localhost:3000")
element = driver.find_element(By.ID, "my-element")
assert element.is_displayed()
```
#[tokio::test]
async fn test_my_feature() -> Result<()> {
let driver = WebDriver::new("http://localhost:4444", DesiredCapabilities::chrome()).await?;
driver.goto("http://localhost:3000").await?;
// Your test logic here
driver.quit().await?;
Ok(())
}
```
4. Run your test:
```bash
pytest tests/test_my_feature.py -v
```
## Debugging
To run tests with visible browser (non-headless):
1. Modify the test to remove `.set_headless()?` from capabilities
2. Use `selenium/standalone-chrome-debug` Docker image with VNC viewer on port 7900
### Running with Visible Browser
To see the browser during tests, modify the driver fixture in `test_basic_load.py`:
```python
# Comment out headless mode
# chrome_options.add_argument("--headless")
```
### Using VNC to Watch Tests
When using the Selenium Docker image, you can watch tests in real-time:
1. Connect to VNC viewer at `http://localhost:7900` (password: `secret`)
2. Remove `--headless` from Chrome options
3. Run tests and watch in VNC viewer
### Verbose Output
Run tests with more verbose output:
```bash
pytest -vv --tb=long
```
### Running Specific Tests
```bash
# Run a single test
pytest tests/test_basic_load.py::test_homepage_loads -v
# Run tests matching a pattern
pytest -k "canvas" -v
```
## Troubleshooting
**Connection refused errors:**
- Ensure Selenium/ChromeDriver is running on port 4444
- Ensure FossFLOW app is running on port 3000
### Connection refused errors
- Ensure Selenium is running: `docker ps | grep selenium`
- Check Selenium status: `curl http://localhost:4444/status`
- Ensure FossFLOW app is running: `curl http://localhost:3000`
**Element not found errors:**
### Element not found errors
- Increase wait times in tests
- Check if the app URL is correct
- Verify the app loaded successfully in browser
**Chrome version mismatch:**
- Update ChromeDriver to match your Chrome version
- Use Selenium Docker image (automatically handles version matching)
### Import errors
- Activate virtual environment: `source venv/bin/activate`
- Install dependencies: `pip install -r requirements.txt`
### Docker container conflicts
- Remove existing container: `docker rm -f fossflow-selenium`
- Check for port conflicts: `lsof -i :4444`
## Dependencies
- **selenium** (4.27.1) - WebDriver automation library
- **pytest** (8.3.4) - Testing framework
- **pytest-xdist** (3.6.1) - Parallel test execution support
e2e-tests/SETUP.md
+138 -54
View File
@@ -2,20 +2,20 @@
## What Was Added
A complete Selenium-based end-to-end testing framework using Rust and the `thirtyfour` WebDriver library.
A complete Selenium-based end-to-end testing framework using Python and pytest with the Selenium WebDriver library.
### File Structure
```
e2e-tests/
├── Cargo.toml # Rust project configuration with thirtyfour dependencies
├── Cargo.lock # Locked dependency versions
├── .gitignore # Ignore target/ and artifacts
├── README.md # Comprehensive testing documentation
├── SETUP.md # This file
├── run-tests.sh # Helper script for local testing
├── requirements.txt # Python dependencies (selenium, pytest)
├── pytest.ini # Pytest configuration
├── .gitignore # Ignore __pycache__, .pytest_cache, venv
├── README.md # Comprehensive testing documentation
├── SETUP.md # This file
├── run-tests.sh # Helper script for local testing
└── tests/
└── basic_load.rs # Initial test suite
└── test_basic_load.py # Initial test suite
```
### Tests Included
@@ -38,23 +38,25 @@ Three basic tests to verify the application loads correctly:
### CI/CD Integration
Created `.github/workflows/e2e-tests.yml` that:
- Runs on push/PR to master/main branches
- Spins up Selenium standalone Chrome in Docker
- Builds the FossFLOW app
- Serves the built app
- Runs all E2E tests
- Uploads test artifacts
Updated `.github/workflows/e2e-tests.yml` to:
- Run on push/PR to master/main branches
- Set up Python 3.11 with pip caching
- Spin up Selenium standalone Chrome in Docker
- Build the FossFLOW app
- Serve the built app with nohup for persistence
- Install Python test dependencies
- Run all E2E tests with pytest
- Upload test artifacts
### Dependencies
**Rust crates:**
- `thirtyfour` v0.34.0 - WebDriver client
- `tokio` v1.47 - Async runtime
- `anyhow` v1.0 - Error handling
**Python packages:**
- `selenium` v4.27.1 - WebDriver automation library
- `pytest` v8.3.4 - Testing framework
- `pytest-xdist` v3.6.1 - Parallel test execution support
**External services:**
- ChromeDriver or Selenium Server
- Selenium Server (via Docker)
- Running FossFLOW instance
## Quick Start
@@ -62,18 +64,36 @@ Created `.github/workflows/e2e-tests.yml` that:
### Local Development
```bash
# 1. Start Selenium (in Docker)
# Easiest: Use the helper script
cd e2e-tests
./run-tests.sh
# Or manually:
docker run -d -p 4444:4444 --shm-size=2g selenium/standalone-chrome
# The script will:
# - Start Selenium container
# - Create Python venv
# - Install dependencies
# - Prompt you to start the app
# - Run tests
# - Clean up
```
### Manual Setup
```bash
# 1. Start Selenium (in Docker)
docker run -d -p 4444:4444 -p 7900:7900 --shm-size=2g selenium/standalone-chrome
# 2. Start FossFLOW dev server (in another terminal)
npm run dev
# 3. Run tests
# 3. Set up Python environment
cd e2e-tests
cargo test
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
# 4. Run tests
pytest -v
```
### CI/CD
@@ -110,50 +130,114 @@ You can now expand the test suite to cover:
## Example: Adding a New Test
Create `tests/diagram_creation.rs`:
Create `tests/test_diagram_creation.py`:
```rust
use anyhow::Result;
use thirtyfour::prelude::*;
```python
import pytest
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#[tokio::test]
async fn test_can_add_node() -> Result<()> {
let driver = WebDriver::new("http://localhost:4444", DesiredCapabilities::chrome()).await?;
driver.goto("http://localhost:3000").await?;
// Wait for app to load
tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
def test_can_add_node(driver):
"""Test that users can add a node to the canvas."""
driver.get("http://localhost:3000")
// Click the add node button
let add_button = driver.find(By::Css("button[aria-label='Add Node']")).await?;
add_button.click().await?;
# Wait for app to load
wait = WebDriverWait(driver, 10)
// Verify node library appears
let library = driver.find(By::ClassName("node-library")).await?;
assert!(library.is_displayed().await?);
# Click the add node button
add_button = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-label='Add Node']"))
)
add_button.click()
driver.quit().await?;
Ok(())
}
# Verify node library appears
library = wait.until(
EC.visibility_of_element_located((By.CLASS_NAME, "node-library"))
)
assert library.is_displayed()
```
Add to `Cargo.toml`:
Run: `pytest tests/test_diagram_creation.py::test_can_add_node -v`
```toml
[[test]]
name = "diagram_creation"
path = "tests/diagram_creation.rs"
## Pytest Features
### Running Tests
```bash
# Run all tests
pytest -v
# Run specific test file
pytest tests/test_basic_load.py -v
# Run specific test
pytest tests/test_basic_load.py::test_homepage_loads -v
# Run tests matching pattern
pytest -k "canvas" -v
# Run with more verbose output
pytest -vv --tb=long
```
Run: `cargo test test_can_add_node`
### Test Fixtures
The `driver` fixture is automatically available to all tests:
```python
def test_example(driver):
driver.get("http://localhost:3000")
# driver is automatically created and cleaned up
```
## Debugging
### Watch Tests with VNC
Connect to `http://localhost:7900` (password: `secret`) to watch tests run in real-time.
### Run Non-Headless
Edit `test_basic_load.py` and comment out:
```python
# chrome_options.add_argument("--headless")
```
### Add Screenshots on Failure
Add to your test:
```python
def test_example(driver):
try:
# Your test code
assert something
except AssertionError:
driver.save_screenshot("failure.png")
raise
```
## Troubleshooting
See `README.md` for common issues and solutions.
See `README.md` for detailed troubleshooting steps including:
- Connection refused errors
- Element not found errors
- Import errors
- Docker container conflicts
## Resources
- [thirtyfour documentation](https://docs.rs/thirtyfour/)
- [thirtyfour GitHub](https://github.com/Vrtgs/thirtyfour)
- [Selenium documentation](https://www.selenium.dev/documentation/)
- [Selenium Python documentation](https://selenium-python.readthedocs.io/)
- [pytest documentation](https://docs.pytest.org/)
- [Selenium WebDriver docs](https://www.selenium.dev/documentation/webdriver/)
- [WebDriver spec](https://w3c.github.io/webdriver/)
## Migration Notes
This test suite was migrated from Rust (thirtyfour) to Python (selenium + pytest) for:
- Simpler syntax and easier maintenance
- Better debugging tools
- Wider community support
- Faster test development
- More reliable WebDriver connections
e2e-tests/pytest.ini
+6
View File
@@ -0,0 +1,6 @@
[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts = -v --tb=short
e2e-tests/requirements.txt
+3
View File
@@ -0,0 +1,3 @@
selenium==4.27.1
pytest==8.3.4
pytest-xdist==3.6.1
e2e-tests/run-tests.sh
+25 -5
View File
@@ -18,10 +18,17 @@ if ! command -v docker &> /dev/null; then
exit 1
fi
# Check if Rust/Cargo is available
if ! command -v cargo &> /dev/null; then
echo "❌ Rust/Cargo is required but not installed."
echo "Please install Rust from https://rustup.rs/"
# Check if Python is available
if ! command -v python3 &> /dev/null; then
echo "❌ Python 3 is required but not installed."
echo "Please install Python 3 from https://www.python.org/"
exit 1
fi
# Check if pip is available
if ! command -v pip3 &> /dev/null; then
echo "❌ pip3 is required but not installed."
echo "Please install pip3"
exit 1
fi
@@ -65,16 +72,29 @@ fi
echo "✅ FossFLOW app is accessible"
echo ""
# Install Python dependencies if needed
if [ ! -d "venv" ]; then
echo "Creating Python virtual environment..."
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
else
source venv/bin/activate
fi
# Run tests
echo "Running E2E tests..."
echo ""
FOSSFLOW_TEST_URL="http://localhost:$APP_PORT" \
WEBDRIVER_URL="http://localhost:$SELENIUM_PORT" \
cargo test -- --test-threads=1 "$@"
pytest -v --tb=short "$@"
TEST_RESULT=$?
# Deactivate venv
deactivate
# Cleanup
echo ""
echo "Cleaning up..."
e2e-tests/tests/basic_load.rs
-132
View File
@@ -1,132 +0,0 @@
use anyhow::Result;
use thirtyfour::prelude::*;
/// Get the base URL from environment variable or use default localhost
fn get_base_url() -> String {
std::env::var("FOSSFLOW_TEST_URL").unwrap_or_else(|_| "http://localhost:3000".to_string())
}
/// Get the WebDriver URL from environment variable or use default
fn get_webdriver_url() -> String {
std::env::var("WEBDRIVER_URL").unwrap_or_else(|_| "http://localhost:4444".to_string())
}
#[tokio::test]
async fn test_homepage_loads() -> Result<()> {
let base_url = get_base_url();
let webdriver_url = get_webdriver_url();
// Configure Chrome options
let mut caps = DesiredCapabilities::chrome();
caps.set_headless()?;
caps.set_no_sandbox()?;
caps.set_disable_dev_shm_usage()?;
// Connect to WebDriver
let driver = WebDriver::new(&webdriver_url, caps).await?;
// Navigate to the homepage
driver.goto(&base_url).await?;
// Wait for the page to load (give it a moment)
tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
// Get the page title
let title = driver.title().await?;
println!("Page title: {}", title);
// Verify the title contains "FossFLOW" or relevant app name
assert!(
title.to_lowercase().contains("fossflow")
|| title.to_lowercase().contains("isometric")
|| !title.is_empty(),
"Page title should contain 'FossFLOW' or 'isometric', or at least not be empty. Got: '{}'",
title
);
// Check that the page body exists
let body = driver.find(By::Tag("body")).await?;
assert!(body.is_present().await?, "Page body should be present");
// Check for React root element (common in React apps)
let root_exists = driver.find(By::Id("root")).await.is_ok();
assert!(root_exists, "React root element should exist");
println!("✓ Homepage loaded successfully");
println!("✓ Title: {}", title);
println!("✓ Body element present");
println!("✓ React root element present");
// Clean up
driver.quit().await?;
Ok(())
}
#[tokio::test]
async fn test_page_has_canvas() -> Result<()> {
let base_url = get_base_url();
let webdriver_url = get_webdriver_url();
let mut caps = DesiredCapabilities::chrome();
caps.set_headless()?;
caps.set_no_sandbox()?;
caps.set_disable_dev_shm_usage()?;
let driver = WebDriver::new(&webdriver_url, caps).await?;
driver.goto(&base_url).await?;
// Wait for the page to load
tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
// Check for canvas element (isometric drawing should have a canvas)
let canvas_exists = driver.find(By::Tag("canvas")).await.is_ok();
assert!(canvas_exists, "Canvas element should exist for diagram drawing");
println!("✓ Canvas element found on page");
driver.quit().await?;
Ok(())
}
#[tokio::test]
async fn test_page_renders_without_crash() -> Result<()> {
let base_url = get_base_url();
let webdriver_url = get_webdriver_url();
let mut caps = DesiredCapabilities::chrome();
caps.set_headless()?;
caps.set_no_sandbox()?;
caps.set_disable_dev_shm_usage()?;
let driver = WebDriver::new(&webdriver_url, caps).await?;
driver.goto(&base_url).await?;
// Wait for the page to fully load
tokio::time::sleep(tokio::time::Duration::from_secs(5)).await;
// Check multiple elements exist to ensure page rendered properly
let body = driver.find(By::Tag("body")).await?;
assert!(body.is_displayed().await?, "Body should be visible");
let root = driver.find(By::Id("root")).await?;
assert!(root.is_displayed().await?, "Root element should be visible");
// Check for canvas (main drawing area)
let canvas = driver.find(By::Tag("canvas")).await?;
assert!(canvas.is_displayed().await?, "Canvas should be visible");
// Verify we can get the page source (ensures no blank/error page)
let source = driver.source().await?;
assert!(source.len() > 1000, "Page source should be substantial (got {} bytes)", source.len());
println!("✓ Page rendered successfully without crashing");
println!("✓ Page source size: {} bytes", source.len());
driver.quit().await?;
Ok(())
}
e2e-tests/tests/test_basic_load.py
+139
View File
@@ -0,0 +1,139 @@
"""
Basic E2E tests for FossFLOW application.
Tests basic page loading, canvas presence, and rendering.
"""
import os
import time
import pytest
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def get_base_url():
"""Get the base URL from environment or use default."""
return os.getenv("FOSSFLOW_TEST_URL", "http://localhost:3000")
def get_webdriver_url():
"""Get the WebDriver URL from environment or use default."""
return os.getenv("WEBDRIVER_URL", "http://localhost:4444")
@pytest.fixture(scope="function")
def driver():
"""Create a Chrome WebDriver instance for each test."""
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--window-size=1920,1080")
webdriver_url = get_webdriver_url()
# Connect to remote WebDriver (Selenium Grid)
driver = webdriver.Remote(
command_executor=webdriver_url,
options=chrome_options
)
driver.implicitly_wait(10)
yield driver
# Cleanup
driver.quit()
def test_homepage_loads(driver):
"""Test that the homepage loads successfully."""
base_url = get_base_url()
# Navigate to homepage
driver.get(base_url)
# Wait for page to load
time.sleep(3)
# Get page title
title = driver.title
print(f"Page title: {title}")
# Verify title contains relevant keywords or is not empty
assert (
"fossflow" in title.lower() or
"isometric" in title.lower() or
len(title) > 0
), f"Page title should contain 'FossFLOW' or 'isometric', or at least not be empty. Got: '{title}'"
# Check that body exists
body = driver.find_element(By.TAG_NAME, "body")
assert body.is_displayed(), "Page body should be visible"
# Check for React root element
try:
root = driver.find_element(By.ID, "root")
assert root is not None, "React root element should exist"
except Exception as e:
pytest.fail(f"React root element not found: {e}")
print("✓ Homepage loaded successfully")
print(f"✓ Title: {title}")
print("✓ Body element present")
print("✓ React root element present")
def test_page_has_canvas(driver):
"""Test that the page has a canvas element for diagram drawing."""
base_url = get_base_url()
# Navigate to homepage
driver.get(base_url)
# Wait for page to load
time.sleep(3)
# Check for canvas element
try:
canvas = driver.find_element(By.TAG_NAME, "canvas")
assert canvas is not None, "Canvas element should exist for diagram drawing"
print("✓ Canvas element found on page")
except Exception as e:
pytest.fail(f"Canvas element not found: {e}")
def test_page_renders_without_crash(driver):
"""Test that the page renders completely without crashing."""
base_url = get_base_url()
# Navigate to homepage
driver.get(base_url)
# Wait for page to fully load
time.sleep(5)
# Check multiple elements to ensure page rendered properly
body = driver.find_element(By.TAG_NAME, "body")
assert body.is_displayed(), "Body should be visible"
root = driver.find_element(By.ID, "root")
assert root.is_displayed(), "Root element should be visible"
# Check for canvas (main drawing area)
canvas = driver.find_element(By.TAG_NAME, "canvas")
assert canvas.is_displayed(), "Canvas should be visible"
# Verify we can get page source (ensures no blank/error page)
source = driver.page_source
source_len = len(source)
assert source_len > 1000, f"Page source should be substantial (got {source_len} bytes)"
print("✓ Page rendered successfully without crashing")
print(f"✓ Page source size: {source_len} bytes")
if __name__ == "__main__":
pytest.main([__file__, "-v"])