Files
cline/.github/workflows/test.yml
T
b6f6358d4a Set up E2E tests with Playwright (#4721)
* Add Playwright E2E tests

Adding end-to-end (E2E) testing capabilities using Playwright. It also updates the `@vscode/test-electron` dependency.

The changes include:

- Adding Playwright as a dev dependency.
- Adding `e2e` and `e2e:build` scripts to `package.json` for running E2E tests.
- Adding `@playwright/test` to the list of dependencies.
- Updating `@vscode/test-electron` from `2.4.1` to `2.5.2`.
- Adding `test-results` to `.gitignore` to exclude test result files.

* wip: github workflow

Adding a new GitHub Actions workflow for running end-to-end (E2E) tests using Playwright. The workflow is triggered on push to the main branch, pull requests, and manual workflow dispatch.

The workflow defines a matrix strategy to run tests on different runners (Ubuntu and Windows) and shards. It also uploads Playwright recordings as artifacts if the tests fail.

* add @vscode/vsce as dev dep

* update workflow

* apply feedback

* fix test workflows

* add command palette helper

This commit improves the reliability and efficiency of the end-to-end tests by:

- Adding a delay to the "Let's go!" button click in the auth test to ensure the action is properly registered.
- Adding an expectation to ensure the "Get Started for Free" button is no longer visible after API key submission.
- Caching the "Use your own API key" button to avoid redundant lookups.
- Introducing a `runCommandPalette` helper function to streamline command execution within the VS Code environment.
- Disabling notifications before running the tests to prevent interference.

* state change

* set TEMP_PROFILE

* v3.18.7 Release Notes

* changeset version bump

* Updating CHANGELOG.md format

* Update CHANGELOG.md for version 3.18.7

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: pashpashpash <nik@cline.bot>

* Remove optimistic loading from organization dropdown (#4746)

* update build script to javascript

* fix match

* Add mode switching to chat test

* expected

---------

Co-authored-by: abeatrix <beatrix@cline.bot>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: pashpashpash <nik@cline.bot>
Co-authored-by: canvrno <46584286+canvrno@users.noreply.github.com>
2025-07-16 07:29:58 +05:30

231 lines
9.0 KiB
YAML

name: Tests
on:
workflow_dispatch:
pull_request:
branches:
- main
workflow_call:
# Set default permissions for all jobs
permissions:
contents: read # Needed to check out code
checks: write # Needed to report test results
pull-requests: write # Needed to add comments/annotations to PRs
jobs:
test:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
runs-on: ${{ matrix.os }}
name: ${{ matrix.os == 'ubuntu-latest' && 'test' || format('test ({0})', matrix.os) }}
defaults:
run:
shell: bash
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js environment
uses: actions/setup-node@v4
with:
node-version: 22
# Setup Python for coverage script
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Install Python dependencies
run: |
python -m pip install --upgrade pip
pip install requests
# Cache root dependencies - only reuse if package-lock.json exactly matches
- name: Cache root dependencies
uses: actions/cache@v4
id: root-cache
with:
path: node_modules
key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}
# Cache webview-ui dependencies - only reuse if package-lock.json exactly matches
- name: Cache webview-ui dependencies
uses: actions/cache@v4
id: webview-cache
with:
path: webview-ui/node_modules
key: ${{ runner.os }}-npm-webview-${{ hashFiles('webview-ui/package-lock.json') }}
- name: Install root dependencies
if: steps.root-cache.outputs.cache-hit != 'true'
run: npm ci
- name: Install webview-ui dependencies
if: steps.webview-cache.outputs.cache-hit != 'true'
run: cd webview-ui && npm ci
- name: Install xvfb on Linux
if: runner.os == 'Linux'
run: sudo apt-get update && sudo apt-get install -y xvfb
- name: Install local modules on windows
if: runner.os == 'Windows' && steps.root-cache.outputs.cache-hit == 'true'
run: |
npm install eslint-plugin-eslint-rules
cd webview-ui/ && npm install eslint-plugin-eslint-rules
- name: Set up NPM on Windows
if: runner.os == 'Windows'
run: |
npm config set script-shell "C:\\Program Files\\Git\\bin\\bash.exe"
- name: Type Check
run: npm run check-types
- name: ESLint Check
run: npm run lint
- name: Prettier / Format Check
run: npm run format
# Build the extension before running tests
- name: Build Tests and Extension
run: npm run pretest
# Unit Tests disabled due to module system conflicts between backend and webview-ui
# - name: Unit Tests
# run: npm run test:unit
# Run extension tests with coverage
- name: Extension Tests with Coverage
id: extension_coverage
continue-on-error: true
run: |
node ./scripts/test-ci.js > extension_coverage.txt 2>&1
# Default the encoding to UTF-8 - It's not the default on Windows
PYTHONUTF8=1 PYTHONPATH=.github/scripts python -m coverage_check extract-coverage extension_coverage.txt --type=extension --github-output --verbose
# Run webview tests with coverage
- name: Webview Tests with Coverage
id: webview_coverage
continue-on-error: true
run: |
cd webview-ui
# Ensure coverage dependency is installed
npm install --no-save @vitest/coverage-v8
npm run test:coverage > webview_coverage.txt 2>&1
cd ..
# Default the encoding to UTF-8 - It's not the default on Windows
PYTHONUTF8=1 PYTHONPATH=.github/scripts python -m coverage_check extract-coverage webview-ui/webview_coverage.txt --type=webview --github-output --verbose
# Save coverage reports as artifacts (workflow-scoped)
- name: Save Coverage Reports
uses: actions/upload-artifact@v4
# Only upload artifacts on Linux - We only need coverage from one OS
if: runner.os == 'Linux'
with:
name: pr-coverage-reports
path: |
extension_coverage.txt
webview-ui/webview_coverage.txt
retention-period: workflow # Artifacts are automatically deleted when the workflow completes
# Set the check as failed if any of the tests failed
- name: Print test results and check for failures
run: |
echo "Extension Tests Result: ${{ steps.extension_coverage.outcome }}"
cat extension_coverage.txt
echo "Webview Tests Result: ${{ steps.webview_coverage.outcome }}"
cat webview-ui/webview_coverage.txt
# Check if any of the test steps failed
# https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/accessing-contextual-information-about-workflow-runs#steps-context
if [ "${{ steps.extension_coverage.outcome }}" != "success" ] || [ "${{ steps.webview_coverage.outcome }}" != "success" ]; then
echo "Tests failed."
exit 1
fi
coverage:
needs: test
runs-on: ubuntu-latest
# Only run on PRs to main branch
if: github.event_name == 'pull_request' && github.base_ref == 'main'
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0 # Fetch all history for accurate comparison
# Setup Python for coverage script
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Install Python dependencies
run: |
python -m pip install --upgrade pip
pip install requests
- name: Setup Node.js environment
uses: actions/setup-node@v4
with:
node-version: 22
# Cache root dependencies - only reuse if package-lock.json exactly matches
- name: Cache root dependencies
uses: actions/cache@v4
id: root-cache
with:
path: node_modules
key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}
# Cache webview-ui dependencies - only reuse if package-lock.json exactly matches
- name: Cache webview-ui dependencies
uses: actions/cache@v4
id: webview-cache
with:
path: webview-ui/node_modules
key: ${{ runner.os }}-npm-webview-${{ hashFiles('webview-ui/package-lock.json') }}
- name: Install root dependencies
if: steps.root-cache.outputs.cache-hit != 'true'
run: npm ci
- name: Install webview-ui dependencies
if: steps.webview-cache.outputs.cache-hit != 'true'
run: cd webview-ui && npm ci
# Build the extension before running tests
- name: Build Extension
run: npm run compile
# Download coverage artifacts from test job
- name: Download Coverage Reports
uses: actions/download-artifact@v4
with:
name: pr-coverage-reports
path: . # Download to root directory to match expected paths
# Process coverage workflow
- name: Process coverage workflow
id: coverage
run: |
# Extract PR number from GITHUB_REF
PR_NUMBER=$(echo "$GITHUB_REF" | sed -e 's/refs\/pull\///' -e 's/\/merge//')
# Run the coverage workflow from root directory
PYTHONPATH=.github/scripts python -m coverage_check process-workflow \
--base-branch ${{ github.base_ref }} \
--pr-number $PR_NUMBER \
--repo $GITHUB_REPOSITORY \
--token ${{ secrets.GITHUB_TOKEN }} \
--verbose
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}