Set up E2E tests with Playwright (#4721)

* Add Playwright E2E tests

Adding end-to-end (E2E) testing capabilities using Playwright. It also updates the `@vscode/test-electron` dependency.

The changes include:

- Adding Playwright as a dev dependency.
- Adding `e2e` and `e2e:build` scripts to `package.json` for running E2E tests.
- Adding `@playwright/test` to the list of dependencies.
- Updating `@vscode/test-electron` from `2.4.1` to `2.5.2`.
- Adding `test-results` to `.gitignore` to exclude test result files.

* wip: github workflow

Adding a new GitHub Actions workflow for running end-to-end (E2E) tests using Playwright. The workflow is triggered on push to the main branch, pull requests, and manual workflow dispatch.

The workflow defines a matrix strategy to run tests on different runners (Ubuntu and Windows) and shards. It also uploads Playwright recordings as artifacts if the tests fail.

* add @vscode/vsce as dev dep

* update workflow

* apply feedback

* fix test workflows

* add command palette helper

This commit improves the reliability and efficiency of the end-to-end tests by:

- Adding a delay to the "Let's go!" button click in the auth test to ensure the action is properly registered.
- Adding an expectation to ensure the "Get Started for Free" button is no longer visible after API key submission.
- Caching the "Use your own API key" button to avoid redundant lookups.
- Introducing a `runCommandPalette` helper function to streamline command execution within the VS Code environment.
- Disabling notifications before running the tests to prevent interference.

* state change

* set TEMP_PROFILE

* v3.18.7 Release Notes

* changeset version bump

* Updating CHANGELOG.md format

* Update CHANGELOG.md for version 3.18.7

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: pashpashpash <nik@cline.bot>

* Remove optimistic loading from organization dropdown (#4746)

* update build script to javascript

* fix match

* Add mode switching to chat test

* expected

---------

Co-authored-by: abeatrix <beatrix@cline.bot>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: pashpashpash <nik@cline.bot>
Co-authored-by: canvrno <46584286+canvrno@users.noreply.github.com>

.github/workflows/e2e.yml

@@ -0,0 +1,88 @@
+name: E2E Tests
+
+on:
+    push:
+        branches:
+            - main
+    pull_request:
+        types: [opened, reopened, synchronize, ready_for_review]
+    workflow_dispatch:
+
+concurrency:
+    group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+    cancel-in-progress: true
+
+jobs:
+    matrix_prep:
+        runs-on: ubuntu-latest
+        outputs:
+            matrix: ${{ steps.set-matrix.outputs.matrix }}
+        steps:
+            - id: set-matrix
+              run: |
+                  echo 'matrix=[{"runner":"ubuntu"},{"runner":"windows"},{"runner":"macos"}]' >> $GITHUB_OUTPUT
+
+    e2e:
+        needs: matrix_prep
+        strategy:
+            fail-fast: false
+            matrix:
+                include: ${{ fromJson(needs.matrix_prep.outputs.matrix) }}
+        runs-on: ${{ matrix.runner }}-latest
+        timeout-minutes: 20
+        permissions:
+            id-token: write
+            contents: read
+        steps:
+            - uses: actions/checkout@v4
+            - name: Setup Node.js environment
+              uses: actions/setup-node@v4
+              with:
+                  node-version: 22
+
+            # Cache root dependencies - only reuse if package-lock.json exactly matches
+            - name: Cache root dependencies
+              uses: actions/cache@v4
+              id: root-cache
+              with:
+                  path: node_modules
+                  key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}
+
+            # Cache webview-ui dependencies - only reuse if package-lock.json exactly matches
+            - name: Cache webview-ui dependencies
+              uses: actions/cache@v4
+              id: webview-cache
+              with:
+                  path: webview-ui/node_modules
+                  key: ${{ runner.os }}-npm-webview-${{ hashFiles('webview-ui/package-lock.json') }}
+
+            - name: Install root dependencies
+              if: steps.root-cache.outputs.cache-hit != 'true'
+              run: npm ci
+
+            - name: Install webview-ui dependencies
+              if: steps.webview-cache.outputs.cache-hit != 'true'
+              run: cd webview-ui && npm ci
+
+            - name: Install xvfb on Linux
+              if: matrix.runner == 'ubuntu'
+              run: sudo apt-get update && sudo apt-get install -y xvfb
+
+            # Build the extension before running tests
+            - name: Build Tests and Extension
+              run: npm run pretest
+
+            - name: Run E2E tests - Linux
+              if: matrix.runner == 'ubuntu'
+              run: xvfb-run -a npm run test:e2e
+
+            - name: Run E2E tests - Non-Linux
+              if: matrix.runner != 'ubuntu'
+              run: npm run test:e2e
+
+            - uses: actions/upload-artifact@v4
+              if: ${{ failure() }}
+              with:
+                  name: playwright-recordings-${{ matrix.runner }}
+                  path: |
+                      test-results/playwright/

.github/workflows/test.yml

@@ -68,6 +68,10 @@ jobs:
               if: steps.webview-cache.outputs.cache-hit != 'true'
               run: cd webview-ui && npm ci
 
+            - name: Install xvfb on Linux
+              if: runner.os == 'Linux'
+              run: sudo apt-get update && sudo apt-get install -y xvfb
+
             - name: Install local modules on windows
               if: runner.os == 'Windows' && steps.root-cache.outputs.cache-hit == 'true'
               run: |