8 Commits
Author SHA1 Message Date
pashpashpashandCline Evaluation 50b43c0559 Diff Evals - Replay feature (#4407)
* wip

* ok replays

* replays cooking

* docs

---------

Co-authored-by: Cline Evaluation <cline@example.com>
2025-06-23 18:25:26 -07:00
pashpashpashandCline Evaluation 430074d0c2 diff evals (#4154)
* cleaning up a bit

* cleaning up some more

* readme

* added max limit

* making it portable

* ignore

* committing plans for now

* strealit hooked up, multi model runs, better db torage

* docs

* VALID attempts

* logging

* more stability

* strategy

* cleaning deps

* docs

* streamlit dashboard work

* dashboard showing bad cases

* better parallelization pt1

* global worker pool for even better more robust parallelization

* bumping up default max parallel requests from 20 -> 80

* better devx

* better docs

* docs

* better devx

* better docs

* better presentation

* dark mode

* removed unused import

---------

Co-authored-by: Cline Evaluation <cline@example.com>
2025-06-20 11:41:16 -07:00
Toshii 336eb46547 replay evals (#4140) 2025-06-10 17:11:06 -07:00
Toshii 335086ef36 diff edit evals (#4112)
* base

* multi parallel

* function registry

* nit

* basic prompt

* support data 1

* types

* types

* input format

* path name

* use helpers

* logs

* claude4 prompt

* handling

* evals history

* verbose

* prints

* var

* cli base

* cli inputs

* sqlite

* v3 diff apply

* thinking tokens

* more metrics

* nit

* new structure

* print
2025-06-09 16:01:27 -07:00
pashpashpashandCline Evaluation 90b0d6a73b evals formatting (#3105)
Co-authored-by: Cline Evaluation <cline@example.com>
2025-04-24 15:26:12 -07:00
pashpashpashandCline Evaluation 9d9e54360b activating extension with evals.env (#3041)
Co-authored-by: Cline Evaluation <cline@example.com>
2025-04-21 15:06:13 -07:00
pashpashpash e70264a56c Running Commands Old-School + Living on the Edge with the Latest VSIX (#2999)
* using ndoe shell instead of vs code terminal for commands + always using latest vsix

* 30s max time for commands in test mode

* removed overwhelming logs

* better 30s termination
2025-04-21 11:00:35 -07:00
pashpashpash ff3c840021 CLI for orchastrating automated evals (#2874)
* cli for evals

* preloading with multiple language extensions

* preloading with multiple language extensions

* moving to my repo

* test server
2025-04-14 19:29:49 -07:00