Commit Graph
4 Commits
Author SHA1 Message Date
Toshii 978a8a0aa6 update e2e evals to use cline cli (#6977)
* remove un-implemented tests and create foundation for running cline in cli for exercism

* running version for python language

* remove unused code and reorder benchmark adapter

* remove optional helper functions from BenchmarkAdapter

* unskipping tests for java and javascript

* updating db schema

* updating output to match schema

* functional tests for all languages

* clean up unused commit and stored result

* nits

* small changes to wording

* adding to the test outputs

* using stdin for cline task send

* adding results dir to gitignore

* updating readme

* small nits for readme
2025-10-24 09:21:21 -07:00
pashpashpashandCline Evaluation 430074d0c2 diff evals (#4154)
* cleaning up a bit

* cleaning up some more

* readme

* added max limit

* making it portable

* ignore

* committing plans for now

* strealit hooked up, multi model runs, better db torage

* docs

* VALID attempts

* logging

* more stability

* strategy

* cleaning deps

* docs

* streamlit dashboard work

* dashboard showing bad cases

* better parallelization pt1

* global worker pool for even better more robust parallelization

* bumping up default max parallel requests from 20 -> 80

* better devx

* better docs

* docs

* better devx

* better docs

* better presentation

* dark mode

* removed unused import

---------

Co-authored-by: Cline Evaluation <cline@example.com>
2025-06-20 11:41:16 -07:00
pashpashpashandCline Evaluation 9d9e54360b activating extension with evals.env (#3041)
Co-authored-by: Cline Evaluation <cline@example.com>
2025-04-21 15:06:13 -07:00
pashpashpash ff3c840021 CLI for orchastrating automated evals (#2874)
* cli for evals

* preloading with multiple language extensions

* preloading with multiple language extensions

* moving to my repo

* test server
2025-04-14 19:29:49 -07:00