Commit Graph
40 Commits
Author SHA1 Message Date
william chang(張仲威) 9738e7ad1f release 0.4.1 (#419) 2024-06-14 20:05:39 +08:00
william chang(張仲威) 38a0d0e255 fix(docker): remove redis (#418) 2024-06-14 20:02:38 +08:00
Andy Yen 291e374580 release 0.4.0 (#416) 2024-06-14 19:07:08 +08:00
Andy Yen b652515de6 release 0.4.0-rc.4 (#415) 2024-06-14 17:07:19 +08:00
Andy Yen b014e686e3 release 0.4.0-rc.3 (#409) 2024-06-14 11:14:45 +08:00
Andy Yen 3c8741f200 release 0.4.0-rc-2 (#405) 2024-06-13 07:56:59 +08:00
Andy YenandShimin f4e4551498 Epic: integrate with ibis-server (#403)
* Chore: integrate with ibis server (#306)

* feat: Add ibis adaptor for querying supported data sources

* feat: Add testIbis resolver for querying supported data sources

* feat: use queryService for preview and describe statement

* feat: Refactor ibisAdaptor to use DataSourceName enum consistently

* test ibis using tpch query

* rm test ibis

* feat: Add previewSql resolver for SQL preview

* rm debug log

* use projectId and manifest to generate hash
Implement getMDL api

* pass mdl to ibis server to preview

* transform manifest data type to json object when retrieved from database

* better error message from Ibis server, rewrite host when development and using docker

* use base64 str to transfer mdl

* update README

* change env name to fit naming style.
update docker-compose.x.yaml and .env.example file

* fix yarn build issue

* update docker-compose file: fix incorrent ibis server endpoint

* port forward to ibis server

* fix incorrect migration rollback function

* update engine version

* Chore: refactor database schema to reflect how we use them. (#341)

* add connectionInfo column and do data migration

* drop deprecated columns that store connection info, use connectionInfo column instead

* do data migration to store tableReference in model's properties

* fix incorrect migration rollback function

* use tableReference in mdl instead of refSql

* chore: Update variable names in ibisAdaptor and queryService

* chore: Update variable names in ibisAdaptor and queryService

* fix lint

* rename variable in ibis adaptor

* - remove duplicate logic to parse string
- add comment

* release 0.4.0-rc.1 (#378)

* fix: fix nested resolver to format sql (#379)

* feat(wren-ui): Provide limit property in preview data API for excel-add-in used (#373)

* fix(wren-ui): add displayName to all view API which share ViewInfo type in graphQL

* feat(wren-ui): provide limit property of preview API for usage in excel-add-in

* Chore: minimized the diversity of data sources from wren-ui  (#384)

* chore: Update ibisAdaptor to include methods for getting tables and constraints

* remove strategies and connectors from codebase, use ibis server and wrenEngine to fetch metadata of data source

* add validation rule api in ibisAdapter

* replace invalid char when generating referenceName

* fix comments

* Feature: support dry run in preview API (#402)

* can dry-run  with previewSql

* update env.example and docker-compose-dev.yaml

---------

Co-authored-by: Shimin <andreashimin@gmail.com>
2024-06-12 21:19:05 +08:00
Pao Sheng b86cd6f3e2 feature(ai-service): wren-ui dry-run SQL API instead of wren-engine API (#400)
* feat(ai-service): change dryrun api

* feat: change the docker env
2024-06-12 20:20:43 +08:00
a362757431 azure openai provider implementation (#390)
Co-authored-by: Siddharth Kumar <ybsid@Siddharths-MacBook-Pro.local>
Co-authored-by: Chih-Yu Yeh <chihyu.jimmy.yeh@gmail.com>
2024-06-12 19:49:04 +08:00
Chih-Yu Yeh 8d8bccfbd7 Chore/ai service/minor update (#382)
* update

* use redis

* update

* fix document store duplicate collection issue

* update timer setting

* fix pre_run and comment out qurey_understanding pipeline

* add redis container

* update redis config

* update redis config

* update redis config

* fix bug

* update dummy api

* add DummyAskUser

* update

* add load test observation notes

* refine wording

* update

* fix missing env

* fix bug
2024-06-06 18:29:30 +08:00
Chih-Yu Yeh e96c928b36 Chore/ai service/minor updates (#377)
* restructure test folder

* fix WREN_ENGINE_ENDPOINT issue

* add async timer

* update

* use logger.info

* use asyncio.sleep

* use asyncio.sleep

* fix timer to async_timer

* add dummy endpoint

* fix time.sleep

* add WORKER

* add timer

* add LOGGING_LEVEL env

* simplify code

* change to async text/document embedder

* simplify code

* update configs and fix timer
2024-06-05 14:40:35 +08:00
HoupusandChih-Yu Yeh 7decf5fbc4 Support 3rd party openai api (#365)
* Support for specifying third-party OPENAI proxy services

* Remove unnecessary changes

* Support for specifying third-party OPENAI proxy services

* update get_generator method

---------

Co-authored-by: Chih-Yu Yeh <chihyu.jimmy.yeh@gmail.com>
2024-06-04 21:38:23 +08:00
Andy Yen ae81e89773 release 0.3.6 (#338) 2024-05-30 13:18:19 +08:00
william chang(張仲威) 2f7cf808de release 0.3.5 (#334) 2024-05-29 16:46:38 +08:00
william chang(張仲威) a60d1be585 release 0.3.5-rc.2 (#333) 2024-05-29 15:36:57 +08:00
william chang(張仲威) 5b9fde34f3 release 0.3.5-rc.1 (#329) 2024-05-28 18:27:58 +08:00
Chih-Yu Yeh e724bd8ad0 add WREN_ENGINE_VERSION (#303) 2024-05-22 18:28:09 +08:00
william chang(張仲威) f6e1746fc9 release 0.3.4 (#293) 2024-05-21 19:31:57 +08:00
william chang(張仲威) 21413553b4 release 0.3.4-rc.5 (#291) 2024-05-21 18:48:53 +08:00
william chang(張仲威) bbb2f743d1 release 0.3.4-rc.4 (#290) 2024-05-21 18:00:51 +08:00
william chang(張仲威) 868c8f5101 release 0.3.4-rc.3 (#286) 2024-05-21 13:06:06 +08:00
william chang(張仲威) 14b9a87566 fix(wren-launcher): fix adding properties to config.properties (#285) 2024-05-21 13:03:18 +08:00
william chang(張仲威) c871a4d6c4 release 0.3.4-rc.2 (#284) 2024-05-21 12:46:35 +08:00
william chang(張仲威) c3e9579766 Release/0.3.4 rc.1 (#282)
* release 0.3.4-rc.1

* update WREN_BOOTSTRAP_VERSION to 0.1.3
2024-05-21 12:25:26 +08:00
william chang(張仲威) 834a953255 feature(wren-launcher): enable dynamic-fields config for Wren Engine (#280) 2024-05-21 12:11:14 +08:00
Chih-Yu Yeh 1fac9b76f3 chore/small refinements to launcher (#269)
* fix missing envs to posthog

* fix .env.example

* update add openai key instructions

* add missing env vars
2024-05-20 16:17:02 +08:00
Chih-Yu Yeh 66bfe8a95e Chore(wren-ui): record openai generation model used by user (#254)
* record openai generation model used by user

* remove unused comments
2024-05-15 11:20:10 +08:00
Chih-Yu Yehandwilliam chang d75f6d1ed7 allow users to change openai model in launcher (#244)
* ..

* add logs for init_providers

* fix ci

* add openai model url instructions

* fix empty generation model if OPENAI_GENERATION_MODEL is not provided

* Update wren-launcher/utils/docker.go

Co-authored-by: william chang(張仲威) <wwwy3y3@gmail.com>

---------

Co-authored-by: william chang(張仲威) <wwwy3y3@gmail.com>
2024-05-14 08:49:15 +08:00
Chih-Yu Yeh e0fa7dcb91 refine demo site for easier debugging (#174) 2024-04-29 11:23:12 +08:00
william chang(張仲威)andonlyjackfrost 3c4a6f28a6 Feature: support PG connector (#135)
* feat(wren-ui): add new migration file to support pg in project

* feat(wren-ui): add new pg connector

* feat(wren-ui): install required deps to test pg connector

* feat(wren-ui): refactor strategy a bit (put strategy of data sources into different files) and add pg strategy

* feat(wren-ui): add pg support in repository

* feat(wren-ui): support pg in graphql schema

* feat(wren-ui): fix connection reusing issue and add finalization stage for testing

* feat(wren-ui): change to a better file names & structure in tests

* feat(wren-ui): comment out postgres part in docker-compose-dev

* fix(wren-ui): column properties be empty object in pg connector

---------

Co-authored-by: onlyjackfrost <onlyjackfrost@gmail.com>
2024-04-19 19:05:01 +08:00
Chih-Yu Yeh cf02536bb9 Feature/wren ai service/demo lab (#128)
* setup duckdb demo dataset testing

* allow demo to use duckdb demo dataset

* remove debug message

* remove unused command

* update wren-ai-service demo setup and README

* reformat sql, prevent duplicating deployment, update sampledata mdl

* update logger setup and fix preview sql issue

* remove psycopg2

* refine demo setup flow

* revert

* remove unused file

* remove unused dep

* change url

* revert

* add TODO for improvement

* update
2024-04-18 15:51:09 +08:00
william chang(張仲威) 08af071345 feat(wren-launcher): add rc utility and ensure pg_password generated when deploying services 2024-04-16 11:52:55 +08:00
william chang(張仲威) 7d66b68ada feat(wren-launcher): put accounts file into volume, fix file format 2024-04-16 11:52:55 +08:00
william chang(張仲威) a59bafc7f4 feat(wren-launcher): add pg_username & password to accounts file at bootstrap 2024-04-16 11:52:55 +08:00
9507838991 Feature(ai service): refine ask/ask details pipelines (#64)
* allow visualization using one file

* remove default argparse option value

* resolve conflict

* resolve conflict

* resolve conflict

* update setup instructions

* resolve conflict

* resolve conflicts

* resolve conflict

* resolve conflict

* resolve conflict

* add anthropic model pricing

* allow change model for anthropic

* resolve conflict

* resolve conflict

* resovle conflict

* resovle conflict

* resolve conflict

* resolve conflict

* add container:ubuntu

* undo

* resolve conflict

* resolve conflict

* resolve conflict

* resolve conflict

* resolve conflict

* update

* resolve conflict

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* remove unused files

* resolve conflict

* resolve conflict

* generate ddl from mdl for indexing

* resolve conflict

* resolve conflicts

* eval ask with semantic description (#45)

* update make eval-ask

* add semantic description to mdl before eval

* refine

* add model semantic generation

* simplify make command

* print timestamp

* minor update

---------

Co-authored-by: qa <qa@qadeMacBook-Pro.local>
Co-authored-by: ChihYu Yeh <chihyu.jimmy.yeh@gmail.com>
Co-authored-by: Aster Sun <aster.sun@cannerdata.com>

* remove ;

* simplify prompt

* revert topk

* update prompt

* make default top_k for retriever 10

* generate multiple results for ask pipeline

* resolve conflict

* add followup_generation_pipeline

* add logging for ask api results

* fix generate_mdl issue

* fix ask_details error message

* add extra standard package for uvicorn

* refine ask

* fix postprocessor for ask details

* fix preview data

* fix test

* update prompts

* refine prompts for ask and ask details

* fix prompt

* refine prompt

* remove slash at the end of api endpoints

* refine prompt

* add custom semantic run option

* resolve conflict

* fix bugs for tests and eval and restructure eval-ask

* fix trainling slash and add logging

* remove unused code

* restructure eval ask pipeline outputs

* add comments to the eval command

* change generation component argument name

* add try/except to handle other errors that might happen

* change WREN_AI_SERVICE_VERSION to nightly

* fix eval ask_details errors

* add error message to indexing

---------

Co-authored-by: imAsterSun <61279528+imAsterSun@users.noreply.github.com>
Co-authored-by: qa <qa@qadeMacBook-Pro.local>
Co-authored-by: Aster Sun <aster.sun@cannerdata.com>
Co-authored-by: Pao Sheng <paooap.oappao@gmail.com>
2024-04-08 15:44:15 +08:00
Chih-Yu Yeh 69263e6dd9 Chore/docker compose update (#69)
* update docker related settings

* change WREN_AI_SERVICE_PORT default 5555

* add pull_policy, restart condition and update wren-ai-service port

* fix default port for wren-ai-service
2024-04-03 15:57:09 +08:00
william chang(張仲威) 4ff1ae0678 Feature/home api (#41)
* feat(wren-ui): wrenAIAdaptor

* feat(wren-ui): migration file to add thread tables

* feat(wren-ui): ask feature & thread graphql resolver

* feat(wren-ui): thread repository

* feat(wren-ui): finish ask & thread api

* docker-compose for dev

* fix(wren-ui): fix some transformation issues in API

* fix(wren-ui/ask-feature): fix business logics in thread API

* fix(wren-ui): revert some configs back

* feat(wren-ui/ask): add follow-up question support

* support update/delete thread

* feat(wren-ui/ask): finish preview data & show full sql apis

* feat(wren-ui/ask): use allSettle to print error message if a job failed

* feat(wren-ui/ask): remove showsql mutation & put sql into threadResponse.detail

* fix(wren-ui/ask): fix lint issue

* fix(apollo): handle error formating

* error handling

* change of to create in error
2024-04-01 18:19:36 +08:00
Chih-Yu YehandPao Sheng 32cd3df311 Feature/ai service/fix failed sql queries + add error code (#50)
* resolve conflict

* allow visualization using one file

* remove default argparse option value

* resolve conflict

* resolve conflict

* resolve conflict

* update setup instructions

* resolve conflict

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* add anthropic model pricing

* allow change model for anthropic

* resolve conflicts

* resolve conflicts

* remove invalid sql generation results

* WIP: add sql correction

* add sql correction pipeline

* update ai service ci yaml

* update eval

* add instructions for running tests

* add comments for cte_name validation for last step

* use wren-engine api to validate sql instead of wren-engine psql

* simplify testing instructions

* update testing step

* create missing dir

* fix commands

* add WREN_ENGINE_ENDPOINT

* update WREN_ENGINE_ENDPOINT

* add container:ubuntu

* undo

* changed to localhost

* adjust docker network

* test ci

* test ci

* update

* update

* update

* revert

* update

* revert

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix bug

* update

* remove contest

* Feature/ai service/add error code (#40)

* resolve conflict

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* resolve conflicts

* refine

* remove unused files

* resolve conflicts

* add error code and post processor to ask details

* add volume for qdrant and WREN_ENGINE_ENDPOINT for wren-ai-service

* add status check for ask service and fix tests

* Update wren-ai-service/tests/services/test_ask_details.py

Co-authored-by: Pao Sheng <paooap.oappao@gmail.com>

* Update wren-ai-service/tests/services/test_ask_details.py

Co-authored-by: Pao Sheng <paooap.oappao@gmail.com>

---------

Co-authored-by: Pao Sheng <paooap.oappao@gmail.com>

---------

Co-authored-by: Pao Sheng <paooap.oappao@gmail.com>
2024-03-28 15:11:30 +08:00
william chang(張仲威) 3234dc218b Feature/wren launcher (#33)
* feat(dockerfile): add standalone option. fix eslint issues at next.config.js

* feat(docker): add docker-compose file

* feat(launcher): init project

* feat(launcher): use docker-compose sdk to run containers

* feat(laucner): wait until checking if CheckWrenAIStarted return without error

* feat(launcher): CheckDockerDaemonRunning

* feat(laucher): format the message

* feat(launcher): fix regex replace api key

* readme
2024-03-25 17:42:20 +08:00
william chang(張仲威) ffbb2c4084 Feature: build wren-ui docker image & docker-compose to start all service we need (#27)
* feat(dockerfile): add standalone option. fix eslint issues at next.config.js

* feat(docker): add docker-compose file

* feat(docker): add QDRANT_HOST

* feat(docker): modify docker-compose to add a bootstrap runner

* feat(docker): bootstrap service

* feat(wren-ui): add dockerfile for wren-ui

* fix(wren-ui): rm docker entry script

* feat(docker): publish ui & bootstrap packages

* feat(docker): remove comment in dockerfile & put up a how to start in readme

* fix(docker): change ai-service port to 80
2024-03-22 16:39:48 +08:00