Jakub Nyckowski ae9eae3df6 Remove usage of tiktoken (#41374) ...

* Remove usage of tiktoken

The token counting mechanism in tokencount.go has been simplified by removing the usage of the tokenizer and instead estimating token count based on character length.

* Update tests

2024-05-09 19:24:02 +00:00

..

embedding

Updated Teleport codebase to AGPL3 license (#35259)

2023-12-01 17:48:14 +00:00

model

Enable errorlint via golangci-lint (#37669)

2024-02-02 15:59:24 +00:00

testutils

Updated Teleport codebase to AGPL3 license (#35259)

2023-12-01 17:48:14 +00:00

tokens

Remove usage of tiktoken (#41374)

2024-05-09 19:24:02 +00:00

chat_test.go

Remove usage of tiktoken (#41374)

2024-05-09 19:24:02 +00:00

chat.go

Updated Teleport codebase to AGPL3 license (#35259)

2023-12-01 17:48:14 +00:00

client_test.go

Updated Teleport codebase to AGPL3 license (#35259)

2023-12-01 17:48:14 +00:00

client.go

Updated Teleport codebase to AGPL3 license (#35259)

2023-12-01 17:48:14 +00:00

embeddingprocessor_test.go

Updated Teleport codebase to AGPL3 license (#35259)

2023-12-01 17:48:14 +00:00

embeddingprocessor.go

Updated Teleport codebase to AGPL3 license (#35259)

2023-12-01 17:48:14 +00:00

mock_embedder.go

Updated Teleport codebase to AGPL3 license (#35259)

2023-12-01 17:48:14 +00:00

simpleretriever_test.go

Updated Teleport codebase to AGPL3 license (#35259)

2023-12-01 17:48:14 +00:00

simpleretriever.go

Updated Teleport codebase to AGPL3 license (#35259)

2023-12-01 17:48:14 +00:00