mirror of
https://github.com/mattermost/mattermost.git
synced 2026-09-21 14:20:41 +08:00
Alejandro García MontoroandMattermost Build
161f0713a4
* Add health flag to fast-fail when ES is offline
When Elasticsearch goes offline, the watcher takes up to 3 health
check cycles (~180 s) to detect the outage and stop the engine.
During that window every search query blocks for 30 s before
falling back to the database, and indexing goroutines pile up
unboundedly — causing server-wide slowness, posting failures,
and duplicate posts from client retries (MM-66612).
Introduce a `healthy` atomic flag on each ES/OpenSearch engine.
The watcher sets it to false on the *first* health-check failure
and back to true on success. `Broker.GetActiveEngines()` now
requires both `IsActive()` and `IsHealthy()`, so all search and
indexing operations skip the unhealthy engine immediately. The
existing 3-failure stop/restart cycle is unchanged and continues
to handle full recovery.
* Fix other tests
* Fix unrelated flaky test
* Use atomic.int32 everywhere
* Revert "Fix unrelated flaky test"
This reverts commit a289015637.
* Improve coverage for ActiveEngine/GetActiveEngines
* Document expectations on SearchEngineInterface
* Use mock.On("call").Unset
* Remove healthCalls to avoid a flaky test
* Be explicit on Unset
* Log any change in the health of the search engine
* Test the healthy<->unhealthy changes are logged
---------
Co-authored-by: Mattermost Build <build@mattermost.com>