docs(P13f): reconcile docs drift — Administration Guide: Scale (#37481)

* docs(P13f): reconcile scale/search/monitoring drift from legacy docs repo Ports content drift from mattermost/docs (Sphinx) into the migrated MDX pages under administration-guide/scale: Elasticsearch v9 support and v7.17 deprecation, corrected AWS OpenSearch access policy guidance (open principal instead of an unsupported IAM-principal policy) plus an OpenSearch v1.x deprecation notice, new search-engine outage health-monitor documentation, and plugin-metrics /metrics aggregation notes. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(P13f): update scale-to-N-users Azure DB SKUs to Intel E*ds v5 Replaces the Azure RDS reader/writer instance SKU (E*as v6, Intel "as v6" family) with E*ds v5 (Intel "Eds v5" family) across all nine scale-to-N-users reference architecture pages, matching the upstream correction in mattermost/docs. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-19 02:06:37 +08:00 · 2026-07-21 10:29:56 +03:00
parent ef7bafc48c
commit fc18ad4e6e
15 changed files with 94 additions and 38 deletions
@@ -38,7 +38,7 @@ For high post volume deployments, we strongly encourage you to read and properly

 <Warning>

-For high post volume deployments, we also strongly recommend *disabling* Database Search once Elasticsearch or AWS OpenSearch is fully configured and running. The Mattermost Server will fall back on Database search if ElasticSearch or OpenSearch are unavailable which can lead to performance degradation on high post volume deployments.
+For high post volume deployments, we also strongly recommend *disabling* Database Search once Elasticsearch or AWS OpenSearch is fully configured and running. The Mattermost Server will fall back on Database search if Elasticsearch or OpenSearch are unavailable, which can lead to performance degradation on high post volume deployments. From Mattermost v11.7, the server proactively detects outages through periodic health checks and falls back to database search on the first health check failure, rather than waiting for requests to time out. See the [outage handling FAQ](/administration-guide/scale/enterprise-search#how-does-mattermost-handle-elasticsearch-or-opensearch-outages) for details.

 </Warning>

@@ -129,6 +129,8 @@ See [this Grafana guide](https://grafana.com/docs/grafana/v7.5/dashboards/export

 Mattermost provides [custom metrics](/administration-guide/scale/performance-monitoring-metrics#custom-mattermost-metrics) and [standard Go metrics](/administration-guide/scale/performance-monitoring-metrics#standard-go-metrics) that can be used to monitor your system's performance.

+When the `AggregatePluginMetrics` feature flag is enabled, plugin-provided metrics are included in the same `/metrics` scrape target and can be filtered by the `plugin_id` label.
+
 ## Next steps

 Once you've set up performance monitoring, you may want to:
@@ -13,7 +13,7 @@ Deploying Elasticsearch includes the following two steps: [setting up Elasticsea

 We highly recommend that you set up Elasticsearch server on a dedicated machine separate from the Mattermost Server.

-1.  Download and install the latest release of [Elasticsearch v8](https://www.elastic.co/guide/en/elasticsearch/reference/8.15/install-elasticsearch.html), or [Elasticsearch v7.17+](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/install-elasticsearch.html). See the Elasticsearch documentation for installation details.
+1.  Download and install the latest release of [Elasticsearch v9](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) or [Elasticsearch v8](https://www.elastic.co/guide/en/elasticsearch/reference/8.19/install-elasticsearch.html). See the Elasticsearch documentation for installation details.
 2.  Set up Elasticsearch with `systemd` by running the following commands:

 > ``` sh
@@ -41,7 +41,7 @@ We highly recommend that you set up Elasticsearch server on a dedicated machine
 > ```

 6.  In this file, replace the `network.host` value of `_eth0_` with your network interface name, and save your changes.
-7.  When using Elasticsearch v8, ensure you set `action.destructive_requires_name` to `false` in `elasticsearch.yml` to allow for wildcard operations to work.
+7.  Ensure you set `action.destructive_requires_name` to `false` in `elasticsearch.yml` to allow for wildcard operations to work.
 8.  Restart Elasticsearch by running the following commands:

 > ``` sh
@@ -29,9 +29,20 @@ The AWS OpenSearch Service is built on the open-source OpenSearch project, which
 Review the following support paths for enterprise search based on the version you're using:

 <Tabs>
+<TabItem value="elasticsearch-v9" label="Elasticsearch v9">
+
+[Elasticsearch v9](https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html) is supported from Mattermost v11.7. We recommend upgrading to Elasticsearch v9 for the latest features and security updates. See the [Elasticsearch upgrade](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html) documentation for upgrade details, and see the [Elasticsearch setup](/administration-guide/scale/elasticsearch-setup) documentation for details on configuring your Mattermost deployment to use Elasticsearch.
+
+</TabItem>
 <TabItem value="elasticsearch-v8" label="Elasticsearch v8">

-[Elasticsearch v8](https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html) is supported from Mattermost v9.11. While Mattermost supports Elasticsearch v7.17+, we recommend upgrading your Elasticsearch v7 instance to v8.x. See the [Elasticsearch upgrade](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html) documentation for upgrade details, and see the [Elasticsearch setup](/administration-guide/scale/elasticsearch-setup) documentation for details on configuring your Mattermost deployment to use Elasticsearch.
+[Elasticsearch v8](https://www.elastic.co/guide/en/elasticsearch/reference/8.19/elasticsearch-intro.html) is supported from Mattermost v9.11. We recommend upgrading to Elasticsearch v9 for the latest features and security updates. See the [Elasticsearch upgrade](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html) documentation for upgrade details, and see the [Elasticsearch setup](/administration-guide/scale/elasticsearch-setup) documentation for details on configuring your Mattermost deployment to use Elasticsearch.
+
+<Note>
+
+Elasticsearch v7.17 is no longer supported as of Mattermost v11.7. If you're still using Elasticsearch v7, you must upgrade to Elasticsearch v8 or v9.
+
+</Note>

 </TabItem>
 <TabItem value="aws-opensearch-service" label="AWS OpenSearch Service">
@@ -43,13 +54,13 @@ See the **AWS Elasticsearch v7.10.x** tab on this page for details on upgrading
 </TabItem>
 <TabItem value="aws-elasticsearch-v7-10-x" label="AWS Elasticsearch v7.10.x">

-If you're using Elasticsearch v7.10.x under AWS’s managed services, you can't use newer Elasticsearch clients like the v8 client without changing backend infrastructure. If you're using AWS Elasticsearch v7.10.x, you must [upgrade to AWS OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) for future compatibility.
+If you're using Elasticsearch v7.10.x under AWS’s managed services, you can't use newer Elasticsearch clients like the v8 or v9 client without changing backend infrastructure. If you're using AWS Elasticsearch v7.10.x, you must [upgrade to AWS OpenSearch](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) for future compatibility.

 The migration path from Elasticsearch v7.10.x to OpenSearch has been designed to be straightforward, minimizing effort:

 1.  Disable "compatibility mode" in OpenSearch.
 2.  Upgrade Mattermost server.
-3.  Update the Mattermost `ElasticsearchSettings.Backend` configuration setting value from `elasticsearch` to `` `opensearch ``<code> manually or using \`mmctl \<mm-ref:administration-guide%2Fmanage%2Fmmctl-command-line-tool%3Ammctl%20config%20set\></code>\_\_. This value cannot be changed using the System Console. See the Mattermost search [backend type](/administration-guide/configure/environment-configuration-settings#backend-type) configuration setting documentation for additional details.
+3.  Update the Mattermost `ElasticsearchSettings.Backend` configuration setting value from `elasticsearch` to `opensearch` manually or using [mmctl](/administration-guide/manage/mmctl-command-line-tool#mmctl-config-set). This value cannot be changed using the System Console. See the Mattermost search [backend type](/administration-guide/configure/environment-configuration-settings#backend-type) configuration setting documentation for additional details.
 4.  Restart the Mattermost server.

 </TabItem>
@@ -114,6 +125,29 @@ From Mattermost v11, [Support Packet generation](/administration-guide/manage/ad

 The enterprise search connection test results appear in the Support Packet and can help identify configuration issues such as network connectivity problems, authentication failures, or server availability issues. If connection errors are present, they will be clearly documented with specific error messages to aid in troubleshooting.

+### How does Mattermost handle Elasticsearch or OpenSearch outages?
+
+From Mattermost v11.7, the server includes an automatic health monitor for Elasticsearch and OpenSearch connections. The health monitor runs periodic health checks and automatically manages the connection lifecycle:
+
+- **Health checks**: The server checks the health of the search engine cluster every 60 seconds. After 3 consecutive health check failures, the engine is stopped and search falls back to the database.
+- **Fast-fail on first failure**: On the very first health check failure, the engine is immediately marked as unhealthy and search requests are routed to the database. This happens before the consecutive failure threshold is reached, so users experience minimal disruption.
+- **Automatic retry**: When the search engine is unavailable, the server retries connecting with exponential backoff, starting at 15 seconds and doubling up to a maximum of 5 minutes between attempts.
+- **Automatic recovery**: When the search engine becomes available again, the server automatically reconnects and resumes using it for search queries. No manual intervention or server restart is required.
+- **Configuration changes**: Changes to Elasticsearch or OpenSearch configuration settings, or license changes, immediately trigger the health monitor to re-evaluate the connection state.
+- **Monitoring**: A `mattermost_search_engine_status` Prometheus metric reports the health of the search engine (`1` = healthy or not configured, `0` = configured but unavailable). Use this metric to [set up alerts](/administration-guide/scale/performance-alerting) for search engine outages. See [performance monitoring metrics](/administration-guide/scale/performance-monitoring-metrics) for details.
+
+During an outage, you may see the following log messages:
+
+| Level | Log message | Meaning |
+|---|---|---|
+| Error | `Search engine health check failed repeatedly; stopping engine` | The failure threshold was reached and the engine has been stopped. Search falls back to the database. |
+| Warn | `Search engine health check failed` | An individual health check failed. Includes a `consecutive_failures` count. |
+| Warn | `Search engine health check failed: it is now marked as unhealthy` | A previously healthy engine failed a health check and has been marked unhealthy. Search requests will fall back to the database immediately. |
+| Warn | `Search engine watcher: Start() failed, will retry` | A reconnection attempt failed. Includes a `next_backoff` field indicating the time until the next retry. |
+| Info | `Search engine health check succeeded: it is now marked as healthy` | The engine passed a health check after being unhealthy and is now handling search requests again. |
+| Info | `Search engine watcher: engine started successfully` | The engine has recovered and is active again. |
+| Info | `Search engine watcher: engine disabled, parking` | The health monitor is idle because the search engine is disabled in configuration. |
+
 ### My search indexes won't complete, what should I do?

 If you have an search indexing job that's paused, it's likely your Elasticsearch or OpenSearch server has restarted. If you restart that server, you must also restart Mattermost to ensure jobs are completed. If restarting the Mattermost server does not resolve the issue, [customers with a Mattermost subscription can contact Mattermost Support](https://mattermost.com/support/) for assistance.
@@ -18,6 +18,12 @@ From Mattermost v9.11, beta support is available for [AWS OpenSearch v1.x and v2

 We highly recommend that you set up an AWS OpenSearch server on a separate machine from the Mattermost server.

+<Important>
+
+Starting in Mattermost v12.0 (October 2026), OpenSearch v1.x is no longer supported. Upgrading to OpenSearch v2.x or later is required.
+
+</Important>
+
 <Tabs>
 <TabItem value="on-premises-opensearch" label="On-Premises OpenSearch">

@@ -198,20 +204,25 @@ Port 9200 is commonly used for local or on-premise OpenSearch. The AWS OpenSearc

 </Note>

-5.  Configure the access policy (JSON):
+5.  Configure the access policy (JSON). Mattermost doesn't sign OpenSearch requests with AWS SigV4, so restrict access at the network layer with the VPC and security group from step 4, and use an open principal in the domain access policy:

 > ``` sh
 > &#123;
->   "Version": "2012-10-17", 
->   "Statement": [&#123; 
->     "Effect": "Allow", 
->     "Principal": &#123; "AWS": 
-> "arn:aws:iam::123456789012:role/MattermostAppRole" &#125;, 
->     "Action": "es:*", 
->     "Resource": "arn:aws:es:us-east-1:123456789012:domain/mattermost-os/*" &#125;] 
+>   "Version": "2012-10-17",
+>   "Statement": [&#123;
+>     "Effect": "Allow",
+>     "Principal": &#123; "AWS": "*" &#125;,
+>     "Action": "es:ESHttp*",
+>     "Resource": "arn:aws:es:us-east-1:123456789012:domain/mattermost-os/*" &#125;]
 > &#125;
 > ```

+<Warning>
+
+IAM principal-based access policies (for example, `"Principal": { "AWS": "arn:aws:iam::<account-id>:role/<role-name>" }`) are not supported. Mattermost's OpenSearch client doesn't sign requests with AWS SigV4, so AWS treats the requests as anonymous and an IAM-restricted policy will reject them with `User: anonymous is not authorized to perform: es:ESHttpGet`. If you need authentication enforced at the OpenSearch layer rather than the network layer, enable [fine-grained access control](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html) on the domain with an internal master user, and enter those credentials as the **Server Username** and **Server Password** in the Mattermost Elasticsearch configuration.
+
+</Warning>
+
 6.  Configure the following advanced settings (JSON):

 > ``` sh
@@ -284,13 +295,13 @@ Port 9200 is commonly used for local or on-premise OpenSearch. The AWS OpenSearc
 >
 >   access_policies = &lt;<POLICY 
 > &#123;
->   "Version": "2012-10-17", 
->   "Statement": [&#123; 
->     "Effect": "Allow", 
->     "Principal": &#123; 
->       "AWS": "arn:aws:iam::123456789012:role/MattermostAppRole" 
+>   "Version": "2012-10-17",
+>   "Statement": [&#123;
+>     "Effect": "Allow",
+>     "Principal": &#123;
+>       "AWS": "*"
 >     &#125;,
->       "Action": "es:*", 
+>       "Action": "es:ESHttp*",
 >       "Resource": "arn:aws:es:us-east-1:123456789012:domain/mattermost-os/*" 
 >     &#125;]
 >   &#125;
@@ -105,6 +105,7 @@ From Mattermost version v9.9, this value includes any potentially unauthenticate
 - `mattermost_search_post_index_total`: The total number of posts indexes carried out.
 - `mattermost_search_posts_searches_total`: The total number of post searches carried out.
 - `mattermost_search_user_index_total`: The total number of user indexes carried out.
+- `mattermost_search_engine_status`: Status of the configured search engine: `1` = healthy or not configured, `0` = configured but unavailable. Use this metric to set up alerts for search engine outages.

 ### WebSocket metrics

@@ -157,6 +158,14 @@ Jobs where the runtime is less than the Prometheus polling interval are unlikely
 - `mattermost_plugin_multi_hook_server_time`: Time for the server to execute multiple plugin hook handlers in seconds.
 - `mattermost_plugin_multi_hook_time`: Time to execute multiple plugin hook handler in seconds.

+The metrics above are measured by the Mattermost server as it executes plugin code. From Mattermost v11.8.0, plugin-provided Prometheus metrics can also be exposed through the standard Mattermost `/metrics` endpoint when the `AggregatePluginMetrics` feature flag is enabled. Aggregated plugin metrics include a `plugin_id` label, based on the plugin's manifest ID, so admins can identify which plugin produced each metric.
+
+<Note>
+
+`AggregatePluginMetrics` is disabled by default and must be enabled before plugin-provided metrics are included in the `/metrics` response.
+
+</Note>
+
 ### Shared metrics

 - `mattermost_shared_channels_sync_collection_duration_seconds`: Duration tasks spend collecting sync data (seconds).
@@ -78,14 +78,14 @@ This page describes the Mattermost reference architecture designed for the load
 <td>1</td>
 <td>16/128</td>
 <td>db.r7g.4xlarge</td>
-<td>E16as v6</td>
+<td>E16ds v5</td>
 </tr>
 <tr>
 <td>RDS Reader</td>
 <td>5</td>
 <td>16/128</td>
 <td>db.r7g.4xlarge</td>
-<td>E16as v6</td>
+<td>E16ds v5</td>
 </tr>
 <tr>
 <td>Elasticsearch cluster</td>
@@ -97,14 +97,14 @@ body:not([data-custom-theme="light"]) .scale-requirements-table td &#123;
 <td>1</td>
 <td>4/32</td>
 <td>db.r7g.xlarge</td>
-<td>E4as v6</td>
+<td>E4ds v5</td>
 </tr>
 <tr>
 <td>RDS Reader</td>
 <td>1</td>
 <td>4/32</td>
 <td>db.r7g.xlarge</td>
-<td>E4as v6</td>
+<td>E4ds v5</td>
 </tr>
 <tr>
 <td>Elasticsearch Node</td>
@@ -94,14 +94,14 @@ body:not([data-custom-theme="light"]) .scale-requirements-table td &#123;
 <td>1</td>
 <td>2/16</td>
 <td>db.r7g.large</td>
-<td>E2as v6</td>
+<td>E2ds v5</td>
 </tr>
 <tr>
 <td>RDS Reader</td>
 <td>0</td>
 <td>2/16</td>
 <td>db.r7g.large</td>
-<td>E2as v6</td>
+<td>E2ds v5</td>
 </tr>
 <tr>
 <td>Elasticsearch Node</td>
@@ -97,14 +97,14 @@ body:not([data-custom-theme="light"]) .scale-requirements-table td &#123;
 <td>1</td>
 <td>2/16</td>
 <td>db.r7g.large</td>
-<td>E2as v6</td>
+<td>E2ds v5</td>
 </tr>
 <tr>
 <td>RDS Reader</td>
 <td>1</td>
 <td>2/16</td>
 <td>db.r7g.large</td>
-<td>E2as v6</td>
+<td>E2ds v5</td>
 </tr>
 <tr>
 <td>Elasticsearch Node</td>
@@ -84,14 +84,14 @@ Results show that this architecture supports logging in up to 150,000 users with
 <td>1</td>
 <td>16/128</td>
 <td>db.r7g.4xlarge</td>
-<td>E16as v6</td>
+<td>E16ds v5</td>
 </tr>
 <tr>
 <td>RDS Reader</td>
 <td>6</td>
 <td>16/128</td>
 <td>db.r7g.4xlarge</td>
-<td>E16as v6</td>
+<td>E16ds v5</td>
 </tr>
 <tr>
 <td>Elasticsearch cluster</td>
@@ -79,14 +79,14 @@ This page describes the Mattermost reference architecture designed for the load
 <td>1</td>
 <td>8/64</td>
 <td>db.r7g.2xlarge</td>
-<td>E8as v6</td>
+<td>E8ds v5</td>
 </tr>
 <tr>
 <td>RDS Reader</td>
 <td>1</td>
 <td>8/64</td>
 <td>db.r7g.2xlarge</td>
-<td>E8as v6</td>
+<td>E8ds v5</td>
 </tr>
 <tr>
 <td>Elasticsearch cluster</td>
@@ -79,14 +79,14 @@ This page describes the Mattermost reference architecture designed for the load
 <td>1</td>
 <td>8/64</td>
 <td>db.r7g.2xlarge</td>
-<td>E8as v6</td>
+<td>E8ds v5</td>
 </tr>
 <tr>
 <td>RDS Reader</td>
 <td>4</td>
 <td>8/64</td>
 <td>db.r7g.2xlarge</td>
-<td>E8as v6</td>
+<td>E8ds v5</td>
 </tr>
 <tr>
 <td>Elasticsearch cluster</td>
@@ -79,14 +79,14 @@ This page describes the Mattermost reference architecture designed for the load
 <td>1</td>
 <td>16/128</td>
 <td>db.r7g.4xlarge</td>
-<td>E16as v6</td>
+<td>E16ds v5</td>
 </tr>
 <tr>
 <td>RDS Reader</td>
 <td>3</td>
 <td>16/128</td>
 <td>db.r7g.4xlarge</td>
-<td>E16as v6</td>
+<td>E16ds v5</td>
 </tr>
 <tr>
 <td>Elasticsearch cluster</td>
@@ -79,14 +79,14 @@ This page describes the Mattermost reference architecture designed for the load
 <td>1</td>
 <td>16/128</td>
 <td>db.r7g.4xlarge</td>
-<td>E16as v6</td>
+<td>E16ds v5</td>
 </tr>
 <tr>
 <td>RDS Reader</td>
 <td>4</td>
 <td>16/128</td>
 <td>db.r7g.4xlarge</td>
-<td>E16as v6</td>
+<td>E16ds v5</td>
 </tr>
 <tr>
 <td>Elasticsearch cluster</td>