mirror of
https://github.com/mattermost/mattermost.git
synced 2026-09-19 02:06:37 +08:00
Doug Lauder
7778bb79d1
* MM-69792: Recover shared channel sync after a brief remote outage When a remote went offline for less than the 5-minute IsOnline() threshold, sync recovery never ran. IsOnline() never flipped, so no connection-state-change event fired and ForceSyncForRemote was never triggered. Posts created during the outage stayed unsynced until the next organic change in the channel. The root cause was that send failures are reported asynchronously through a callback that sendSyncMsgToRemote discarded. On a failed send syncForRemote still returned nil, so processTask saw success: the retry loop and the failure-notification hook were effectively dead code for delivery failures. Changes: - sendSyncMsgToRemote now propagates the delivery error reported via the callback instead of swallowing it. A failed send is retried rather than silently dropped. This applies to every sync type (posts, users, memberships, reactions, statuses); the success path is unchanged. - remotecluster tracks a per-remote "sync failed since last ping" flag (NotifySyncFailed). On the next successful ping, PingNow fires a connection-state-change event even when IsOnline() never flipped, driving a single ForceSyncForRemote on recovery. - Sync retries are now spaced by SyncRetryDelay (15s) instead of retrying immediately, giving a short outage time to recover and guaranteeing a failed ping before retries exhaust. Also adds a post/offline-recovery case to the sharedchannel-test integration tool (stop and restart Server B mid-run) and disables metrics on the two test servers so they do not collide on the metrics port when the loaded config has metrics enabled. * fix linter error * Fix Rabbit nit * fix CI * Delete pr37499-fix-findings-1-and-4.md * address review comments * MM-69792: Keep a per-remote retry task when a shared channel sync fans out A sync task with no specific remote (remoteID == "") fans out to every remote sharing the channel. On failure each remote's retry was a copy of the original task that still carried the original, remote-less task id. Because addTask merges on task id, all of those per-remote retries collapsed into a single entry, so only one failed remote was ever retried and the rest were silently dropped. Delivery failures were partly masked by the ping-driven ForceSyncForRemote recovery, but non-delivery errors for the other remotes had no path back. Give each per-remote retry a remote-specific id (task.id + remoteId) when the originating task had no remote, so every failed remote keeps its own retry task. Single-remote tasks already have a remote-specific id and are left untouched, since recomputing would grow the id on each retry. Also harden the tests: assert the retry is scheduled the full SyncRetryDelay out (no tolerance), assert the result callback is not invoked on delivery failures or unconfirmed responses (so the cursor stays un-advanced), and add a regression test covering fan-out to multiple failing remotes. * fix one more pre-existing edge case. * fix linter