chore: send notification to #dev on any CI failure on main (#16102)

We've had a [few failures in
main](https://github.com/coder/coder/actions?query=branch%3Amain+is%3Afailure)
of late, and unless the committer of the change has CI notifications
enabled we may not be aware of the failure.

This PR sends a Slack notification to the #dev channel so everyone has
visibility.

Signed-off-by: Danny Kopping <danny@coder.com>

.github/workflows/ci.yaml

@@ -1248,3 +1248,52 @@ jobs:
       - name: Setup and run sqlc vet
         run: |
           make sqlc-vet
+
+  notify-slack-on-failure:
+    runs-on: ubuntu-latest
+    if: always() && failure() && github.ref == 'refs/heads/main'
+
+    steps:
+      - name: Send Slack notification
+        run: |
+          curl -X POST -H 'Content-type: application/json' \
+          --data '{
+            "blocks": [
+              {
+                "type": "header",
+                "text": {
+                  "type": "plain_text",
+                  "text": "❌ CI Failure in `main`",
+                  "emoji": true
+                }
+              },
+              {
+                "type": "section",
+                "fields": [
+                  {
+                    "type": "mrkdwn",
+                    "text": "*Workflow:*\n${{ github.workflow }}"
+                  },
+                  {
+                    "type": "mrkdwn",
+                    "text": "*Failed Job:*\n${{ github.job }}"
+                  },
+                  {
+                    "type": "mrkdwn",
+                    "text": "*Committer:*\n${{ github.actor }}"
+                  },
+                  {
+                    "type": "mrkdwn",
+                    "text": "*Commit:*\n${{ github.sha }}"
+                  }
+                ]
+              },
+              {
+                "type": "section",
+                "text": {
+                  "type": "mrkdwn",
+                  "text": "*View failure:* <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|Click here>"
+                }
+              }
+            ]
+          }' ${{ secrets.CI_FAILURE_SLACK_WEBHOOK }}