mirror of
https://github.com/gravitational/teleport.git
synced 2026-11-01 21:52:02 +08:00
Trent Clarke
ea176c2b3c
Actually tracking down the cause of a failure in the integration tests can
be hard:
* It's hard to get an overall summary of what failed
* The tests sometimes emit no output before timing out, meaning any
diagnostic info is lost
* The emitted logs are too voluminous for a human to parse
* The emitted logs can present information out of order
* It's often hard to tell where the output from one test ends
and the next one begins
This patch attempts to address these concerns without attempting to rewrite
any of the underlying teleport logging.
* It improves the render-tests script to (optionally) report progress per-
test, rather than on a per-package basis. My working hypothesis on the
tests that time out with no output is that go test ./integration is
waiting for the entire set of integration tests tests to be complete
before reporting success or failure. Reporting on a per-test cycle gives
faster feedback and means that any timed-out builds should give at least
some idea of where they are stuck.
* Adds the render-tests filter to the integration and integration-root make
targets. This will show an overall summary of test results, as well as
- Discarding log output from passing tests to increase signal-to-noise
ratio, and
- Strongly delimiting the output from each failed test, making failures
easier to find.
* Removes the notion of a failure-only logger in favour of post-processing
the log events with render-tests. The failure-only logger catches log
output from the tests and only forwards it to the console if the test
fails. Unfortunately, not all log output is guaranteed to pass through
this logger (some teleport packages do not honour the configured logger,
and reports from the go race detector certainly don't), meaning some
output is presented at the time it happens, and other output is batched
and displayed at the end of the test. This makes working out what
happened where harder than it need be.
In addition, this patch also promotes the render-tests script into a fully-
fledged program, with appropriate makefile targets, make clean support, etc.
It is now also more robust in the face on non-JSON output from go test
(which happens if a package fails to compile).
91 lines
2.4 KiB
Go
91 lines
2.4 KiB
Go
// Copyright 2022 Gravitational, Inc
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package main
|
|
|
|
import "sort"
|
|
|
|
type packageOutput struct {
|
|
output []string
|
|
subtestOutput map[string][]string
|
|
failedSubtests map[string][]string
|
|
}
|
|
|
|
func (pkg *packageOutput) FailedTests() []string {
|
|
result := []string{}
|
|
for testName := range pkg.failedSubtests {
|
|
result = append(result, testName)
|
|
}
|
|
sort.Strings(result)
|
|
return result
|
|
}
|
|
|
|
type outputMap struct {
|
|
packages map[string]*packageOutput
|
|
actionCounts map[string]int
|
|
}
|
|
|
|
func newOutputMap() *outputMap {
|
|
return &outputMap{
|
|
packages: make(map[string]*packageOutput),
|
|
actionCounts: make(map[string]int),
|
|
}
|
|
}
|
|
|
|
func (m *outputMap) record(event TestEvent) {
|
|
m.actionCounts[event.Action]++
|
|
|
|
var pkgOutput *packageOutput
|
|
var exists bool
|
|
if pkgOutput, exists = m.packages[event.Package]; !exists {
|
|
pkgOutput = &packageOutput{
|
|
subtestOutput: make(map[string][]string),
|
|
failedSubtests: make(map[string][]string),
|
|
}
|
|
m.packages[event.Package] = pkgOutput
|
|
}
|
|
|
|
switch event.Action {
|
|
case actionOutput:
|
|
pkgOutput.output = append(pkgOutput.output, event.Output)
|
|
if event.Test != "" {
|
|
pkgOutput.subtestOutput[event.Test] = append(pkgOutput.subtestOutput[event.Test], event.Output)
|
|
}
|
|
|
|
case actionFail:
|
|
// If this is a single test result
|
|
if event.Test != "" {
|
|
pkgOutput.failedSubtests[event.Test] = pkgOutput.subtestOutput[event.Test]
|
|
} else {
|
|
// If this is a package result, we only want to preserve the package output if
|
|
// there are no failed subtests
|
|
if len(pkgOutput.failedSubtests) > 0 {
|
|
pkgOutput.output = nil
|
|
}
|
|
}
|
|
fallthrough
|
|
|
|
case actionPass, actionSkip:
|
|
delete(pkgOutput.subtestOutput, event.Test)
|
|
}
|
|
}
|
|
|
|
func (m *outputMap) getPkg(pkgName string) *packageOutput {
|
|
return m.packages[pkgName]
|
|
}
|
|
|
|
func (m *outputMap) deletePkg(pkgName string) {
|
|
delete(m.packages, pkgName)
|
|
}
|