## Manual Testing Plan Below are the items that should be manually tested with each release of Teleport. These tests should be run on both a fresh install of the version to be released as well as an upgrade of the previous version of Teleport. - [ ] Adding nodes to a cluster - [ ] Adding Nodes via Valid Static Token - [ ] Adding Nodes via Valid Short-lived Tokens - [ ] Adding Nodes via Invalid Token Fails - [ ] Revoking Node Invitation - [ ] Labels - [ ] Static Labels - [ ] Dynamic Labels - [ ] Trusted Clusters - [ ] Adding Trusted Cluster Valid Static Token - [ ] Adding Trusted Cluster Valid Short-lived Token - [ ] Adding Trusted Cluster Invalid Token - [ ] Removing Trusted Cluster - [ ] RBAC Make sure that invalid and valid attempts are reflected in audit log. - [ ] Successfully connect to node with correct role - [ ] Unsuccessfully connect to a node in a role restricting access by label - [ ] Unsuccessfully connect to a node in a role restricting access by invalid SSH login - [ ] Allow/deny role option: SSH agent forwarding - [ ] Allow/deny role option: Port forwarding - [ ] Users With every user combination, try to login and signup with invalid second factor, invalid password to see how the system reacts. - [ ] Adding Users Password Only - [ ] Adding Users OTP - [ ] Adding Users U2F - [ ] Login Password Only - [ ] Login OTP - [ ] Login U2F - [ ] Login OIDC - [ ] Login SAML - [ ] Login GitHub - [ ] Deleting Users - [ ] Backends - [ ] Teleport runs with etcd - [ ] Teleport runs with dynamodb - [ ] Teleport runs with boltdb - [ ] Teleport runs with dir - [ ] Session Recording - [ ] Session recording can be disabled - [ ] Sessions can be recorded at the node - [ ] Sessions in remote clusters are recorded in remote clusters - [ ] Sessions can be recorded at the proxy - [ ] Sessions on remote clusters are recorded in the local cluster - [ ] Enable/disable host key checking. - [ ] Audit Log - [ ] Failed login attempts are recorded - [ ] Interactive sessions have the correct Server ID - [ ] Server ID is the ID of the node in regular mode - [ ] Server ID is randomly generated for proxy node - [ ] Exec commands are recorded - [ ] `scp` commands are recorded - [ ] Subsystem results are recorded - [ ] Interact with a cluster using `tsh` These commands should ideally be tested for recording and non-recording modes as they are implemented in a different ways. - [ ] tsh ssh \ - [ ] tsh ssh \ - [ ] tsh ssh -A \ - [ ] tsh ssh -A \ - [ ] tsh ssh \ ls - [ ] tsh ssh \ ls - [ ] tsh join \ - [ ] tsh join \ - [ ] tsh play \ - [ ] tsh play \ - [ ] tsh scp \ - [ ] tsh scp \ - [ ] tsh ssh -L \ - [ ] tsh ssh -L \ - [ ] tsh ls - [ ] tsh clusters - [ ] Interact with a cluster using `ssh` Make sure to test both recording and regular proxy modes. - [ ] ssh \ - [ ] ssh \ - [ ] ssh -A \ - [ ] ssh -A \ - [ ] ssh \ ls - [ ] ssh \ ls - [ ] scp \ - [ ] scp \ - [ ] ssh -L \ - [ ] ssh -L \ - [ ] Interact with a cluster using the Web UI - [ ] Connect to a Teleport node - [ ] Connect to a OpenSSH node - [ ] Check agent forwarding is correct based on role and proxy mode. ### Combinations For some manual testing, many combinations need to be tested. For example, for interactive sessions the 12 combinations are below. - [ ] Connect to a OpenSSH node in a local cluster using OpenSSH. - [ ] Connect to a OpenSSH node in a local cluster using Teleport. - [ ] Connect to a OpenSSH node in a local cluster using the Web UI. - [ ] Connect to a Teleport node in a local cluster using OpenSSH. - [ ] Connect to a Teleport node in a local cluster using Teleport. - [ ] Connect to a Teleport node in a local cluster using the Web UI. - [ ] Connect to a OpenSSH node in a remote cluster using OpenSSH. - [ ] Connect to a OpenSSH node in a remote cluster using Teleport. - [ ] Connect to a OpenSSH node in a remote cluster using the Web UI. - [ ] Connect to a Teleport node in a remote cluster using OpenSSH. - [ ] Connect to a Teleport node in a remote cluster using Teleport. - [ ] Connect to a Teleport node in a remote cluster using the Web UI. ### Teleport with EKS/GKE * [ ] Deploy Teleport on a single EKS cluster * [ ] Deploy Teleport on two EKS clusters and connect them via trusted cluster feature * [ ] Deploy Teleport Proxy outside of GKE cluster fronting connections to it (this feature is not yet supported for EKS) ### Teleport with FIPS mode * [ ] Perform trusted clusters, Web and SSH sanity check with all teleport components deployed in FIPS mode. ### Migrations * [ ] Migrate trusted clusters from 2.4.0 to 2.5.0 * [ ] Migrate auth server on main cluster, then rest of the servers on main cluster SSH should work for both main and old clusters * [ ] Migrate auth server on remote cluster, then rest of the remote cluster SSH should work ### Command Templates When interacting with a cluster, the following command templates are useful: #### OpenSSH ``` # when connecting to the recording proxy, `-o 'ForwardAgent yes'` is required. ssh -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %r@proxy.example.com -s proxy:%h:%p" \ node.example.com # the above command only forwards the agent to the proxy, to forward the agent # to the target node, `-o 'ForwardAgent yes'` needs to be passed twice. ssh -o "ForwardAgent yes" \ -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %r@proxy.example.com -s proxy:%h:%p" \ node.example.com # when connecting to a remote cluster using OpenSSH, the subsystem request is # updated with the name of the remote cluster. ssh -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %r@proxy.example.com -s proxy:%h:%p@foo.com" \ node.foo.com ``` #### Teleport ``` # when connecting to a OpenSSH node, remember `-p 22` needs to be passed. tsh --proxy=proxy.example.com --user= --insecure ssh -p 22 node.example.com # an agent can be forwarded to the target node with `-A` tsh --proxy=proxy.example.com --user= --insecure ssh -A -p 22 node.example.com # the --cluster flag is used to connect to a node in a remote cluster. tsh --proxy=proxy.example.com --user= --insecure ssh --cluster=foo.com -p 22 node.foo.com ``` ### Teleport with SSO Providers - [ ] G Suite install instructions work - [ ] G Suite Screenshots are up to date - [ ] ActiveDirectoy install instructions work - [ ] Active Directoy Screenshots are up to date - [ ] Okta install instructions work - [ ] Okta Screenshots are up to date - [ ] OneLogin install instructions work - [ ] OneLogin Screenshots are up to date - [ ] OIDC install instructions work - [ ] OIDC Screenshots are up to date ### Teleport Plugins - [ ] Test receiving a message via Teleport Slackbot - [ ] Test receiving a new Jira Ticket via Teleport Jira ## Web UI - [ ] Creating a new SSH session. Open a terminal to existing server - [ ] Verify that input/output works as expected by typing on the keyboard. - [ ] Verify that participant indicator is shown. - [ ] Verify that window resize works. - [ ] Verify that terminal is automatically closed after session ends (by typing `exit`) Open a terminal to non-existing server - [ ] Verify that connection error is shown to the user. - [ ] Joining an existing SSH session. Start a new session then copy the session URL and open it in another tab. - [ ] Should be able to join an existing session. - [ ] Should be able to see what another participant is typing. - [ ] Should display the correct number of participants. - [ ] Verify that resize works in both terminals. Use midnight commander (`apt-get install mc`) as it clearly shows the actual screen of the terminal. Start a new session and then open the Sessions Page in another tab: - [ ] Should see 1 active session and participant. - [ ] Should be able to join this session by clicking on "Join" button. Join a session that has ended: - [ ] Should see a message saying that this session has ended. - [ ] Should see 2 buttons: "Start new" and "Replay". Verify that both work as expected. - [ ] Node List. Open the Node List page. - [ ] Verify that sorting works. - [ ] Verify that dynamic labels are getting updated. - [ ] Verify that login dropdown control shows the correct list of available logins. - [ ] Verify that search works. - [ ] Verify that ssh control works (input validation). - [ ] Verify that cluster dropdown control shows the correct list of clusters. - [ ] Session List (stored sessions). Open the Sessions page. - [ ] Verify that it displays the list of stored sessions. - [ ] Should be able to open a session player by clicking on "Play" button. - [ ] Verify that all columns display correct information. - [ ] Verify that date-picker works. - [ ] Verify that search works. Create a new session and open the Session List in another browser window. - [ ] Verify that this session is shown as active in the Session List. - [ ] Verify that "Join" button works. - [ ] Invite Create user invite URL and use this URL to: - [ ] Verify that invite works with OTP enabled. - [ ] Verify that invite works with U2F enabled. - [ ] Verify that invite works with 2nd factor disabled. - [ ] Expired Invite Create invalid or expired invite URL. - [ ] Verify that "Invite code has expired" message is displayed. - [ ] User lock Open a login screen. - [ ] Verify that a user gets locked after several unsuccessful attempts. - [ ] Login redirects. Log out and then navigate to Nodes list. - [ ] Verify that a user is redirected to the login page. - [ ] Verify that after successful login, a user is redirected to the Node List. ## Performance/Soak Test Using `tsh bench` tool, perform the soak tests and benchmark tests on the following configurations: * Cluster with 10K nodes in normal (non-IOT) node mode with ETCD * Cluster with 10K nodes in normal (non-IOT) mode with DynamoDB * Cluster with 1K IOT nodes with ETCD * Cluster with 1K IOT nodes with DynamoDB * Cluster with 500 trusted clusters with ETCD * Cluster with 500 trusted clusters with DynamoDB **Soak Tests** Run 4hour soak test with a mix of interactive/non-interactive sessions: ``` tsh bench --duration=4h --threads=10 user@teleport-monster-6757d7b487-x226b ls tsh bench -i --duration=4h --threads=10 user@teleport-monster-6757d7b487-x226b ps uax ``` Observe prometheus metrics for goroutines, open files, RAM, CPU, Timers and make sure there are no leaks **Breaking load tests** Load system with tsh bench to the capacity and publish maximum numbers of concurrent sessions with interactive and non interactive tsh bench loads.