{"token_count": 4256} # Teleport Networking Reference A Teleport cluster is a distributed system that may comprise a number of networks. On Teleport Enterprise (Cloud), for example, the Auth Service and Proxy Service run in Teleport-managed infrastructure, while Teleport users manage Agents and `tbot` instances. This reference guide describes the networking requirements of a Teleport cluster. ## Architecture overview A Teleport cluster is a distributed system consisting of components that can run in both public and private networks. Teleport also supports fully air-gapped environments. The **Teleport Auth Service**, which manages backend data and issues certificates, typically runs in a private network. The **Teleport Proxy Service** should be the only component of your Teleport cluster that is addressable from the public internet, and can run in a private network as long as end-users can dial it. We expect all other components, including Teleport Agents and Machine & Workload ID Bots, to run in private networks. Components in private networks connect to the Teleport Proxy Service and establish reverse tunnels that the Proxy Service uses to communicate with them. For a comprehensive explanation of how a Teleport cluster works, see [Teleport Architecture](https://goteleport.com/docs/reference/architecture.md). For a glossary of terms, see [Core Concepts](https://goteleport.com/docs/core-concepts.md). ## Public addresses **Self-Hosted** All Teleport services (e.g., the Proxy Service, Auth Service, and agents) have an optional `public_addr` property that you can modify in each service's configuration file. The public address can take an IP or a DNS name. It can also be a list of values: ``` public_addr: ["service1.example.com", "service2.example.com"] ``` --- NOTE Only a single Proxy Service `public_addr` should be configured. Attempting to have multiple addresses can result in redirects to the first listed address that may not be available to the client. --- Specifying a public address for a Teleport service may be useful in the following use cases: - You have multiple identical services, e.g., Proxy Service instances, behind a load balancer. - You want Teleport to issue an SSH certificate for the service with additional principals, e.g., host names. **Cloud-Hosted** On Teleport Enterprise (Cloud) the Teleport Agent services always connect using reverse tunnels so there is no need to set a public address for an Agent. ## HTTP CONNECT proxies Some networks funnel all connections through a proxy server where they can be audited and access control rules can be applied. For these scenarios, Teleport supports HTTP CONNECT tunneling. HTTP CONNECT applies to: - `tsh` in all cases. - Teleport services, such as the SSH Service and Database Service, that dial back to the Teleport Proxy Service. To use HTTP CONNECT tunneling, set the `HTTPS_PROXY` and `HTTP_PROXY` environment variables when running Teleport. You can also optionally set the `NO_PROXY` environment variable to avoid use of the proxy when accessing specified hosts/netmasks/ports. By default, Teleport installations based on package managers (such as `apt` and `yum`) configure the `teleport` systemd unit to read environment variables from the file `/etc/default/teleport` by using the `EnvironmentFile` field: ``` [Unit] Description=Teleport Service After=network.target [Service] Type=simple Restart=always RestartSec=5 EnvironmentFile=-/etc/default/teleport ExecStart=/usr/local/bin/teleport start --config /etc/teleport.yaml --pid-file=/run/teleport.pid # systemd before 239 needs an absolute path ExecReload=/bin/sh -c "exec pkill -HUP -L -F /run/teleport.pid" PIDFile=/run/teleport.pid LimitNOFILE=524288 [Install] WantedBy=multi-user.target ``` To configure HTTP CONNECT tunneling, you can assign these environment variables within `/etc/default/teleport` on machines that run Teleport binaries. Use the following example, replacing `proxy.example.com` with the address of your proxy: ``` HTTP_PROXY=http://proxy.example.com:8080/ HTTPS_PROXY=http://proxy.example.com:8080/ NO_PROXY=localhost,127.0.0.1,192.168.0.0/16,172.16.0.0/12,10.0.0.0/8 ``` When Teleport builds and establishes the reverse tunnel to the main cluster, it will funnel all traffic through the proxy. Specifically, if using the default configuration, Teleport will tunnel ports `3024` (SSH, reverse tunnel) and `3080` (HTTPS, establishing trust) through the proxy. If you don't want to proxy some of this traffic (for example, proxying HTTPS but not SSH), assign `NO_PROXY` to the address of the Teleport Proxy Service endpoint you want to exclude from HTTP\_CONNECT tunneling in `host:port` format. For example, you can modify the environment file at `/etc/default/teleport` on each machine that runs a Teleport binary to resemble the following: ``` HTTP_PROXY=http://httpproxy.example.com:8080/ HTTPS_PROXY=http://httpproxy.example.com:8080/ NO_PROXY=teleportproxy.example.com:3024 ``` The value of `HTTPS_PROXY` or `HTTP_PROXY` should be in the format `scheme://[user[:password]@]host:port` where scheme is either `https` or `http` . If the value is `host:port` , Teleport will prepend `http` . --- NOTE `localhost` and `127.0.0.1` are invalid values for the proxy host. If for some reason your proxy runs locally, you'll need to provide some other DNS name or a private IP address for it. --- --- NOTE The Proxy Service also respects `HTTPS_PROXY` and `HTTP_PROXY` when connecting to a local Kubernetes cluster, which may not work. To fix this, add `kube.teleport.cluster.local` to `NO_PROXY`. --- ## Ports This section describes the ports you should open on your Teleport instances. **Self-Hosted** ### Proxy Service ports --- NOTE To get a listing of the assigned ports for an instance of the Teleport Proxy Service, use the following command: ``` $ curl https://teleport.example.com:443/webapi/ping | jq ``` Note that if `auth_service.proxy_listener_mode` is set to `multiplex` in your Teleport configuration, that means only a single port is used for multiple services through the Proxy. --- #### Ports with TLS routing TLS routing is enabled by default. In this mode, all connections to a Teleport service (e.g., the Teleport SSH Service or Kubernetes) are routed through the Proxy Service's public web address. Read more in our [TLS Routing](https://goteleport.com/docs/reference/architecture/tls-routing.md) guide. | Port | Downstream Service | Description | | ---- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------- | | 443 | Proxy Service | In TLS Routing mode, the Proxy handles all protocols, including Web UI, HTTPS, Kubernetes, SSH, and all databases on a single port. | | 3021 | Proxy Service | Port used by Teleport Proxy Service instances to dial agents in Proxy Peering mode. | #### Ports without TLS routing In some cases, administrators may want to use separate ports for different services. In those cases, they can set up separate listeners in the config file. | Port | Downstream Service | Description | | ----------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 3021 | Proxy Service | Port used by Teleport Proxy Service instances to dial agents in Proxy Peering mode. | | 3023 | All clients | SSH port clients connect to. The Proxy Service will forward this connection to port `3022` on the destination service or use a reverse tunnel connection. | | 3024 | Auth Service | SSH port used to create reverse SSH tunnels from behind-firewall environments into a trusted Proxy Service instance. All Teleport services (e.g., the SSH Service and Database Service) connecting through the Proxy Service will use this port to form their reverse tunnel connections. | | 3080 or 443 | Proxy Service | HTTPS connection to authenticate `tsh` users into the cluster. The same connection is used to serve a Web UI. | | 3036 | Database Service | Traffic to MySQL databases. | | 5432 | Database Service | Traffic to Postgres databases. | | 27017 | Database Service | Traffic to MongoDB instances. | | 6379 | Database Service | Traffic to Redis instances. | ### Auth Service ports | Port | Downstream Service | Description | | ---- | --------------------- | ------------------------------------------------------------------------------------------------ | | 3025 | All Teleport services | TLS port used by the Auth Service to serve its gRPC API to other Teleport services in a cluster. | **Cloud-Hosted** ### Proxy Service ports Cloud-hosted Teleport deployments allocate a different set of ports to each tenant's Proxy Service. To see which ports are available for your Teleport tenant, run a command similar to the following, replacing `example.teleport.sh` with your tenant domain: ``` $ curl https://example.teleport.sh/webapi/ping | jq '.proxy' ``` The output should resemble the following, including the unique ports assigned to your tenant: ``` { "kube": { "enabled": true, "listen_addr": "0.0.0.0:3080" }, "ssh": { "listen_addr": "0.0.0.0:3080", "tunnel_listen_addr": "0.0.0.0:3080", "web_listen_addr": "0.0.0.0:3080", "public_addr": "example.teleport.sh:443", "dial_timeout": 30000000000 }, "db": { "postgres_listen_addr": "0.0.0.0:3080", "mysql_listen_addr": "0.0.0.0:3080" }, "tls_routing_enabled": true } ``` This output also indicates whether TLS routing is enabled for your tenant. When TLS routing is enabled, connections to a Teleport service (e.g., the Teleport SSH Service) are routed through the Proxy Service's public web address, rather than through a port allocated to that service. In this case, you can see that TLS routing is enabled, and that the Proxy Service's public web address (`ssh.public_addr`) is `mytenant.teleport.sh:443`. Read more in our [TLS Routing](https://goteleport.com/docs/reference/architecture/tls-routing.md) guide. ### Agent ports Teleport Agents dial the Teleport Proxy Service to establish a reverse tunnel. Client traffic flows via the Proxy Service to the agent, and the agent forwards traffic to resources in your infrastructure. As a result, for Teleport processes running agents, e.g., instances of the SSH Service, Kubernetes Service, and other services that protect resources in your infrastructure, there is no need to open ports on the machines running the agents to the public internet. ### Direct connections to agents If you run a self-hosted Teleport cluster, you can join an agent [directly to the Teleport Auth Service](https://goteleport.com/docs/installation/agents/join-token.md). In this setup, certain Teleport services open their own listeners rather than accepting connections via reverse tunnel. The Proxy Service connects to these agent services by dialing them directly. The table below describes the ports that each Teleport service opens for proxied traffic: | Port | Service | Traffic Type | | ---- | ----------------------- | -------------------------------------------------------- | | 3022 | SSH Service | Incoming SSH connections. | | 3026 | Kubernetes Service | HTTPS traffic to a Kubernetes API server. | | 3028 | Windows Desktop Service | Teleport Desktop Protocol traffic from Teleport clients. | You can only access enrolled applications and desktops through the Teleport Proxy Service. The Teleport Application Service and Teleport Database Service use reverse tunnel connections through the Teleport Proxy Service and cannot expose ports directly. --- IMPORTANT Direct Auth Service joining is only supported for Teleport Agents running the Teleport SSH Service, Kubernetes Service, and Windows Desktop Service. Using Auth Service joining to enroll applications, databases, and for other discovery-based features, is not supported. --- ## Troubleshooting Logs emitted by a Teleport process can include the following messages if you have set up your Teleport cluster network incorrectly. ### Connection reset by peer If you see the message, `connection reset by peer`, it means that the remote server (usually the Teleport control plane) closed the client's connection. There is a variety of reasons why an upstream server may reset the connection, and they may or may not have to do with the components of a Teleport cluster. For example, a `tsh` client can emit a `connection reset by peer` message if it is behind the Teleport Proxy Service, which closes a connection if it receives traffic from an outdated `tsh` client. It could also be that a Teleport Agent proxying a connection exited without closing the connection. A device between the Teleport component and the other side of the connection, such as a load balancer, may also be closing the connection. To investigate: 1. If the connection involves two Teleport components, make sure their versions abide by the Teleport [version compatibility guarantees](https://goteleport.com/docs/upgrading/overview.md). 2. Examine the logs of the remote Teleport process and check whether the process was healthy at the time you saw the error. ### First record does not look like a TLS handshake You may see an error similar to the following, indicating an issue with a TLS handshake: ``` transport: authentication handshake failed: tls: first record does not look like a TLS handshake ``` The library that Teleport uses for TLS prints this error if the server and client have not negotiated a TLS version and the TLS record in question is not a handshake. In other words, the Teleport process that printed the log was expecting a TLS handshake from the other side of the connection and did not receive one. This can happen if the other side was replying to the Teleport process in a different protocol. In this case, check the address of the upstream host and make sure it is one that is configured to use TLS, rather than (for example) HTTP or unencrypted TCP. ### Authentication handshake failed Your `teleport` or `tbot` logs may report a certificate with a subject that does not match the expected domain name, a subdomain of `teleport.cluster.local`. The following example assumes your cluster's address is `example.teleport.sh`: ``` transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for *.example.teleport.sh, example.teleport.sh, not 6578616d706c652e74656c65706f72742e7368.teleport.cluster.local ``` As explained in [Troubleshooting](https://goteleport.com/docs/zero-trust-access/management/diagnostics/troubleshooting.md#teleportclusterlocal), Teleport issues certificates with `teleport.cluster.local` as a Subject Alternative Name (SAN) for the Teleport Auth Service. In this case, the certificate your Teleport component received does not include the intended SAN. One reason this situation could take place is that the cluster is configured to use [TLS multiplexing](https://goteleport.com/docs/zero-trust-access/management/tls-routing.md), but you are running the Teleport Proxy Service behind a Layer 7 load balancer that only returns the load balancer's main certificate instead of allowing the Teleport internal certificate. Your infrastructure may be interfering with TLS connections by altering the certificate that your Teleport instance presents. Investigate whether the certificate your client receives is signed by, for example, a corporate TLS proxy, or has otherwise been transformed between the time the Teleport Auth Service issued it and it reached your client during the attempted TLS handshake.