Zpa Troubleshoot App Connector

zpa_list_app_connectors(search="<connector_name>")```text
Get detailed info for a specific connector:
```textzpa_get_app_connector(connector_id="<connector_id>")```text
**Check for:**
- `runtime_status` -- `ZPN_STATUS_AUTHENTICATED` (healthy), `ZPN_STATUS_DISCONNECTED`, `ZPN_STATUS_NOT_ENROLLED`- `current_version` -- software version running- `expected_version` -- version it should be running (upgrade needed if different)- `last_broker_connect_time` -- when it last connected to ZPA cloud- `last_broker_disconnect_time` -- when it last disconnected- `control_channel_status` -- control connection state- `connector_group_id` -- which group this connector belongs to- `private_ip` / `public_ip` -- network addresses- `platform` -- OS/hypervisor platform- `enrollment_cert` -- certificate details and expiry
**Then inspect the connector group** for group-level settings:
```textzpa_list_app_connector_groups(search="<connector_group_name>")zpa_get_app_connector_group(group_id="<group_id>")```text
**Check group settings:**
- `enabled` -- is the group enabled?- `upgrade_day` and `upgrade_time_in_secs` -- maintenance window- `version_profile` -- version track (Default, Previous Default, New Release)
**Connector status values:**
| Status | Meaning ||--------|---------|| `ZPN_STATUS_AUTHENTICATED` | Healthy, control connection established || `ZPN_STATUS_DISCONNECTED` | Lost connection to ZPA cloud || `ZPN_STATUS_NOT_ENROLLED` | Never enrolled or enrollment failed || `ZPN_STATUS_PENDING` | Enrollment in progress |
---
### Step 3: Check Provisioning Keys
Provisioning key issues are the #1 cause of enrollment failures.
```textzpa_list_provisioning_keys(key_type="connector")```text
For a specific key:
```textzpa_get_provisioning_key(key_id="<key_id>", key_type="connector")```text
**Check for:**
- `max_usage` vs current enrollment count -- if equal, no new enrollments can use this key- `enabled` -- is the key active?- The `component_id` must match the target App Connector Group- Key must not be expired
**If max usage is reached:**
```textThe provisioning key has reached its maximum enrollment count.
Resolution: Increase the max_usage value or create a new provisioning keyfor this connector group.```text
```textzpa_update_provisioning_key(  key_id="<key_id>",  key_type="connector",  max_usage=<current + needed>)```text
Or create a new key:
```textzpa_create_provisioning_key(  name="<name>",  key_type="connector",  max_usage=10,  component_id="<connector_group_id>",  enrollment_cert_id="<cert_id>")```text
---
### Step 4: Check Server Groups and Application Segments
If applications are unreachable through the connector, verify the infrastructure chain.
```textzpa_list_server_groups()```text
Find server groups that reference the affected connector group:
```textzpa_get_server_group(group_id="<server_group_id>")```text
**Check for:**
- `app_connector_groups` -- does it reference the affected connector group?- `enabled` -- is the server group active?- `servers` -- are application servers configured?
Then check which application segments use this server group:
```textzpa_list_application_segments()```text
**If applications are unreachable:**
- Verify the domain names in the application segment match what users access- Verify ports are correct- Confirm the server group → connector group → connector chain is intact- Check that connectors can reach the internal application servers (network-level)
---
### Step 5: Diagnose by Symptom Category
#### 5A: Enrollment Failure
If the connector shows `ZPN_STATUS_NOT_ENROLLED`:
1. **Provisioning key issues** (most common):   - Key max_usage reached (Step 3 above)   - Key copied incorrectly -- administrator should re-copy the exact key   - Key associated with wrong connector group
2. **DNS resolution failure:**   - Connector must resolve `co2br.<cloudname>.net`   - Check DNS servers in `/etc/resolv.conf` on the connector   - Common clouds: `prod.zpath.net`, `zpatwo.net`
3. **Network connectivity:**   - Connector needs outbound TCP 443 to ZPA Public Service Edges   - Check firewall rules per `config.zscaler.com/<cloudname>/zpa`   - SSL interception on the path will break enrollment (connector logs show "self signed certificate in certificate chain")
4. **Certificate/time issues:**   - If connector clock is skewed, certificate validation fails   - Logs show: "certificate is not yet valid" or similar   - Resolution: sync NTP on the connector host
5. **VM migration issues:**   - If connector was cloned/migrated (changed MAC/hardware ID), enrollment breaks   - Logs show: "Cannot decrypt data from instance_id.crypt"   - Resolution: wipe and re-provision the connector
**Wipe and re-provision procedure:**
```text1. Stop: sudo systemctl stop zpa-connector2. Wipe: sudo rm /opt/zscaler/var/*3. Create key file: sudo touch /opt/zscaler/var/provision_key && sudo chmod 644 /opt/zscaler/var/provision_key4. Paste provisioning key into the file5. Start: sudo systemctl start zpa-connector```text
#### 5B: Upgrade Failure
If connector upgrade failed:
1. **Check current version** via the connector group API response (Step 2)2. **Network connectivity to upgrade servers:**   - Connector needs to reach `dist.private.zscaler.com` and `yum.private.zscaler.com`   - Test: `ping dist.private.zscaler.com` and `telnet dist.private.zscaler.com 443`
3. **Revert to default version:**
```text1. sudo systemctl stop zpa-connector2. sudo rm /opt/zscaler/var/image.bin3. sudo rm /opt/zscaler/var/version4. sudo rm /opt/zscaler/var/metadata5. sudo systemctl start zpa-connector```text
The connector will re-download a clean default version from the CDN.
4. **Full rebuild** if revert doesn't work (follow wipe procedure in 5A)
#### 5C: Public Service Edge Connectivity
If connector shows `ZPN_STATUS_DISCONNECTED`:
1. **DNS check:** Verify `co2br.<cloudname>.net` resolves correctly2. **ICMP check:** `ping co2br.<cloudname>.net`3. **TCP check:** `telnet <broker_ip> 443`4. **TLS check:** `openssl s_client -servername <company>.com.server1.net -connect <broker_ip>:443`   - Should return a Zscaler certificate (CN=broker*.*.prod.zpath.net)   - If it returns a different CA, SSL interception is occurring5. **Packet capture:** `sudo tcpdump -i any -w output.pcap`   - Look for TCP RSTs or connection timeouts   - Compare TTL of RST packets with expected hop count from MTR
**Firewall requirements:**
- All ZPA IP ranges must be allowed: check `config.zscaler.com/<cloudname>/zpa`- Common ports: TCP 443 outbound to brokers- SSL interception devices must allowlist all ZPA domains
#### 5D: High CPU / Memory / Disk
1. **CPU:**   - Check if `zpa-connector-child` is the top consumer: `ps aux --sort=-pcpu | head -5`   - If another process is high, the issue is not Zscaler-related   - Normalize CPU % by core count (4 cores = 400% max)   - If consistently >80% normalized CPU, collect logs and contact Zscaler Support
2. **Memory:**   - Check process memory: `ps aux --sort=-pmem | head -5`   - Check connector internal memory: `curl -s 127.0.0.1:9000/memory/status`   - Consider adding more connectors to the group to distribute load
3. **Disk:**   - Check space: `sudo df -h`   - Find large consumers: `sudo du -a / | sort -n -r | head -n 20`   - Common culprit: `/var/log` and `/var/cache/yum`   - Clean journals: `sudo journalctl --vacuum-size=1G`   - Minimum 100MB free required; connector auto-restarts below this
4. **Scaling recommendations:**   - Increase VM resources (CPU/RAM) per Zscaler sizing guide   - Add more connectors to the group to distribute users   - Create dedicated connector groups for high-traffic application segments
---
### Step 6: Present Diagnosis
#### Report Format
```textApp Connector Troubleshooting Report======================================
**Connector Group:** <name> (ID: <id>)**Affected Connector(s):** <connector_name(s)>**Symptom:** <enrollment failure / disconnected / upgrade failed / high resource>**Status:** <ZPN_STATUS_*>
---
## Diagnosis: <ROOT CAUSE>
---
## API Findings
### Connector Group Status- Group enabled: Yes/No- Total connectors: X- Healthy (AUTHENTICATED): Y- Disconnected: Z- Version: X.Y.Z- Upgrade window: <day> at <time>
### Provisioning Keys- Key "<name>": <current_usage>/<max_usage> enrollments used- Key status: Active/Exhausted/Expired
### Server Groups Using This Connector Group- "<server_group_name>": enabled, X application servers- Application segments: <list>
---
## Root Cause
<Explanation based on API findings and symptom analysis>
---
## Remediation Steps
1. <Actionable step with specific commands/API calls>2. <Next step>3. <Verification step>
---
## If Issue Persists
Collect the following and contact Zscaler Support:- Connector journalctl logs: sudo journalctl -u zpa-connector > connector-logs.txt- Memory status: curl -s 127.0.0.1:9000/memory/status- Memory argo: curl -s 127.0.0.1:9000/memory/argo- System info: lscpu, free -h, df -h```text
---
## Healthy Connector Indicators
A healthy App Connector shows these in its journalctl status block (repeated every 60 seconds):
- `Control connection state: fohh_connection_connected` -- connected to ZPA cloud- Certificate expiry > 30 days- `uptime` continuously incrementing (no resets)- `Broker data connection count` > 0 with `backed_off connections = 0`- `Registered apps count = N, alive app = N` (all apps alive)
---
## Quick Reference
**API Tools -- App Connectors (individual):**
- `zpa_list_app_connectors(search)` -- list connectors with runtime status, version, health- `zpa_get_app_connector(connector_id)` -- detailed connector info (status, version, IPs, cert)- `zpa_update_app_connector(connector_id, enabled)` -- enable/disable a connector- `zpa_delete_app_connector(connector_id)` -- remove a connector from ZPA cloud- `zpa_bulk_delete_app_connectors(connector_ids)` -- remove multiple connectors
**API Tools -- App Connector Groups:**
- `zpa_list_app_connector_groups(search)` -- find connector groups- `zpa_get_app_connector_group(group_id)` -- group settings (upgrade window, version profile)
**API Tools -- Provisioning & Infrastructure:**
- `zpa_list_provisioning_keys(key_type="connector")` -- list provisioning keys- `zpa_get_provisioning_key(key_id, key_type)` -- key details including usage count- `zpa_create_provisioning_key(...)` -- create new provisioning key- `zpa_update_provisioning_key(...)` -- update key (e.g., increase max_usage)- `zpa_list_server_groups()` -- find server groups using this connector group- `zpa_get_server_group(group_id)` -- server group details- `zpa_list_application_segments()` -- find affected application segments
**CLI Commands (run on the connector host):**
- `sudo journalctl -u zpa-connector -f` -- live connector logs- `dig co2br.<cloud>.net` -- DNS resolution check- `openssl s_client -connect <broker_ip>:443` -- TLS connectivity check- `curl -s 127.0.0.1:9000/memory/status` -- internal memory stats- `ps aux --sort=-pcpu | head -5` -- top CPU consumers- `sudo df -h` -- disk usage