> ## Documentation Index
> Fetch the complete documentation index at: https://api-docs.rhombus.community/llms.txt
> Use this file to discover all available pages before exploring further.

# WebSocket Troubleshooting

> Diagnose and resolve common Rhombus WebSocket issues — connection failures, STOMP errors, missed heartbeats, authentication problems, and dropped events.

This guide covers common issues when working with Rhombus WebSocket connections and how to resolve them.

## Connection Issues

### Unable to Connect (Connection Refused)

**Symptoms**: WebSocket handshake fails immediately with a connection error.

**Possible Causes**:

* Firewall blocking outbound connections on port 8443
* Incorrect hostname

**Solutions**:

1. Verify outbound access to `ws.rhombussystems.com` on port `8443`:
   ```bash theme={null}
   # Test connectivity
   curl -v https://ws.rhombussystems.com:8443/websocket

   # Test TCP connectivity directly
   nc -zv ws.rhombussystems.com 8443
   ```
2. Check that your network/firewall allows outbound WSS on port 8443
3. If behind a corporate proxy, ensure WebSocket upgrade requests are permitted

### HTTP 401 Unauthorized

**Symptoms**: WebSocket handshake fails with HTTP 401.

**Possible Causes**:

* Invalid API token
* Expired API token
* Missing `x-auth-apikey` header

**Solutions**:

1. Verify your API token works with the REST API:
   ```bash theme={null}
   curl -X POST https://api2.rhombussystems.com/api/org/getOrgV2 \
     -H "x-auth-apikey: YOUR_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{}'
   ```
2. Regenerate the token in the Rhombus console under **Settings > API Access**
3. Ensure the `x-auth-apikey` header is set (not just the query parameter)

### HTTP 403 Forbidden

**Symptoms**: WebSocket handshake fails with HTTP 403.

**Possible Causes**:

* API token lacks permissions for WebSocket access
* Partner token used without specifying the target organization
* Token does not have access to the specified organization

**Solutions**:

1. Check token permissions in the Rhombus console
2. If using a partner token, send the target organization's UUID in the `x-auth-org` **header** (the server reads `x-auth-org` from request headers, not the query string), while `x-auth-scheme=partner-api-token` stays in the connection URL:

```text theme={null}
   Header: x-auth-org: CLIENT_ORG_UUID
   URL:    wss://ws.rhombussystems.com:8443/websocket?x-auth-scheme=partner-api-token
```

### Certificate-Based Auth Not Working

**Symptoms**: Connection fails when using mTLS certificates.

**Cause**: WebSocket connections **do not support** certificate-based authentication.

**Solution**: Generate an API token for WebSocket connections. Certificate auth is only supported for the REST API.

## STOMP Protocol Issues

### CONNECTED Frame Never Received

**Symptoms**: WebSocket connects successfully, but no STOMP `CONNECTED` response after sending `CONNECT`.

**Possible Causes**:

* Malformed STOMP `CONNECT` frame
* Missing null terminator (`\x00`)
* Incorrect `accept-version` value

**Solutions**:

1. Verify your frame includes the null byte terminator:
   ```python theme={null}
   # Correct
   frame = "CONNECT\naccept-version:1.2\nheart-beat:10000,10000\n\n\x00"

   # Wrong - missing \x00
   frame = "CONNECT\naccept-version:1.2\nheart-beat:10000,10000\n\n"
   ```
2. Ensure there is an empty line (`\n\n`) between headers and body
3. Use `accept-version:1.2` (not `1.0` or `1.1`)

### No Messages Received After Subscribing

**Symptoms**: Connection and subscription succeed, but no MESSAGE frames arrive.

**Possible Causes**:

* Incorrect organization UUID in the topic
* No events occurring in the organization
* Subscription frame malformed

**Solutions**:

1. Verify your org UUID by calling the REST API:
   ```bash theme={null}
   curl -X POST https://api2.rhombussystems.com/api/org/getOrgV2 \
     -H "x-auth-apikey: YOUR_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{}'
   ```
2. Trigger a test event (e.g., adjust a camera setting) to confirm the stream is active
3. Verify the topic format is exactly `/topic/change/{orgUuid}` with no trailing slashes or spaces

## Connection Stability Issues

### Connection Drops Every \~30 Seconds

**Symptoms**: WebSocket connection closes after approximately 30 seconds of inactivity.

**Cause**: Heartbeats are not being sent.

**Solution**: Implement the heartbeat sender that sends `\n` every 10 seconds:

```python theme={null}
import threading

def heartbeat_sender(ws, stop_event):
    while not stop_event.is_set():
        ws.send("\n")
        stop_event.wait(10)

stop = threading.Event()
thread = threading.Thread(target=heartbeat_sender, args=(ws, stop), daemon=True)
thread.start()
```

### Connection Drops Intermittently

**Symptoms**: Connection works for a while then drops unpredictably.

**Possible Causes**:

* Network instability
* Server-side maintenance
* Load balancer timeout

**Solutions**:

1. Implement automatic reconnection with backoff:
   ```python theme={null}
   while True:
       try:
           run_websocket_monitor()
       except ConnectionError:
           print("Reconnecting in 5 seconds...")
           time.sleep(5)
   ```
2. Log disconnection reasons to identify patterns
3. Monitor heartbeat receipt to detect dead connections proactively

### High Memory Usage Over Time

**Symptoms**: Application memory grows steadily while connected.

**Possible Causes**:

* Events accumulating in an unbounded queue
* Not processing messages fast enough

**Solutions**:

1. Use bounded message buffers:
   ```python theme={null}
   # Use a bounded queue
   from queue import Queue
   event_queue = Queue(maxsize=1000)
   ```
2. Process events asynchronously to prevent backpressure
3. Drop or log events if the processing queue is full

## Event Processing Issues

### Missing Events

**Symptoms**: Some events that appear in the Rhombus console don't appear in the WebSocket stream.

**Possible Causes**:

* Event filtering is too aggressive
* Brief disconnection caused missed events
* Event occurred before subscription was established

**Solutions**:

1. Temporarily enable all events to verify the stream:
   ```python theme={null}
   # Don't filter by entity type
   if frame["command"] == "MESSAGE":
       print(json.loads(frame["body"]))
   ```
2. WebSocket events are real-time only. For historical events, use the REST API
3. Ensure SUBSCRIBE is sent before expecting events

### Malformed JSON in Event Body

**Symptoms**: `json.loads()` fails on MESSAGE body.

**Possible Causes**:

* Frame parser not correctly separating body from headers
* Null bytes or extra whitespace in the body

**Solutions**:

1. Strip null bytes before parsing:
   ```python theme={null}
   body = frame["body"].strip("\x00").strip()
   payload = json.loads(body)
   ```
2. Verify your frame parser splits on `\n\n` correctly (first occurrence only)

## Debugging Tools

### Enable Raw Frame Logging

Add logging to see exactly what's being sent and received:

```python theme={null}
import logging

logging.basicConfig(level=logging.DEBUG)

# In your message loop:
raw = ws.recv()
logging.debug(f"Received: {repr(raw)}")
```

### Test with websocat

Use `wscat` to manually interact with the WebSocket endpoint:

```bash theme={null}
# Install
npm install -g wscat

# Connect (note: wscat does not support custom headers natively,
# so use a tool like websocat for full header support)
websocat -H "x-auth-apikey: YOUR_TOKEN" \
  "wss://ws.rhombussystems.com:8443/websocket?x-auth-scheme=api-token"

# Then manually type STOMP frames:
CONNECT
accept-version:1.2
heart-beat:10000,10000

^@
```

<Note>
  `^@` represents the null byte (`\x00`). In most terminals, type `Ctrl+@` or `Ctrl+Shift+2` to produce it.
</Note>

### Rhombus CLI Monitor

Use the [Rhombus CLI](/rhombus-cli) to verify your account and WebSocket endpoint work:

```bash theme={null}
# Install the CLI (macOS/Linux)
brew install RhombusSystems/tap/rhombus

# Authenticate
rhombus login

# Test WebSocket monitoring
rhombus monitor --all-events --json
```

If the CLI works but your code doesn't, compare the connection parameters your code uses against the CLI's behavior. See the [Rhombus CLI documentation](/rhombus-cli) for full installation and usage details.

## Getting Help

If you've tried the steps above and are still experiencing issues:

1. Check the [Rhombus API documentation](https://api-docs.rhombus.community) for updates
2. Verify your Rhombus organization has WebSocket access enabled
3. Contact Rhombus support with:
   * Your organization UUID
   * The error message or HTTP status code
   * The timestamp of the failed connection attempt
   * Your client language and WebSocket library version