WebSocket Troubleshooting

Documentation Index

Fetch the complete documentation index at: https://api-docs.rhombus.community/llms.txt

Use this file to discover all available pages before exploring further.

​

Connection Issues

​

Unable to Connect (Connection Refused)

• Firewall blocking outbound connections on port 8443
• Incorrect hostname

1. Verify outbound access to ws.rhombussystems.com on port 8443:
   # Test connectivity
   curl -v https://ws.rhombussystems.com:8443/websocket
   
   # Test TCP connectivity directly
   nc -zv ws.rhombussystems.com 8443
   
2. Check that your network/firewall allows outbound WSS on port 8443
3. If behind a corporate proxy, ensure WebSocket upgrade requests are permitted

​

HTTP 401 Unauthorized

• Invalid API token
• Expired API token
• Missing x-auth-apikey header

1. Verify your API token works with the REST API:
   curl -X POST https://api2.rhombussystems.com/api/org/getOrgV2 \
     -H "x-auth-apikey: YOUR_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{}'
   
2. Regenerate the token in the Rhombus console under Settings > API Access
3. Ensure the x-auth-apikey header is set (not just the query parameter)

​

HTTP 403 Forbidden

• API token lacks permissions for WebSocket access
• Partner token used without specifying the target organization
• Token does not have access to the specified organization

1. Check token permissions in the Rhombus console
2. If using a partner token, ensure you include x-auth-org query parameter:

   ?x-auth-scheme=partner-api-token&x-auth-org=CLIENT_ORG_UUID

​

Certificate-Based Auth Not Working

​

STOMP Protocol Issues

​

CONNECTED Frame Never Received

• Malformed STOMP CONNECT frame
• Missing null terminator (\x00)
• Incorrect accept-version value

1. Verify your frame includes the null byte terminator:
   # Correct
   frame = "CONNECT\naccept-version:1.2\nheart-beat:10000,10000\n\n\x00"
   
   # Wrong - missing \x00
   frame = "CONNECT\naccept-version:1.2\nheart-beat:10000,10000\n\n"
   
2. Ensure there is an empty line (\n\n) between headers and body
3. Use accept-version:1.2 (not 1.0 or 1.1)

​

No Messages Received After Subscribing

• Incorrect organization UUID in the topic
• No events occurring in the organization
• Subscription frame malformed

1. Verify your org UUID by calling the REST API:
   curl -X POST https://api2.rhombussystems.com/api/org/getOrgV2 \
     -H "x-auth-apikey: YOUR_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{}'
   
2. Trigger a test event (e.g., adjust a camera setting) to confirm the stream is active
3. Verify the topic format is exactly /topic/change/{orgUuid} with no trailing slashes or spaces

​

Connection Stability Issues

​

Connection Drops Every ~30 Seconds

import threading

def heartbeat_sender(ws, stop_event):
    while not stop_event.is_set():
        ws.send("\n")
        stop_event.wait(10)

stop = threading.Event()
thread = threading.Thread(target=heartbeat_sender, args=(ws, stop), daemon=True)
thread.start()

​

Connection Drops Intermittently

• Network instability
• Server-side maintenance
• Load balancer timeout

1. Implement automatic reconnection with backoff:
   while True:
       try:
           run_websocket_monitor()
       except ConnectionError:
           print("Reconnecting in 5 seconds...")
           time.sleep(5)
   
2. Log disconnection reasons to identify patterns
3. Monitor heartbeat receipt to detect dead connections proactively

​

High Memory Usage Over Time

• Events accumulating in an unbounded queue
• Not processing messages fast enough

1. Use bounded message buffers:
   # Use a bounded queue
   from queue import Queue
   event_queue = Queue(maxsize=1000)
   
2. Process events asynchronously to prevent backpressure
3. Drop or log events if the processing queue is full

​

Event Processing Issues

​

Missing Events

• Event filtering is too aggressive
• Brief disconnection caused missed events
• Event occurred before subscription was established

1. Temporarily enable all events to verify the stream:
   # Don't filter by entity type
   if frame["command"] == "MESSAGE":
       print(json.loads(frame["body"]))
   
2. WebSocket events are real-time only. For historical events, use the REST API
3. Ensure SUBSCRIBE is sent before expecting events

​

Malformed JSON in Event Body

• Frame parser not correctly separating body from headers
• Null bytes or extra whitespace in the body

1. Strip null bytes before parsing:
   body = frame["body"].strip("\x00").strip()
   payload = json.loads(body)
   
2. Verify your frame parser splits on \n\n correctly (first occurrence only)

​

Debugging Tools

​

Enable Raw Frame Logging

import logging

logging.basicConfig(level=logging.DEBUG)

# In your message loop:
raw = ws.recv()
logging.debug(f"Received: {repr(raw)}")

​

Test with websocat

# Install
npm install -g wscat

# Connect (note: wscat does not support custom headers natively,
# so use a tool like websocat for full header support)
websocat -H "x-auth-apikey: YOUR_TOKEN" \
  "wss://ws.rhombussystems.com:8443/websocket?x-auth-scheme=api-token"

# Then manually type STOMP frames:
CONNECT
accept-version:1.2
heart-beat:10000,10000

^@

​

Rhombus CLI Monitor

# Install the CLI (macOS/Linux)
brew install RhombusSystems/tap/rhombus

# Authenticate
rhombus login

# Test WebSocket monitoring
rhombus monitor --all-events --json

​

Getting Help

1. Check the Rhombus API documentation for updates
2. Verify your Rhombus organization has WebSocket access enabled
3. Contact Rhombus support with:
   • Your organization UUID
   • The error message or HTTP status code
   • The timestamp of the failed connection attempt
   • Your client language and WebSocket library version