BetaFree-to-use OpAMP fleet management for Windows & LinuxQuick Start →
Get Started

OpenTelemetry Collector Troubleshooting & FAQ

Diagnose and resolve issues across your CollectorCtrl deployment. This reference covers network connectivity, configuration policy reconciliation, remote supervisor upgrades, edge resource consumption, log diagnostics, and common architectural questions.

Quick Diagnostics Matrix

Identify your symptom below to jump directly to the verified debugging workflow:

Symptom / Observed StateLikely CauseRecommended Action
Agent Missing or OfflineOpAMP port 4320 blocked, wrong scheme, or token mismatchVerify Port & WebSocket URL
Status Stuck in "Reconciling"YAML syntax error, wrong otelcol path, or port 13133 closedInspect Preflight Syntax & Paths
Upgrade Shows "Rolled Back"Binary architecture mismatch or 45s connection watchdog timeoutDiagnose Upgrade & Watchdog
Collector Crashes on StartPort conflict (4317/4318) or missing component in binaryCheck Port Bindings & Receiver Config
High CPU / Memory UsageMissing memory limiter processor or debug log levelApply Resource Limits & Throttle Logs

1. Supervisor Agent Won't Connect to Server

Symptoms

  • The agent does not appear in the Fleet Overview or Supervisors dashboard.
  • Supervisor logs show WebSocket connection failed, dial tcp: connection refused, or i/o timeout.
  • The connection status remains offline indefinitely.
Outbound Only

Supervisors establish an outbound persistent WebSocket connection to the Management Server on port 4320. You never need to open inbound ports on target nodes.

Debugging Steps

1

Step 1: Test Network Reachability on Port 4320

Confirm that the target node can reach the Management Server's OpAMP gateway (port 4320):

Windows (PowerShell):

Test-NetConnection -ComputerName YOUR_SERVER_IP -Port 4320

Linux (Bash):

nc -zv YOUR_SERVER_IP 4320
# Or test using curl
curl -v http://YOUR_SERVER_IP:4320/v1/opamp

If the test fails, check security groups, cloud VPC firewall rules, and host firewalls (ufw, firewalld, or Windows Defender Firewall) on the Management Server.

2

Step 2: Check Endpoint Protocol Scheme

In supervisor.yaml, verify the server.endpoint configuration:

  • Unencrypted Gateway (HTTP / local testing): Must use ws:// (e.g. ws://YOUR_SERVER_IP:4320/v1/opamp).
  • TLS-Terminated Gateway (Production): Must use wss:// (e.g. wss://collectorctrl.internal:4320/v1/opamp).
Common Configuration Pitfall

Do not use http:// or https:// for the OpAMP endpoint. It must specify the WebSocket protocol (ws:// or wss://) and include the /v1/opamp path suffix.

3

Step 3: Validate API Enrollment Token

Ensure the token in supervisor.yaml matches the API token generated in the Admin Console (Settings → API Tokens):

  • Windows Path: C:\Program Files\CollectorCtrl Supervisor\supervisor.yaml
  • Linux Path: /etc/collectorctrl/supervisor.yaml
server:
  endpoint: "ws://YOUR_SERVER_IP:4320/v1/opamp"
  token: "your-valid-api-token-here"
4

Step 4: Check Reverse Proxy / Ingress Settings

If your server sits behind Nginx, Caddy, or an AWS Application Load Balancer (ALB), ensure WebSocket proxy headers are forwarded:

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

2. Configuration Policy Updates Are Not Applying

Symptoms

  • The agent status displays an amber Reconciling badge that never turns green (In Sync).
  • Pipeline modifications made in the Web UI do not take effect on the local OpenTelemetry Collector.

Debugging Steps

1

Step 1: Dry-Run Syntax Verification

Before applying any configuration to the running collector, the Supervisor executes a dry-run syntax check (otelcol validate).

If the syntax fails (e.g., malformed YAML indentation, missing exporter, unknown receiver), the Supervisor refuses to apply it to prevent taking down telemetry pipelines.

Check the Supervisor log for validation errors:

Failed to validate new config: error reading configuration: yaml: line 14: did not find expected key
2

Step 2: Verify Collector Executable Path

Ensure the agent.executable parameter inside supervisor.yaml points to a valid OpenTelemetry Collector binary:

  • Windows: Must be an absolute path. Remember to escape backslashes: 
    agent:
  executable: "C:\\Program Files\\otelcol\\otelcol.exe"
  • Linux: Must be an executable binary with proper permissions: 
    ls -l /usr/bin/otelcol-contrib
# Ensure executable bit is set
sudo chmod +x /usr/bin/otelcol-contrib
3

Step 3: Verify Local Health Check Port (13133)

The Supervisor monitors Collector health via HTTP on http://localhost:13133.

Verify that your base configuration includes the health_check extension:

extensions:
  health_check:
    endpoint: "0.0.0.0:13133"

service:
  extensions: [health_check]

3. Supervisor Upgrade Failures & Rollback

Symptoms

  • Triggering an upgrade from Settings → Supervisor Management results in the agent reverting to its previous version.
  • The agent displays a Rolled Back state in the console.

Root Causes & Solutions

45-Second Safety Watchdog

When an upgrade is initiated, the Supervisor spawns an independent watchdog process and replaces the binary. If the new binary fails to start, crashes, or cannot reconnect to the Management Server within 45 seconds, the watchdog automatically restores supervisor.exe.bak and restarts the old version.

  1. CPU Architecture Mismatch: Verify you uploaded the correct architecture binary:

    • For 64-bit Intel/AMD systems, ensure the package tag and binary are amd64 (not arm64).

  2. Missing OS Dependencies: On minimal Windows Server installations, ensure required C++ runtime components are present if using custom collector binaries.

  3. Inspect the .bad File: If an upgrade fails, the Supervisor writes an error marker:

    • Windows: C:\ProgramData\CollectorCtrlSupervisor\upgrade.bad
    • Linux: /var/log/collectorctrl/upgrade.bad

    View this file to see the exact crash reason recorded by the watchdog.

4. OpenTelemetry Collector Crashes on Startup

Symptoms

  • The Supervisor service is running, but the child otelcol process continuously stops and restarts.
  • Supervisor log shows: process exited with status 1 or process exited with status 2.

Root Causes & Solutions

  1. Port Conflict on OTLP Ports (4317 / 4318): Another process (e.g. an existing standalone collector, Jaeger, or Datadog agent) may already be listening on the standard OTLP ports:

    Windows:

    Get-NetTCPConnection -LocalPort 4317, 4318 -ErrorAction SilentlyContinue

    Linux:

    sudo ss -tulpn | grep -E '4317|4318'

  2. Core Binary vs. Contrib Binary: If your pipeline configuration uses contrib components (e.g., hostmetricsreceiver, filelogreceiver, kafkaexporter), you must use the otelcol-contrib distribution rather than the minimal otelcol core binary.

  3. Insufficient File Permissions: If using the filelog receiver to tail /var/log/* or Windows Event Logs, verify that the service user account has read permissions for the target log files.

5. High CPU or Memory Usage on Edge Nodes

Symptoms

  • The collector or supervisor process consumes excessive CPU or memory on virtualization hosts or containers.

Recommended Optimizations

1

1. Configure the Memory Limiter Processor

Always include the memory_limiter processor as the first processor in your telemetry pipelines to prevent out-of-memory crashes under load spikes:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20

  batch:
    send_batch_size: 8192
    timeout: 5s
2

2. Enforce systemd Resource Boundaries (Linux)

Set strict cgroup limits in /etc/systemd/system/collectorctrl-supervisor.service:

[Service]
MemoryMax=512M
CPUQuota=30%

Then reload and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart collectorctrl-supervisor
3

3. Reduce Logging Verbosity

Running collectors with level: debug generates high I/O overhead. Lower the log level in production to warn or info:

service:
  telemetry:
    logs:
      level: "warn"

6. System Log Locations & Live Diagnostics

When diagnosing issues, consult the system log files or use the live streaming commands below:

Default Log Locations

ComponentOperating SystemDefault Log File Path
Management ServerWindowsC:\ProgramData\CollectorCtrl\logs\server.log
Management ServerLinux/var/log/collectorctrl/server.log
Supervisor ServiceWindowsC:\ProgramData\CollectorCtrlSupervisor\supervisor.log
Supervisor DaemonLinux/var/log/collectorctrl/supervisor.log
OTel CollectorAll PlatformsStreamable live in Admin Console (Fleet → Agents → Logs)

Live Tail Commands

Live Stream Supervisor Logs on Windows:

Get-Content -Path "C:\ProgramData\CollectorCtrlSupervisor\supervisor.log" -Wait -Tail 50

Live Stream Supervisor Logs on Linux:

journalctl -u collectorctrl-supervisor -f --no-tail

Live Stream Management Server Logs (Docker):

docker logs -f collectorctrl --tail 100

7. Frequently Asked Questions (FAQ)

Do I need inbound ports on collector hosts?

No. Supervisor agents initiate an outbound WebSocket connection to port 4320 on the Management Server. No inbound firewall rules or SSH keys are required on target nodes.

What if the Management Server goes offline?

Supervisors and local OpenTelemetry Collectors continue running uninterrupted. Telemetry data continues flowing directly to your backends (Honeycomb, Coralogix, Datadog, etc.).

Can I manage custom collector distributions?

Yes. CollectorCtrl is vendor-neutral and OpAMP-standard. The supervisor can manage standard OpenTelemetry core, contrib, or custom binaries created with OpenTelemetry Builder (OCB).

Does CollectorCtrl work in air-gapped environments?

Yes. CollectorCtrl runs completely self-hosted with no external phone-home dependencies. You can upload custom collector and supervisor packages into the internal repository.

Related Documentation