How to Fix Portainer Error: agent not found in cluster

Users of Portainer CE, a popular management UI for Docker environments, have recently reported intermittent connectivity problems after upgrading to version 2.27.0. The error message, "unable to redirect request to specified node: agent not found in cluster," consistently disrupts access to container information, such as statistics and execution consoles. This issue, absent in version 2.6, significantly impacts operational efficiency.

The problem manifests as an inability to consistently access container details within a Docker Swarm cluster. While the setup previously functioned correctly, the upgrade triggered this persistent connectivity disruption. The frequency of the error suggests a fundamental incompatibility between Portainer's new version and the underlying Docker Swarm infrastructure.

The issue’s reproducibility is straightforward. A three-node Docker Swarm cluster, deployed using a standard Docker Compose file (detailed below), reliably demonstrates the error when attempting to access container information like statistics or the execution console. The vast majority of attempts result in failure, highlighting the severity and consistency of the problem.

A key aspect of the reported issue is its dependence on the Portainer version. Downgrading resolves the problem, while upgrading invariably reproduces it. This firmly establishes a causal relationship between the Portainer CE 2.27.0 update and the connectivity disruption.

The Docker Compose file employed for deployment is as follows:

      version: "3.8"

services:
  agent:
    image: portainer/agent:2.27.0-alpine
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock # Mount the Docker socket for agent access
      - /var/lib/docker/volumes:/var/lib/docker/volumes # Mount Docker volumes
    environment:
      - LOG_LEVEL=INFO # Set logging level to INFO for detailed logs
      # - AGENT_CLUSTER_ADDR=localhost  # Removed: This line was causing the issue.  The agent should automatically discover the cluster.
    networks:
      - portainer-network
    deploy:
      mode: global # Deploy the agent globally across all Swarm nodes

  ui:
    image: portainer/portainer-ce:2.27.0-alpine
    command: -H "tcp://agent.portainer-network:9001" --tlsskipverify # Connect to the agent via the specified network and port
    ports:
      - "9443:9443" # Expose Portainer UI on port 9443
      - "9000:9000" # Expose another port (purpose unclear, potentially for internal communication)
    volumes:
      - portainer_data:/data # Persistent storage for Portainer data
    networks:
      - portainer-network
    deploy:
      mode: replicated # Replicate the UI service for high availability
      replicas: 1 # Run only one replica of the UI

networks:
  portainer-network:
    name: portainer-network # Define the network for inter-service communication
    driver: overlay # Use the overlay driver for Docker Swarm networking
    attachable: true # Allow services to explicitly connect to the network

volumes:
  portainer_data:
    

The environment variables within the agent service configuration are crucial. The LOG_LEVEL setting enables detailed logging, useful for debugging. The commented-out AGENT_CLUSTER_ADDR variable is particularly significant. Initial investigations suggested that explicitly setting this to localhost was the root cause of the connectivity issues. Removing this line allows the agent to automatically discover its peers within the Swarm cluster, resolving the reported "agent not found" error.

This solution, while simple, highlights a critical architectural oversight in Portainer CE 2.27.0. The need to manually specify the cluster address, and the resulting connectivity problems when doing so, point to a deficiency in the agent's cluster discovery mechanism. In a well-designed system, this automatic discovery should be robust and reliable, eliminating the need for manual configuration and the ensuing complications.

The reported system configuration includes: Portainer CE 2.27.0, Docker Engine 27.3.1, and a three-node Docker Swarm cluster running on Debian 5.10.226-1. The browser used, Microsoft Edge 133, is unlikely to be a contributing factor. The issue's persistence across different browsers further confirms that the problem lies within the Docker Swarm and Portainer interaction, not the client-side browser environment.

The consistent failure to access container details emphasizes the need for a comprehensive review of the Portainer agent's cluster discovery and communication protocols. The current implementation clearly suffers from fragility and requires immediate attention from the development team. A proper fix should ensure reliable communication between the Portainer UI and the agents across all Swarm nodes, irrespective of configuration settings.

In conclusion, the "unable to redirect request to specified node: agent not found in cluster" error in Portainer CE 2.27.0 is directly linked to the manual specification of the AGENT_CLUSTER_ADDR environment variable within the agent's configuration. Removing this line enables the agent to correctly discover the Docker Swarm cluster, resolving the connectivity issues. This incident underscores the importance of thorough testing before releasing major software updates and highlights the potential consequences of flawed auto-discovery mechanisms in distributed systems.

Portainer, Docker Swarm, Docker, Container Management, Microservices, Agent Discovery, Connectivity Issues, Docker Compose, Deployment, Debugging