Challenge
Enterprises often have multiple AI agents performing specialized tasks — for instance, monitoring IT infrastructure, managing tickets, and analysing logs. Coordinating these agents efficiently is difficult: agents work in silos, data flow is slow, observability is limited, and maintaining session context across multiple agents is error-prone.
Data Size: Millions of structured and unstructured events per day, spanning logs, tickets, metrics, and configuration data.
Our Solution
Implemented an MCP (Model Context Protocol) server as the central orchestration layer for all AI agents. The MCP server:
- Registers all agents with metadata (capabilities, endpoints, schemas)
- Routes requests dynamically based on orchestrator decisions
- Maintains session context and conversation history across multiple agents
- Enables real-time observability, logging, and metrics
- Supports agentic workflows where multiple agents collaborate on a single task
Features
- Centralized Agent Registry: All agents (IT, DevOps, Observability, Incident Management) are discoverable and invokable.
- Domain-Specific Sub-Orchestrators: Lightweight orchestrators route queries to relevant agent clusters.
- Python-Based Direct Router: Fast payload forwarding, bypassing heavy middleware, reducing latency.
- Observability & Telemetry Layer: Traces each agent call, measures latency, logs failures, and tracks system health.
- Agentic Workflow Support: Multi-agent pipelines can execute in sequence or parallel while preserving context.
- Interactive Human-in-the-Loop Validation: Ensures correctness before external system calls (e.g., ticket creation).
Benefits
- Faster execution: Direct routing and session-aware orchestration reduces end-to-end latency.
- Scalability: Agents and orchestrators scale independently; new agents can be added without affecting existing workflows.
- Improved observability: Full traceability and metrics for debugging and monitoring.
- Reliability: HITL validation reduces failed external API calls.
- Maintainability: Modular architecture separates orchestration, routing, and agent execution logic.
Tech Stack
- MCP Server: Central orchestration and routing layer
- Sub-Orchestrators & Python Routers: Domain-specific routing
- Agents: Python | LangChain | LangGraph | Pydantic | FASTAPI
- Observability: Prometheus | Grafana | Structured Logging
- LLM Integration: Google Gemini | Anthropic Claude v3 | Azure OpenAI | Meta LLaMA 3 | Self-hosted models (Deepseek R3)