Lessons in Agentic Workflow Design

Introduction

BAA Black Sheep was an early experiment of a large scale agentic workflow using LangGraph. The project served as a testing ground for exploring LLM capabilities and agentic pattterns in a realistic use case. There are severak flaws in its design, to be discussed later, but ultimately it was shelved to make way for newer projects as the world of LLMs and Agents has rapidly evolved.

The Project

BAA Black Sheep was designed as a system for managing user stories and process maps through AI agents. The architecture employed multiple specialized agents, each handling specific tasks such as:

• Analyzing user story details
• Determining if a story was new or existing
• Generating and modifying user stories
• Creating and updating process maps
• Providing reflective feedback on story quality

Technical Implementation

The system was built using:

• LangChain for agent orchestration
• LangGraph for workflow management
• Multiple LLM integrations (Cohere, Groq)
• Vector stores for document retrieval
• BPMN standards for process mapping

Node Structure Analysis

The BAA Black Sheep project employs a complex nodal structure to manage its agentic workflow, comprising 18 different nodes and 14 'proto-agents'. At the core of this system is the task_router, which functions as the primary decision-making node, directing tasks to specialized agents based on the type of user story or process map involved. This node is crucial for maintaining the flow of information and ensuring that each task is handled by the appropriate agent, such as analyse_user_story for new stories or get_existing_user_story for existing ones. The workflow also integrates human input through the ask_human node, which serves as a critical checkpoint for tasks that require human judgment or when the system is uncertain about the next steps. This node allows for a feedback loop, ensuring that the system can adapt and refine its processes based on human insights. Additionally, nodes like generate_user_story and rewrite_user_story highlight the system's capability to autonomously create and modify content, showcasing the balance between AI-driven automation and human oversight.

Overall, the nodal structure of BAA Black Sheep reflects a thoughtful integration of AI and human elements, leveraging the strengths of both to create a robust and flexible workflow. This design not only facilitates efficient task management but also ensures that the system can evolve and improve over time, adapting to new challenges and requirements in the realm of business analysis and process mapping.

Key Learnings

The Power and Pitfalls of Agentic Design

While the project successfully demonstrated the potential of AI agents in workflow automation, it also highlighted a crucial insight: not every component of a system benefits from being agentic.

For example, our initial design relied heavily on natural language processing for task routing and classification. The task_router_agent would analyze user input to determine whether it related to:

• New user stories
• Existing user stories
• Process maps
• Project questions
• Gap analysis

In retrospect, many of these routing decisions could have been more effectively handled through traditional UI elements like dropdown menus or radio buttons, offering:

• Faster user interaction
• Zero-error classification
• Clearer user expectations
• Reduced system complexity

Finding the Right Balance

The project taught us to be more judicious in deciding which elements truly benefit from AI agency. The most effective components were those where AI added unique value, such as:

• Generating creative content for user stories
• Providing nuanced feedback on story quality
• Analyzing complex relationships between requirements

Meanwhile, structural decisions and categorical choices were often better served by conventional UI patterns.

Impact on Our Development Philosophy

This early experiment has significantly influenced our approach to AI integration in subsequent projects. We now follow a hybrid model where:

1. AI agents handle tasks requiring creativity, analysis, and natural language understanding
2. Traditional UI patterns manage structural and categorical decisions
3. The two approaches complement each other, creating more robust and user-friendly systems

Conclusion

BAA Black Sheep, while experimental, provided valuable insights into the practical application of AI agents in software development workflows. It demonstrated that the future of AI integration lies not in wholesale replacement of traditional patterns, but in thoughtful combination of AI capabilities with proven software design principles.

The project continues to influence our development decisions, reminding us that sometimes the most innovative solution isn't about maximizing AI usage, but about finding the right balance between artificial and traditional intelligence.