How We Improved Our AI Agent in Slack by Making It Dumber

Our Slack agent got worse every time we made it smarter. More tools, longer prompts, broader memory — each addition felt like progress. In the channel, it felt like a colleague who had read every wiki page and still could not answer a simple question without a monologue.

The fix was not a better model. We made the main agent dumber on purpose — reduced it to a simple index. Most questions never leave that index. The rest get handed to a sub-agent that already knows how to do one job. The main agent never creates anything. It serves what exists, or it passes a packet of context to something else.

The failure mode of a helpful agent

Helpful agents fail in a specific way. They infer intent you did not state, reach for tools you did not ask for, and invent answers when the right move is to look something up or stay quiet.

In Slack that reads as noise. A thread about a broken deploy does not need a recap of your company values. It needs one of three things: an answer from the runbook, a handoff to the right specialist, or silence.

We had built the opposite — one agent that tried to be the whole company. It searched, summarized, drafted, filed tickets, and improvised policy on the fly. Every message became a reason to create something new.

Before fixing architecture, name exactly how it fails in the channel.

The main agent is an index

We collapsed the main agent down to a single job: hold an index of what the company already knows, and know when to stop answering.

We seeded that index with our LLM — not as an open-ended brain, but as a retrieval surface. Runbooks, policies, client notes, deploy steps, who owns what. The model does not freestyle from training data. It answers from the index first, the way you would answer from an internal wiki you had actually read.

That covers most of what people ask in Slack. “How do we deploy?” “Who handles billing for X?” “What is our refund window?” — lookup, reply, done. No tools, no sub-agents, no thread takeover.

Decision matrices for everything else

Not every question is a lookup. Some need action — file a ticket, pull live data, draft a client email, run a scrape. Those do not belong in the main agent.

The index also holds decision matrices: if the request looks like this, spawn that sub-agent. Not vibes. Rows and columns we can read without opening the model weights.

The matrix is a product surface, not hidden prompt lore.

request_type: deploy_incident
  signals: ["down", "500", "prod", "rollback"]
  sub_agent: incident-runner
  main_agent_may: [lookup_runbook, pass_thread_url, pass_env]

request_type: client_ticket
  signals: ["bug report", "customer name", "broken"]
  sub_agent: ticket-filer
  main_agent_may: [lookup_client_index, pass_summary]

request_type: unknown
  main_agent_may: [lookup_index, reply_with_owner]
  sub_agent: none

Main agent permissions stay narrow: answer from index or pass context.

The main agent does not plan the work. It classifies the request, checks the matrix, and either answers from the index or spawns the sub-agent with exactly the context that agent expects. Creation happens downstream. Upstream is read and route.

What we removed from the main agent

Making it an index meant stripping everything that looked like intelligence but was really improvisation.

• Creation. No drafts, no tickets, no generated policy. If it is not in the index and no matrix row matches, the agent says so and points to a human.
• Open-ended tool use. The main agent does not browse Slack, query APIs, or chain five calls together. Sub-agents own that — with narrow permissions and one output shape.
• Multi-step plans in public. The channel sees a lookup result or a single handoff line. Planning runs inside the sub-agent, not in the thread.
• Personality. We deleted the friendly preamble and the performative empathy. Slack already has humans for that.

Each removal felt like a step backward in a demo. In production it was a step forward in trust — because the main agent stopped guessing.

Serve or send — never invent

The rule is simple enough to put on a sticky note: serve what exists, or send to someone who can act. The main agent is a librarian and a switchboard, not an author.

That constraint sounds limiting until you notice what it buys you. Answers get faster because most requests never spawn anything. Failures get clearer because when something breaks, you know whether the index was wrong, the matrix was wrong, or the sub-agent misbehaved. You are not debugging a single omniscient blob.

When the model drifts, we do not tune temperature. We update the index, add a row to a matrix, or tighten what a sub-agent is allowed to return.

Serve-or-send flow

This is the whole contract. Every turn follows one branch: serve from index, or send a context packet to a specialist.

sequenceDiagram
    participant User
    participant Main as main_agent
    participant Index as memory_index
    participant Worker as sub_agent
    participant Slack

    User->>Slack: mention
    Slack->>Main: normalized request
    Main->>Index: lookup + classify
    alt answer exists in index
        Index-->>Main: answer + source
        Main->>Slack: short factual reply
    else matrix route exists
        Main->>Worker: context packet
        Worker->>Slack: specialist output
    else unknown
        Main->>Slack: owner handoff
    end

What changed in the channel

Fewer wrong-thread replies. Fewer “let me think about that…” messages that never land. More @mentions that actually wanted the bot — because people learned it would either quote something real or kick off a specialist, not riff.

The lesson generalizes. Your main agent does not need to be smart. It needs a good index, honest decision matrices, and sub-agents that do one thing each. Intelligence belongs at the edges — in the seeded knowledge and in the workers — not in a single Slack face trying to do everything at once.

If your agent gets worse every time you add a tool, try the opposite. Reduce the main agent to an index. Seed it with what you already know. Let everything else spawn a specialist. Make it dumber until the channel trusts it again.