Workshop 3

Agentic AI

From a bare API call to a harness: tools, context, guardrails, verification, and an autonomous loop.

NCRM LLMs course - 2026

Prompttask enters

Contextstate + memory

Modelreasoning

Toolsactions

Verifychecked state

Starting point · bare-call-demo

Basic API call: one request, one response, no harness

request

response

The requestPOST /chat/completions

OpenRouter API key

Model

Temperature

Prompt

// exactly what the button runs const res = await fetch("https://openrouter.ai/api/v1/chat/completions", { method: "POST", headers: { "Authorization": "Bearer " + key }, body: JSON.stringify({ model: "openai/gpt-4o-mini", temperature: 0.7, messages: [{ role: "user", content: "What's the difference between an API and a SDK. Rela..." }] }) });

Use sample response

Raw responseopenai/gpt-4o-mini · chat.completion

{
  "id": "gen-1730000000-Xa9",
  "provider": "OpenAI",
  "model": "openai/gpt-4o-mini",
  "object": "chat.completion",
  "created": 1730000000,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "An LLM API call sends your text straight to a model on a remote server and returns its generated response as plain data: no chat interface, no memory, no tools. For a survey researcher, it means you can script model outputs and log every parameter, rather than clicking through a chatbot you cannot reproduce."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 22,
    "completion_tokens": 72,
    "total_tokens": 94
  }
}

tokens in 22 tokens out 72 total 94 took 503 ms

Clean responsechoices[0].message.content

An LLM API call sends your text straight to a model on a remote server and returns its generated response as plain data: no chat interface, no memory, no tools. For a survey researcher, it means you can script model outputs and log every parameter, rather than clicking through a chatbot you cannot reproduce.

The important absence: no memory, no tools, no loop, no verification. The call ends when generation stops.

The opposite case · n8n demo

Embedd the API call in a workflow

single prompts

->

workflows

->

single agent

->

multi-agent systems

Graph firstIn n8n the workflow graph decides the order of steps, branches, credentials, retries, and external calls. Here we control the entire workflow.

LLM nodeThe LLM can sit inside the graph, but it does not own the graph. The workflow stays explicit and inspectable.

Teaching contrastDeterministic orchestration first, agentic autonomy second. n8n is the controlled opposite of an agent.

Triggernew row

Fetch dataAPI call

LLM nodefixed slot

Ifbranch

Notifyemail / slack

Logsheet / db

The graph owns order, branching, credentials, retries, and integrations. The model is one node inside a deterministic workflow.

Autonomy frame

What if the LLM had autonomy?

Goal

The user names the desired outcome, not the steps.

Brain

The LLM is the brain. It reasons about the next action.

Harness

What does the brain need to get there? A harness that turns reasoning into bounded work.

goal: upvote the item

choose next step

call browser tool

check final state

Agent
loop + tools + checks

The surrounding software starts deciding what the model can see, remember, do, and verify.

What an agent needs

The agent loop

Perceive

Reason

Act

Observe

2023: compact ReActThink, choose a tool, observe the result, repeat. Useful, but mostly a thin loop around short reasoning.

Extended reasoningThe model spends more time building and revising an internal plan before it commits to an action.

Self-correctionFailed checks and tool errors are fed back as evidence, not treated as conversational inconvenience.

Planning and discoveryThe agent can decompose work, look up relevant files or capabilities, and load tools progressively instead of starting with everything in context.

Same loop shape; heavier machinery inside each turn.

The four components

The brain, the tools, the memory, and the runtime

LLMThe reasoning engine that selects the next move.

ToolsThe named actions the model may request.

MemoryConversation, trace, files, and durable facts carried forward.

Runtime

The brain, the tools, the memory, the environment

Agent EnvironmentRuntime is the operational surface: what process is running, where files live, what credentials are mounted, and what side effects are possible.

Runtime shapes capabilityA browser-only runtime can click and read pages. A repo runtime can edit files and run tests. A locked sandbox can make reset cheap and secrets inaccessible.

Diagnostic questionWhen the agent fails, ask whether the model lacked reasoning, the tool was missing, memory was stale, or the runtime could not support the action.

runtime = { browser: isolated session, filesystem: mounted project, shell: allowed commands, network: policy boundary, credentials: explicitly mounted }

basic-ai-harness · the motivating failure

We give the LLM a browser. Is a tool enough to accomplish the task?

The task

Use a browser to open Hacker News, upvote a target story, handle login if needed, and only claim success when the trace proves the upvote landed.

Hypothesis

If a model can use a browser, maybe it can accomplish the task.

Simple task, one tool...is it enough?

A browser-capable agent can still report success without evidence. It reaches a login or redirect, infers success, and says the work is done.

browser_click(a[id="up_..."]) · now at /vote?id=... "since the page redirected ... the upvote likely succeeded" Verification: none

What do we build around it?

A comprehensive harness

Tool registryDefine the actions and bind them to a real runtime.

Context managerTrack what happened and trim stale history.

Guardrails

Cap loops, block bad actions, require approvals.

VerifierCheck whether the result is actually true.

Owned runtimeBrowser, filesystem, sandbox, credentials, cleanup.

Under the hood · 1. tool registry

Every tool has two parts: a schema for the LLM and code for the harness

agent/1-tools.tsTool

export type Tool = { definition: ChatCompletionTool; // the model sees this execute: (args) => Promise<string>; // the harness owns this }; { definition: { type: "function", function: { name: "browser_navigate", parameters: { required: ["url"] } } }, execute: async ({ url }) => session.navigate(url), }

The model choosesBrowser, filesystem, sandbox, credentials, cleanup.

The harness executes

Every tool has two parts: a schema for the LLM and code for the harness

The trace recordsThis boundary is what makes a run inspectable. The tool result is stored, not just narrated.

Under the hood · 2. the agent loop

The loop stops in exactly two places

agent/5-loop.tsstop conditions

// 1. a guardrail says stop if (!guardrail({ iterations, messages }).ok) { return { answer: reason, stoppedBy: "guardrail" }; } const choice = await callModel(model, messages, tools); // 2. the model gives a final answer if (choice.finish_reason === "stop") { return { answer: choice.message.content, stoppedBy: "model" }; } // otherwise: run the tools, append results, loop again for (const call of choice.message.tool_calls) { result = await tools.byName.get(call.name).execute(args); toolEvents.push({ tool, args, result }); // trace messages.push({ role: "tool", content: result }); // context }

Two exits onlyA guardrail halts a structurally bad run, or the model returns a final answer. Nothing else ends the loop.

If it asks for tools, it does not stop

The loop stops in exactly two places

One result, two destinationsEvery tool result becomes both context (a role:"tool" message the next call sees) and trace (a stored record humans and verifiers can inspect). The trace is not an afterthought.

Under the hood · 3. verify & recover

Verification reads the trace

agent/6-harness.tsverifySuccessfulUpvote

const events = result.trace.flatMap(i => i.toolEvents); // a valid upvote is EVIDENCE in the browser trace: const ok = events.find(e => e.tool === "browser_click" && /up_/.test(JSON.stringify(e.args)) && /ycombinator\.com\/(news)?$/.test(landedUrl(e)) ); // rejects unrecovered /vote or /login redirects if (!verification.passed && attempt < maxAttempts) { console.log("failed - retrying..."); // harness retries }

Semantic success is a deterministic predicate

Verification reads the trace

agent/6-harness.tsverifySuccessfulUpvote

const events = result.trace.flatMap(i => i.toolEvents); // a valid upvote is EVIDENCE in the browser trace: const ok = events.find(e => e.tool === "browser_click" && /up_/.test(JSON.stringify(e.args)) && /ycombinator\.com\/(news)?$/.test(landedUrl(e)) ); // rejects unrecovered /vote or /login redirects if (!verification.passed && attempt < maxAttempts) { console.log("failed - retrying..."); // harness retries }e

The harness can handle known failures without asking the model to improvise. A login redirect is repaired by harness_auto_login and recorded as a trace event. The model still decides what to do next; the brittle operational step is deterministic code.

Two different checks

Guardrails ask "should this run continue?"

Verification asks "did it succeed?"

Guardrails — structural failure

Catch too many steps, forbidden actions, unsafe file changes, or missing approval. They stop structurally bad runs before they spiral. They do not make the final answer true.

Guardrails ask "should this run continue?"

Verification asks "did it succeed?"

Guardrails: structural failure

You need both. A run can obey every guardrail and still be semantically wrong.

Class demo flow

Testing the harness

$ npm run basic-agent
browser tools: yes
harness verifier: no
auto-login: no

Can claim success after /vote redirect.

$ npm run agent:upvote:login
browser tools: yes
harness verifier: yes
auto-login: yes

Passes only when trace proves success.

Optional: npm run trace:md -- --latest turns the JSON trace into a student-readable walkthrough.

harness-loop-demo · native

When failure modes are missing tools...

task ->

Contextmemory

Modelweights

Toolsact

Guardrailbound

Verifycheck

-> answer

The taskdeterministic teaching model

Task

Toolslet the model run the query instead of guessing

Memorykeep earlier turns so follow-ups make sense

Verifycheck the final answer against the data

Guardrailcap the steps so a bad loop cannot run away

max stepsbefore halt

Loop tracedone

harnessTask received: What is the mean of stflife (life satisfaction) for Germany?; memory kept 3 item(s)

modelstep 1: thinking about the task

modelwants to act -> query_data("stflife", "DE")

toolmean of life satisfaction (stflife) for Germany = 7.20

modelstep 2: thinking about the task

modelanswers: "The mean of life satisfaction (stflife) for Germany is 7.20."

verifyPASS - matches the data (7.20)

Result2 steps used

The mean of life satisfaction (stflife) for Germany is 7.20.

verified

Suggested reveal: everything off; then Verify + Guardrail; then Tools + Verify. The fix is a missing capability, not a better pep talk.

Part 2

Testing the harness

Optional: npm run trace:md -- --latest turns the JSON trace into a readable walkthrough.

Definition & anatomy

Everything we talked about, falling into one system

A complete harness

The LLM, tools, a planning loop, context engineering, a sandbox, memory, an orchestration layer, and a serving layer — the scaffolding that wraps an LLM to make it useful for real work, not just answering one prompt.

The diagnostic questionWhen something fails, ask what capability is missing, and how to make it legible and enforceable. Harness engineering moves attention from better wording to better runtime design.

Tool registry

The registry is the action surface

Without toolsThe model is just a chatbot. It can discuss actions but cannot perform them.

With toolsThe harness exposes specific actions: read, write, search, browse, query, run code, call APIs.

BoundaryNaming, arguments, permissions, and errors all shape behaviour. The early failure mode is exposing too many tools.

tools = { "read_file": { "path": "string" }, "search": { "query": "string" }, "browser_click": { "selector": "string" }, "run_tests": { "target": "string" } } The model chooses. The harness executes. The trace records.

Context & state

Long runs fail through compounding errors

Reliability frameStale assumptions, lost intermediate decisions, context overflow, and workers that disagree about hidden choices. At the core of reliability is context engineering.

Principle 1Share context, and share full agent traces, not just individual messages. The trace lets later steps understand why earlier steps happened.

Principle 2Actions carry implicit decisions, and conflicting decisions carry bad results. The system must preserve the moments that shaped the run.

TradeoffCompression improves longer tasks, but summaries can erase the decisions that matter.

Still unreliable long-running agent diagram

Simple and reliable sequential agent diagram

Context compression for longer tasks diagram

Cognition, "A Theory of Building Long-running Agents."

Guardrails

Guardrails sit outside the model and stay deterministic when the boundary matters

Block

One harness, multiple agents?

Warn

Surface risk without stopping low-risk work.

Cap

Limit iterations, messages, spend, and time so a bad loop cannot run away.

PlacementGuardrails answer "should this run continue?" not "did the task succeed?"

Scope

Guardrails sit outside the model and stay deterministic

Runtime & sandbox

The real lever is the runtime (& permissions)

Folder accessWhich directories can the agent read, write, or execute in? Read is safer than write, but secrets and credentials may still be visible.

Tool accessWhich actions: shell, Python, Git, package installs, browser tools, MCP servers, long-running jobs, network calls?

Worst caseThe practical safety question is what damage is possible if the model is wrong for a long time.

RecipeMount only the project folder, omit home-directory secrets, drop capabilities, disable network after setup, make reset cheap. Sandboxing moves the hard thinking ex ante — so agents can run longer without approving every step.

Terminal backend picker: Local, Docker, Modal, SSH, Daytona

docker run --rm -it \ -v "$PWD":/work:rw -w /work \ --cap-drop=ALL --security-opt=no-new-privileges \ --network=none agent-workbench Agent sees /work, not your whole home directory; approved CLIs, not every host tool; no API keys unless deliberately mounted; git diff as rollback evidence.

Based on Paul Goldsmith-Pinkham, "Permissions, Sandboxes, and Autonomous Agents."

Verification

The call ends when the goal is checked, not when the model stops talking

The real lever is the environment (& the permissions)

The model says, "done."

Better signal

The harness checks the actual final state against the goal.

Verification can be

A test, a query, a browser check, a structured scorer, or a human approval gate. Whatever turns "success" into a deterministic predicate the model cannot narrate away.

See Paul Goldsmith-Pinkham, "Permissions, Sandboxes, and Autonomous Agents."

·

The call ends when the goal is checked

(not when the model stops talking)

·

browser check

·

scorere

·

approval

Control surface

Which part of the harness do we actually control?

Layer

Mostly sealed

Usually configurable

Model product

weights, core runtime, product loop

settings, prompts, files, tool permissions

Coding agent

Which part of the harness do we actually control?

skills, hooks, MCPs, commands, project rules

Primitive harness

less is sealed by default

runtime, tools, loop, verification

If we use existing software — Claude Code, Codex, Cursor — we only control part of the harness. That configurable part has a name for each piece.

Part 3

Configurable surfaces

Skills, progressive disclosure, commands, hooks, MCPs, plugins — the parts of the harness you configure.

Mental model

Commands, skills, MCP: three layers that compose

Layer

Primary job

Token pattern

Commands (guardrails)

for most existing software (Claude Code, Codex, Cursor), we only control part of the harness.

Configurable harness

Skills, progressive disclosure

A skill is a prompt? o.O

small idle metadata; full body when loaded

MCP (actions)

call an API, run a query, fetch a resource

definitions in context; work often server-side

System prompt = instinct. AGENTS.md = project README. Tools/MCP = hands. RAG = library. Skills = the runbook.

Skills

A skill is a prompt? o.O

my-skill/ |-- SKILL.md # required |-- scripts/ # deterministic code |-- references/ # load on demand `-- assets/ # templates, schemas --- name: literature-review description: Use when the user asks to... --- 1. Clarify the target output. 2. Load only the needed references. 3. Run the prescribed tools or scripts. 4. Check the result against the rubric.e

Reusable instruction bundleBuilt around SKILL.md: a stable interface, opinionated workflow, and bundled resources that can be installed, versioned, tested, and reused across projects.

Trigger metadataThe description is doing double duty. The agent uses it to decide when to load the skill, so "Use when the user asks to..." beats vague marketing like "Helps with Git."

Procedural memorySkills are the first credible procedural-memory primitive for agents: the runbook an experienced colleague hands you on day one.

PortabilityA markdown folder works anywhere with filesystem access — Claude Code, Codex, Cursor — with no vendor lock-in at the artifact layer.

Progressive disclosure

Context bloat? Metadata first, full skill on demand.

Level

What loads

When

L1 metadata

name + description

every turn (~50 tokens/skill)

L2 body

full SKILL.md

Commands, skills, MCP: three layers that compose

L3 bundle

references, scripts, assets

the body references or executes

Token economyCommands (guardrails)

A skill without a test is a hopeSkillsBench found 19% of tasks performed worse with a skill than without. Failure modes: wrong trigger, wrong execution, token-budget bloat, and regressions that break existing routing.

TargetRoughly 90% trigger accuracy on descriptions. Vercel data showed a 56% non-invocation rate for skills expected to fire consistently.

Commands

A command is deterministic; a skill is a suggestion

Command

A user-initiated shortcut. You type /command, the tool injects the prompt, and the agent executes it. Commands are for ergonomics, so keep them stable.

Skill

The agent decides whether it needs that context and loads it when appropriate. Skills are for policy; they need to be reviewable.

/refactor component.tsx tanstack load the tanstack and panda-css skills, then refactor the component. $1 = file, $2 = target

The strongest pattern: keep complex, long-lived instructions as skills; use commands as ergonomic shortcuts to trigger any combination of them. Update a skill and you change behaviour without a new incantation; update a command and you change the incantation itself.

Hooks

Event-driven automations that fire before or after tool executions

UserPromptSubmit

->

PreToolUse

->

tool call

->

PostToolUse

->

Stop

Before toolsA PreToolUse hook can block (exit code 2) or warn: stop a dangerous command, protect .env files, require approval, or rewrite tool input before it reaches the shell.

After toolsA PostToolUse hook can inspect output, run a formatter after edits, compress verbose logs, or record telemetry — it cannot block.

LifecycleStop hooks run after each response; SessionStart/End at boundaries; PreCompact before context compaction, useful for saving state.

Where they liveClaude Code: .claude/settings.json. Codex: .codex/hooks.json. Structured event data enters deterministic code; the model does not enforce the boundary itself.

Hook lifecycle diagram: session, main loop, tool execution, subagent

MCP · Model Context Protocol

MCP is a wrapped API...

ClientThe host — Claude Code, Codex, Cursor — talks to the MCP server and passes tool options back into the LLM loop.

ServerExposes tools, resources, and prompts, typically wrapping an API endpoint. If an organisation ships it for their service, it becomes the single thing LLMs plug into.

Tool sprawlThe early failure mode, flagged by Gumloop, Cloudflare, and Anthropic: we exposed LLMs to too many tools. A good server curates capabilities and keeps arguments legible.

MCP architecture: client, FastMCP server, tools, API calls

Client · server · tools · API. Build one with FastMCP: a name, instructions, then @mcp.tool / @mcp.resource / @mcp.prompt.

Plugins

Plugins bundle skills, subagents, commands, hooks, and MCPs as one installable unit

Bundle part

Example

MCP is a wrapped API

Skills

review, research, security

procedure and examples

Hooks

PreToolUse, Stop

policy and automation

MCP

Gmail, Drive, Slack

external capabilities

Assets

templates, scripts

shared implementation files

Distribution unitInstead of scattered markdown and install scripts, a plugin turns everything into a versioned, installable package. The install script detects your directory and copies files to the right locations.

Decision treeRemember X automatically → skill. Automate a workflow step-by-step → subagent. Frequent shortcut → command. Share your setup → plugin, which bundles all three.

Trust boundaryBundled hooks and MCP servers deserve code review. Installing a plugin can extend what the agent is allowed to do.

Types of software harness

From vertically integrated to primitives: Claude to Pi, OpenCode, and Letta

Shape

Examples

Tradeoff

Vertical product

Claude Code, Codex, Cursor

fast start; loop, context, and model are bundled and mostly sealed

Controlled workflow

n8n, Gumloop graphs

observable and deterministic; autonomy stays inside explicit nodes

Primitive harness

Pi, OpenCode, Aider, Roo Code, Letta / Letta Code

more exposed control; more assembly and judgment required

The question becomes concrete: which part of the harness are we changing, and which part is hidden inside the product?

Vertical product

Fast start, but loop, memory, tools, and verification are mostly bundled.

Claude Code

Codex

Cursor

Controlled workflow

Explicit nodes, inspectable transitions, and bounded autonomy.

n8n

Gumloop graphs

Primitive harness

More exposed control over models, memory, tools, runtime, and local workflow.

Pi

OpenCode

Aider

Roo Code

Letta / Letta Code

Demos & where to find skills

What changes when we use skills?

Skills

Load a playbook on demand.

Hooks

Run checks around tools.

Commands

Start repeatable workflows.

MCPs

What changes when we use skills?

Matt PocockSkills as reusable discipline — TDD, planning, architecture — with the user-invoked / model-invoked taxonomy.

Scott CunninghamResearch, teaching, and workflow skills as reusable craft.

skills.sh · first-party reposVercel's directory to browse public skills; google/skills, anthropics/skills as safer defaults. Pin versions and audit like code.

Part 4 · discussion

Rethinking what we learnt

Agents make us reconsider context, prompting, and the human in the loop.

1 · feeding context

grep > semantic search. Is the vector database still needed? Is RAG dead?

Retrieval habitThe old reflex was to build RAG: chunk documents, embed them, retrieve semantically similar passages, paste them into context.

Agent contextAgents need more than documents: full traces, tool outputs, diffs, errors, decisions, runtime state, and why earlier actions happened.

Exact evidenceSkills as reusable discipline: TDD, planning, architecture — with the user-invoked / model-invoked taxonomy.

Current questionVector databases are still useful for fuzzy corpora. They are no longer the automatic answer to every context problem.

Feed

Rethinking what we learnt

Risk

grep / rg

symbols, filenames, logs, tests, exact claims

misses unstated similarity

semantic search

large fuzzy corpora, unknown phrasing

plausible but weak evidence

trace

preserving actions, observations, decisions

noise without compression

state manager

keeping the next model call coherent

bad trimming erases decisions

2 · prompting

Prompt engineering is (sort of) dead: what matters is skills and workflows

Single promptA one-off instruction tries to carry intent, context, procedure, quality bar, style, examples, and evaluation in one fragile message.

Skills and workflowsNot what we tell the agent, but the skills and tools it has available. The durable part becomes the workflow: instructions, scripts, examples, rubrics, tool choices, checks — versioned and reused.

Tools and loopWhat matters is what the agent can use to engage the task, execute it, evaluate it, and loop until it satisfies the goal.

Goal -> select relevant skills -> expose tools and resources -> act in owned runtime -> observe trace and state -> verify against success condition -> retry, escalate, or stop Prompt engineering becomes less about "find the magic wording" and more about building the repeatable workflow around the model call.

3 · the human in the loop

Human in the loop? The /goal is to remove it

Human in the loopThe agent pauses and asks before every significant action. Safe but slow — right for high-stakes tasks: production databases, emails, payments, irreversible actions.

Human on the loopThe agent works autonomously while a human monitors and can intervene, like a self-driving car with a hand near the wheel. No permission per step, but a kill switch.

Harness designPrompt engineering is dead: what matters are skills and workflows

Autonomy targetGive the agent a goal and enough autonomy to pursue it. Human intervention becomes exceptional rather than the default unit of progress.

Responsibility moves into the harnessPolicy, logging, auditability, rollback, evaluation, and clear boundaries for where the agent may act.

Workshop questionIf verification is part of the loop, what should still require a person — and what should be automated because the harness can check it better?

Close

The model is the brain.

The product is the loop, the tools, the context, the environment, and the checks, everything except the weights.