← All insights
Share this insight
LinkedIn

In this insight

A useful agent has a clear job

An AI agent becomes useful when its responsibilities can be described in operational terms. What information does it receive, what result should it prepare, which actions may it take, and when should a person intervene? Those questions create a more practical starting point than a broad request to automate a department.

Language-based systems can help organize requests, retrieve relevant material, and prepare outputs for review. They can also connect with tools that perform defined actions. Their usefulness depends on the workflow, the information available, and the controls around execution. Treat the first implementation as a bounded system to evaluate, with explicit expectations and a responsible owner.

Understand the existing workflow first

Imagine a hypothetical support team receiving requests through a shared inbox. An assistant might classify a request, find relevant internal guidance, and draft a response for a person to approve. This is a concrete workflow with recognizable inputs and outputs. It also leaves the team able to handle unusual or sensitive cases.

Before implementation, map how that work happens today. Identify where people wait, what information they search for, and which decisions require judgment. Record exceptions as well as the usual path. This often reveals that some steps need a simple integration or a clearer process, while others are suitable for AI assistance.

Connect knowledge with context

An assistant needs relevant information to work on a company-specific task. Retrieval can make selected documents available during a request, but the source material still needs ownership and review. Conflicting instructions, outdated policies, or missing context can make a plausible response unhelpful or incorrect.

Decide which information belongs in the workflow and which users are authorized to access it. Preserve enough source context for a reviewer to assess the answer. Define what the assistant should do when information is unavailable or inconsistent, such as ask a question or route the request to a person. Access to more documents does not automatically improve the result.

Separate preparation from consequential action

Drafting a summary and changing a business record involve different responsibilities. An agent that proposes an action can be reviewed before execution. An agent allowed to act directly needs clearly defined permissions, validation, and a way to recognize when a request falls outside its intended scope.

Begin by documenting permitted actions and their conditions. Consider which changes require approval, how duplicate requests are handled, and what should happen after a failure. Keep tool access appropriate to the task. For consequential workflows, involve the relevant process owners in deciding where human review belongs. Avoid treating a model's confidence in its response as authorization to make a change.

Design the handoff to people

Human review works best when it is a defined part of the workflow. A reviewer needs the original request, the information used, the proposed output or action, and a clear explanation of what remains uncertain. A vague instruction to check everything can simply move the original workload into a different interface.

Identify the conditions that should trigger a handoff. These might include missing information, conflicting guidance, an unsupported request, or a decision outside the agent's permission boundary. Provide a route for the reviewer to correct the result and continue the work. The system should make exceptions visible and keep responsibilities understandable when the normal path cannot be completed.

Evaluate with representative work

A successful demonstration shows that one example worked. Evaluation needs a broader set of requests, including incomplete inputs, unusual wording, conflicting documents, and tasks the assistant should decline or escalate. Use examples that reflect the actual workflow, with appropriate handling of any sensitive information.

Review output quality, action correctness, latency, operating cost, and the effort required from people. Compare these observations with the existing process. Keep a record of failure patterns and changes made in response. Revisit evaluation when prompts, models, source material, or integrations change, because the assumptions behind an earlier result may no longer apply.

Expand responsibility deliberately

The first release should have an owner who can monitor the workflow, review exceptions, and decide when to adjust or pause it. Define what a useful result looks like and how the team will notice deterioration. Include operating costs and review effort when assessing value rather than counting only the steps handled automatically.

A practical starting exercise is to choose one recurring request, map its current path, and mark where information gathering, drafting, and action occur. Select a bounded portion for a pilot. Expand only when evaluation supports the next responsibility and the organization is prepared to operate it. Useful automation remains accountable to the process it serves.

Further reading

NIST's AI Risk Management Framework provides voluntary guidance for considering AI risk during design, development, use, and evaluation. Explore its broader risk framing at https://www.nist.gov/itl/ai-risk-management-framework.

NIST's Generative Artificial Intelligence Profile discusses considerations including governance and pre-deployment testing. It is a useful companion when planning evaluation and oversight for an AI workflow: https://doi.org/10.6028/NIST.AI.600-1.

Share this insight
LinkedIn