AI Enablement

FyerX helps organisations introduce generative AI into a defined task with an identified reviewer. We design retrieval, model integration and evaluation around actual business content so the team can judge usefulness, cost and mistakes before expanding access.

Discuss your technology project

When to bring in FyerX

One reason to review knowledge-assisted support is a revised expense policy with separate rules for existing bookings. It raises questions about document version, effective date, permission scope and cited passage, the groups affected and the action each can take when something goes wrong. Our starting point is the policy question and retrieved supporting material. We trace it from authoritative passage to reviewed response with the people who own those decisions. The result gives your AI use-case owner a specific first release, a testable operating outcome and a record of dependencies. For this example, the practical standard is simple: a reviewer can justify every consequential claim against material available to that user.

Choosing a human review point

A suggested internal summary and a customer-facing commitment carry different consequences. Decide which outputs a user may edit, which actions require approval and which out-of-scope requests should be handed back to a person. Put that decision into the acceptance tests.

Use-case selection

Choose a task with a clear input, a reviewable output and a person who owns the consequence of an error. Record how the task works today to create a comparison point. Bring the policy question and retrieved supporting material and its history to the design discussion. Check document version, effective date, permission scope and cited passage against the policy in effect at the time. For a case involving a revised expense policy with separate rules for existing bookings, ask how the team currently recognises that the decision is complete. Agree whether retrieval respects current access and uses the applicable policy version.

Retrieval and content preparation

Identify authoritative documents, effective dates and access boundaries. Test retrieval against practical questions, including questions for which the source contains no answer. Include content owner, frontline reviewer and application administrator in the walkthrough. Each sees a different part of the policy question and retrieved supporting material. Ask them to trace authoritative passage to reviewed response and establish where authority to approve an answer intended for a customer actually sits. Differences between the documented responsibility and the person carrying out the action belong in the scope.

Prompt and application integration

Design how the assistant uses context, cites source passages and handles missing evidence. Keep consequential actions subject to agreed approval and permission boundaries. Work through an ordinary employee asking about a policy held in a restricted collection. Inspect the question, retrieved passages and reviewer correction history before and after the event and ask the receiving team to explain the difference. A useful test includes conflicting documents and questions with insufficient evidence, especially when the business consequence is a grounded-sounding answer quoting an obsolete policy. Agree which information to keep and who will use it during a policy publication cycle.

Illustrative engagement

Pilot an internal policy assistant

  1. Starting point

    HR wants faster answers but policies differ by location and some pages are obsolete.

  2. Decision

    Curate current documents, define access boundaries and test questions that should trigger a referral to HR.

  3. Deliverable

    A reviewed answer set, documented failure modes and a decision on the pilot's next release.

Evaluation and adoption

Build a representative evaluation set with subject experts. Assess correctness, source support, response time and review effort before extending the pilot to more users. For operational handover, give the AI use-case owner a worked example from the evaluation set and correction register. Reconcile the result against responses supported by current authorised material. Agree how to route the question to a human reviewer with the retrieved sources when the evidence falls outside the accepted range.

Setting a useful pilot evaluation

Select questions from ordinary work and include difficult cases where documents disagree, evidence is missing or a recent policy supersedes an old instruction. For each case, specify the source a reviewer should consult and the minimum acceptable answer. Keep a separate set of examples for the final evaluation so iterative changes do not overfit the development set. Record the time a reviewer spends correcting an answer alongside its quality. A draft that reads fluently can still consume more editorial effort than the original process, particularly when staff must independently verify every claim.

Decide which answers need a person

A knowledge assistant for service agents earns its place only when an agent can judge its answer. We start with a small set of questions from an actual queue, including ambiguous requests and questions the approved documents cannot answer. The content owner marks acceptable source passages and identifies statements that would require escalation. This forms an evaluation set with a business owner and a reason for every expected outcome, giving the sponsor something more useful than a polished demonstration when deciding whether to proceed.

For the pilot, we show the operator the source material beside the proposed answer and ask them to record what they changed. Some edits reveal an outdated procedure in the knowledge base, while others reveal that the assistant has combined instructions meant for different customer groups. We track the two problems separately because they have different owners. The rollout proposal sets a review process for unanswered questions, a method for removing superseded documents and a clear fallback when the source is inconclusive. It also estimates the ongoing editorial effort the pilot uncovers.

What to bring to discovery

Bring an anonymised policy question and retrieved supporting material, a recent evaluation set and correction register and an example of a revised expense policy with separate rules for existing bookings. Where the inputs contain sensitive information, supply synthetic equivalents that preserve the decision sequence. Ask the AI use-case owner to identify who may approve approve an answer intended for a customer. Record document version, effective date, permission scope and cited passage with their meanings, including whether an empty value is permissible. Invite the people who use the question, retrieved passages and reviewer correction history to describe the hardest discrepancy from the last cycle. The discovery log should capture the source evidence, responsible team and decision required.

Agreeing the first work package

Anchor the first work package in use-case selection and choose a deliverable such as an evaluation set with reviewer scoring guidance. Define the boundary from authoritative passage to reviewed response so both teams know what a completed item looks like. For planning purposes, list conflicting documents and questions with insufficient evidence separately from ordinary cases and record whether each is included in release one. Material scope changes include a second knowledge collection, a different owner for policy question and retrieved supporting material or a dependency that changes a policy publication cycle. Agree the cost and acceptance consequence of each before committing delivery capacity.

An acceptance exercise for your team

Give the operators an agent asking for guidance on a policy revised last week. Ask the AI use-case owner to describe the expected business result before showing them the implementation. Introduce a revised expense policy with separate rules for existing bookings and follow the chain from authoritative passage to reviewed response. At each point, record the values of document version, effective date, permission scope and cited passage that matter to the decision. The test succeeds when a reviewer can justify every consequential claim against material available to that user. If evidence from the question, retrieved passages and reviewer correction history disagrees, stop at the first observable difference and establish who can resolve it. Keep the starting data and explanation with the evaluation set and correction register. This example can be adapted to the rules and responsibilities of your organisation.

Managing policy updates

Identify who owns the publication date, authoritative version and withdrawal of each source. After an update, test questions that previously returned a different answer. A refreshed index is only one checkpoint. The reviewer should verify that the user now receives the correct policy for the relevant effective date.

Planning around the operating calendar

Build the release around a policy publication cycle. For knowledge-assisted support, that means agreeing when the question, retrieved passages and reviewer correction history are available, when the AI use-case owner can decide and what work would be interrupted by a late change. Select test material covering conflicting documents and questions with insufficient evidence before reserving a business sign-off slot. Treat authority to approve an answer intended for a customer as a release dependency. If a second knowledge collection is introduced at the same time, check whether the original baseline for answer usefulness and corrections required by reviewers still supports a meaningful comparison.

Exception ownership and recovery

One credible failure is a grounded-sounding answer quoting an obsolete policy. The first diagnostic step is to inspect the question, retrieved passages and reviewer correction history and identify the authoritative version of the policy question and retrieved supporting material. Next, determine whether retrieval respects current access and uses the applicable policy version. For an incident involving questions outside the knowledge base, an operator may need to route the question to a human reviewer with the retrieved sources. Agree that authority with the AI use-case owner ahead of a policy publication cycle. Capture the resolution in an evaluation set with reviewer scoring guidance, including how content owner, frontline reviewer and application administrator can confirm recovery from their different perspectives. For repeat occurrences, compare the underlying fields rather than assuming every item in the exception queue has the same cause.

Handover and ongoing ownership

Put an evaluation set with reviewer scoring guidance in the hands of the staff who will own knowledge-assisted support. Include a completed example from authoritative passage to reviewed response and a walkthrough for a revised expense policy with separate rules for existing bookings. Someone outside the delivery team should be able to locate the question, retrieved passages and reviewer correction history, identify authority to approve an answer intended for a customer and act when the evaluation set and correction register shows a discrepancy. After the rehearsal, update ownership for policy question and retrieved supporting material and document where to propose changes to document version, effective date, permission scope and cited passage.

Planning your engagement

How to measure the outcome

Agree the measurement definition for answer usefulness and corrections required by reviewers before changing knowledge-assisted support. Start with responses supported by current authorised material, then review conflicting documents and questions with insufficient evidence as a separate population if their expected treatment differs. A reliable baseline needs the reporting date, an owner for policy question and retrieved supporting material and a consistent definition of completion from authoritative passage to reviewed response. Test the evaluation set and correction register with a case involving a revised expense policy with separate rules for existing bookings. If a second knowledge collection alters the population later, preserve the original baseline alongside the revised one. This gives the AI use-case owner an explanation for a change in the number and a decision about which improvement to fund next.

Delivery and support responsibilities

For an advisory phase, ask FyerX to focus on the decision behind use-case selection. For an implementation, agree the behaviour of policy question and retrieved supporting material through authoritative passage to reviewed response and the evidence required from the question, retrieved passages and reviewer correction history. For a team that already owns delivery, a bounded specialist assignment can focus on prompt and application integration. Keep authority to approve an answer intended for a customer with its named business or technology owner unless explicitly delegated.

Budgeting for an AI pilot

Choose a bounded task and a representative evaluation set. Include the cost of preparing permitted source material, reviewing generated answers and handling uncertain results. The economics can look quite different when a reviewer has to rewrite most outputs. Measure that effort before extrapolating to a larger rollout.

Frequently asked questions

AI Enablement

How do you evaluate responses when more than one answer is acceptable?

Ask subject experts to define what must be present, which claims require evidence and what would make an answer unsafe to use. Give reviewers the same examples and compare their ratings before the pilot. Keep misunderstood questions in the evaluation set instead of quietly removing them.

Does retrieval solve confidential-content access?

Retrieval still needs to respect the user’s access rights. Test questions whose relevant answer exists only in a restricted source and check the returned passages as well as the generated text. Confirm how revoked access and superseded documents flow through the indexing process.

What would you need to assess knowledge-assisted support?

Bring one policy question and retrieved supporting material, an example from the evaluation set and correction register and the people responsible for the question, retrieved passages and reviewer correction history. A useful discovery exercise follows authoritative passage to reviewed response under both ordinary conditions and a revised expense policy with separate rules for existing bookings. The resulting scope should identify the business decision, environment access and operating owner required for the first work package.

Can we begin with use-case selection?

Yes. Define the first deliverable in terms of a reviewer can justify every consequential claim against material available to that user. Agree with the AI use-case owner how to test it using the question, retrieved passages and reviewer correction history. The first phase can stop after the decision and evidence are delivered, or continue into retrieval and content preparation with scope and responsibilities already documented.

How does retrieval and content preparation fit into delivery?

It addresses the point in knowledge-assisted support where the team must understand document version, effective date, permission scope and cited passage. Use a revised expense policy with separate rules for existing bookings to establish how information, permissions and ownership cross that boundary. The resulting acceptance case should demonstrate retrieval respects current access and uses the applicable policy version and identify who takes over if the outcome is uncertain.

What happens to cases involving superseded policies?

Use policy question and retrieved supporting material as the starting reference and examine the question, retrieved passages and reviewer correction history. The key question is whether retrieval respects current access and uses the applicable policy version. Agree a treatment with the AI use-case owner, preserve evidence for the evaluation set and correction register and make the authorised next action visible to the team. Include the case in later release checks, especially after a second knowledge collection.

What should the assistant do when the knowledge base has no answer?

Define an explicit response and handoff for insufficient evidence. Test a realistic question with no authorised source and have reviewers check whether the assistant identifies the gap. Capture the request for the content owner rather than using the absence of a document as permission to invent guidance.

What should we expect at handover?

Expect the agreed delivery evidence and an evaluation set with reviewer scoring guidance. A useful sign-off is for the receiving team to trace authoritative passage to reviewed response, read the relevant entry in the evaluation set and correction register and respond to a revised expense policy with separate rules for existing bookings using the material supplied. Document ongoing responsibility for updates to policy question and retrieved supporting material before closing the engagement.

How will our team decide whether to go ahead?

Ask whether a reviewer can justify every consequential claim against material available to that user. Check the question, retrieved passages and reviewer correction history with the AI use-case owner and verify that authority to approve an answer intended for a customer is available during a policy publication cycle. If a critical dependency remains open, identify its effect on responses supported by current authorised material and the owner who can accept, resolve or defer it.

Which measures are useful after release?

Use answer usefulness and corrections required by reviewers alongside the evaluation set and correction register. For knowledge-assisted support, reconcile a sample against responses supported by current authorised material and review the influence of conflicting documents and questions with insufficient evidence. Confirm that a second knowledge collection or a new reporting definition has not changed the population before crediting an improvement to the work.

Who can operate the outcome internally?

Agree responsibilities among the content owner, frontline reviewer and application administrator. Give the receiving operator an unfamiliar example involving questions outside the knowledge base and ask them to investigate it through the question, retrieved passages and reviewer correction history. Their ability to explain authoritative passage to reviewed response and identify authority to approve an answer intended for a customer is a practical test of the handover.

What has to be ready before we pilot?

Identify a content owner, permitted documents and reviewers who know a useful answer when they see it. Collect difficult cases as well as routine questions so the pilot tests meaningful limits.

Talk to FyerX about AI Enablement

Contact Technology