Lesson 02: Managing your AI assistant: who does it, how do you go about it and how much time does it take?

This is lesson 2 in our series on working with an AI assistant on your website. Lesson 1 was about the context that shapes the question. We have now completed an onboarding with more than forty organisations. Municipalities and healthcare institutions, but also trade publishers, industry associations, construction firms and technical service providers. The question that comes up most often is about managing the AI assistant: how do you go about it? That is what this lesson is about.

Who manages the AI assistant?

The first reaction is often that AI is something for IT. We see it as a marketing tool and something for the web team: the content manager, web editor or product owner of the website. Those are the colleagues who already decide what is on the site and how it is presented.

When an answer is wrong, it is usually down to the content, and that is exactly the web team's expertise. They know the pages, they know which information is out of date and which term the organisation would rather not use. An AI assistant is a new channel onto the knowledge on your website. If something on your website is wrong, the web team puts it right, and it works the same way here.

At vragen.ai, we take care of the technology, the model, availability and the safety rules. What we cannot do is judge whether an answer is correct in substance. An answer from a language model quickly sounds plausible and well phrased, but if you know the subject, you see straight away whether it is also right.

The division of roles I see working well at clients is this:

• one owner (two at most) who manages the assistant and may change it
• a core team of three or four people who help decide
• a wider group of colleagues who now and then check a sample and give feedback

The smaller the core team, the faster you can act.

vragen.ai =

• AI harness The engine. A language model of your choice, in the Netherlands or the EU.vragen.ai
• Knowledge Your own sources, always up to date.You
• Instructions Tone, guidelines and behaviour of your agents.You
• Monitoring Follow live what your AI does and how it performs.You
• Evaluation Measure quality and adjust for results.You

We manage the model and the functionality. • You manage the rest.

How much time does it take?

Fortunately, not much. You do not need half an FTE to keep an AI assistant running. Most of the work happens before go-live: deciding which content goes in, the tone, setting boundaries and testing with real questions. Once that is in place, you can start monitoring and adjusting where needed.

Right after go-live you will probably spend a quarter of an hour every morning on a quick sample. You get to know the assistant, you see which questions come in and you discover where your content falls short. It is also the most fun phase, because for the first time you see, word for word, what visitors ask your website.

After a few weeks that drops to a weekly look of half an hour to an hour. You get a better feel for what your assistant does and why it gives certain answers. On top of that, you set aside an hour or two every month to make changes, so a few hours a month.

The time grows with the number of conversations. A site with two hundred conversations a month takes half an hour to review. A site with thousands of conversations relies more on the automatic signals in the vragen.ai admin environment. What does not change is that you have to schedule this work, or it does not get done.

Compare it to your web analytics: you do not look at those every day either, but you do look at them on a fixed rhythm.

These are the hours you spend monitoring and steering the AI assistant. If that shows content is missing or out of date, writing or updating that page also takes time. How much depends on what you find. It is not new work; your web team does this already. You just get a tool that shows what that content should be about, because your visitors are literally asking you.

How do I go about managing it?

Open the analysis in the admin environment before you start reading individual conversations. There you see at a glance which topics came up most this week, which topics shifted compared to the previous period and how answer quality is developing. You also see the conversations vragen.ai itself has labelled “missing content” or “improve content”. So you do not have to read every conversation; that takes far too much time. The analysis tells you which topics and labelled conversations to include in your sample.

Reading and assessing conversations

During your sample you open the conversations for the period you choose and read a handful. Not all of them, and not only the ones with a signal: the conversations without a label are what show you whether you can trust the automatic assessment. A quarter of an hour to half an hour is enough.

If you see a conversation that needs attention, you flag it and add one sentence saying why. “Promises to look something up that it cannot.” “Used an old version of the guideline.” Adding a comment quickly shows you whether there is a common thread in your findings. Colleagues can also follow along this way and see where you think action is needed. And not only colleagues: on request, Improve rules with AI can read your comments and suggest changes based on them, which you then only need to review.

Making a change

After a few days of flagging, you have a pile of conversations with a comment attached, and then comes the adjusting. Before you change anything, it helps to know which two dials you can turn.

An answer from the AI assistant is determined by:

1. Knowledge (the sources) Which sources the assistant uses and what is in them. What is missing, what is wrong, what should carry more weight?
2. Expertise (behaviour and tone) How it answers: behaviour, tone and search strategy. How should the agent answer?

You discuss the flagged conversations with your team. For each conversation you decide whether it is a content action or a change to the instructions. This flowchart helps you make that choice:

1. 1 Does this happen more often?

   No One-off No action, keep an eye on it.

   YesGo on to question 2

2. 2 Is the response correct and does it fit the question? Also when the AI deliberately gives no answer.

   Yes Deliberate choice No action.

   NoGo on to question 3

3. 3 Does this question belong with the assistant?

   No Instructions Set down how it should refer people on.

   YesGo on to question 4

4. 4 Is the answer (the information) on your website?

   No Content Put the information on your website.

   YesGo on to question 5

5. 5 Is the page in the AI's knowledge base?

   No Knowledge base The AI cannot read the page properly. Check whether it is being fetched, or ask us to take a look.

   YesGo on to question 6

6. 6 The right source was found, but is the answer badly put together?

   Yes Instructions Adjust them with Improve rules with AI.

The first question in the flowchart is also the most important one: does this happen more often? A language model never gives exactly the same answer twice, so one poor answer to your own favourite question says little. Look for patterns across several conversations before you change anything.

If you are changing a lot at once, you duplicate the agent (that is what your assistant is called in the admin environment), give the copy a version number and change that. You test it together first, with the same questions as before, and only when it is better do you open the new version to visitors, just as you prepare a page in your CMS before you publish it. For every conversation you can see afterwards which version answered it.

So the rhythm is: look and flag every week, and decide and adjust with the team every month.

What do you look at in vragen.ai?

Not at everything: in practice, you answer three questions.

What do people come for? The topics and themes show which questions are on people's minds. This is the part teams find most interesting, and rightly so. You no longer have to guess the intent behind a page visit; you read it. Do watch out for distortion from example questions: a topic that scores high because everyone clicks the same suggestion says little.

Where does it go wrong? You have two tools for that. The first is the scores: vragen.ai rates every answer automatically on three points, and each score calls for a different action.

• Reliability Do the claims match the sources? If this is low, the assistant is saying things that are not in your sources. Look at the content first: is the information missing, out of date or contradictory? If the content is right, it is an instruction action: it is too assertive or fills in gaps itself.
• Relevance Does the answer fit the question? If this is low, the information is often missing from your website. Usually a content action, or the question does not belong with the assistant.
• Source relevance Do the sources found fit the question? If this is low, the assistant mostly found something that looks similar. Often missing information too, or the right page is not properly in the knowledge base.

The second is the labels. Conversations that score low, or where the visitor gave a thumbs down, get one. Two labels deserve your attention:

• “Missing content” means there is no page that answers this question.
• “Improve content” means the page does exist, but the assistant did not understand it well or the information is spread over six places.

See a label as good news. A question the assistant has no answer to shows exactly what people come for and what is still missing from your site.

What does the trend say? A single bad answer is an incident. A score that slowly drops over weeks is a signal: the content no longer matches what visitors ask, or a change has broken something. That is why the monthly session also looks at the trend, not just at this week.

Can I change the AI instructions?

The instructions feel like the engine: you turn one screw and who knows what changes on the other side. A client put it like this recently: how do I know a new rule will not clash with the rules that are already there?

Look at the content first, and only then at the instructions. In eight out of ten cases where an answer is not right, the content is the cause. The page is missing, out of date, or the information is spread over six places. So always check first where the assistant got its information; you can see that for each conversation, step by step. Usually the fix is a better page, and your web team works on those every day. The instructions are for the exceptions: a campaign running from 3 to 6 October, a technical term the assistant does not recognise, a follow-up question that sounds too assertive.

What is more, there is little you can break. You try out a rule without saving, so the live assistant does not change, and you run the same question again to compare. If you like it, you put it live. Bigger changes go into a copy with a version number, as described above. And anyone can create their own test assistant to practise with, which never goes online.

You do not have to write a prompt either. You describe in plain language what you saw (“the assistant promises to look up things it cannot find”) and with Improve rules with AI vragen.ai suggests a rule, checks it against the existing rules and shows what will change. You decide.

One more thing I tell teams when they start making changes: write rules in general terms. A rule for a single technical term is quickly one rule too many; after a while you have twenty and the assistant cannot see the wood for the trees. What does work well: one general rule with a few examples. You do not need to supply an exhaustive list of terms; with two or three examples the assistant gets the pattern and also applies the rule to terms you did not mention.

Right after go-live you will adjust the instructions regularly. After a few months, for most clients, that drops to one change a month.

How to start

You do not have to arrange everything in this lesson at once. Start with two agreements: who the owner is, and when you are satisfied. Teams forget the second one most often. Which score do you want to see, how many conversations do you expect, which questions must be answered well in any case? Without your own criteria, an assistant stays stuck in the pilot phase forever.

The rest follows once you take that quarter of an hour every morning. If you are not sure who should take this on in your organisation, ask us for advice.