Training and Proactive Learning

Training is how your bot gets smarter over time. As your bot handles real conversations, the AI observes patterns — repeated questions, gaps in its answers, corrections from users — and generates learning insights you can review and apply. You control which suggestions are accepted, which are rejected, and how aggressively the system looks for improvements.

This page covers both the Training tab (AI-generated insights and your review queue) and the Proactive Learning sub-page (the dedicated review interface for AI-generated knowledge suggestions).

What training does

Training in Balchemy is not fine-tuning the underlying language model. Instead, it works by enriching two things that directly influence the bot's responses:

Knowledge base entries — new Q&A pairs and facts that improve retrieval-augmented generation (RAG). When the bot encounters a question it answered poorly, the system can suggest a better answer as a knowledge base entry for you to approve.
Response patterns — observations about how the bot's persona, tone, or response structure could be improved. These appear as "improvement" type insights in the review queue.

When you approve an insight, it is added to the bot's knowledge base and becomes available for retrieval on the next conversation. Rejecting an insight teaches the system that this pattern is not worth surfacing again.

The Training tab

Navigate to your bot in Studio and click the Training tab. The page opens on the Insights sub-tab by default.

Insights sub-tab

The Insights sub-tab shows the review queue — all pending AI-generated suggestions waiting for your decision. Each insight card contains:

Type badge — one of three types:
- Improvement — a suggestion to refine how the bot phrases or structures a response
- Pattern — a recurring question or topic the bot has encountered multiple times; a new knowledge entry could handle this better
- Correction — a case where the bot gave an incorrect or incomplete answer and the system detected it
Title and description — a plain-English summary of what was learned and why the system is suggesting it
Confidence score — expressed as a percentage (e.g., 87%). Higher confidence means the system observed this pattern frequently or with high certainty. Lower confidence suggestions are worth examining before approving.
Age — how long ago the insight was generated, shown as relative time (e.g., "3 hours ago")
Approve / Dismiss buttons — visible only on pending insights. Applied insights show a green "applied" badge; rejected ones show a neutral "dismissed" badge.

To review an insight, read the description and decide whether the suggested change aligns with your intent for the bot. If the suggestion is accurate and useful, click Apply. If it is wrong, off-brand, or not relevant, click Dismiss.

An empty insights queue means the bot is performing well or has not yet accumulated enough conversation history to generate suggestions. New suggestions appear automatically as usage grows.

Learning Settings sub-tab

Click Learning Settings to configure how aggressively the system generates insights.

Setting	Description
Proactive Learning	Master toggle. When off, no new insights are generated. Existing pending insights remain in the queue.
Learning Aggressiveness	Controls the confidence threshold for surfacing insights. See details below.
Background Learning	When on, the system continues analyzing conversations even when the Training tab is not open. Recommended to leave this enabled.
Share Knowledge Across Bots	When on, approved insights from this bot can inform suggestions for your other bots. Useful if you manage multiple bots covering the same domain. Off by default.

After changing any settings, click Save Settings. Changes take effect on the next analysis cycle.

Aggressiveness levels

The aggressiveness setting determines which confidence threshold is required before an insight appears in your queue.

Level	Behavior	Best for
Cautious	Only surfaces insights with very high confidence — patterns seen many times across many conversations. Your queue stays short and clean.	High-stakes bots where accuracy is critical and false suggestions are costly. Start here if you are new to training.
Balanced	Moderate threshold. Most recurring patterns and clear corrections are surfaced.	Most bots. A good default that generates useful insights without overwhelming you.
Aggressive	All detected patterns are surfaced, including low-confidence and one-off observations. Your queue will be longer and require more review effort.	Bots in active development where you want maximum signal, even at the cost of more noise.

You can change aggressiveness at any time. Switching from Aggressive to Cautious does not delete existing pending insights — it only affects what is generated going forward.

Proactive learning sub-page

The Proactive Learning page (/studio/bots/<botId>/training/proactive) is a dedicated review interface focused specifically on knowledge base suggestions — the Q&A pairs and facts the AI believes should be added to help the bot answer future questions better.

This page differs from the main Training tab in that it shows raw knowledge document proposals rather than the broader category of behavioral improvements.

What you see

At the top of the page, three KPI cards show:

Pending Review — how many knowledge suggestions are waiting for your decision
Approved Today — how many you have approved in the current day
Rejected Today — how many you have rejected in the current day

Below the KPIs, each pending insight card shows:

Question — the question or topic this knowledge entry would address
Confidence badge — color-coded: green (80%+), amber (60–79%), red (below 60%). Red suggestions should be reviewed carefully before approving.
Source badge — where the insight came from: AI Generated, Conversation, or another source label
Suggested Answer — the answer text the system proposes adding to the knowledge base
Timestamp — when the insight was generated
Approve / Reject buttons

Approving an insight

When you click Approve, the suggested Q&A pair is added to the bot's knowledge base as a new document. The bot can immediately use this entry in RAG retrieval for future conversations.

Approval is a one-way action for the insight card — approved insights move out of the pending queue and cannot be reverted from this page. If you later decide an approved entry is wrong, you can edit or delete it directly in the Knowledge Base tab.

Rejecting an insight

When you click Reject, the insight is dismissed. The rejection is recorded so the system can learn not to surface the same pattern again. Rejecting is appropriate when the suggested answer is incorrect, duplicates something already in the knowledge base, or is too narrow to be generally useful.

How training affects bot responses

Training improves bot responses through the knowledge base retrieval path. When a user sends a message, Balchemy performs a semantic similarity search across all knowledge base entries. Approved insights become searchable entries — the more high-quality entries the knowledge base contains, the more precisely the bot can match incoming questions.

This means the impact of training is gradual. A single approved insight may not noticeably change responses, but approving dozens of well-targeted Q&A pairs over several weeks of usage typically produces measurable improvements in response relevance and accuracy.

Training does not change the bot's core persona, language style, or trading capabilities. Those are controlled by the AI Configuration (system prompt and model settings) and the Strategy tab respectively.

Best practices

Start with Cautious mode, then broaden. When you first enable proactive learning on a new bot, use the Cautious aggressiveness setting. Review the first batch of insights carefully. Once you develop a sense of the system's signal quality, you can move to Balanced or Aggressive.

Review your queue weekly. Insights left in the queue too long become stale — the conversations that generated them may no longer represent the bot's current usage patterns. A weekly review session of 10–15 minutes is sufficient for most bots.

Use the Knowledge Base for core facts, training for edge cases. Your bot's foundational facts — what it does, what chains it supports, how to use specific features — belong in the Knowledge Base as manually curated entries. Let the training system surface edge cases and gaps that you did not anticipate.

Do not approve low-confidence suggestions blindly. A red confidence badge (below 60%) means the system saw a weak signal. Read the suggested answer carefully. If it is wrong or misleading, reject it. A few bad knowledge entries can cause the bot to give incorrect information to users.

Use the Dismiss button freely. Rejecting an insight is not a wasted action. Each rejection helps the system filter better suggestions in future cycles. If you find yourself rejecting the same type of suggestion repeatedly, consider lowering the aggressiveness setting.

Knowledge Base — directly manage the knowledge documents that training inserts into
AI Configuration — configure the model, system prompt, and context retrieval settings
Bot Analytics — use error rate and response metrics to identify where training is most needed
Troubleshooting — what to do if training insights stop appearing or approvals are not taking effect

Bot Analytics

Trade History

Edit this page on GitHub