# AI Setup

The AI layer should feel reliable to buyers, not just available. Configure provider status, channel policies, knowledge sources, prompt versions, QA logs, and handoff rules before presenting the app as production-ready.

## Provider Configuration

Open the AI settings area and configure:

- Provider.
- Model.
- API key.
- Temperature and response settings.
- Retrieval limits.
- Confidence threshold.
- Fallback message.
- Handoff behavior.

After saving, run the provider test action. The test should record:

- Last tested time.
- Status.
- Latency.
- Error message when failed.

For a public demo, keep a clear state: either a working provider test or an obvious demo-mode explanation.

## Channel AI Policies

AI behavior can differ by channel. Configure policies for:

- Widget.
- Email.
- API.
- WhatsApp.
- Telegram.
- Slack.
- Facebook Messenger.
- Instagram.
- Future connector channels.

Each policy should define:

- Whether AI is enabled.
- Tone or behavior instructions.
- Confidence threshold.
- Retrieval threshold.
- Whether citations are required.
- Handoff condition.
- Fallback message.

## Prompt Version History

Use prompt versions before major edits. A prompt version should capture:

- Model/provider settings.
- Main behavior prompt.
- Channel policies.
- Confidence thresholds.
- Retrieval settings.
- Admin who changed it.
- Timestamp.

Admins should be able to review previous settings and restore a known-good version.

## Knowledge Sources

Knowledge sources power cited answers. Keep source titles and content clear because buyers will inspect citations.

Use Knowledge Diagnostics to find:

- Stale sources.
- Failed processing.
- Empty chunks.
- Low-match answers.
- Top unanswered topics.
- Sources that need retraining.

Recommended buyer-demo sources:

- Product overview.
- Pricing and plan limits.
- Installation notes.
- Billing FAQ.
- Envato support policy.
- E-commerce order support examples.

## Citations And Confidence

AI answers should show:

- Source title.
- Citation link or reference.
- Confidence score.
- Fallback reason when applicable.
- Whether the answer used retrieved knowledge.

Buyer value:

- "Why AI answered this" is explained by source and retrieval match.
- "Which source was used" is shown through citation data.
- "When human handoff happens" is controlled by confidence and retrieval thresholds.
- "How usage is counted" is tied to the plan `ai_messages` counter.

## Low-Confidence Handoff

When confidence is low or retrieval is below threshold, the app should create a human handoff instead of pretending certainty.

Expected handoff behavior:

- Conversation remains visible in the inbox.
- Handoff event is recorded.
- Agent is notified where notification settings allow it.
- Webhook event `ai.handoff.requested` is emitted.
- AI QA log records fallback reason.
- Usage counter still reflects the attempted AI message when plan rules require it.

## AI QA Console

The AI QA console should let admins test:

- Provider connection.
- Model latency.
- Retrieval quality.
- Confidence thresholds.
- Channel policies.
- Citation presence.
- Fallback behavior.
- Handoff creation.

Use the QA console before recording marketplace videos or enabling a public demo.

## Evaluation Datasets

Create evaluation sets for common buyer scenarios:

- General support questions.
- SaaS billing questions.
- E-commerce product/order questions.
- Envato support and license questions.
- Installation/onboarding questions.

Each evaluation should track:

- Question.
- Expected answer.
- Sources used.
- Citation accuracy.
- Confidence.
- Fallback reason.
- Handoff outcome.
- Provider latency.

## Reports Buyers Should See

For marketplace proof, show:

- Deflection rate.
- AI message usage.
- Saved agent time estimate.
- Low-confidence handoff count.
- Unanswered-topic queue.
- Stale-source warnings.
- Provider latency and error trends.

## Troubleshooting

- If AI does not answer, check provider status, plan `ai_messages` usage, channel policy, and knowledge source availability.
- If citations are missing, check source processing and retrieval settings.
- If handoff never happens, lower thresholds or confirm handoff rules are enabled.
- If every answer falls back, check stale or empty knowledge sources.
- If provider tests fail, confirm API key, model name, outbound HTTP, and account billing.