🤓 User Experience analysis

<aside> 💡

The Verified Answers feature ensures that responses from Sybil are reviewed and validated by experts, maintaining a reliable knowledge base over time. Users can submit responses for verification manually if they find them unreliable, or automatically if Sybil detects low confidence in its response.

Experts receive flagged responses in a dedicated review queue, where they can modify and approve answers before they become part of the trusted knowledge base. Administrators have control over enabling the feature, choosing between manual or automatic review modes, and configuring anonymity settings.

Sybil assesses response reliability based on usefulness, groundedness, and freshness, ensuring that the information provided is relevant, well-supported, and up to date.

For the user, there are two ways to submit a response for expert verification:

• Manual Mode: If manual mode is enabled, the user will have a button available in each conversation turn to submit a response for verification if they deem it unreliable or in need of further expert review.

• Automatic Mode: If automatic mode is enabled, every interaction between the user and Sybil undergoes a reliability analysis, which automatically calculates the confidence level of the response. If the confidence is high, the response is displayed on screen as shown below.

If the confidence calculated by Sybil does not meet the threshold set by the administrator, the conversation turn is automatically flagged for review, alerting experts to verify the response.

Once the expert verifies the response, the user is notified, and the correction will be available for Sybil in the future!

</aside>

🥼 Expert Review Process

<aside> 💡

The "Verified Answers" feature enables experts to receive flagged interactions directly in their designated Desk.

Experts will have a queue of responses to verify and can assign responses to themselves.

Once in editor mode, the expert can directly modify the response content, remove cited documents, and refine the response as needed. Once approved, the verified response enriches the associated Desk, ensuring an immediate and controlled update to the knowledge base.

</aside>

🔧 Admin Configuration

<aside> 💡

The "Verified Answers" feature must be enabled by SylloTips system administrators. To do this, they need to access the admin panel and navigate to the Verified Answers section.

In this section, administrators can enable or disable the feature, configure the review request mode (manual or automatic), and choose whether review requests should be anonymous.

Below is an overview of how Sybil automatically conducts reliability analysis and the key factors considered in assessing a response’s reliability.

</aside>

⚖️ Reliability Analysis

<aside> 💡

The reliability (confidence) of Sybil’s generated responses is estimated using advanced Self-Reasoning approaches and Natural Language Processing techniques. The analysis focuses on three key aspects:

• Usefulness: Evaluates whether Sybil’s response provides precise information relevant to the user's question.
  • Helps identify knowledge gaps in the company’s knowledge base.
• Groundedness: Assesses how well Sybil’s response is supported by and rooted in the company’s knowledge base.
  • Helps detect potential hallucinations in the response.
• Freshness: Determines whether Sybil’s response is based on documents at risk of becoming outdated.
  • Obsolescence risk is measured using the last modified date of documents, or, if unavailable, the date the documents were added to the system.
  • Helps identify areas where documentation needs updates.

Each of these three aspects—usefulness, groundedness, and freshness—is categorized as low, medium, or high.

The partial results are then dynamically combined to calculate an overall reliability category, which determines whether the response should be sent for expert verification.

</aside>