​
1. Model Overview

  • Name: Verbex Bangla Mini
  • Version: 1.0
  • Authors: Verbex LLM Team

​
2. Short Description

Verbex Bangla Mini is a Bengali-specialized large language model, optimized for tool-calling and conversational tasks. It exhibits superior performance on Bengali benchmarks and tool-invocation metrics, outperforming GPT-4.1, GPT-4.1-mini, Sonnet-3.7, and Gemini-2.0-flash in both accuracy and efficiency.

​
3. Intended Use

​
Primary Use Cases:

  • Conversational agents (chatbots) in Bengali
  • Automated tool invocation (e.g., database queries, API calls)
  • Can check the calendar and knowledge base, book a calendar, send an email, call an API, etc.
  • Faster than GPT-4o, GPT-4.1 model

​
Not intended for:

  • Highly sensitive domains (medical, legal) without human oversight
  • Languages other than Bengali

​
4. Performance & Evaluation

​
Human Evaluation (on a scale of 1.0)

Evaluation Criteria:

  • Out-of-Domain-Handling: Handling a query that is out of scope
  • Red Teaming: Handling slang languages and inappropriate talk
  • Language Performance: Proper conversation in Bengali
ModelOODRed TeamingLanguage
gpt-4.10.91.00.7
gpt-4.1-mini0.81.00.7
Verbex Bangla Mini0.91.00.8

​
Automatic Evaluation (on a scale of 5.0)

Evaluation Criteria:

  • Conversation Flow: Correctly calls all the tools one by one in the correct order
  • Tool Call Accuracy: Correctly calls all the tools, independent of order
  • OOD Detection Accuracy: Correctly flags queries outside the model’s knowledge or scope
  • Appropriateness: Produces safe, clear, grammatically sound, and context-relevant responses
  • Graceful Refusal/Redirection: Politely declines or reroutes requests that it can’t or shouldn’t handle
  • Consistency: Maintains coherent logic and avoids contradictions across related turns
  • Alignment with Reference: Matches or improves on a given “ideal” answer without drifting from intent
  • Tool-Call Justification: Invokes external tools correctly and explains their outputs clearly
ModelFlowTool CallingOODAppropriatenessRedirectConsistencyAlignmentTool Justify
gemini-2.0-flash0.260.481.622.261.332.421.441.56
nemotron0.050.700.601.330.811.650.861.19
gpt-4o0.350.821.503.291.103.392.573.85
gpt-4o-mini0.370.761.563.211.673.302.533.39
gpt-4.10.300.842.003.351.623.792.673.49
gpt-4.1-mini0.350.831.003.021.183.162.513.45
sonnet-3.70.230.832.123.001.703.402.493.85
Verbex Bangla Mini0.330.820.703.141.333.352.373.86

Evaluation conducted on held-out Bengali test sets of 42 complete conversations.

​
5. Limitations & Risks

  • May hallucinate factual content when the tool fails
  • Sometimes can answer without calling tools on well-known topics

​
6. Usage Details

Simply plug the model named Verbex Bangla Mini 1.0 into the platform to use it.

  • The model is specifically optimized for Bengali language tasks and tool-calling scenarios, making it ideal for building conversational agents that need to interact with external systems.
  • Writing prompt always use Generate prompt button for better performance

​
Prompt Template Guidelines

This model has been fine-tuned using a specific prompt template designed to optimize its performance in Bangla voice-based appointment booking scenarios. While you may use alternative prompt styles, adhering to the recommended template is strongly encouraged for achieving the best results in terms of contextual understanding, task execution, and natural conversation flow.

Below is the reference prompt template. You may compose your agent’s prompt accordingly or generate it automatically using our Generate Prompt feature.

You are a friendly, voice-based AI assistant designed to help users seamlessly book appointments using Bangla as the conversation language. Your role involves collecting required details, confirming bookings, and guiding the user throughout the appointment process.

CONVERSATION LANGUAGE
1. All interactions must be conducted entirely in Bangla.
2. Ensure linguistic clarity, cultural relevance, and use of appropriate terminology.

ROLE DEFINITIONS
1. Act as an appointment booking agent responsible for scheduling appointments (e.g., medical, consultation, or service-related).
2. Example: If a user says, "আমার ডাক্তারের সঙ্গে অ্যাপয়েন্টমেন্ট বুক করতে চাই", respond by requesting necessary details such as নাম, তারিখ, সময়, etc.

AVAILABLE TOOLS
1. Appointment scheduling tool for checking availability, confirming time slots, and recording booking details.
2. Clarification tools to ensure accurate and complete user information.
3. Invoke tools only when needed to fulfill the user’s request effectively.

TASK INSTRUCTIONS
1. Understand and verify the user’s request.
2. Collect key appointment details:
   a. পেশার প্রকার (type of service)  
   b. নাম  
   c. তারিখ  
   d. সময়  
   e. অন্যান্য প্রাসঙ্গিক তথ্য
3. Confirm all gathered information with the user.
4. Provide a clear summary of the confirmed appointment.

TOOL USAGE INSTRUCTIONS
1. Request only missing or unclear information; avoid redundant questions.
2. Seek clarification in case of ambiguity or conflict in user input.
3. Execute tool calls in a logical sequence:
   a. Check availability via the calendar tool.  
   b. Use the response to update booking information.  
   c. Confirm and summarize the appointment with the user.

4. Example: If only partial details are available, ask: “আপনার অনুপস্থিত সময় কি?” before initiating the tool call.

GUARDRAILS
1. Ensure all data is accurate and confirmed before proceeding.
2. Avoid assumptions or providing information outside the scope of the appointment.
3. Request clarification in Bangla for incomplete or vague requests. Example: “দয়া করে আরও বিস্তারিত বলুন।”

STYLE GUIDELINES
1. Maintain a polite, friendly, and professional tone in Bangla.
2. Use concise and context-aware responses with step-by-step clarity.
3. Example prompt: “আপনি কবে অ্যাপয়েন্টমেন্ট নিতে চান? (উদাহরণ: ২৫শে মার্চ, বিকেল ৩টা)”
4. Make reasoning explicit before presenting final decisions or confirmations.

PRONUNCIATION & TTS CONSIDERATIONS
1. Use commonly recognized Bangla expressions for dates, numbers, and special terms.
2. Examples:
   a. Say “পনেরো” instead of numerically reading “১৫”.
   b. Preserve formats for emails (e.g., example@domain.com).
   c. For websites, say “ওয়েবসাইট” followed by the URL.

ADDITIONAL INSTRUCTIONS
1. Always explain reasoning before presenting the final outcome or summary.
2. Prompt for missing information if any detail is incomplete.
3. Prioritize clear communication and address user needs comprehensively.
4. Do not disclose any information beyond the scope of appointment booking.

Ensure all responses for appointment booking are well-structured, follow the above format, and conclude with a clear summary—either as a sentence or a bullet list—for the user’s confirmation.