Building a Custom Open-Source LLM-Based Chatbot

Client: A Cybersecurity Company

Industry: Cybersecurity

Client Description:

A prominent player in the cybersecurity sector, renowned for its commitment to protecting data and systems. The company prioritizes innovation and efficiency, especially in securely managing sensitive recruitment data.

Project Overview:

Challenge:

The client required a robust solution to automate the early stages of the recruitment process. The challenge was to streamline candidate screening and initial engagement efficiently, ensuring data security and adapting to specific organizational needs while handling sensitive recruitment data.

Solution:

We developed a generative AI chatbot that operates seamlessly within the client's infrastructure. This AI-driven solution utilizes advanced Large Language Models (LLMs) to automate candidate identification and preliminary interactions, significantly reducing the administrative load on recruitment teams.

Key Features:

• Autonomous Agent Capabilities: Fully functional within the client's private infrastructure, ensuring complete data control and security.
• State-of-the-Art Technologies: Features input validation, corrective guardrails, Retrieval-Augmented Generation (RAG), and dynamic prompting. Entirely open-source, enabling customization without dependency on commercial tools.
• Comprehensive Development: From initial fine-tuning based on human feedback to deployment in a serverless environment, including performance evaluations, LLM-as-judge metrics, and LLMOps.

Benefits:

• Full Data Control: Keeps all sensitive data within the client's control, enhancing security and compliance.
• Customization and Flexibility: Utilizes open-source LLMs for a customizable approach, allowing clients to choose models and fine-tune system settings to their specific needs.

Impact:

• Efficiency in Recruitment: Automates initial phases of recruitment, reducing time and resource expenditure.
• Customizable AI Solutions: Demonstrates the potential and effectiveness of open-source AI in adapting to and fulfilling diverse business needs.

Technologies Used:

• Pulumi: Infrastructure as Code (IaC) SDK.
• MLFlow Tracking: Data versioning, model versioning, and experiment tracking and validation.
• Hugging Face Transformers: Fine-tuning and inference of additional models (e.g., zero-shot classifiers for sentiment analysis as part of a custom guardrail validator).
• Langchain: LLM integration framework.
• vLLM: Model inference engine, in conjunction with MLFlow Deployments Server, deployed on serverless inference infrastructure (RunPod).
• FastAPI: Backend framework for building the system API.
• Postgres: Metadata and conversational data store.
• MinIO: Object storage system, used as an artifact store for the MLFlow Tracking instance.

Outcome:

The chatbot excels in the initial recruitment stages, efficiently narrowing down applicant pools and facilitating faster and more effective candidate screening processes. This leads to improved recruitment efficiency, better candidate fit, and significant time and cost savings while being able to keep control of the data generated.