Professor in a Box: Personalized CS Tutoring via Fine-Tuned LLMs and Retrieval-Augmented Generation

Abstract

Professor in a Box (PiAB) is a production-grade AI tutoring system that addresses two fundamental limitations of off-the-shelf large language models in educational settings: weak performance on low-frequency computer science sub-domains and hallucination on student-specific study material. The system fine-tunes open-weight 7B-parameter models using QLoRA and serves them via a real-time retrieval-augmented generation pipeline that dynamically injects student-uploaded document context into every inference call.

Students interact through a React dashboard that streams token-by-token responses from LoRA-adapted Mistral-7B and DeepSeek-R1-7B models running on serverless NVIDIA A10G GPUs via Modal. When fine-tuned inference is unavailable, a three-tier fallback chain (OpenAI, DeepSeek, Gemini) maintains uptime transparently. The backend orchestrates JWT-authenticated file ingestion through LlamaCloud, semantic chunking, dense embedding via Gemini text-embedding-004, and ChromaDB retrieval all within a single Node.js API gateway.

Fine-tuning domain-adapted LoRA adapters on curated CS Q&A corpora combined with per-user RAG memory reduces hallucination on personal study material and shifts LLM behavior from generic text completion toward grounded, pedagogically consistent tutoring.

System Architecture

Research Context and Motivation

Large language models achieve strong average-case performance across general benchmarks but exhibit systematic degradation on low-frequency CS sub-domains such as competitive programming and advanced algorithms. They also hallucinate when queried about student-specific materials that fall outside the training distribution. PiAB addresses both failure modes with two complementary techniques that operate at different stages of the inference pipeline.

Supervised fine-tuning with LoRA adapts model behavior at training time by exposing the network to curated CS tutoring-style Q&A pairs. This shifts the base distribution toward pedagogically grounded responses and improves accuracy on domain-specific questions. Retrieval-augmented generation operates at inference time by fetching the most semantically relevant passages from a student's own uploaded notes and injecting them as grounded context, directly counteracting hallucination on personal study material.

Models and Fine-Tuning

QLoRA Training Configuration

Both models were quantized to 4-bit NF4 precision using BitsAndBytes with double quantization enabled, reducing each 14 GB model to approximately 4 GB VRAM. PEFT LoRA adapters were applied to the query and value projection matrices with rank 16, targeting the instruction-following heads most responsible for response style. The training data consists of multi-source CS Q&A pairs: LeetCode problems, Codeforces editorial-style explanations, the APPS dataset, and CS textbook chapter excerpts. An Apache Airflow ETL pipeline (Astronomer runtime) orchestrated raw ingestion from Hugging Face Hub, schema normalization, deduplication, and prompt formatting into the instruction-tuning template before uploading versioned Parquet files to the training bucket.

Inference Configuration (Modal Serverless)

Fine-tuned adapters are loaded onto NVIDIA A10G GPUs (24 GB VRAM) via Modal serverless functions with min_containers=0, eliminating GPU cost when idle. The batch endpoint (POST /ask) awaits full generation and returns JSON; the streaming endpoint (POST /ask-stream) emits SSE tokens as data: chunks for real-time display. Cold starts take approximately 60 to 90 seconds as weights load from Modal Volume cache. Warm requests complete in 5 to 15 seconds depending on response length. API keys for cloud providers are stored as Modal Secrets and fetched by the backend at runtime, removing secrets from source code and environment variables entirely.

RAG Pipeline

Document Ingestion

Students upload study materials (PDFs or images) through an authenticated file upload endpoint. LlamaCloud (LlamaParse) handles multi-modal OCR, converting scanned PDFs and handwritten notes into structured Markdown. The extracted text is split into 800-character semantic chunks with 100-character overlap to preserve cross-sentence context at boundaries. Each chunk is encoded using Gemini text-embedding-004, a dense embedding model optimized for retrieval, and stored in a per-user ChromaDB collection keyed by userId. This namespace isolation ensures that every student's document memory remains private and independent.

Retrieval and Grounded Generation

At query time, the student's question is embedded with the same Gemini text-embedding-004 model and matched against their ChromaDB namespace using cosine similarity. The top-5 nearest chunks are retrieved and injected into a structured system prompt alongside the question. The resulting prompt is forwarded to the selected inference provider; the response streams back via SSE with direct citations to the retrieved passages. If the primary provider returns a 401, 429, or 402 status, the gateway automatically retries through OpenAI, DeepSeek, and Gemini in sequence without surfacing an error to the user.

API Design

Deployment Infrastructure

Render — Web Services

The Node.js API gateway and React frontend are deployed on Render via a render.yaml manifest. The backend service runs in a Docker container with a 1 GB persistent disk for SQLite and ChromaDB state. The frontend compiles to a Vite static bundle with SPA rewrite rules. Push to main triggers automatic redeploy from the manifest. GitHub Actions CI builds the Vite bundle and deploys to Render static hosting on every push to main.

Modal — Serverless GPU Inference

Modal provides elastic GPU compute that scales to zero between requests. On first cold start, model weights download from Hugging Face Hub and cache to a Modal Volume (~15 to 20 minutes). Subsequent cold starts load from the volume cache (~60 to 90 seconds). A /warmup endpoint lets demonstrators pre-heat the container roughly 90 seconds before a live demo. All cloud provider API keys are stored in Modal Secrets and fetched by the backend at runtime via a /config endpoint, rotating without requiring a redeploy.

Conclusion

Professor in a Box demonstrates how combining parameter-efficient fine-tuning with retrieval-augmented generation produces a tutoring system that is simultaneously more accurate on domain-specific CS content and more grounded in each student's personal study materials. The serverless deployment model on Modal eliminates idle GPU cost while maintaining sub-15-second warm inference latency, and the three-tier fallback chain ensures continuous availability independent of any single provider.

The system architecture separates concerns cleanly: ETL and data quality live in Airflow, fine-tuning happens offline on cloud GPUs, inference runs on Modal, semantic memory lives in ChromaDB, and the API gateway handles authentication, routing, and fallback logic without coupling any of these layers. This design makes each component independently replaceable as the underlying model landscape evolves.