Langfuse Lead Qualifier

In V1, both classification and reasoning were delegated to the LLM, so results changed from one run to another. Accuracy began at 66.7%; a more explicit rubric raised it to 86.7�93.3%, but variability remained. V3 moved signals, segment classification and recommended action into deterministic code. The LLM keeps only fit_score and qualitative reasoning. On a benchmark of 15 simulated companies, the architecture produced 15/15 correct classifications.

Why Langfuse

Once an LLM enters a workflow, a second question appears: how do you know whether its behaviour is improving or getting worse?

The project simulates B2B lead qualification for a hypothetical SaaS company selling API infrastructure to travel businesses. Each company is evaluated for travel or hospitality relevance, B2B model and API, integration or infrastructure orientation. The output is segment, fit_score, reason and next_action.

Langfuse provides tracing, prompt versioning, a benchmark dataset, experiments, automated evaluators and quantitative comparison between prompt versions.

• Hybrid deterministic and LLM architecture: code determines stable business signals, segment and next action; the LLM supplies only fit score and qualitative reasoning.
• Node.js and JavaScript ES Modules keep the runtime light.
• Controlled OpenAI layer: gpt-4o-mini runs with temperature: 0.
• Structured output: a strict JSON Schema restricts the model to fit_score and reason.
• External prompt management: the production prompt is versioned in Langfuse rather than hard-coded.
• OpenTelemetry and Langfuse tracing make model calls observable.
• Separate benchmark dataset: test companies remain separate from application logic.

In the V3 benchmark the workflow achieved 15 correct classifications out of 15. The key result is the separation of deterministic business rules from probabilistic reasoning.