Skip to main content
Back to projects

Evaluation design / private enterprise work

Model evaluation and right-sizing

How I structured an evidence-led comparison for a Kubernetes diagnostic agent without exposing private prompts, infrastructure, or measurements.

Methodology published; measured results are not publicly disclosed.

Workload and evidence boundary

Evaluate whether a smaller self-hosted language model can diagnose representative Kubernetes incidents while choosing and using investigation tools appropriately. This mirrors the multi-step behavior required by an internal SRE agent.

Scenario set
approximately 40, sourced from an internal set of real kubernetes diagnostic scenarios, created with assistance from holmesgpt.
Provenance and license
Private scenario content and its license are not publicly disclosed.
Data split
Any train, development, and test separation is not publicly disclosed.
Baseline
A strong production baseline was part of the internal comparison; its identity and configuration are not publicly disclosed here.

Evaluation flow

Evaluation flow from private Kubernetes scenarios through candidate responses, structured scoring, expert review, and shadow validation

Metrics intended

  • diagnostic correctness
  • tool-call correctness
  • reasoning quality
  • latency
  • cost

Human review was part of the evaluation design, but the sample size and reviewer protocol are confidential.

Judge limitations

DeepEval, Langfuse, and an LLM-as-a-Judge workflow supported structured review.

  • An LLM judge can prefer fluent answers over operationally safe ones and can reproduce its own model biases.
  • Judge scores are not a substitute for expert review of diagnoses and tool use.
  • Private judge identity, prompt, and calibration data prevent public replication of measured scores.

Error taxonomy

Incorrect diagnosis

Synthetic example: Attributes a scheduling failure to networking despite evidence that a resource request cannot be satisfied.

Unnecessary tool use

Synthetic example: Requests unrelated cluster-wide logs after the supplied event already identifies the failing workload.

Missed evidence

Synthetic example: Repeats a generic remediation without incorporating a synthetic warning event returned by a tool.

Unsafe recommendation

Synthetic example: Suggests a disruptive change before checking workload scope or rollback options.

Right-sizing decision framework

Quality, latency, and cost decision framework with safety as a gate and shadow validation before cutover
  1. 01Reject candidates that do not meet the workload's quality and safety bar.
  2. 02Among acceptable candidates, compare tail latency and operational behavior, not only aggregate quality.
  3. 03Estimate total serving cost using the intended traffic and infrastructure context.
  4. 04Validate the selected candidate in shadow mode before considering production cutover.

No public claim is made about which model matched the baseline, where a candidate failed, whether fine-tuning improved it, or which measured configuration should be deployed.

What can and cannot be reproduced

The flow, metric categories, review questions, error taxonomy, and decision gates can be reused as an experiment template. The benchmark itself is not publicly reproducible: The private scenarios, model configurations, judge setup, hardware, decoding settings, and measured outputs are not approved for disclosure.

Not publicly disclosed

  • model configurations
  • hardware and serving configuration
  • decoding settings
  • quality measurements
  • latency measurements
  • cost measurements
  • measurement date