“LLM evaluation”

Conduct expert heuristic evaluations using Nielsen's heuristics and domain-specific criteria.

Emit structured self-evaluation telemetry and report artifacts for the current agent session.

Build, validate, and package reusable evaluation datasets for agent and LLM systems from prompts, traces, tool-call logs, metadata, and expected outcomes.

Evaluate the value of staging, deferral, reversible commitment, and information-gathering options under uncertainty; return decision-relevant option value and…

Create or update the reusable evaluation contract that future Agents use to review product and UI design consistently.

Recalculate product unit economics and sensitivities from explicit quote, BOM, fee, and logistics inputs.

Systematic content evaluation framework progressing through Critique → Reinforcement → Risk Analysis → Growth.

Build a protocol-difference matrix and estimate which differences can materially change measured performance.

To systematically assess the performance, accuracy, and safety of LLM outputs using quantitative metrics and "LLM-as-a-Judge" patterns, ensuring production…

Evaluate and design data warehouse schemas — star, snowflake, data vault, OBT — with grain definition, SCD strategies, and normalization trade-offs

Deprecated alias for novel-craft-draft-review.

Run bounded spike work to test feasibility before committing to a normal implementation path.

Process multiple artifacts efficiently with comparative analysis and trend reporting

Use when running client-side security stress harnesses against security challenge packs, measuring leak posture, Agent Vault routing, or HashiCorp Vault…

Use when evaluating local or open-weight language models with the AI Lab OS benchmark workflow, including raw prompt evidence, dashboard-compatible scores…

Discounted cash flow valuation and intrinsic value analysis for public companies.

Design and validate Product Factory evaluation corpora, benchmark adapters, scorecards, model routes, local/cloud comparisons, skill or prompt promotion, and…

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

Evaluate factual claims against inspected sources and identify unsupported assertions, contradictions and missing evidence.

Eval-driven skill tuning.

Use when designing and implementing evaluation loops for AI agents, including reflection, evaluator-optimiser patterns, rubric scoring, LLM-as-judge review…

Run the optional deterministic local writing evaluation composite without model calls.

Design agent evaluations that separate execution success from correctness, scope discipline, safety and recovery behavior.

Extract metrics, score traces, and evaluate benchmark task results.