scholar-evaluation

Files (3)

• SKILL.md
• evaluation_framework.md · references/
• calculate_scores.py · scripts/

SKILL.md (first lines)

# Scholar Evaluation

## Overview

Apply the ScholarEval framework to systematically evaluate scholarly and research work. This skill provides structured evaluation methodology based on peer-reviewed research assessment criteria, enabling comprehensive analysis of academic papers, research proposals, literature reviews, and scholarly writing across multiple quality dimensions.

## When to Use This Skill

Use this skill when:
- Evaluating research papers for quality and rigor
- Assessing literature review comprehensiveness and quality
- Reviewing research methodology design
- Scoring data analysis approaches
- Evaluating scholarly writing and presentation
- Providing structured feedback on academic work
- Benchmarking research quality against established criteria
- Assessing publication readiness for target venues
- Providing quantitative evaluation to complement qualitative peer review

## Visual Enhancement with Scientific Schematics

**When creating documents with this skill, always consider adding scientific diagrams and schematics to enhance visual communication.**

If your document does not already contain schematics or diagrams:
- Use the **scientific-schematics** skill to generate AI-powered publication-quality diagrams
- Simply describe your desired diagram in natural language
- Nano Banana Pro will automatically generate, review, and refine the schematic

**For new documents:** Scientific schematics should be generated by default to visually represent key concepts, workflows, architectures, or relationships described in the text.

**How to generate schematics:**
```bash
python scripts/generate_schematic.py "your diagram description" -o figures/output.png
```

The AI will automatically:
- Create publication-quality images with proper formatting
- Review and refine through multiple iterations
- Ensure accessibility (colorblind-friendly, high contrast)
- Save outputs in the figures/ directory

**When to add schematics:**
- Evaluation framework diagrams
- Quality assessment criteria decision trees
- Scholarly workflow visualizations
- Assessment methodology flowcharts
- Scoring rubric visualizations
- Evaluation process diagrams
- Any complex concept that benefits from visualization

For detailed guidance on creating schematics, refer to the scientific-schematics skill documentation.

---

## Evaluation Workflow

### Step 1: Initial Assessment and Scope Definition

Begin by identifying the type of scholarly work being evaluated and the evaluation scope:

**Work Types:**
- Full research paper (empirical, theoretical, or review)
- Research proposal or protocol
- Literature review (systematic, narrative, or scoping)
- Thesis or dissertation chapter
- Conference abstract or short paper

**Evaluation Scope:**
- Comprehensive (all dimensions)
- Targeted (specific aspects like methodology or writing)
- Comparative (benchmarking against other work)

Ask the user to clarify if the scope is ambiguous.

### Step 2: Dimension-Based Evaluation

Systematically evaluate the work across the ScholarEval dimensions. For each applicable dimension, assess quality, identify strengths and weaknesses, and provide scores where appropriate.

Refer to `references/evaluation_framework.md` for detailed criteria and rubrics for each dimension.

**Core Evaluation Dimensions:**

1. **Problem Formulation & Research Questions**
   - Clarity and specificity of research questions
   - Theoretical or practical significance
   - Feasibility and scope appropriateness
   - Novelty and contribution potential

2. **Literature Review**
   - Comprehensiveness of coverage
   - Critical synthesis vs. mere summarization
   - Identification of research gaps
   - Currency and relevance of sources
   - Proper contextualization

3. **Methodology & Research Design**
   - Appropriateness for research questions
   - Rigor and validity
   - Reproducibility and transparency
   - Ethical considerations
   - Limitations acknowledgment

4. **Data Collection & Sources**
   - Quality and appropriateness of data
   - Sample size and representativeness
   - Data collection procedures
   - Source credibility and reliability

5. **Analysis & Interpretation**
   - Appropriateness of analytical methods
   - Rigor of analysis
   - Logical coherence
   - Alternative explanations considered
   - Results-claims alignment

6. **Results & Findings**
   - Clarity of presentation
   - Statistical or qualitative rigor
   - Visualization quality

From qinyan-ai/qinyan-academic-skills, MIT.

More from qinyan-ai/qinyan-academic-skills