literature-review

Files (3)

• SKILL.md
• routing.yaml · evals/
• literature-review.skill

SKILL.md (first lines)

# literature-review

## Core Philosophy
A literature review is not a chronological laundry list of paper summaries. Writing *"Author A found X. Then Author B found Y. Then Author C found Z"* is an annotated bibliography, not a literature review. A true literature review is a **critical synthesis**: it maps the intellectual topography of an academic field. It groups disparate studies into coherent thematic clusters, exposes underlying methodological contradictions, evaluates statistical and experimental validity, identifies scholarly consensus, and highlights the precise research gap that your work fills.

---

## 4-Step Literature Review Synthesis Framework

### Step 1: Systematic Search Strategy & PRISMA Protocols
1. **Search Query Formulation**:
   - Construct boolean search strings using controlled vocabulary (MeSH terms, IEEE keywords):
     - `("large language models" OR "LLMs") AND ("hallucination" OR "factual inconsistency") AND ("retrieval-augmented generation" OR "RAG")`
2. **Inclusion & Exclusion Criteria (PRISMA Flow)**:
   - *Inclusion*: Peer-reviewed conference/journal papers published 2020-2026, empirical benchmark evaluations, sample size $N \ge 100$.
   - *Exclusion*: Non-peer-reviewed blog posts, opinion pieces, non-English publications.

### Step 2: The 5-Column Synthesis Matrix
1. **Cross-Study Comparative Grid**:
   - Never write prose before completing the synthesis matrix:
     - **Column 1**: Citation & Year (e.g. Lewis et al., 2020).
     - **Column 2**: Theoretical Framework / Core Hypothesis.
     - **Column 3**: Methodology & Dataset ($N$, sample characteristics, benchmarks).
     - **Column 4**: Key Empirical Findings.
     - **Column 5**: Methodological Limitations & Research Gap.

### Step 3: Thematic Synthesis & Scholarly Dialectics
1. **Structuring by Thematic Clusters**:
   - Group papers by conceptual approach rather than author name:
     - *Theme 1: Retrieval-Augmented Grounding Approaches*.
     - *Theme 2: Reinforcement Learning from Human/AI Feedback (RLHF/RLAIF)*.
     - *Theme 3: Decoding-Time Factuality Interventions*.
2. **Surfacing Intellectual Contradictions**:
   - Highlight where leading researchers disagree:
     - *"While Zhang et al. (2023) argue that hallucinations stem primarily from out-of-distribution training data, Bender et al. (2021) maintain that the autoregressive objective itself fundamentally precludes factual comprehension."*

### Step 4: Methodological Quality Appraisal & The Gap
1. **Critical Quality Appraisal**:
   - Scrutinize empirical rigor: Were sample sizes adequate? Were control groups utilized? Did the study suffer from p-hacking or publication bias?
2. **The Research Gap Formulation**:
   - Conclude each thematic section by framing what remains unknown:
     - *"Despite widespread adoption of RAG architectures, current literature exhibits a critical gap in multi-hop reasoning over temporal knowledge graphs where historical facts conflict."*

---

## Deliverable Format: Scholarly Synthesis Matrix (LLM Factuality Research)

| Author & Year | Methodology / Model | Benchmark / Sample | Key Empirical Finding | Critical Limitation / Gap |
|---|---|---|---|---|
| **Lewis et al. (2020)** | Dense Passage Retrieval (DPR) + BART generator | Natural Questions, TriviaQA | RAG reduces hallucination rates by 42% compared to parametric-only models on open-domain QA. | Fails on cross-document multi-hop reasoning; assumes static non-conflicting corpora. |
| **Ji et al. (2023)** | Comprehensive Taxonomy Survey | 120+ NLP Papers Reviewed | Formalized hallucinations into intrinsic (conflicting with source) vs extrinsic (unverifiable). | Purely qualitative taxonomy; lacked unified quantitative benchmarking standard. |
| **Mallen et al. (2023)** | Probing QA memorization across model scales | PopQA (14k entities), EntityQuestions | LMs memorize high-popularity facts well but fail catastrophically (<15% accuracy) on long-tail tail entities. | Did not test whether fine-tuning improves long-tail retrieval without forgetting heads. |
| **Gao et al. (2024)** | Modular RAG Architecture Analysis | MultiHop-RAG, HotpotQA | Pre-retrieval routing and post-retrieval reranking improve factual precision by 28%. | Evaluated only synthetic English benchmarks; real-world noisy enterprise data unaddressed. |

---

## Worked Example: Transforming a Disjointed Draft into an Integrated Review

- **Original Fragmented Text**:
  - *"Smith (2021) investigated remote worker burnout and found high fatigue. Jones (2022) studied zoom meetings and found fatigue. Brown (2023) surveyed 500 managers about productivity."*
- **Synthesized Thematic Revision**:
  - *"Recent scholarship on post-pandemic remote organizational dynamics converges on cognitive fatigue as a primary barrier to sustained knowledge worker productivity. While early investigations focused on broad psychological stressors (Smith, 2021), subsequent empirical work isolated synchronous video communication—termed 'Zoom fatigue'—as the specific driver of neurological depletion (Jones, 2022). However, a persistent contradiction emerges in managerial appraisals: despite documented employee exhaustion, large-scale managerial surveys indicate stable or rising net output (Brown, 2023). This paradox suggests current literature conflates subjective worker well-being with immediate transactional output, leaving the long-term attrition costs of digital presenteeism unexamined."*

---

## Verification Checklist

- [ ] Papers organized into coherent conceptual themes rather than sequential author summaries.
- [ ] Synthesis matrix completed before drafting body paragraphs.
- [ ] Methodological strengths and limitations evaluated, not just study conclusions.
- [ ] Conflicting findings and academic debates highlighted and contextualized.
- [ ] Clear research gap articulated that logically motivates future work.

---

## Anti-Patterns

- **The Serial Abstract Dump**: Paraphrasing paper abstracts one after another in chronological order without connective tissue.
- **Uncritical Acceptance of Claims**: Treating authors' self-reported conclusions as objective fact without examining sample size, methodology, or confounders.
- **Missing the "So What?" (The Absent Gap)**: Writing 30 pages summarizing a field without ever concluding what questions remain unsolved.

From poorvith-mp/skills-education, MIT.

More from poorvith-mp/skills-education