paper-analyzer

Files (5)

• README.md
• README.zh-CN.md
• SKILL.md
• hypothesis.example.md · references/
• taxonomy.example.md · references/

SKILL.md (first lines)

# Paper Analyzer

A structured methodology for critical paper analysis. The goal is not summarization — it's understanding what a paper actually demonstrates (which may differ from what it claims), extracting the numbers that matter, and connecting findings to your research framework.

## When to Use

- User gives an arXiv ID, URL, or paper title and wants deep analysis
- User is building a related work section and needs to understand a specific paper's contribution
- User wants to evaluate whether a paper supports or challenges their hypothesis

## Workflow

### Step 1: Obtain the Paper and Set Up NotebookLM

The primary reading channel is **NotebookLM** — it provides source-grounded answers from Gemini, handles tables/figures reliably, and supports iterative deep questioning. Every paper analyzed should end up in NotebookLM.

**If the user specifies a notebook ID** (e.g., "in notebook 9ef789c8"):
```bash
notebooklm use <notebook-id>
notebooklm source list  # check if the paper is already a source
```

**If the paper is not yet in NotebookLM**, add it:
```bash
# Prefer arXiv HTML — it preserves equations, tables, and structure
notebooklm source add "https://arxiv.org/html/<ID>"
# If HTML unavailable, fall back to PDF (abs page only has abstract, useless for deep reading)
notebooklm source add "https://arxiv.org/pdf/<ID>"
```

If no notebook exists for this line of research, create one:
```bash
notebooklm create "Research Topic Name"
notebooklm source add "https://arxiv.org/html/<ID>"
# fallback
notebooklm source add "https://arxiv.org/pdf/<ID>"
```

**Supplementary sources** (only when NotebookLM is insufficient):
- Semantic Scholar API — for citation count, related papers, and metadata:
  `https://api.semanticscholar.org/graph/v1/paper/ArXiv:<ID>?fields=title,abstract,year,citationCount,authors`
- `/web-fetcher` on arXiv HTML — fallback if NotebookLM source processing fails
- Direct PDF via Read tool — if the user provides a local path

**How to use NotebookLM during analysis**: Use `notebooklm ask` throughout all steps — for understanding the paper's narrative (Step 2), extracting numbers (Step 3), and verifying claims. Example questions:

- Step 2: "What problem does this paper claim to solve, and what is their core hypothesis?"
- Step 2: "Walk me through their evidence structure — what experiments support their main claim?"
- Step 3: "What exact numbers does Table 3 show? Include all conditions and metrics."
- Step 3: "How is [metric name] defined? What's the denominator?"
- Step 4: "Does Section 4.2 actually control for X when making claim Y?"
- Step 4: "Is the model checkpoint in Table 3 the same as in Table 5?"

### Step 2: Deconstruct the Paper's Own Narrative

Before applying any external framework, respect the authors' story. Map out:

1. **Motivation**: What problem do they claim to solve? What gap do they identify?
2. **Core claim/hypothesis**: What's their main argument? (State it in one sentence)
3. **Evidence structure**: How do they support the claim? (Experiments, ablations, theoretical analysis)
4. **Proposed solution**: If they propose a method, what's the key mechanism?
5. **Validation**: How do they measure success? What baselines do they compare against?
6. **Limitations they acknowledge**: What do they say doesn't work or isn't covered?

This step matters because jumping straight to criticism without understanding the paper on its own terms leads to shallow analysis. The interesting insights come from the gap between what a paper shows and what it claims to show — and you can only see that gap if you first understand the claims.

### Step 3: Extract Key Numbers and Definitions

For every core metric the paper introduces or relies on:
- **Name**: The metric/concept name
- **Definition**: Precise definition including what the numerator and denominator are, how it's measured, and by whom (human? LLM judge? automatic?)
- **Key values**: The actual numbers, with enough context (which model, which benchmark, which condition) to interpret them
- **Comparison anchor**: How these numbers relate to prior work

Format as a table when there are multiple conditions:

```markdown
| Condition | Metric A | Metric B | Notes |
|-----------|----------|----------|-------|
| Baseline  | X%       | Y%       | ...   |
| Proposed  | X'%      | Y'%      | ...   |
```

The standard for definitions is: a reader should understand what the number means without looking at the original paper.

### Step 4: Write the Causal Chain Analysis

This is the core output. The structure follows a causal reasoning chain — each section flows from the previous one:

1. **现象**: What did they observe? What's the surprising or counterintuitive finding? Lead with the punchline.
2. **实验设置**: How did they observe it? What experimental design, models, data, and metrics made this phenomenon visible? The setup serves the phenomenon, not the other way around.
3. **归因**: Why do they think this happens? What's their causal explanation for the phenomenon?
4. **解法**: Based on their attribution, what do they propose to fix it? What's the mechanism of the fix, and does it actually address the attributed cause?

```markdown
### Paper-Name (arXiv-ID) — "one-line characterization"

**现象**：
- The key experimental finding, with actual numbers
- What's surprising or counterintuitive about it
- Note: lead with what they *saw*, not what they *did*

**实验设置**：Base model | Method | Data | Key metrics — how the above phenomenon was observed

**归因**：Authors' explanation for why the phenomenon occurs.

**解法**（if applicable）：Based on the attribution above, what they propose and how it addresses the cause.
```

The causal chain matters because it exposes logical gaps: if the 解法 doesn't actually follow from the 归因, or if the 归因 doesn't fully explain the 现象, that's where the interesting analysis lives.

### Step 4.5: Tri-dimensional Tagging

If a taxonomy is configured (i.e., `references/taxonomy.md` exists), add a tag line at the top of each paper's analysis, mapping it to the classification framework:

```markdown
### Paper-Name (arXiv-ID) — "大白话一句话概括"

> 📍 诊断：[用什么方法发现的] | 归因：[为什么会这样] | 修复：[怎么修的/没修]

From jiahao-shao1/sjh-skills, MIT.

More from jiahao-shao1/sjh-skills