machine-learning-mitchell

Files (1)

• SKILL.md

SKILL.md (first lines)

# Machine Learning (Tom M. Mitchell)

Apply this skill to ground ML design, evaluation, and algorithm-selection questions in the classical framework — useful for foundational reasoning even when the actual implementation uses modern deep learning libraries, since the underlying tradeoffs (bias/variance, evaluation validity, inductive bias) haven't changed.

## Core concepts

**The well-posed learning problem (T, P, E)** — Any learning problem should be stated precisely as: the Task (what the system should get better at), the Performance measure (how "better" is quantified), and the Experience (what data/interaction it learns from). Vague ML proposals ("build a model to predict churn") usually resolve once forced through this triple — especially P, which is often left implicit and later turns out to be the wrong thing to optimize.

**Hypothesis space & inductive bias** — A learner can only find hypotheses within the space its algorithm/representation can express (e.g. decision trees express axis-aligned splits; linear models express linear boundaries). Every learner has an inductive bias — the assumptions it uses to generalize beyond the training data (e.g. "prefer shorter trees," "prefer maximum-margin boundaries"). When a model generalizes badly, ask first whether the hypothesis space can even represent the true pattern, and second whether its bias matches the domain.

**Overfitting & the bias-variance tradeoff** — A hypothesis overfits if it fits training data better than the true underlying distribution, usually from a hypothesis space too expressive relative to the data available. High-bias (too simple) models underfit; high-variance (too flexible) models overfit. Diagnosing a performance problem should identify which side of this tradeoff it's on before reaching for "more data" or "a bigger model" as a default fix.

**Proper evaluation methodology** — Never evaluate on training data. Use held-out test sets or k-fold cross-validation to estimate true error, and treat that estimate as having a confidence interval, not a single fixed number — small test sets or single train/test splits produce noisy comparisons. When comparing two algorithms/models, check whether the observed difference is actually statistically significant given sample size, not just "which number is higher."

**Bayesian perspective (MAP, ML, naive Bayes, MDL)** — Learning can be framed as picking the hypothesis with Maximum A Posteriori probability given data and a prior, or Maximum Likelihood if no prior is assumed. This gives a principled reading of regularization: a prior favoring simpler hypotheses is a Bayesian justification for Occam's razor, formalized via the Minimum Description Length principle (prefer the hypothesis that minimizes encoding-length-of-model plus encoding-length-of-data-given-model). Naive Bayes (assuming conditional independence of features given the class) is a specific, often-effective simplification worth reaching for when features are numerous and roughly independent given the label.

**PAC learning & sample complexity** — Computational learning theory gives a way to reason about *how much data is enough*: a hypothesis space's complexity (its size, or VC dimension for continuous spaces) determines the number of examples needed to probably-approximately-correctly learn a good hypothesis. Use this as intuition, not literal computation, when a user asks "do I have enough data" — richer/more flexible hypothesis spaces need more data to avoid overfitting, which connects directly back to the bias-variance tradeoff.

**Matching algorithm family to problem structure** — Decision tree learning (e.g. ID3, information gain splits) suits problems needing interpretable, discrete, rule-like decisions, but needs pruning/depth limits to avoid overfitting. Artificial neural networks (trained via gradient descent/backpropagation) suit problems with complex, high-dimensional, non-linear patterns but need more data and are less interpretable. Instance-based learning (k-nearest neighbors, locally weighted regression) defers generalization until query time and suits problems where the "true function" is locally simple but globally complex, at the cost of expensive inference and poor performance in high dimensions (curse of dimensionality). Rule-learning / sequential covering suits problems needing explicit, auditable if-then rules. Reinforcement learning (e.g. Q-learning) suits sequential decision problems with delayed reward rather than i.i.d. labeled examples — don't reach for it if the problem is really supervised learning in disguise.

## How to apply this

1. **Force the T/P/E framing** when a proposed ML task is vague — especially pin down the performance measure, since that's what optimization will actually chase.
2. **Diagnose generalization problems via bias-variance** — ask whether the model underfits (too simple/biased) or overfits (too flexible/high-variance) before recommending a fix; the fixes are opposite (more capacity/features vs. more regularization/data/simpler model).
3. **Check the evaluation methodology** before trusting a reported accuracy/metric — was it measured on held-out data, with cross-validation or a big-enough test set, and is a claimed improvement over a baseline actually significant?
4. **When choosing an algorithm**, match it to the problem's structure (interpretability needs, dimensionality, data volume, whether it's sequential/delayed-reward vs. i.i.d. supervised) rather than defaulting to whatever's trendiest.
5. **When the user asks "how much data do I need" or "why doesn't this generalize"**, connect the answer to hypothesis-space complexity/inductive bias rather than giving a generic rule of thumb.
6. **For classification with independent-ish features or a Bayesian angle (priors, regularization-as-prior)**, surface naive Bayes / MAP framing as an option worth considering, especially when interpretability of the probabilistic reasoning matters.

Keep the discussion tied to the user's actual problem and data — this is a framework for sound reasoning about learning, not a lecture on classical algorithms.

From DamiMartinez/book-skills, MIT.

More from DamiMartinez/book-skills