Cover

About This Book

Category Theory for Tiny ML in Rust is a working draft that develops a small, explicit machine-learning system through the lens of category theory and Rust.

The book is designed for readers who want to understand machine learning not only as numerical computation, but as a structured pipeline of objects, transformations, composition, and constraints.

Rather than treating category theory as decorative abstraction, this book uses it as an engineering tool:

• domain objects become Rust types,
• morphisms become typed transformations,
• composition becomes executable program structure,
• training becomes repeated transformation of model state,
• and tiny ML systems become a way to make mathematical structure concrete.

This is not a finalized edition. Chapters, examples, terminology, diagrams, code, and references may evolve as the work continues.

The current public edition is still worth publishing: readers can use the existing chapters, run the Rust examples, and send evidence-shaped feedback while later completion passes continue.

The public source repository is available at github.com/hghalebi/category_theory_transformer_rs.

Public Workshop

The first public workshop for this book and Rust lab is hosted through AI Reading Club. It introduces the tiny ML pipeline as typed Rust structure and uses the working draft as the shared study material.

Register for Event

Coauthors

Hamze Ghalebi

Hamze Ghalebi is a Paris-based AI architect, CTO, and software builder associated with Remo Lab. His work focuses on production GenAI, regulated AI systems, auditable AI products, Rust systems, and the transition from AI prototypes to reliable production architectures.

His background includes advanced study at Institut Polytechnique de Paris across statistics, optimization, machine learning, artificial intelligence, distributed systems, cloud computing, and data science.

Hamze brings the engineering and product perspective of the book: how to turn mathematical and machine-learning ideas into understandable, typed, maintainable systems. His current work is especially concerned with AI systems that can be evaluated, monitored, audited, and kept under human accountability in real operational environments.

In this book, his role is to connect tiny ML, Rust implementation, and production-minded software architecture — because apparently making category theory executable was not ambitious enough already.

Farzad Jafarranmani

Farzad Jafarranmani is a researcher and engineer in the Paris area, associated with Huawei and the Lagrange Mathematics and Computing Research Center. His work sits at the intersection of mathematics, computer science, logic, semantics, proof theory, and category theory.

He holds a PhD in Mathematics and Computer Science from Université Paris Cité, where his doctoral work focused on fixpoints of types in linear logic from a Curry–Howard–Lambek perspective. He also studied Mathematics and Computer Science at ENS Paris-Saclay, with work including induction in fibred multicategories and denotational semantics of linear logic with least and greatest fixpoints.

His previous research experience includes postdoctoral work at LIP6, Laboratoire d’Informatique de Sorbonne Université / CNRS, as well as a visiting research position at the University of Cambridge.

Farzad brings the mathematical and theoretical foundation of the book: category theory, denotational semantics, proof theory, type-theoretic structure, and the discipline required to keep abstractions precise instead of merely fashionable.

Public Feedback

Public feedback is welcome while the book is still growing.

Useful feedback includes unclear explanations, broken examples, missing references, awkward terminology, incorrect or overloaded mathematical language, Rust examples that could be clearer or more idiomatic, and places where the connection between Rust, machine learning, and category theory should be made more explicit.

Feedback is easiest to act on when it is opened in the GitHub repository with a specific chapter, command, or source file.

Use the public review path if you want a short route for reviewing the book. If you want the shareable public reviewer call, use Reviewers Needed. If several readers are reviewing together, use the public review sprint to split reports across Rust, ML, category-theory, educator, and beginner perspectives. Use the chapter clarity form when you can name the first unclear sentence, output line, table row, code block, or exercise.

If you are reading the online book and cannot clone the repository right now, you can still review one public page. Read Welcome, Course Map, Domain Objects, or Morphism and Composition, then report the first visible sentence, heading, diagram, table row, code block, or exercise prompt that becomes unclear. Put public book path in the command or page field and include one evidence signal from the page. A no-clone report is useful only when it names a public page and one visible signal; broad praise or broad confusion is not enough.

This edition is intentionally public before it is final.

Citation, Reuse, And Support

Short version:

• The public book will always remain open access at hghalebi.github.io/category_theory_transformer_rs.
• The source repository is available at github.com/hghalebi/category_theory_transformer_rs.
• Cite both the public book URL and the source repository URL where reuse is allowed.
• Personal and individual study are allowed with clear citation.
• One reader may study, cite, link, clone, and run the project for personal learning.
• Commercial or organizational group reuse involving more than one person requires written permission before reproducing, adapting, distributing, or teaching material from the book or repository beyond short quotation, linking, review, and individual-study allowances. This includes company workshops, internal team workshops, classes, cohorts, courses, and training programs.
• Company workshops, internal team workshops, paid workshops, commercial training programs, course packs, adapted slide decks, handouts, labs, and workshop packets require written permission when they reuse substantial material from this project.
• When Kindle or hard copy editions are available, buying the Kindle version or a hard copy supports continued public work. Paid editions are support editions, not access gates.

The public book will always remain open access at hghalebi.github.io/category_theory_transformer_rs.

The source repository is available at github.com/hghalebi/category_theory_transformer_rs.

These are custom citation-and-permission terms. Open access means the public book remains free to read online; it does not mean unrestricted commercial redistribution, commercial training use, company workshop use, or organizational group reuse.

The source code is published so readers can inspect, run, test, and contribute to the examples. Substantial reproduced code, prose, exercises, diagrams, or adapted teaching material used in a commercial or organizational group setting follows the same written-permission rule.

Original material from this book may be linked, quoted in short form, reviewed, or discussed for personal, academic, or noncommercial educational use when the source is clearly referenced and both coauthors are credited.

Suggested reference format. Use both the public book URL and the source repository URL:

Ghalebi, H., & Jafarranmani, F.
Category Theory for Tiny ML in Rust.
Open-access working draft.
Book: https://hghalebi.github.io/category_theory_transformer_rs/
Source: https://github.com/hghalebi/category_theory_transformer_rs

Use this citation in permitted reuse contexts such as papers, posts, slides, course notes, workshop pages, repositories, and public references.

Keep both URLs in public references. The book page is the open-access reading surface; the source repository is the executable Rust source for the examples.

Reproducing, adapting, distributing, or teaching material from this book or repository in a commercial or organizational setting that involves more than one person requires written permission from the project owners, except for short quotation, linking, review, and individual-study allowances. This includes company workshops, company-sponsored workshops, internal team workshops, company reading groups based on copied or adapted material, paid training material, course packs, adapted slide decks, handouts, labs, and workshop packets. If the material is reused by or for a company, team, class, workshop, cohort, course, or training program with more than one person, request written permission first. Citation is required where reuse is allowed, but citation alone is not permission and does not replace written permission for commercial or organizational group reuse. See the repository license and reuse terms.

Company workshops, internal team workshops, paid workshops, and workshop packets count as commercial or organizational group reuse when they reproduce, adapt, distribute, or teach substantial material from the book or repository.

Permission requests should start through the source repository, for example by opening an issue or contacting the maintainers from the repository page.

Opening an issue or sending a request does not itself grant permission. Only an explicit written approval from a project owner or maintainer grants permission for the requested commercial or organizational group use.

When Kindle or hard copy editions are available, buying the Kindle version or a hard copy is a way to support continued work on the project. Paid editions are support editions, not access gates. Paid editions will not remove free public access to the online book.

External works cited or referenced by this book remain under their own licenses, terms, and attribution requirements.

Category Theory for Tiny ML in Rust

This is a public working draft. The current edition is published so readers can learn from it now, run the examples, and send precise feedback. It is not the completed textbook yet; later passes will keep revising the chapters from source review, exercises, and direct reader reports.

First Win

From the repository root, run:

cargo run --example 01_token_sequence

That command turns one sentence into typed training material:

Text
  -> TokenSequence
  -> TrainingPairs

The point of starting there is practical. Before the book asks you to care about category theory, it lets you run a small program and inspect the shape it prints.

First Output Transfer Checklist

Use the first run as a reading test. Do not treat the output as a demo banner. Treat each printed block as evidence for a boundary.

The first win is not that the tokenizer is impressive. It is deliberately tiny. The win is that you can point at the output and say:

this raw value became this domain object,
then this domain object became training examples,
and the transformation path is visible.

That is the reading habit the rest of the book repeats at larger scales.

Source-Backed Reading Contract

This welcome chapter uses sources to keep the first session practical. Each source supports one local rule for how the reader should move from the first command to the rest of the book.

The transfer pattern is:

run one small example -> name the visible boundary -> reuse the reading habit

For this chapter, the first command is evidence for a small claim:

Text becomes TokenSequence.
TokenSequence becomes TrainingPairs.
The path is visible in terminal output.

It is not evidence that the whole book is easy for every reader yet. That is why the public review path asks for exact evidence signals when a sentence, output line, table row, code block, or exercise breaks the learning path.

Then run the guided walkthrough:

cargo run --bin category_ml

That command walks through the larger pipeline: domain objects, morphisms, composition, prediction, loss, repeated training, functors, monoids, and the small chain-rule example.

The repository is public at github.com/hghalebi/category_theory_transformer_rs. Use it for source files, runnable examples, issues, and contribution work.

Help Improve This Book

If you want to help as a reader, use the public review path. If you want the shareable public call for the five reviewer perspectives, use Reviewers Needed. If you are reviewing with a group, use the public review sprint to collect one report from each reader perspective.

The most useful report is small:

Command or page tried:
Evidence signal:
Last clear idea:
First unclear sentence, output line, table row, code block, or exercise:
What would have helped:

If you are reading the online book and cannot clone the repository right now, use the same shape. Put public book path in Command or page tried, name the page you read, and quote or describe one visible evidence signal: a sentence, heading, diagram, table row, code block, or exercise prompt.

Open the chapter clarity form when the learning path breaks. One exact blocked step is more useful than a broad review.

What This Book Is About

Most machine-learning education starts with frameworks.

Frameworks are useful. They let us train real models quickly. But they also hide the small structure underneath: the types of values moving through the system, the transformations between those values, the loss that measures error, and the update step that changes the model.

This book takes the opposite path.

It builds a tiny learning system slowly enough that every important shape can be read in Rust:

Text
  -> TokenSequence
  -> TrainingSet
  -> Prediction
  -> Loss
  -> Updated Parameters

The goal is to make the hidden structure easier to see.

Executable structure, not AI magic.

The Central Thesis

This book is built around one claim:

A useful ML system is a chain of typed transformations.

Rust gives those transformations compile-checked boundaries. Category theory gives names to recurring shapes such as objects, morphisms, products, composition, endomorphisms, functors, and monoids. Tiny ML keeps the system small enough to inspect completely.

The book uses all three, but in this order:

intuition
  -> small Rust example
  -> ML meaning
  -> category-theory name
  -> runnable exercise

The category-theory words should arrive after the reader has seen the shape in code.

What You Already Know

If you have written a Rust function, you already know the first shape. A function has an input type, an output type, and a body that explains how to move from one to the other.

Worked Example: Naming One Raw Value

Start with the deliberately unsafe version:

#![allow(unused)]
fn main() {
fn token_to_position(token_id: usize) -> usize {
    token_id + 100
}

assert_eq!(token_to_position(3), 103);
}

This is a transformation from one type to another:

usize -> usize

The problem is that both sides are too vague. A raw usize might mean a token index, a vocabulary size, a vector dimension, a training step, or a row number. Those are different concepts, even if the machine representation is the same.

The book’s first move is to give those concepts names.

pub struct TokenId(usize);
pub struct VocabSize(usize);
pub struct ModelDimension(usize);

Now the reader can ask better questions:

Can this token be embedded?
Does this vector have the expected dimension?
Is this probability distribution valid?
Can this loss be accumulated?
Can this training update be repeated?

That is where the Rust type system starts to become part of the explanation.

Self-check

Before continuing, explain what changed when token_id: usize became TokenId. Did the machine representation change, or did the program gain a clearer boundary?

The Three Readings

Every important idea in the book is read three ways.

Rust Reading

The Rust reading asks:

What type is this?
What function or trait connects it to another type?
What invariant does the constructor protect?
What error can happen at the boundary?

For example:

pub struct TokenSequence(Vec<TokenId>);

This is not only “a struct containing a vector.” In the real source, it is a controlled domain object. Other code can use a TokenSequence, but it cannot freely reach inside and mutate the raw representation.

ML Reading

The ML reading asks:

What stage of the learning pipeline is this?
Is it data, prediction, loss, or an update?
What would a larger framework usually hide here?

A token sequence is not the model yet. It is data after tokenization and before training pairs. A distribution is not just a vector of floats. It is a vector of non-negative probabilities that should sum to one.

Category-Theory Reading

The category-theory reading asks:

What object is this?
What morphism starts here or ends here?
Can two transformations compose?
Is this update an endomorphism?
Which law is the code trying to make visible?

The point is not to make the code sound more abstract. The point is to name the same shape that the Rust and ML readings already revealed.

The Main Picture

The tiny model is organized around this chain:

TokenSequence -> TrainingSet
TokenId       -> Vector
Vector        -> Logits
Logits        -> Distribution
Distribution x TokenId -> Loss
Parameters    -> Parameters

Read it left to right.

The first line prepares examples.

The middle lines make a prediction and measure error.

The last line updates the model.

The Rust reading is:

types + constructors + traits + errors + tests

The ML reading is:

data + scores + probabilities + loss + training

The category-theory reading is:

objects + morphisms + products + composition + laws

Learning Contract

Use the same loop in every chapter. Start with the practical problem, read the smallest example, and then inspect the relevant source snapshot. Translate the Rust type or function into plain English before connecting it to the ML pipeline. Only after the code is concrete should the chapter name the category-theory shape. Then run the example and answer the retrieval questions without looking back.

The chapters are deliberately repetitive in structure. That repetition is part of the learning design. The pattern should become familiar:

raw representation
  -> validated domain object
  -> typed transformation
  -> composed pipeline
  -> checked law

What This Book Is Not

This is not a production ML framework.

This is not a performance-first Rust implementation.

This is not category theory as decoration.

This is not a promise that every advanced mathematical idea has been fully formalized in the code.

The examples are intentionally small. They are designed to make structure visible before speed, scale, or completeness enter the conversation.

Reading Path

Read the chapters in order on the first pass.

The Course Map gives the whole pipeline shape.

Domain Objects names the typed nouns.

Morphism and Composition names the typed arrows between them.

The Tiny ML Pipeline turns those arrows into prediction and loss.

Training as an Endomorphism shows why one optimizer step has the repeatable shape:

Parameters -> Parameters

After the core pipeline, Functors, Naturality, Monoids, and Chain Rule introduces reusable structure, and Seven Sketches Through Rust widens the method to applied category theory.

Use the Exercises for practice, the Glossary for terms, the References for chapter-specific sources, and the Transformer Roadmap for the path toward attention.

Live Study

The first public workshop for the project is available through Luma registration.

The workshop is a guided study path through the same tiny pipeline. It is useful if you want to see the code, diagrams, and vocabulary connected live.

The public session plan is available in the repository: First online workshop curriculum.

What To Remember

The central discipline is:

Do not let raw values travel farther than they should.

A raw usize becomes TokenId.

A raw Vec<TokenId> becomes TokenSequence.

A raw Vec<f32> becomes Distribution only after probability validation.

A raw optimizer update becomes TrainStep, a typed endomorphism:

Parameters -> Parameters

The result is a small codebase where every concept has a name, every boundary has a type, and every composition has to make sense before Rust lets it run.

Where This Leaves Us

This welcome chapter sets the reading contract. You will see the same idea through Rust syntax, tiny ML behavior, and category-theory shape. The next chapter, Course Map, gives the full map before the book starts reading individual source files.

Practice After This Chapter

Do one small check before moving on: run cargo run --example 01_token_sequence and explain one output line using the three-lens shape from this chapter. If you want a written prompt, use the first-output transfer checklist above and Beginner Exercise 3 in Exercises.

Retrieval Practice

Recall

What is the central pipeline shape this book keeps returning to?

Explain

Why does the book connect every concept to Rust syntax, ML meaning, and category-theory shape?

Apply

Pick one raw value from the pipeline, such as a token index or probability vector. Give it a domain-type name and explain what confusion the name prevents.

Course Map

The problem this chapter solves is:

Before reading individual source files, you need one map that connects the tiny ML pipeline, the Rust modules, and the category-theory vocabulary.

The repository is intentionally small, but it still has layers. One layer names the values. Another layer names transformations between values. A third layer uses those transformations to make predictions, measure loss, and update model parameters.

This chapter gives you the whole map before the book zooms in.

Chapter Outcomes

By the end of this chapter, you should be able to:

• place each printed line from cargo run --bin category_ml into domain value, typed transformation, or training update,
• explain how src/domain.rs, src/category.rs, src/ml.rs, and src/training.rs divide responsibility,
• translate the book’s first pipeline into objects, morphisms, product input, loss, and endomorphism language.

Choose Your Path

Use the book-first path if you want the concepts introduced in order:

Welcome
  -> Course Map
  -> Domain Objects
  -> Morphism and Composition
  -> Tiny ML Pipeline
  -> Training as an Endomorphism

Use the code-first path if you learn faster by running something first:

cargo run --example 01_token_sequence
cargo run --bin category_ml

Then come back to this map and place each printed line in one of three locations:

domain value
typed transformation
training update

Both paths are valid. The book-first path reduces surprise. The code-first path reduces abstraction anxiety. The important thing is not to open every file at once. Start with one path, run one command, and attach each new word to one visible Rust shape.

What You Already Know

If you read a program from top to bottom, you already know how to follow a flow. If you read a Rust function signature, you already know that a step has an input type and an output type. If you have seen any ML pipeline, you already know that raw data eventually becomes predictions, loss, and updates.

The map in this chapter puts those familiar habits together:

value
  -> transformation
  -> composed transformations
  -> measured error
  -> repeated update

The category-theory vocabulary is not a separate layer pasted on top. It names shapes that are already present in the Rust and ML readings.

Worked Example: From One Function To A Pipeline

Start with one ordinary function:

#![allow(unused)]
fn main() {
fn token_to_vector_id(token_id: usize) -> usize {
    token_id + 100
}

assert_eq!(token_to_vector_id(7), 107);
}

This has the shape:

usize -> usize

That is a transformation, but it is not yet a good teaching boundary. Both sides use the same raw type, so the signature does not tell us whether the number is a token, a vector row, a dimension, or something else.

The book replaces that vague movement with named stages:

TokenId -> Vector

Then it composes more stages:

TokenId -> Vector -> Logits -> Distribution

That is the basic move for the whole book. Start with a familiar function, give the meaningful values names, then ask which typed transformations can compose safely.

Self-check

Before continuing, explain why TokenId -> Vector carries more information than usize -> Vec<f32>. A strong answer should mention both reader clarity and compiler-checked boundaries.

The Whole Pipeline

The first mental model is:

Text -> Tokens -> TrainingPairs -> ModelState -> Prediction -> Loss -> Updated ModelState

Read it as one question:

What object do we have now, and what typed transformation moves us to the next object?

The same diagram with the first concrete Rust names is:

Text
  |
  | tokenize
  v
TokenSequence
  |
  | adjacent pairs
  v
TrainingSet
  |
  | train with current Parameters
  v
Parameters
  |
  | predict
  v
Distribution
  |
  | compare with target token
  v
Loss
  |
  | optimizer step
  v
Parameters

The public names and Rust names are close, but not identical:

The central book pipeline is:

Text
  -> TokenSequence
  -> TrainingSet
  -> Prediction
  -> Loss
  -> Updated Parameters

The concrete Rust shape is slightly more detailed:

TokenSequence -> TrainingSet
TokenId       -> Vector
Vector        -> Logits
Logits        -> Distribution
Distribution x TokenId -> Loss
Parameters    -> Parameters

The same map can be drawn as a learner-facing flow:

raw text
   |
   v
TokenSequence --DatasetWindowing--> TrainingSet
   |
   v
TokenId --Embedding--> Vector --LinearToLogits--> Logits --Softmax--> Distribution
                                                                        |
                                                                        v
                                                       Product<Distribution, TokenId>
                                                                        |
                                                                        v
                                                               CrossEntropy -> Loss

Parameters --TrainStep--> Updated Parameters

The same course map as a compact rendered math view:

[ \begin{array}{ccccccccc} \mathrm{Text} & \to & \mathrm{TokenSequence} & \xrightarrow{\mathrm{DatasetWindowing}} & \mathrm{TrainingSet} & \leadsto & \mathrm{Product}\langle\mathrm{Distribution},\mathrm{TokenId}\rangle & \xrightarrow{\mathrm{CrossEntropy}} & \mathrm{Loss} \ &&& &&& \uparrow \mathrm{Softmax \circ LinearToLogits \circ Embedding} && \ &&& &&& \mathrm{TokenId} && \ \mathrm{Parameters} & \xrightarrow{\mathrm{TrainStep}} & \mathrm{UpdatedParameters} &&&&&& \end{array} ]

Read the top path as prediction and evaluation. Read the bottom path as training state. The two meet because TrainStep uses the training set, current parameters, prediction path, and loss to produce updated parameters.

If the text diagram is easier to read first, use it first. If the rendered view is easier to track, redraw it and label the Rust object behind every mathematical name. Both views are teaching aids; the proof that the map is real is still the code and the commands.

Read that map in three ways.

The Rust reading is about named types, trait implementations, constructors, fallible boundaries, and tests. The ML reading is about data preparation, embeddings, scores, probabilities, error measurement, and parameter updates. The category-theory reading is about objects, morphisms, products, composition, endomorphisms, and laws.

These are not three different books. They are three readings of the same small program.

Module Map

The source tree follows the learning path. Each file owns one part of the conceptual load, so the reader does not have to learn every abstraction at the same time.

This table is not something to memorize. Use it as a navigation tool. When a later chapter names a concept, you should be able to place it in one file and one row of the pipeline.

The Library Surface

The library root collects the modules and re-exports the teaching types. That is why the examples can import clear names instead of reaching through deep paths.

The important design is:

domain nouns
category arrows
ML arrows
training update
structure patterns
calculus rule
applied sketches
demo

That order is also the reading path. The book first asks “what are the values?” Then it asks “what transformations are allowed?” Only after those two questions does it build prediction, loss, and training.

How The Files Fit Together

src/domain.rs defines the nouns. A TokenId is not a VocabSize, a Distribution is not a raw vector, and a LearningRate is not any other floating-point number. This file protects meaning at the boundary where raw machine values enter the tutorial.

src/category.rs defines the arrows. The central trait is:

pub trait Morphism<Input, Output> {
    fn name(&self) -> &'static str;
    fn apply(&self, input: Input) -> CtResult<Output>;
}

That trait says: a morphism is something that transforms an Input into an Output, and the transformation may fail with a typed course error.

src/ml.rs makes the arrows concrete. It implements dataset windowing, embedding lookup, linear projection, softmax, and cross entropy. This is where the abstract phrase “typed transformation” becomes a tiny learning pipeline.

src/training.rs defines the update step:

TrainStep : Parameters -> Parameters

Because the output type is the same as the input type, the update can be repeated:

Parameters0 -> Parameters1 -> Parameters2 -> ... -> ParametersN

That is why the training chapter teaches one optimizer step as an endomorphism. It is a transformation from a type back to itself.

src/structure.rs gives names to reusable patterns that appear after the pipeline works: mapping inside a container, converting one wrapper shape to another, and combining traces with an identity value. These ideas are useful because real systems accumulate logs, batches, optional results, gradients, and workflow traces.

src/calculus.rs keeps backpropagation deliberately small. It shows the local rule for:

z = x * y
dL/dx = dL/dz * y
dL/dy = dL/dz * x

This is not a full automatic-differentiation engine. It is the smallest local chain-rule shape the later training story can point at.

src/sketches.rs connects the tutorial to applied category theory beyond the tiny ML pipeline. It models orders, resources, databases, co-design, signal flow, circuits, and behavior logic as typed Rust values with law-checking tests.

Guided Walkthrough Snapshot

The terminal demo is the spine of the book. It gives a learner one command that uses every major idea in a concrete order.

use crate::calculus::{LocalGradient, MulOp, Scalar};
use crate::category::{Compose, StepCount, apply_endomorphism_n_times};
use crate::domain::{
    LearningRate, Logits, ModelDimension, Parameters, Product, TokenId, TokenSequence, Vector,
    VocabSize,
};
use crate::error::CtResult;
use crate::ml::{
    CrossEntropy, DatasetWindowing, Embedding, LinearToLogits, Softmax, average_loss,
    composed_prediction_matches_direct_prediction,
};
use crate::structure::{
    Functor, Monoid, OptionFunctor, PipelineTrace, TraceStep, VecFunctor,
    monoid_laws_hold_for_pipeline_trace, naturality_square_holds_for_first_option,
};
use crate::training::TrainStep;
use crate::{Identity, Morphism};

/// Run the full terminal walkthrough used by `cargo run --bin category_ml`.
pub fn run_demo() -> CtResult<()> {
    println!("Category theory concepts implemented in Rust 2024");
    println!("=================================================\n");

    let vocab = ["<pad>", "I", "love", "Rust", "."];
    let raw_text = TokenSequence::from_indices([1, 2, 3, 4, 1, 2, 3, 4])?;

    println!("1. Object examples");
    println!("   TokenId(1) means {:?}\n", vocab[1]);

    println!("2. Dataset morphism: TokenSequence -> TrainingSet");
    let dataset = DatasetWindowing.apply(raw_text)?;
    for example in dataset.examples() {
        println!(
            "   {:?} -> {:?}",
            vocab[example.first().index()],
            vocab[example.second().index()]
        );
    }
    println!();

    println!("3. Identity morphism: id_Vector : Vector -> Vector");
    let v = Vector::new(vec![1.0, 2.0, 3.0]);
    let same_v = Identity::<Vector>::new().apply(v.clone())?;
    println!("   input  = {:?}", v);
    println!("   output = {:?}\n", same_v);

    println!("4. Composition: Softmax after Linear after Embedding");
    let params = Parameters::init(VocabSize::new(vocab.len())?, ModelDimension::new(4)?);
    let embedding = Embedding::from_parameters(&params);
    let linear = LinearToLogits::from_parameters(&params);
    let token_to_logits = Compose::<_, _, Vector>::new(embedding, linear);
    let token_to_distribution = Compose::<_, _, Logits>::new(token_to_logits, Softmax);
    let distribution = token_to_distribution.apply(TokenId::new(1))?;
    println!("   P(next token | 'I') = {:?}\n", distribution.as_slice());

    println!("5. Product object: Prediction x Target -> Loss");
    let loss = CrossEntropy.apply(Product::new(distribution, TokenId::new(2)))?;
    println!("   loss for target 'love' = {:.6}\n", loss.value());

    println!("6. Endomorphism: TrainStep : Parameters -> Parameters");
    let before = average_loss(&params, &dataset)?;
    let train_step = TrainStep::new(dataset.clone(), LearningRate::new(1.0)?);
    let trained_params =
        apply_endomorphism_n_times(&train_step, params.clone(), StepCount::new(80))?;
    let after = average_loss(&trained_params, &dataset)?;
    println!("   average loss before training = {:.6}", before.value());
    println!("   average loss after  training = {:.6}\n", after.value());

    println!("7. Functor: fmap over Vec and Option");
    let xs = vec![1, 2, 3];
    let ys = VecFunctor::fmap(xs, |x| x * x);
    let maybe = OptionFunctor::fmap(Some(7), |x| x + 1);
    println!("   VecFunctor fmap square: {:?}", ys);
    println!("   OptionFunctor fmap +1: {:?}\n", maybe);

    println!("8. Natural transformation: Vec<A> -> Option<A>");
    println!(
        "   naturality square holds: {}\n",
        naturality_square_holds_for_first_option()
    );

    println!("9. Monoid: pipeline traces compose associatively with identity");
    let trace = PipelineTrace::from_steps(vec![TraceStep::new("embedding")])
        .combine(&PipelineTrace::from_steps(vec![TraceStep::new("linear")]))
        .combine(&PipelineTrace::from_steps(vec![TraceStep::new("softmax")]));
    println!("   trace = {:?}", trace.names());
    println!(
        "   monoid laws hold for this trace type: {}\n",
        monoid_laws_hold_for_pipeline_trace()
    );

    println!("10. Commutative diagram check");
    println!(
        "   composed prediction == direct prediction: {}\n",
        composed_prediction_matches_direct_prediction(&params)?
    );

    println!("11. Chain rule / local derivative morphism");
    let mul = MulOp;
    let x = Scalar::new(2.0)?;
    let y = Scalar::new(3.0)?;
    let z = mul.forward(x, y)?;
    let upstream = LocalGradient::new(1.0)?;
    let (dl_dx, dl_dy) = mul.backward(x, y, upstream)?;
    println!("   z = x * y = {}", z.value());
    println!(
        "   if dL/dz = {}, then dL/dx = {}, dL/dy = {}\n",
        upstream.value(),
        dl_dx.value(),
        dl_dy.value()
    );

    println!("Compressed categorical training view:");
    println!("   Dataset x Parameters -> Prediction -> Loss -> Gradients -> Updated Parameters");
    println!("   TrainStep is repeated as Parameters0 -> Parameters1 -> ... -> ParametersN");

    Ok(())
}

How To Read The Demo

The demo output is a miniature course outline.

It starts with an object:

TokenId(1)

Then it applies a data-preparation morphism:

TokenSequence -> TrainingSet

Then it shows identity and composition:

Vector -> Vector
TokenId -> Vector -> Logits -> Distribution

Then it uses a product object to measure loss:

Distribution x TokenId -> Loss

Then it repeats an endomorphism:

Parameters -> Parameters

The later demo sections add functors, naturality, monoids, a commutative diagram check, and a local chain-rule example. By the time you finish the demo, you have seen each major term at least once in executable form.

Demo Output Wayfinding Checklist

After running cargo run --bin category_ml, use the numbered output as a map instead of reading it as one long printout.

This table gives you a safe next action. If the output line is clear, continue reading. If it is not clear, open the source file in the second column and look for the type or function named by the line. The goal is not to memorize the demo. The goal is to use it as a routing table from terminal output to chapter, source file, ML role, and category-theory shape.

Source-Backed Wayfinding Rules

This chapter uses sources to keep the opening map practical. Each source supports one local rule for how a first session should move from command output to source files and then to vocabulary.

The transfer pattern is:

source rule -> route through files -> command/output evidence

For this chapter, that means using cargo run --bin category_ml as more than a demo. Treat it as a table of contents whose output routes you to src/domain.rs, src/category.rs, src/ml.rs, src/training.rs, src/structure.rs, src/calculus.rs, and src/sketches.rs.

The table is not evidence that the book has solved every learner’s route through the material. It is evidence that the first-session map is grounded in named sources, real files, and executable output.

Binary Entrypoint

The binary entrypoint is deliberately tiny:

fn main() -> category_theory_transformer_rs::CtResult<()> {
    category_theory_transformer_rs::run_demo()
}

The whole file delegates to the library walkthrough:

fn main() -> category_theory_transformer_rs::CtResult<()> {
    category_theory_transformer_rs::run_demo()
}

The binary returns CtResult, so fallible work can propagate through Rust’s ordinary Result path. The binary stays short because this book is teaching the typed pipeline, not command-line interface design.

First Run

Start with the smallest visible pipeline:

cargo run --example 01_token_sequence

That command turns text into token IDs and next-token training pairs before any model weights appear.

Then run the full guided demo:

cargo run --bin category_ml

The exact floating-point values are less important than the shape. You should see a loss before training, a lower loss after repeated training, and the same typed pipeline used throughout the walkthrough.

Core Mental Model

Every chapter after this one zooms into one row of the map.

An object is a typed thing the program can talk about precisely.

A morphism is a typed transformation from one object to another.

Composition is a legal connection between transformations, where the output type of one step matches the input type of the next.

An endomorphism is a transformation from a type back to itself.

A law is a property the code checks so the reader can trust the shape, not only the example output.

Checkpoint

Explain this line in your own words:

TokenId -> Vector -> Logits -> Distribution

A strong answer should mention token lookup, the embedding vector, vocabulary scores, the probability distribution, and the fact that the whole path is a composition of typed morphisms.

Where This Leaves Us

This chapter gave the whole shape before the details. You now know the names of the source files, the major pipeline objects, and the difference between objects, morphisms, composition, endomorphisms, and laws.

The next chapter, Domain Objects, slows down and studies the objects themselves. Before a pipeline can compose arrows safely, it needs values whose meanings are clear enough for arrows to start and end at them.

Further Reading

Do not leave this chapter with only a list of links. Use the next sources to practice the map.

Start from this local evidence:

cargo run --example 01_token_sequence
cargo run --bin category_ml
src/domain.rs
src/category.rs
src/ml.rs
src/training.rs

Then read the sources in this order:

After reading one external source, ask four questions:

1. Which command output line did it make easier to place?
2. Which source file should you inspect next?
3. Which category word did it clarify?
4. Which later chapter should you read after this map?

For this chapter, the commands are:

cargo run --example 01_token_sequence
cargo run --bin category_ml

For terminology recovery, use the Glossary entries for object, morphism, composition, and endomorphism. For source depth, use References and the Seven Sketches Through Rust companion chapter after you can already route the first demo output.

If an external source does not help you connect one terminal line to one source file and one category-theory word, it has not transferred back into the map yet.

Practice After This Chapter

Use Exercise 2 to change the demo input and Exercise 8 to connect one source file back to the course map. Use the demo-output wayfinding checklist above to decide which file to inspect. Those exercises check whether the map is active knowledge rather than only a diagram you read once.

Retrieval Practice

Recall

Name the three readings used throughout the book.

Explain

Why does the book start with a whole-pipeline map before reading individual source files?

Apply

Write a one-line diagram for a pipeline you already know, then label the input object, arrow, and output object.

Domain Objects

The problem this chapter solves is:

A machine-learning pipeline should not pass raw numbers around and hope everyone remembers what each number means.

Before this code talks about arrows, composition, loss, or training, it defines the objects those arrows will connect.

In this course, a domain object means:

raw representation
  + a meaningful name
  + optional validation
  + controlled access

For example:

usize

could mean:

• a token index
• a vocabulary size
• a model dimension
• a matrix row count
• a training step count

Those are different concepts.

So the code creates different types.

Reader orientation: In this chapter, focus on why a type exists before focusing on its syntax. A tuple struct, private field, constructor, or accessor is not decoration. It is a small boundary that tells the rest of the pipeline which states it may trust.

Chapter Outcomes

By the end of this chapter, you should be able to:

• explain why TokenId, VocabSize, ModelDimension, and StepCount should not all be raw usize values at the teaching boundary,
• separate semantic wrappers from validated objects,
• name one invalid ML state that each constructor prevents before prediction, loss, or training sees it.

What You Already Know

If you have used a Rust struct, you already know that a value can carry a name instead of floating around as raw data. If you have used an ML pipeline, you already know that a token index, a vector, a probability distribution, and a loss value play different roles. This chapter turns that familiar separation into explicit domain types.

The important move is not “wrap everything because wrappers are nice.” The important move is to ask what the rest of the pipeline is allowed to trust. Some types only separate meanings. Other types also reject invalid values before they can enter prediction, loss, or training.

Worked Example: Naming One Number

The smallest version of the pattern looks like this:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct TokenId(usize);

impl TokenId {
    fn new(index: usize) -> Self {
        Self(index)
    }

    fn index(self) -> usize {
        self.0
    }
}

assert_eq!(TokenId::new(3).index(), 3);
}

The real source file repeats that pattern with stronger validation where the value has an invariant, such as “a distribution must contain probabilities that sum to one.”

Self-Check

Before reading the full source snapshot, explain why TokenId(3) communicates more than the raw number 3.

Two Kinds Of Domain Objects

Read the file with this distinction in mind.

Both kinds matter. A TokenId is useful even though any usize can become a token ID at this layer, because it prevents accidental mixing with dimensions or row counts. A Distribution needs a stronger boundary, because not every Vec<f32> is a valid probability distribution.

This is the Rust API idea behind the chapter: put meaning and validation near construction, then expose small accessors for the raw representation when lower level code really needs it.

The domain-boundary diagram is:

[ \begin{array}{ccccc} \mathrm{usize} & \xrightarrow{\mathrm{TokenId::new}} & \mathrm{TokenId} & \xrightarrow{\mathrm{Embedding}} & \mathrm{Vector} \ \mathrm{Vec}\langle\mathrm{TokenId}\rangle & \xrightarrow{\mathrm{TokenSequence::new}} & \mathrm{TokenSequence} & \xrightarrow{\mathrm{DatasetWindowing}} & \mathrm{TrainingSet} \ \mathrm{Vec}\langle f32\rangle & \xrightarrow{\mathrm{Distribution::new}} & \mathrm{Distribution} & \xrightarrow{\mathrm{Product(-, target)}} & \mathrm{Product}\langle\mathrm{Distribution},\mathrm{TokenId}\rangle \end{array} ]

How to read this diagram:

• the left column is raw representation,
• the first arrow is the constructor or naming boundary,
• the middle object is what downstream code is allowed to trust,
• the last arrow is the first later stage that benefits from the boundary,
• redrawing the diagram should tell you which rows are semantic wrappers and which rows validate an invariant.

The diagram is deliberately modest. It does not claim that TokenId::new checks membership in a real tokenizer vocabulary. It does claim that once code asks for a TokenId, a reader no longer has to wonder whether the value is a model dimension, loop index, or training step count.

Mistakes These Types Prevent

Before reading the whole file, scan the reason each type exists. The point is not to wrap values for style. The point is to make common pipeline mistakes harder to express.

Use this table as the chapter’s review checklist. When a later section shows syntax, ask which mistake the syntax blocks.

Source-Backed Precision Rules

This chapter uses Rust sources to keep the “domain object” claim precise. Each source supports one local teaching rule, and each rule is tied to a concrete constructor, accessor, example, or test. The chapter does not claim that every wrapper is fully validated. Some types only separate meanings; other types reject invalid states at construction.

The transfer pattern is:

source rule -> local domain type -> constructor, accessor, or test evidence

For this chapter, that means reading cargo run --example 01_domain_objects and cargo test domain::tests as evidence for the small boundary claims:

TokenSequence is non-empty
Distribution is non-empty, finite, non-negative, and normalized
shape and training configuration values are not interchangeable

It is not evidence that every future ML value has already been modeled. It is evidence that the chapter’s first layer of objects has explicit names, construction boundaries, and validation where the later pipeline depends on an invariant.

Primitive-To-Domain Responsibility Ledger

Use this ledger whenever a raw value crosses into the tiny ML pipeline. The question is not only “what type wraps this value?” The question is “which boundary now owns the meaning, and what is downstream code allowed to trust?”

The first row is intentionally different from the third row. TokenId::new only gives a number a role. Distribution::new rejects invalid probability mass. Both are domain boundaries, but they own different kinds of responsibility.

This distinction protects the rest of the book from two common mistakes:

mistake 1: "Every wrapper validates everything."
mistake 2: "If a type stores a primitive, it is only decoration."

The right reading is narrower:

semantic wrapper:
  prevents role confusion at typed boundaries

validated object:
  prevents a specific invalid state before later code can trust the value

Source Snapshot

This is the domain layer used by the whole tutorial.

use crate::error::{CtError, CtResult};

/// A vocabulary index. It is intentionally not a raw `usize` in public APIs.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct TokenId(usize);

impl TokenId {
    pub fn new(index: usize) -> Self {
        Self(index)
    }

    pub fn index(&self) -> usize {
        self.0
    }
}

impl From<usize> for TokenId {
    fn from(value: usize) -> Self {
        Self::new(value)
    }
}

/// A sequence of tokens before it has been converted into training pairs.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TokenSequence(Vec<TokenId>);

impl TokenSequence {
    pub fn new(tokens: impl IntoIterator<Item = TokenId>) -> CtResult<Self> {
        let tokens = tokens.into_iter().collect::<Vec<_>>();

        if tokens.is_empty() {
            return Err(CtError::EmptyInput("token sequence"));
        }

        Ok(Self(tokens))
    }

    pub fn from_indices(indices: impl IntoIterator<Item = usize>) -> CtResult<Self> {
        Self::new(indices.into_iter().map(TokenId::new))
    }

    pub fn as_slice(&self) -> &[TokenId] {
        &self.0
    }
}

/// A dense feature vector.
#[derive(Debug, Clone, PartialEq)]
pub struct Vector(Vec<f32>);

impl Vector {
    pub fn new(values: Vec<f32>) -> Self {
        Self(values)
    }

    pub fn as_slice(&self) -> &[f32] {
        &self.0
    }
}

/// Unnormalized model scores.
#[derive(Debug, Clone, PartialEq)]
pub struct Logits(Vec<f32>);

impl Logits {
    pub fn new(values: Vec<f32>) -> Self {
        Self(values)
    }

    pub fn as_slice(&self) -> &[f32] {
        &self.0
    }
}

/// A validated probability distribution.
#[derive(Debug, Clone, PartialEq)]
pub struct Distribution(Vec<f32>);

impl Distribution {
    pub fn new(probabilities: Vec<f32>) -> CtResult<Self> {
        if probabilities.is_empty() {
            return Err(CtError::EmptyInput("distribution"));
        }

        let sum: f32 = probabilities.iter().sum();
        let all_valid = probabilities
            .iter()
            .all(|probability| probability.is_finite() && *probability >= 0.0);

        if !all_valid || !approx_eq(sum, 1.0, 1e-4) {
            return Err(CtError::InvalidProbability("distribution constructor"));
        }

        Ok(Self(probabilities))
    }

    pub fn as_slice(&self) -> &[f32] {
        &self.0
    }
}

/// A non-negative scalar objective value.
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct Loss(f32);

impl Loss {
    pub fn new(value: f32) -> CtResult<Self> {
        if !value.is_finite() || value < 0.0 {
            return Err(CtError::InvalidLoss(value));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> f32 {
        self.0
    }
}

/// Number of vocabulary entries.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct VocabSize(usize);

impl VocabSize {
    pub fn new(value: usize) -> CtResult<Self> {
        if value == 0 {
            return Err(CtError::EmptyInput("vocabulary"));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Width of each embedding vector.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ModelDimension(usize);

impl ModelDimension {
    pub fn new(value: usize) -> CtResult<Self> {
        if value == 0 {
            return Err(CtError::EmptyInput("model dimension"));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Positive optimizer step size.
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct LearningRate(f32);

impl LearningRate {
    pub fn new(value: f32) -> CtResult<Self> {
        if !value.is_finite() || value <= 0.0 {
            return Err(CtError::InvalidLearningRate(value));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> f32 {
        self.0
    }
}

/// Categorical product object: `A x B`.
#[derive(Debug, Clone, PartialEq)]
pub struct Product<A, B> {
    first: A,
    second: B,
}

impl<A, B> Product<A, B> {
    pub fn new(first: A, second: B) -> Self {
        Self { first, second }
    }

    pub fn first(&self) -> &A {
        &self.first
    }

    pub fn second(&self) -> &B {
        &self.second
    }

    pub fn into_parts(self) -> (A, B) {
        (self.first, self.second)
    }
}

pub type TrainingExample = Product<TokenId, TokenId>;

/// Non-empty next-token training pairs.
#[derive(Debug, Clone, PartialEq)]
pub struct TrainingSet(Vec<TrainingExample>);

impl TrainingSet {
    pub fn new(examples: impl IntoIterator<Item = TrainingExample>) -> CtResult<Self> {
        let examples = examples.into_iter().collect::<Vec<_>>();

        if examples.is_empty() {
            return Err(CtError::EmptyInput("training set"));
        }

        Ok(Self(examples))
    }

    pub fn examples(&self) -> &[TrainingExample] {
        &self.0
    }

    pub fn len(&self) -> usize {
        self.0.len()
    }

    pub fn is_empty(&self) -> bool {
        self.0.is_empty()
    }
}

/// Tiny model parameters for an embedding plus language-model head.
#[derive(Debug, Clone, PartialEq)]
pub struct Parameters {
    pub(crate) embedding: Vec<Vec<f32>>,
    pub(crate) lm_head: Vec<Vec<f32>>,
    pub(crate) bias: Vec<f32>,
}

impl Parameters {
    pub fn init(vocab_size: VocabSize, d_model: ModelDimension) -> Self {
        let vocab_size = vocab_size.value();
        let d_model = d_model.value();

        Self {
            embedding: init_matrix(vocab_size, d_model, 0.2, 1),
            lm_head: init_matrix(d_model, vocab_size, 0.2, 2),
            bias: vec![0.0; vocab_size],
        }
    }

    pub fn vocab_size(&self) -> usize {
        self.bias.len()
    }

    pub fn d_model(&self) -> usize {
        self.embedding.first().map_or(0, Vec::len)
    }

    pub fn embedding_table(&self) -> &[Vec<f32>] {
        &self.embedding
    }

    pub fn lm_head(&self) -> &[Vec<f32>] {
        &self.lm_head
    }

    pub fn bias(&self) -> &[f32] {
        &self.bias
    }
}

pub(crate) fn init_matrix(rows: usize, cols: usize, scale: f32, seed: usize) -> Vec<Vec<f32>> {
    let mut out = vec![vec![0.0; cols]; rows];

    for (row_index, row) in out.iter_mut().enumerate() {
        for (col_index, value) in row.iter_mut().enumerate() {
            let raw = ((row_index * cols + col_index) * 37 + seed * 101) % 1000;
            let unit = raw as f32 / 1000.0;
            *value = (unit - 0.5) * scale;
        }
    }

    out
}

pub(crate) fn approx_eq(a: f32, b: f32, eps: f32) -> bool {
    (a - b).abs() <= eps
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn distribution_rejects_non_normalized_values() {
        let result = Distribution::new(vec![0.4, 0.4]);

        assert!(matches!(result, Err(CtError::InvalidProbability(_))));
    }

    #[test]
    fn token_sequence_rejects_empty_input() {
        let result = TokenSequence::new(vec![]);

        assert!(matches!(result, Err(CtError::EmptyInput("token sequence"))));
    }
}

The Whole File

src/domain.rs defines the nouns in the tiny ML system:

TokenId
TokenSequence
Vector
Logits
Distribution
Loss
VocabSize
ModelDimension
LearningRate
Product
TrainingExample
TrainingSet
Parameters

The ML pipeline needs all of them:

TokenSequence -> TrainingSet
TokenId       -> Vector
Vector        -> Logits
Logits        -> Distribution
Distribution x TokenId -> Loss
Parameters    -> Parameters

The category-theory reading is:

These are the objects that morphisms start from and end at.

The Rust reading is:

These are wrapper types that prevent raw representation from leaking through the whole program.

Each major block below is meant to be read through three lenses:

Rust syntax:
what does the code literally declare?

ML concept:
why does the training pipeline need this value?

Category theory concept:
what object, product, list, distribution, or morphism endpoint does it model?

The chapter follows the same order as the model pipeline. First it names token data. Then it names hidden representations and probabilities. Then it names loss, configuration, paired inputs, training data, and model state.

TokenId

The problem this block solves is:

A token index should not be confused with any other usize.

The block:

/// A vocabulary index. It is intentionally not a raw `usize` in public APIs.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct TokenId(usize);

impl TokenId {
    pub fn new(index: usize) -> Self {
        Self(index)
    }

    pub fn index(&self) -> usize {
        self.0
    }
}

impl From<usize> for TokenId {
    fn from(value: usize) -> Self {
        Self::new(value)
    }
}

Rust Syntax

TokenId is a tuple struct with one private field:

pub struct TokenId(usize);

The struct is public, but the field is private.

That means other modules can name TokenId, pass it around, and call its methods, but they cannot directly reach inside and mutate the raw usize.

Why new Cannot Fail

pub fn new(index: usize) -> Self

Every usize is a valid token index at this layer.

The code does not know yet whether the token is inside a particular vocabulary. That check happens later when a morphism tries to look up an embedding row.

So TokenId::new is infallible.

Why index Exists

pub fn index(&self) -> usize {
    self.0
}

This accessor gives read-only access to the raw index when low-level code needs it.

The type still prevents accidental mixing at the API boundary.

ML Concept

In ML terms, TokenId is a vocabulary position.

If the vocabulary is:

0 = <pad>
1 = I
2 = love
3 = Rust
4 = .

then:

TokenId::new(3)

means the token Rust.

Category Theory Concept

TokenId is one object in the category of this program’s typed values.

Arrows such as Embedding start from this object:

TokenId -> Vector

TokenSequence

The problem this block solves is:

A language model does not train directly on raw text. First, text becomes a sequence of token IDs. Then that sequence becomes input-target training pairs.

This block represents the middle stage:

raw text
  -> tokens
  -> token sequence
  -> training examples
  -> model training

The block:

/// A sequence of tokens before it has been converted into training pairs.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TokenSequence(Vec<TokenId>);

impl TokenSequence {
    pub fn new(tokens: impl IntoIterator<Item = TokenId>) -> CtResult<Self> {
        let tokens = tokens.into_iter().collect::<Vec<_>>();

        if tokens.is_empty() {
            return Err(CtError::EmptyInput("token sequence"));
        }

        Ok(Self(tokens))
    }

    pub fn from_indices(indices: impl IntoIterator<Item = usize>) -> CtResult<Self> {
        Self::new(indices.into_iter().map(TokenId::new))
    }

    pub fn as_slice(&self) -> &[TokenId] {
        &self.0
    }
}

Rust Syntax: Documentation Comment

/// A sequence of tokens before it has been converted into training pairs.

This tells you the pipeline stage.

TokenSequence is not raw text.

It is also not yet training data.

It is the ordered token stream before adjacent pairs are created.

Example:

[TokenId(1), TokenId(2), TokenId(3)]

can later become:

TokenId(1) -> TokenId(2)
TokenId(2) -> TokenId(3)

Rust Syntax: Derived Traits

#[derive(Debug, Clone, PartialEq, Eq)]

Debug allows test and debugging output.

Clone allows an explicit copy of the sequence.

PartialEq allows equality checks.

Eq says equality is total and well-behaved.

Order matters. These are not equal:

[TokenId(1), TokenId(2)]
[TokenId(2), TokenId(1)]

Rust Syntax: Private Vector

pub struct TokenSequence(Vec<TokenId>);

This wraps:

Vec<TokenId>

but does not expose the vector directly.

That is important because the type’s invariant is:

TokenSequence is non-empty.

If the field were public, a caller could construct:

TokenSequence(vec![])

and bypass validation.

The private field forces construction through TokenSequence::new or TokenSequence::from_indices.

Rust Syntax: Constructor

pub fn new(tokens: impl IntoIterator<Item = TokenId>) -> CtResult<Self>

This accepts any input that can produce TokenId values:

• a vector
• an array
• a mapped iterator

The return type is:

CtResult<TokenSequence>

So construction can succeed or fail.

Rust Syntax: Collection

let tokens = tokens.into_iter().collect::<Vec<_>>();

This turns the flexible input into the concrete representation stored inside the struct.

The _ means Rust infers the element type as TokenId.

Rust Syntax: Empty Check

if tokens.is_empty() {
    return Err(CtError::EmptyInput("token sequence"));
}

This is the invariant boundary.

An empty token stream cannot carry useful sequence information.

The error happens immediately, before invalid data enters the rest of the pipeline.

Rust Syntax: Successful Construction

Ok(Self(tokens))

Inside the impl, Self means TokenSequence.

So this is equivalent to:

Ok(TokenSequence(tokens))

The vector has already been validated, so the object is safe for later code to trust.

Rust Syntax: Convenience Constructor

pub fn from_indices(indices: impl IntoIterator<Item = usize>) -> CtResult<Self> {
    Self::new(indices.into_iter().map(TokenId::new))
}

This accepts raw indices and converts each one into TokenId.

The important design choice is delegation:

from_indices -> new

Validation is not duplicated.

All construction still passes through the same non-empty check.

Rust Syntax: Read-Only Access

pub fn as_slice(&self) -> &[TokenId] {
    &self.0
}

This returns a borrowed slice.

Callers can inspect the sequence, but they cannot clear it, push to it, or replace the internal vector.

That preserves the invariant after construction.

ML Concept

TokenSequence is tokenized text before next-token examples are created.

A sequence of length n can produce n - 1 adjacent prediction pairs.

Category Theory Concept

TokenSequence behaves like:

List+ TokenId

where List+ means a non-empty finite list.

Its constructor is not:

List TokenId -> TokenSequence

because the empty list is invalid.

It is:

List TokenId -> Result TokenSequence CtError

Rust turns the partial construction into a total function by using Result.

Vector and Logits

The problem these blocks solve is:

A dense hidden vector and raw vocabulary scores are both Vec<f32>, but they do not mean the same thing.

The blocks:

#[derive(Debug, Clone, PartialEq)]
pub struct Vector(Vec<f32>);

impl Vector {
    pub fn new(values: Vec<f32>) -> Self {
        Self(values)
    }

    pub fn as_slice(&self) -> &[f32] {
        &self.0
    }
}

#[derive(Debug, Clone, PartialEq)]
pub struct Logits(Vec<f32>);

impl Logits {
    pub fn new(values: Vec<f32>) -> Self {
        Self(values)
    }

    pub fn as_slice(&self) -> &[f32] {
        &self.0
    }
}

Rust Syntax

Vector means hidden features.

Logits means unnormalized scores.

Both wrap Vec<f32>.

The distinction matters because only this arrow should produce logits:

Vector -> Logits

and only this arrow should normalize logits:

Logits -> Distribution

If both were plain Vec<f32>, the compiler could not help keep those stages separate.

These types derive PartialEq, but not Eq, because they contain f32.

Floating-point values do not have total equality because NaN is not equal to itself.

ML Concept

A Vector is the dense representation used after embedding lookup.

Example:

TokenId(3) -> [0.12, -0.44, 0.88, 0.03]

Logits are raw vocabulary scores.

Example:

[3.0, 1.0, -2.0]

They can be negative, larger than one, and do not need to sum to one.

The pipeline is:

TokenId -> Vector -> Logits -> Distribution

Category Theory Concept

If the model dimension is d, a vector lives in a vector-space-like object:

R^d

If the vocabulary size is V, logits live in:

R^V

The output projection is an arrow:

R^d -> R^V

and softmax maps:

R^V -> probability distributions over TokenId

Distribution

The problem this block solves is:

Probabilities are not just floats. A probability distribution must be non-empty, finite, non-negative, and sum to one.

The core block:

#[derive(Debug, Clone, PartialEq)]
pub struct Distribution(Vec<f32>);

impl Distribution {
    pub fn new(probabilities: Vec<f32>) -> CtResult<Self> {
        if probabilities.is_empty() {
            return Err(CtError::EmptyInput("distribution"));
        }

        let sum: f32 = probabilities.iter().sum();
        let all_valid = probabilities
            .iter()
            .all(|probability| probability.is_finite() && *probability >= 0.0);

        if !all_valid || !approx_eq(sum, 1.0, 1e-4) {
            return Err(CtError::InvalidProbability("distribution constructor"));
        }

        Ok(Self(probabilities))
    }
}

Rust Syntax: Why Construction Can Fail

This is invalid:

[]

This is invalid:

[0.4, 0.4]

because it sums to 0.8, not 1.0.

This is invalid:

[1.2, -0.2]

because probabilities cannot be negative.

So Distribution::new returns CtResult<Self>.

Rust Syntax: The Sum Check

let sum: f32 = probabilities.iter().sum();

This computes the total probability mass.

The code uses approximate equality:

approx_eq(sum, 1.0, 1e-4)

because floating-point arithmetic is not exact.

ML Concept

This is the output of softmax:

Logits -> Distribution

The rest of the model can treat a Distribution as real probabilities because the constructor checked the rule.

Category Theory Concept

Distribution is an object with a stronger invariant than Vec<f32>.

The softmax morphism lands in this object only if it can produce valid probability mass.

Loss

The problem this block solves is:

A loss value must be a real, non-negative scalar.

The block:

#[derive(Debug, Clone, Copy, PartialEq)]
pub struct Loss(f32);

impl Loss {
    pub fn new(value: f32) -> CtResult<Self> {
        if !value.is_finite() || value < 0.0 {
            return Err(CtError::InvalidLoss(value));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> f32 {
        self.0
    }
}

Rust Syntax

Loss::new rejects:

• infinity
• not-a-number values
• negative values

Cross entropy should not produce a negative loss. If it does, something has gone wrong before or during loss calculation.

Loss derives Copy because it wraps one small scalar.

Calling value() returns the raw f32 for printing, comparison, or averaging.

ML Concept

Loss is the training signal.

For next-token prediction:

loss = -log(probability assigned to the correct token)

Lower loss means the model assigned more probability to the correct answer.

Training tries to reduce this value.

Category Theory Concept

Loss is the codomain of an evaluation morphism:

Distribution x TokenId -> Loss

It maps prediction plus truth into a non-negative scalar objective.

Shape and Training Hyperparameter Types

The problem these blocks solve is:

Dimensions and learning rates need boundary checks before they are used to allocate matrices or update parameters.

The types are:

VocabSize
ModelDimension
LearningRate

Rust Syntax

VocabSize::new(0) fails because a vocabulary with zero entries is unusable.

ModelDimension::new(0) fails because an embedding vector with zero width cannot carry features.

LearningRate::new(value) fails when the value is not finite or is not positive.

These checks keep bad configuration from becoming strange matrix behavior later.

Worked Example: Configuration Values Are Not Interchangeable

The raw representation for all three values is small:

VocabSize       -> usize
ModelDimension  -> usize
LearningRate    -> f32

That can make them look like ordinary numbers. They are not ordinary once they cross the model boundary.

Parameters::init in src/domain.rs makes the distinction concrete:

let parameters = Parameters::init(
    VocabSize::new(5)?,
    ModelDimension::new(2)?,
);

The first argument chooses how many vocabulary rows and output scores exist. The second argument chooses how wide each hidden vector is. Swapping those meanings would create a different model shape, even though both values are stored as usize underneath.

The same rule applies to LearningRate. It is not a loss value, probability, or model dimension. It controls how far one update moves the parameters:

new parameter = old parameter - learning_rate * gradient

If the learning rate were zero, negative, infinite, or not-a-number, the update would stop being the small controlled movement the training chapter needs. That is why construction fails early.

ML reading:

VocabSize      -> how many token classes the model can score
ModelDimension -> how much hidden capacity each token receives
LearningRate   -> how large each optimizer step is

Category-theory reading:

VocabSize helps choose the finite token object, ModelDimension helps choose the intermediate representation object, and LearningRate selects one update from a family of possible training endomorphisms. The values are configuration for different parts of the typed system, not interchangeable numbers.

Checkpoint question:

If you see the raw value 5, what extra information tells you whether it is a
vocabulary size, model dimension, token id, or step count?

ML Concept

VocabSize controls:

embedding rows
logit length
distribution length
bias length

ModelDimension controls embedding width:

R^d

LearningRate controls optimizer step size:

parameter = parameter - learning_rate * gradient

Category Theory Concept

VocabSize describes the cardinality of the finite token object.

ModelDimension chooses the intermediate vector-space-like object.

LearningRate chooses one update morphism from a family of training endomorphisms.

Product<A, B>

The problem this block solves is:

Some ML operations need two inputs that belong together.

The block:

#[derive(Debug, Clone, PartialEq)]
pub struct Product<A, B> {
    first: A,
    second: B,
}

This is a generic pair.

It is used in two important places:

pub type TrainingExample = Product<TokenId, TokenId>;

and:

Product<Distribution, TokenId> -> Loss

Rust Syntax: Why Not A Tuple Everywhere?

Rust tuples like (A, B) would work mechanically.

Product<A, B> makes the category-theory idea visible:

A x B

It also gives named methods:

first()
second()
into_parts()

Those methods make call sites easier to read during the course.

ML Concept

Product<TokenId, TokenId> is one supervised next-token example:

input token x target token

Product<Distribution, TokenId> is the input to cross entropy:

prediction x target

Category Theory Concept

Product<A, B> is the course’s named version of:

A x B

The accessors are projection-like operations:

first  ~ pi_1
second ~ pi_2

TrainingSet

The problem this block solves is:

Training should not run on an empty collection of examples.

The block:

#[derive(Debug, Clone, PartialEq)]
pub struct TrainingSet(Vec<TrainingExample>);

impl TrainingSet {
    pub fn new(examples: impl IntoIterator<Item = TrainingExample>) -> CtResult<Self> {
        let examples = examples.into_iter().collect::<Vec<_>>();

        if examples.is_empty() {
            return Err(CtError::EmptyInput("training set"));
        }

        Ok(Self(examples))
    }
}

This mirrors TokenSequence.

The internal vector is private.

Construction validates non-emptiness.

Callers get read-only access through:

pub fn examples(&self) -> &[TrainingExample]

Rust Syntax: Why is_empty Exists If Empty Is Impossible

TrainingSet includes:

pub fn is_empty(&self) -> bool {
    self.0.is_empty()
}

For values constructed through TrainingSet::new, this should always be false.

The method exists because collection-like types conventionally expose both len and is_empty, and tests or generic code may use it.

The invariant is still protected by private storage and the constructor.

ML Concept

A TrainingSet is a non-empty list of next-token examples.

For:

[10, 25, 31, 7]

the training set is:

(10, 25)
(25, 31)
(31, 7)

Category Theory Concept

The shape is:

non-empty list of (TokenId x TokenId)

or:

List+ (TokenId x TokenId)

Parameters

The problem this block solves is:

Training needs one object that owns all trainable model state.

The block:

#[derive(Debug, Clone, PartialEq)]
pub struct Parameters {
    pub(crate) embedding: Vec<Vec<f32>>,
    pub(crate) lm_head: Vec<Vec<f32>>,
    pub(crate) bias: Vec<f32>,
}

The model has three pieces:

embedding table
lm head matrix
bias vector

The fields are pub(crate), not fully public.

That means code inside this crate can update parameters during training, but external callers use accessors.

Rust Syntax: Initialization

pub fn init(vocab_size: VocabSize, d_model: ModelDimension) -> Self

This takes validated domain values, not raw usize.

That means matrix allocation starts from:

non-empty vocabulary
positive model dimension

The initialized shapes are:

embedding: vocab_size x d_model
lm_head:   d_model x vocab_size
bias:      vocab_size

ML Concept

Parameters is the trainable state.

Prediction reads it.

Training maps it back to a new Parameters value:

Parameters -> Parameters

Category Theory Concept

Parameters is the object of the training endomorphism.

The important point is not that the numbers change.

The important point is that the type remains the same.

Utility Functions

The file ends with:

pub(crate) fn init_matrix(...)
pub(crate) fn approx_eq(...)

init_matrix is local deterministic initialization for the teaching model.

approx_eq is a small floating-point helper used by probability checks and composition tests.

Both are crate-internal implementation details, not learner-facing domain objects.

Runnable Example

The domain example shows token IDs becoming training pairs:

use category_theory_transformer_rs::{
    CtResult, DatasetWindowing, Morphism, TokenId, TokenSequence,
};

fn main() -> CtResult<()> {
    let tokens = TokenSequence::from_indices([1, 2, 3, 4])?;
    let dataset = DatasetWindowing.apply(tokens.clone())?;

    println!("TokenSequence:");
    println!("{}", format_token_sequence(tokens.as_slice()));
    println!();
    println!("TrainingSet:");
    for example in dataset.examples() {
        println!(
            "({} -> {})",
            format_token_id(example.first()),
            format_token_id(example.second())
        );
    }
    println!();
    println!("Typed boundaries:");
    println!("usize -> TokenId");
    println!("Vec<TokenId> -> TokenSequence");
    println!("TokenSequence -> TrainingSet");
    println!("TrainingExample = Product<TokenId, TokenId>");

    Ok(())
}

fn format_token_sequence(tokens: &[TokenId]) -> String {
    let formatted = tokens
        .iter()
        .map(format_token_id)
        .collect::<Vec<_>>()
        .join(", ");

    format!("[{formatted}]")
}

fn format_token_id(token: &TokenId) -> String {
    format!("TokenId({})", token.index())
}

Run:

cargo run --example 01_domain_objects

Expected shape:

TokenSequence:
[TokenId(1), TokenId(2), TokenId(3), TokenId(4)]

TrainingSet:
(TokenId(1) -> TokenId(2))
(TokenId(2) -> TokenId(3))
(TokenId(3) -> TokenId(4))

Typed boundaries:
usize -> TokenId
Vec<TokenId> -> TokenSequence
TokenSequence -> TrainingSet
TrainingExample = Product<TokenId, TokenId>

Example Output Transfer Checklist

Use the example output to test whether the chapter’s boundary idea is working.

This is why the example prints TokenId(...) instead of only 1 -> 2. Display can use raw numbers at the edge, but the teaching output should remind you that the program is moving through named objects.

Why This Matters

The main design rule is:

Use raw primitives only at the edge where they are created or displayed.

After that, use domain types.

This prevents mistakes like:

passing a model dimension where a token ID was expected
passing logits where probabilities were expected
training on an empty dataset
using a negative learning rate

Types do not prove that a model is good, that optimization will always converge, or that the tiny implementation is production-ready. They do something narrower and very useful: they make the wrong wiring harder to write. That is the first step toward a pipeline that can be explained, tested, and extended.

Core Mental Model

src/domain.rs turns raw storage into trustworthy objects.

In Rust terms:

private fields + smart constructors + accessors

In ML terms:

tokens, vectors, probabilities, loss, and model weights

In category-theory terms:

objects that morphisms can safely connect

Checkpoint

Pick one type from this file and explain:

1. What raw representation it wraps.
2. What invalid state it prevents.
3. Which morphism consumes or produces it.

Example:

Distribution wraps Vec<f32>, rejects invalid probability mass, and is produced
by Softmax before CrossEntropy consumes it.

Where This Leaves Us

This chapter gave names to the values in the system. A token id is not a model dimension, logits are not probabilities, and a training set is not just any vector of pairs. Each type marks a stage where raw storage becomes a meaningful object.

The next chapter, Morphism and Composition, adds arrows between those objects. Once the arrows exist, the book can talk about identity, composition, and repeated transformations without falling back to loose wiring conventions.

Further Reading

Do not read these sources as generic Rust advice. Read them as a way to answer one question:

what is the pipeline allowed to trust after this value is constructed?

Start from the local Rust evidence:

TokenId::new(index)              -> TokenId
TokenSequence::new(tokens)       -> Result<TokenSequence, CtError>
Distribution::new(probabilities) -> Result<Distribution, CtError>
LearningRate::new(value)         -> Result<LearningRate, CtError>
Parameters::init(vocab, dim)     -> Parameters

Then read the sources in this order:

After reading one external source, ask four questions:

1. Which domain type did it clarify?
2. Does that type only separate meaning, or does it also validate an invariant?
3. Which downstream morphism is allowed to trust the value?
4. Which command would you run to see the evidence?

For this chapter, the commands are:

cargo run --example 01_domain_objects
cargo test domain::tests --lib

For terminology recovery, use the Glossary entries for object, product object, invariant, and smart constructor. For source depth, use References and follow the Rust struct, error-handling, and API design entries.

If a source does not help you explain why Distribution::new rejects invalid probability mass before CrossEntropy sees it, it has not transferred back into the chapter yet.

Practice After This Chapter

Use Exercise 1 to explain one domain type and Exercise 7 to explain one constructor boundary. Together they test the chapter’s main distinction: some types separate meaning, while others also reject invalid states.

Retrieval Practice

Recall

What is a domain object in this book?

Explain

Why does Distribution::new validate probability mass at construction time instead of leaving that check to CrossEntropy?

Apply

Design a one-field newtype for a future SequenceLength. State one invariant its constructor should protect.

Morphism and Composition

The problem this chapter solves is:

Once the system has typed objects, it needs typed transformations between them.

In the previous chapter, the code created objects such as:

TokenId
Vector
Logits
Distribution
Loss
Parameters

This chapter explains the arrows that connect them.

The central category-theory sentence is:

A morphism is a typed transformation from one object to another.

The central Rust sentence is:

A morphism is a trait implementation with an input type, output type, and typed error result.

Reader orientation: The previous chapter defined the objects of the tiny ML system. This chapter explains how values move between those objects. That movement is the bridge between ordinary Rust functions and the categorical idea of morphisms.

Chapter Outcomes

By the end of this chapter, you should be able to:

• read Morphism<Input, Output> as a typed transformation contract,
• explain why Compose<F, G, Middle> requires the first target object to match the second source object,
• diagnose why Embedding followed directly by Softmax is illegal without weakening either stage.

What You Already Know

If you know Rust functions, you already know that computation moves from an input type to an output type. If you know ML pipelines, you already know that a prediction path is built from stages. This chapter gives that familiar movement a shared interface: Morphism<Input, Output>.

Category Terms As Rust Shapes

Before reading the generic source file, pin each category-theory word to a Rust shape and one tiny ML example.

Use the table as a translation layer. When a formal word appears later, ask which Rust trait, type parameter, or implementation makes it concrete. If no Rust shape is nearby, the explanation is probably moving too fast.

Source-Backed Precision Rules

This chapter uses external sources to keep the word morphism small enough to teach. Each source supports a limited claim, and each claim is tied to one local Rust boundary. The chapter does not claim that this crate implements a general category-theory library; it models typed transformations, identity, composition, and one repeated endomorphism helper for the tiny ML system.

The transfer pattern is:

source idea -> local typed boundary -> compiler, output, or test evidence

For this chapter, that means reading cargo run --example 02_morphism_composition, cargo test category::tests, and the failed-shape diagnostic as evidence for a small claim:

two arrows compose only when the first target object matches the second source
object

It is not evidence that every categorical law has been formalized. It is evidence that this tiny Rust interface makes the relevant middle object hard to ignore.

Source Snapshot

This file defines the typed arrow interface and the composition adapter.

use std::marker::PhantomData;

use crate::error::CtResult;

/// A typed category-theory arrow: `Input -> Output`.
pub trait Morphism<Input, Output> {
    fn name(&self) -> &'static str;
    fn apply(&self, input: Input) -> CtResult<Output>;
}

/// Identity morphism: `id_A : A -> A`.
#[derive(Debug, Clone, Copy)]
pub struct Identity<T> {
    _marker: PhantomData<T>,
}

impl<T> Identity<T> {
    pub fn new() -> Self {
        Self {
            _marker: PhantomData,
        }
    }
}

impl<T> Default for Identity<T> {
    fn default() -> Self {
        Self::new()
    }
}

impl<T> Morphism<T, T> for Identity<T> {
    fn name(&self) -> &'static str {
        "identity"
    }

    fn apply(&self, input: T) -> CtResult<T> {
        Ok(input)
    }
}

/// Composition of two morphisms: if `f : A -> B` and `g : B -> C`, this is
/// `g after f : A -> C`.
#[derive(Debug, Clone)]
pub struct Compose<F, G, Middle> {
    first: F,
    second: G,
    _middle: PhantomData<Middle>,
}

impl<F, G, Middle> Compose<F, G, Middle> {
    pub fn new(first: F, second: G) -> Self {
        Self {
            first,
            second,
            _middle: PhantomData,
        }
    }
}

impl<Input, Middle, Output, F, G> Morphism<Input, Output> for Compose<F, G, Middle>
where
    F: Morphism<Input, Middle>,
    G: Morphism<Middle, Output>,
{
    fn name(&self) -> &'static str {
        "composition"
    }

    fn apply(&self, input: Input) -> CtResult<Output> {
        let middle = self.first.apply(input)?;
        self.second.apply(middle)
    }
}

/// Endomorphism: a morphism from a type back to itself.
pub trait Endomorphism<T>: Morphism<T, T> {}

impl<T, M> Endomorphism<T> for M where M: Morphism<T, T> {}

/// How many times to repeat an endomorphism.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct StepCount(usize);

impl StepCount {
    pub fn new(value: usize) -> Self {
        Self(value)
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Repeatedly apply an endomorphism: `A0 -> A1 -> ... -> An`.
pub fn apply_endomorphism_n_times<T, E>(endo: &E, mut value: T, count: StepCount) -> CtResult<T>
where
    E: Endomorphism<T>,
{
    for _ in 0..count.value() {
        value = endo.apply(value)?;
    }

    Ok(value)
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::error::CtError;

    #[derive(Debug, Clone, Copy)]
    struct AddOne;

    impl Morphism<i32, i32> for AddOne {
        fn name(&self) -> &'static str {
            "add_one"
        }

        fn apply(&self, input: i32) -> CtResult<i32> {
            Ok(input + 1)
        }
    }

    #[derive(Debug, Clone, Copy)]
    struct Double;

    impl Morphism<i32, i32> for Double {
        fn name(&self) -> &'static str {
            "double"
        }

        fn apply(&self, input: i32) -> CtResult<i32> {
            Ok(input * 2)
        }
    }

    #[derive(Debug, Clone, Copy)]
    struct Fail;

    impl Morphism<i32, i32> for Fail {
        fn name(&self) -> &'static str {
            "fail"
        }

        fn apply(&self, _input: i32) -> CtResult<i32> {
            Err(CtError::InvalidQuantity {
                kind: "test morphism",
                value: -1,
            })
        }
    }

    #[test]
    fn identity_returns_the_same_value() -> CtResult<()> {
        let value = String::from("same");

        assert_eq!(Identity::<String>::new().apply(value.clone())?, value);
        Ok(())
    }

    #[test]
    fn identity_composes_without_changing_behavior() -> CtResult<()> {
        let left_identity = Compose::<_, _, i32>::new(Identity::<i32>::new(), AddOne);
        let right_identity = Compose::<_, _, i32>::new(AddOne, Identity::<i32>::new());

        assert_eq!(left_identity.apply(41)?, AddOne.apply(41)?);
        assert_eq!(right_identity.apply(41)?, AddOne.apply(41)?);
        Ok(())
    }

    #[test]
    fn composition_applies_first_then_second() -> CtResult<()> {
        let add_then_double = Compose::<_, _, i32>::new(AddOne, Double);

        assert_eq!(add_then_double.apply(4)?, 10);
        Ok(())
    }

    #[test]
    fn composition_returns_the_first_error() {
        let composed = Compose::<_, _, i32>::new(Fail, AddOne);

        assert!(matches!(
            composed.apply(4),
            Err(CtError::InvalidQuantity {
                kind: "test morphism",
                value: -1,
            })
        ));
    }
}

The Whole File

src/category.rs defines:

Morphism<Input, Output>
Identity<T>
Compose<F, G, Middle>
Endomorphism<T>
StepCount
apply_endomorphism_n_times

These are the abstract shapes used by the ML code.

Without this file, prediction could still be written as ordinary functions.

With this file, the course can name and test the structure:

identity
composition
endomorphism
repeated application

Read each block through the same three lenses:

Rust syntax:
what trait, struct, generic parameter, or bound is declared?

ML concept:
which model pipeline behavior does the shape support?

Category theory concept:
which arrow, identity, composition, or endomorphism idea is being modeled?

Worked Example: A Function As An Arrow

Before reading the generic trait, start with an ordinary Rust function:

#![allow(unused)]
fn main() {
fn add_one(input: i32) -> i32 {
    input + 1
}

assert_eq!(add_one(41), 42);
}

That function already has an arrow shape:

i32 -> i32

The real Morphism<Input, Output> trait makes that shape explicit, gives the arrow a name, and lets the arrow fail with a typed error when the input cannot be transformed safely.

Self-Check

Before reading the trait, explain why i32 -> i32 and TokenId -> Vector have the same arrow shape even though they mean very different things.

Worked Example: Where Composition Breaks

Now look at a tiny ML path:

TokenId -> Vector -> Logits -> Distribution

Each arrow has a job:

Embedding       : TokenId -> Vector
LinearToLogits  : Vector -> Logits
Softmax         : Logits -> Distribution

A legal composition connects the target of one arrow to the source of the next arrow:

TokenId --Embedding--> Vector --LinearToLogits--> Logits --Softmax--> Distribution

The middle object is not decoration. It is the reason the pipeline is legal. Embedding produces a Vector, and LinearToLogits consumes a Vector. LinearToLogits produces Logits, and Softmax consumes Logits.

Now remove the middle step:

TokenId --Embedding--> Vector --Softmax--> Distribution

This looks tempting if you only think in English: “turn the token into probabilities.” But the types say something more precise. Softmax does not consume a Vector. It consumes Logits. The missing arrow is:

Vector -> Logits

That is why composition is not just “run functions in order.” Composition means:

the previous output type equals the next input type

In this chapter, the word “morphism” gives that rule a handle. A morphism has a source object and a target object. Two morphisms compose only when the first target object is the second source object.

Runnable Example: The Middle Type Is The Contract

Run the example for this chapter:

cargo run --example 02_morphism_composition

The example builds the legal path in two composed steps:

let token_to_logits = Compose::<_, _, Vector>::new(embedding.clone(), linear.clone());
let token_to_distribution = Compose::<_, _, Logits>::new(token_to_logits, Softmax);

Read the third type argument as the middle object being checked:

Embedding then LinearToLogits:
TokenId -> Vector -> Logits
middle object: Vector

token_to_logits then Softmax:
TokenId -> Logits -> Distribution
middle object: Logits

The example output ends with the composition rule:

first target must equal second source
Embedding then LinearToLogits is legal because Vector == Vector
Embedding then Softmax is illegal because Vector != Logits

This is the concrete Rust reason a morphism is more than a metaphor. The type signature tells you which object an arrow produces and which object the next arrow expects. If those do not match, the composed pipeline is not a valid pipeline.

Composition Debugging Checklist

When a composition fails, do not start by changing type signatures. Name the three objects first:

first source  -> first target
second source -> second target

Then ask whether the middle objects match:

first target == second source ?

For the legal path:

The legal middle objects are:

Vector
Logits

The same legal path as a rendered math view:

[ \mathrm{TokenId} \xrightarrow{\mathrm{Embedding}} \mathrm{Vector} \xrightarrow{\mathrm{LinearToLogits}} \mathrm{Logits} \xrightarrow{\mathrm{Softmax}} \mathrm{Distribution} ]

How to read this diagram:

• the objects are the Rust domain types,
• the arrows are morphism implementations,
• composition is legal only when the target object of one arrow is the source object of the next arrow,
• the diagram is a reading aid, not a claim that Rust proves every category law.

For the broken shortcut:

The fix is not to make Softmax accept Vector. That would erase the model stage that turns hidden features into vocabulary scores. The fix is to restore the missing morphism:

Vector -> Logits

The broken shortcut is useful to draw because it exposes the missing middle object:

[ \begin{array}{ccccc} \mathrm{TokenId} & \xrightarrow{\mathrm{Embedding}} & \mathrm{Vector} & \not!\xrightarrow{\mathrm{Softmax}} & \mathrm{Distribution} \ && \downarrow \mathrm{LinearToLogits} && \ && \mathrm{Logits} & \xrightarrow{\mathrm{Softmax}} & \mathrm{Distribution} \end{array} ]

Reconstruct this diagram by hand when a composition error appears. Label the first target, the second source, and the repair arrow before changing code.

This checklist is useful beyond this chapter. Most pipeline bugs can be read as one of three failures:

The category-theory word “composition” is doing practical engineering work here. It tells you to debug the boundary, not the individual matrix multiplication, softmax formula, or display output first.

Source-Target-Middle Repair Ledger

When a composition breaks, write a small ledger before changing code. The ledger forces the abstract word “composition” back into source object, target object, middle object, and repair.

Use this audit card when the compiler, a diagram, or a reader’s intuition says two stages should connect:

composition attempt:
first arrow:
second arrow:
claimed middle object:
actual first target:
actual second source:
repair:
unsafe shortcut rejected:
validation command or output:

Worked audit:

composition attempt: Embedding then Softmax
first arrow: Embedding : TokenId -> Vector
second arrow: Softmax : Logits -> Distribution
claimed middle object: Vector
actual first target: Vector
actual second source: Logits
repair: insert LinearToLogits : Vector -> Logits
unsafe shortcut rejected: changing Softmax to accept Vector
validation command or output:
  cargo run --example 02_morphism_composition
  Embedding then Softmax is illegal because Vector != Logits

The source-backed limit is important. The Rust compiler is not proving every theorem about categories. It is checking the local trait bounds that make this pipeline composition legal or illegal.

Compiler Error As Evidence

The example does not include a broken composition because examples in this repository are expected to run. But the failed shape is still worth naming.

If you try to compose Embedding directly with Softmax, the intended shape would be:

Embedding : TokenId -> Vector
Softmax   : Logits -> Distribution

For Compose<F, G, Middle> to implement Morphism<Input, Output>, Rust needs these two facts:

F: Morphism<Input, Middle>
G: Morphism<Middle, Output>

With Embedding followed by Softmax, choosing Middle = Vector asks Rust for:

Embedding : Morphism<TokenId, Vector>
Softmax   : Morphism<Vector, Distribution>

The first fact is true. The second fact is false. Softmax is implemented for Logits -> Distribution, not Vector -> Distribution.

That failed trait bound is not noise. It says the missing middle object is:

Logits

and the missing morphism is:

LinearToLogits : Vector -> Logits

So the repair is not to make Softmax accept Vector. The repair is to restore the stage that turns hidden features into vocabulary scores.

From Function To Morphism

An ordinary Rust function already has the outline:

Input -> Output

The course’s Morphism<Input, Output> trait adds three things to that outline. It gives the transformation a stable name, makes failure explicit with CtResult<Output>, and lets different transformation structs share one composition API.

That is why this chapter uses the word “morphism” carefully. In this codebase, read it first as:

a named, fallible, typed transformation

Only after that concrete reading should you attach the category-theory word.

Morphism<Input, Output>

The problem this block solves is:

The code needs one shared contract for typed transformations.

The block:

/// A typed category-theory arrow: `Input -> Output`.
pub trait Morphism<Input, Output> {
    fn name(&self) -> &'static str;
    fn apply(&self, input: Input) -> CtResult<Output>;
}

Rust Syntax: Documentation Comment

/// A typed category-theory arrow: `Input -> Output`.

This tells you how to read the trait.

For example:

Embedding : TokenId -> Vector

means:

impl Morphism<TokenId, Vector> for Embedding

Rust Syntax: Trait Definition

pub trait Morphism<Input, Output>

Input and Output are type parameters.

They are not values.

They describe the type-level shape of the arrow.

This allows the same trait to model:

TokenSequence -> TrainingSet
TokenId -> Vector
Vector -> Logits
Logits -> Distribution
Distribution x TokenId -> Loss
Parameters -> Parameters

Rust Syntax: name

fn name(&self) -> &'static str;

This gives a stable human-readable name.

It is useful for demonstrations, diagnostics, and teaching.

The return type &'static str means the string is known for the whole program lifetime. Names such as "softmax" and "embedding" are static literals.

Rust Syntax: apply

fn apply(&self, input: Input) -> CtResult<Output>;

This is the actual transformation.

It consumes an Input and returns either:

Ok(Output)

or:

Err(CtError)

This is important because many arrows can fail. Embedding can receive an out-of-range token, softmax can receive empty logits, cross entropy can receive an invalid target, and training can receive malformed parameters. The shared return type keeps those failures explicit instead of hiding them behind a panic.

Read One Concrete Implementation

The abstract trait becomes concrete when a stage implements it. In src/ml.rs, the embedding stage has this shape:

impl Morphism<TokenId, Vector> for Embedding {
    fn name(&self) -> &'static str {
        "embedding"
    }

    fn apply(&self, token: TokenId) -> CtResult<Vector> {
        // lookup and validation happen here
    }
}

Read the first line slowly:

Embedding is a morphism from TokenId to Vector.

That one line gives the reader all four pieces requested by the public starter issue:

The implementation does not say that every usize can become features. It says that a validated TokenId can be applied to this embedding table, and the result is either a Vector or a typed error.

ML Concept

Every ML stage becomes an implementation of the same contract.

That makes the pipeline inspectable as arrows, not just function calls.

Category Theory Concept

This trait is the course’s concrete model of a morphism.

It is not trying to implement all category theory. It gives enough structure to talk about typed arrows and composition in ordinary Rust.

Identity<T>

The problem this block solves is:

Every object should have an arrow that returns the object unchanged.

The block:

/// Identity morphism: `id_A : A -> A`.
#[derive(Debug, Clone, Copy)]
pub struct Identity<T> {
    _marker: PhantomData<T>,
}

Rust Syntax: Why The Struct Has No Real Data

Identity<T> does not need to store a T.

It only needs to remember the type T.

That is why it stores:

_marker: PhantomData<T>

PhantomData<T> tells Rust:

This struct is logically connected to T, even though it does not own a real T value.

Rust Syntax: Constructor

pub fn new() -> Self {
    Self {
        _marker: PhantomData,
    }
}

This creates the identity arrow for a type.

Example:

Identity::<Vector>::new()

means:

id_Vector : Vector -> Vector

Rust Syntax: Default

impl<T> Default for Identity<T> {
    fn default() -> Self {
        Self::new()
    }
}

This follows Rust convention: if a type has an obvious empty constructor, it can implement Default.

Rust Syntax: Morphism Implementation

impl<T> Morphism<T, T> for Identity<T> {
    fn name(&self) -> &'static str {
        "identity"
    }

    fn apply(&self, input: T) -> CtResult<T> {
        Ok(input)
    }
}

This is the key:

T -> T

The input and output type are the same.

The implementation simply returns the input.

ML Concept

Identity is a no-op transformation.

In a model pipeline, no-op stages are useful for tests and for understanding what it means for composition to have a neutral element.

Category Theory Concept

Identity matters because composition has laws:

id after f = f
f after id = f

This code does not prove those laws generally, but it gives the object you need to talk about them in Rust.

The tests in src/category.rs check the executable version of this idea: composing identity on either side of a simple morphism leaves the behavior unchanged.

Compose<F, G, Middle>

The problem this block solves is:

If one morphism produces the type another morphism consumes, the code should be able to build a larger morphism.

The block:

/// Composition of two morphisms: if `f : A -> B` and `g : B -> C`, this is
/// `g after f : A -> C`.
#[derive(Debug, Clone)]
pub struct Compose<F, G, Middle> {
    first: F,
    second: G,
    _middle: PhantomData<Middle>,
}

Rust Syntax: The Shape

The category-theory shape is:

f : A -> B
g : B -> C
g after f : A -> C

The Rust type is:

Compose<F, G, Middle>

where:

• F is the first morphism
• G is the second morphism
• Middle is the bridge type

The middle type is explicit because Rust needs to know what connects the two arrows.

This is the most important learner habit in the chapter: when composition feels abstract, look for the middle type.

Rust Syntax: Fields

first: F,
second: G,
_middle: PhantomData<Middle>,

first stores the first arrow.

second stores the second arrow.

_middle records the bridge type without storing a value of that type.

Rust Syntax: Constructor

pub fn new(first: F, second: G) -> Self

This builds the composed morphism.

It does not run the morphisms yet.

It only stores them.

Rust Syntax: Morphism Implementation

impl<Input, Middle, Output, F, G> Morphism<Input, Output>
    for Compose<F, G, Middle>
where
    F: Morphism<Input, Middle>,
    G: Morphism<Middle, Output>,
{
    fn apply(&self, input: Input) -> CtResult<Output> {
        let middle = self.first.apply(input)?;
        self.second.apply(middle)
    }
}

This is the most important block in the chapter.

The where clause says:

F must be Input -> Middle
G must be Middle -> Output

Only then can Compose<F, G, Middle> be:

Input -> Output

Rust Syntax: The ? Operator

let middle = self.first.apply(input)?;

This applies the first arrow.

If it fails, the error returns immediately.

If it succeeds, the successful value is bound to middle.

Then the second arrow runs:

self.second.apply(middle)

So composition preserves failure.

It does not hide invalid states.

The category tests also check this behavior directly. A composed morphism that fails in its first step returns that error immediately instead of pretending the second step ran.

ML Concept

Prediction uses composition:

TokenId -> Vector -> Logits -> Distribution

The code builds that in two steps:

let token_to_logits = Compose::<_, _, Vector>::new(embedding, linear);
let token_to_distribution = Compose::<_, _, Logits>::new(token_to_logits, Softmax);

The bridge types are:

Vector
Logits

The legal diagram is:

TokenId
   |
   | Embedding
   v
Vector
   |
   | LinearToLogits
   v
Logits
   |
   | Softmax
   v
Distribution

The important detail is not the vertical layout. The important detail is that every arrow’s output object is exactly the next arrow’s input object.

If you try to compose Embedding directly with Softmax, the middle type does not match:

Embedding : TokenId -> Vector
Softmax   : Logits -> Distribution

Vector is not Logits, so Rust rejects the composition.

This is the practical win. A diagram that skips LinearToLogits is not only conceptually wrong; it has the wrong type boundary.

Category Theory Concept

Compose is function composition with types made explicit.

It is the course’s main example of:

small legal arrows -> larger legal arrow

The code is deliberately modest. It models enough composition to make the pipeline inspectable and testable; it is not claiming to encode every categorical law in Rust’s type system.

Endomorphism<T>

The problem this block solves is:

Some arrows start and end at the same type, and those arrows can be repeated.

The block:

/// Endomorphism: a morphism from a type back to itself.
pub trait Endomorphism<T>: Morphism<T, T> {}

impl<T, M> Endomorphism<T> for M where M: Morphism<T, T> {}

An endomorphism has shape:

T -> T

The trait has no methods of its own.

It is a marker trait:

if something implements Morphism<T, T>, it is an Endomorphism<T>

The blanket implementation says exactly that:

impl<T, M> Endomorphism<T> for M where M: Morphism<T, T> {}

ML Concept

Training has this shape:

Parameters -> Parameters

One training step consumes parameters and returns updated parameters.

The model changes, but the type stays the same.

Category Theory Concept

Endomorphisms are important because they can be iterated:

A -> A -> A -> A

That is the categorical shape of repeated training.

StepCount

The problem this block solves is:

Repetition count should have a semantic name instead of being a random usize at the call site.

The block:

/// How many times to repeat an endomorphism.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct StepCount(usize);

This wraps a raw usize.

It means:

number of repeated endomorphism applications

StepCount::new(80) reads better than a bare 80 because it names the role of the number.

Rust Syntax

StepCount is a newtype around usize.

It has a constructor and a value() accessor.

ML Concept

It controls how many optimizer steps are applied.

Category Theory Concept

It controls how many times an endomorphism is iterated.

apply_endomorphism_n_times

The problem this block solves is:

Given an endomorphism, repeatedly apply it in a type-safe loop.

The block:

pub fn apply_endomorphism_n_times<T, E>(
    endo: &E,
    mut value: T,
    count: StepCount,
) -> CtResult<T>
where
    E: Endomorphism<T>,
{
    for _ in 0..count.value() {
        value = endo.apply(value)?;
    }

    Ok(value)
}

Rust Syntax: Type Parameters

T is the object being updated.

E is the endomorphism type.

The bound:

E: Endomorphism<T>

means:

E must be a T -> T arrow

Rust Syntax: Mutable Value

mut value: T

The function owns the current value.

Each loop iteration replaces it with the next value:

value = endo.apply(value)?;

This is not mutation of shared global state.

It is ownership passing through a repeated transformation.

Rust Syntax: Failure Behavior

If any application fails, the whole repeated process fails immediately.

This is the correct behavior for training too: if a step discovers invalid parameters or an out-of-range token, the loop should not pretend everything is fine.

ML Concept

For training:

T = Parameters
E = TrainStep

The function becomes:

repeat TrainStep on Parameters

Category Theory Concept

This is iteration of an endomorphism:

value0
  -> value1
  -> value2
  -> ...
  -> valueN

Runnable Example

The composition example builds:

TokenId -> Vector -> Logits -> Distribution

use category_theory_transformer_rs::{
    Compose, CtResult, Distribution, Embedding, LinearToLogits, Logits, ModelDimension, Morphism,
    Parameters, Softmax, TokenId, Vector, VocabSize,
};

fn main() -> CtResult<()> {
    let params = Parameters::init(VocabSize::new(5)?, ModelDimension::new(4)?);
    let token = TokenId::new(1);
    let embedding = Embedding::from_parameters(&params);
    let linear = LinearToLogits::from_parameters(&params);

    let token_to_logits = Compose::<_, _, Vector>::new(embedding.clone(), linear.clone());
    let token_to_distribution = Compose::<_, _, Logits>::new(token_to_logits, Softmax);

    let vector = embedding.apply(token)?;
    let logits = linear.apply(vector.clone())?;
    let distribution = Softmax.apply(logits.clone())?;
    let composed_distribution = token_to_distribution.apply(token)?;

    println!("Input object:");
    println!("TokenId({})", token.index());
    println!();
    println!("Stage outputs:");
    println!("Embedding : TokenId -> Vector");
    println!("{}", format_vector(&vector));
    println!("LinearToLogits : Vector -> Logits");
    println!("{}", format_logits(&logits));
    println!("Softmax : Logits -> Distribution");
    println!("{}", format_distribution(&distribution));
    println!();
    println!("Composed morphism:");
    println!("TokenId -> Distribution");
    println!(
        "next-token probabilities: {}",
        format_values(composed_distribution.as_slice())
    );
    println!();
    println!("Middle objects kept visible:");
    println!("Vector");
    println!("Logits");
    println!();
    println!("Composition rule:");
    println!("first target must equal second source");
    println!("Embedding then LinearToLogits is legal because Vector == Vector");
    println!("Embedding then Softmax is illegal because Vector != Logits");

    Ok(())
}

fn format_vector(vector: &Vector) -> String {
    format!(
        "Vector(dim={}, values={})",
        vector.as_slice().len(),
        format_values(vector.as_slice())
    )
}

fn format_logits(logits: &Logits) -> String {
    format!(
        "Logits(vocab={}, values={})",
        logits.as_slice().len(),
        format_values(logits.as_slice())
    )
}

fn format_distribution(distribution: &Distribution) -> String {
    format!(
        "Distribution(vocab={}, sum={:.6}, values={})",
        distribution.as_slice().len(),
        distribution.as_slice().iter().sum::<f32>(),
        format_values(distribution.as_slice())
    )
}

fn format_values(values: &[f32]) -> String {
    let formatted = values
        .iter()
        .map(|value| format!("{value:.6}"))
        .collect::<Vec<_>>()
        .join(", ");

    format!("[{formatted}]")
}

Run:

cargo run --example 02_morphism_composition

Expected shape:

Input object:
TokenId(1)

Stage outputs:
Embedding : TokenId -> Vector
Vector(dim=4, values=[...])
LinearToLogits : Vector -> Logits
Logits(vocab=5, values=[...])
Softmax : Logits -> Distribution
Distribution(vocab=5, sum=1.000000, values=[...])

Composed morphism:
TokenId -> Distribution
next-token probabilities: [...]

Middle objects kept visible:
Vector
Logits

Example Output Transfer Checklist

The example prints stage outputs and then prints the composed arrow. Read that output as a composition report, not only as a numeric demo.

This is the chapter’s most important transfer move. The user-facing output is compact:

next-token probabilities: [...]

The typed explanation is larger:

TokenId -> Vector -> Logits -> Distribution

A strong reader can connect both views. The numeric output tells you what the pipeline produced. The typed path tells you why the pipeline was legal.

The stage outputs also explain the ML meaning of the middle objects:

Vector       = hidden features
Logits       = vocabulary scores
Distribution = normalized next-token probabilities

The category-theory discipline is to keep those middle objects visible. A composite arrow can be named TokenId -> Distribution, but the legal route is still built from the two bridge objects Vector and Logits.

Why This API Is Good Design

The code does not make composition a loose runtime convention.

It puts composition into the type system.

That means the compiler checks the bridge type:

F output == G input

This is the core practical value of the category-theory framing in this repo.

It turns:

remember to wire the stages correctly

into:

make invalid wiring fail to compile

Core Mental Model

In Rust terms:

Morphism<Input, Output> = fallible typed transformation
Compose<F, G, Middle> = legal connection of two transformations
Endomorphism<T> = repeatable T -> T transformation

In ML terms:

small prediction stages compose into a model path
training is a repeatable update step

In category-theory terms:

objects are connected by arrows, arrows compose when their endpoints match

Checkpoint

Why does this composition compile:

TokenId -> Vector -> Logits

but this one does not:

TokenId -> Vector -> Distribution

A strong answer should mention that Softmax expects Logits, not Vector.

Where This Leaves Us

This chapter turned ordinary transformations into named arrows. Identity<T> leaves a value unchanged, Compose<F, G, Middle> connects compatible arrows, and Endomorphism<T> names the special case where the input and output object are the same.

The next chapter, The Tiny ML Pipeline, fills those arrow shapes with concrete ML behavior: token windowing, embedding lookup, linear projection, softmax, and cross entropy.

Further Reading

Do not use these sources to make the word “morphism” sound larger. Use them to debug one concrete question:

what is the source object, target object, and middle object?

Start from the local Rust evidence:

Morphism<Input, Output>
Compose<F, G, Middle>
F: Morphism<Input, Middle>
G: Morphism<Middle, Output>
Embedding      : TokenId -> Vector
LinearToLogits : Vector -> Logits
Softmax        : Logits -> Distribution

Then read the sources in this order:

After reading one external source, ask four questions:

1. Which local boundary did it clarify?
2. Which type relationship did it help protect?
3. Which illegal composition does it help reject?
4. Which command would you run to see the evidence?

For this chapter, the commands are:

cargo run --example 02_morphism_composition
cargo test category::tests --lib
cargo test ml::tests::composed_and_direct_prediction_match --lib

Use Glossary when a term becomes slippery. Use References when you want the full source list.

If a source does not help you explain why Embedding can compose with LinearToLogits but not directly with Softmax, it has not transferred back into the chapter yet.

Practice After This Chapter

Use Exercise 4 to intentionally break a composition and explain the missing middle type. This is the chapter’s most important transfer check: a type error should become evidence about the pipeline boundary.

Retrieval Practice

Recall

Recover the shape of the API before explaining the pipeline.

1. What two methods must Morphism<Input, Output> provide?
2. Which type in Compose<F, G, Middle> records the bridge between two arrows?
3. What shape makes a morphism an endomorphism?

Explain

Use the middle object to explain why composition is legal or illegal.

1. Why does Compose<F, G, Middle> require F: Morphism<Input, Middle> and G: Morphism<Middle, Output>?
2. Why is TokenId -> Vector -> Distribution not a legal version of the prediction path?
3. Why does composition return the first error instead of trying to run the second arrow?

Apply

Use the output from cargo run --example 02_morphism_composition as the working path.

1. Write the legal path from TokenId to Distribution, naming both middle objects.
2. If you insert Identity<Vector> between Embedding and LinearToLogits, why should the behavior stay the same?
3. If you try to repeat Embedding with apply_endomorphism_n_times, which shape rule blocks the attempt?

Debug

For each invalid shortcut, name the missing or mismatched middle type:

Embedding followed directly by Softmax
Embedding followed by Identity<TokenId>
repeating Embedding as an endomorphism

A strong answer should identify the source and target object of each arrow, then state which object fails to line up. Do not answer only with “the compiler rejects it”; explain the typed boundary the compiler is protecting.

The Tiny ML Pipeline

The problem this chapter solves is:

The abstract Morphism trait needs concrete machine-learning arrows that turn token data into predictions and loss.

The whole prediction-and-loss path is:

TokenSequence -> TrainingSet
TokenId       -> Vector
Vector        -> Logits
Logits        -> Distribution
Distribution x TokenId -> Loss

In ordinary ML language, the path turns a token stream into adjacent training pairs, looks up an embedding vector for the current token, uses a linear layer to score every possible next token, normalizes those scores with softmax, and then measures surprise with cross entropy.

In category-theory language:

Each stage is a morphism, and the legal stages compose.

Reader orientation: This is the first chapter where all three subjects meet at once. When the code feels dense, follow the pipeline order: data preparation first, prediction second, loss third.

First Mental Model

The public shorthand for the whole project is:

Text -> Tokens -> TrainingPairs -> ModelState -> Prediction -> Loss -> Updated ModelState

This chapter zooms into the middle of that path. It explains how token pairs, current parameters, predictions, and loss become separate typed boundaries.

flowchart LR
    A["Text"] --> B["Tokens / TokenSequence"]
    B --> C["TrainingPairs / TrainingSet"]
    C --> F["Loss"]
    M["ModelState / Parameters"] --> P["Prediction / Distribution"]
    P --> F
    M --> U["Updated ModelState / Updated Parameters"]
    F --> U

Read the diagram as orientation, then use the Rust types for precision. The loss boundary needs both current model state and training data:

Parameters x TrainingSet -> Loss

The update boundary returns a complete next model state:

Parameters -> Updated Parameters

The same orientation as a compact rendered math view:

[ \begin{array}{ccccccc} \mathrm{Text} & \to & \mathrm{TokenSequence} & \to & \mathrm{TrainingSet} & \to & \mathrm{Loss} \ &&&& \uparrow && \downarrow \ &&&& \mathrm{Parameters} & \to & \mathrm{UpdatedParameters} \end{array} ]

How to read this diagram:

• the top row is the data path from text into measured loss,
• Parameters enters the prediction and loss boundary as model state,
• the returned object is a complete updated parameter object,
• the diagram is a map of responsibilities; the later sections name the exact Rust functions that own each arrow.

Chapter Outcomes

By the end of this chapter, you should be able to:

• trace TokenId -> Vector -> Logits -> Distribution through the concrete Rust morphisms,
• explain why cross entropy consumes both a prediction and the target token,
• distinguish the production shortcut CrossEntropyLoss(logits, target) from this book’s explicit Logits -> Distribution -> Product<Distribution, TokenId> -> Loss teaching path.

What You Already Know

If you know ML, you already know the rough path: prepare data, make a prediction, and measure the error. If you know Rust, you already know that each step can have a concrete input and output type. This chapter combines those two habits by making each ML step implement the same morphism interface.

Prediction Trace Before Source

Before reading src/ml.rs, keep this trace in view. It separates raw scores, probabilities, the target token, and the final loss.

The target index is the key detail. Cross entropy does not punish every probability equally. It first selects the probability assigned to the correct next token, then computes:

loss = -ln(probability assigned to target)

So this chapter’s core mental model is:

Logits
  -> Distribution
  -> target probability
  -> Loss

That order matters. If a reader treats logits as probabilities, or forgets that loss uses the target token index, the pipeline becomes hard to debug.

Source Snapshot

This file owns the concrete ML arrows.

use crate::category::{Compose, Morphism};
use crate::domain::{
    Distribution, Logits, Loss, Parameters, Product, TokenId, TokenSequence, TrainingSet, Vector,
    approx_eq,
};
use crate::error::{CtError, CtResult};

/// Turns adjacent tokens into next-token training examples.
#[derive(Debug, Clone)]
pub struct DatasetWindowing;

impl Morphism<TokenSequence, TrainingSet> for DatasetWindowing {
    fn name(&self) -> &'static str {
        "dataset_windowing"
    }

    fn apply(&self, tokens: TokenSequence) -> CtResult<TrainingSet> {
        if tokens.as_slice().len() < 2 {
            return Err(CtError::EmptyInput(
                "dataset windowing requires at least 2 tokens",
            ));
        }

        TrainingSet::new(
            tokens
                .as_slice()
                .windows(2)
                .map(|pair| Product::new(pair[0], pair[1])),
        )
    }
}

/// Morphism from token id to embedding vector.
#[derive(Debug, Clone)]
pub struct Embedding {
    table: Vec<Vec<f32>>,
}

impl Embedding {
    pub fn from_parameters(params: &Parameters) -> Self {
        Self {
            table: params.embedding_table().to_vec(),
        }
    }
}

impl Morphism<TokenId, Vector> for Embedding {
    fn name(&self) -> &'static str {
        "embedding"
    }

    fn apply(&self, token: TokenId) -> CtResult<Vector> {
        let Some(row) = self.table.get(token.index()) else {
            return Err(CtError::OutOfRange {
                kind: "token",
                index: token.index(),
                limit: self.table.len(),
            });
        };

        Ok(Vector::new(row.clone()))
    }
}

/// Linear projection from hidden vector to vocabulary logits.
#[derive(Debug, Clone)]
pub struct LinearToLogits {
    weight: Vec<Vec<f32>>,
    bias: Vec<f32>,
}

impl LinearToLogits {
    pub fn from_parameters(params: &Parameters) -> Self {
        Self {
            weight: params.lm_head().to_vec(),
            bias: params.bias().to_vec(),
        }
    }

    pub(crate) fn from_parts(weight: Vec<Vec<f32>>, bias: Vec<f32>) -> Self {
        Self { weight, bias }
    }
}

impl Morphism<Vector, Logits> for LinearToLogits {
    fn name(&self) -> &'static str {
        "linear_to_logits"
    }

    fn apply(&self, input: Vector) -> CtResult<Logits> {
        let d_model = input.as_slice().len();
        let vocab_size = self.bias.len();

        if self.weight.len() != d_model {
            return Err(CtError::ShapeMismatch {
                op: "linear layer",
                expected: format!("weight rows == input dim {d_model}"),
                got: format!("weight rows {}", self.weight.len()),
            });
        }

        let mut out = self.bias.clone();

        for (feature, input_value) in input.as_slice().iter().enumerate() {
            if self.weight[feature].len() != vocab_size {
                return Err(CtError::ShapeMismatch {
                    op: "linear layer",
                    expected: format!("weight cols == vocab size {vocab_size}"),
                    got: format!("weight cols {}", self.weight[feature].len()),
                });
            }

            for (vocab_id, output_value) in out.iter_mut().enumerate() {
                *output_value += input_value * self.weight[feature][vocab_id];
            }
        }

        Ok(Logits::new(out))
    }
}

/// Converts logits to a probability distribution.
#[derive(Debug, Clone)]
pub struct Softmax;

impl Morphism<Logits, Distribution> for Softmax {
    fn name(&self) -> &'static str {
        "softmax"
    }

    fn apply(&self, logits: Logits) -> CtResult<Distribution> {
        if logits.as_slice().is_empty() {
            return Err(CtError::EmptyInput("softmax"));
        }

        let max_value = logits
            .as_slice()
            .iter()
            .copied()
            .fold(f32::NEG_INFINITY, f32::max);
        let mut exps = Vec::with_capacity(logits.as_slice().len());
        let mut sum = 0.0;

        for value in logits.as_slice() {
            let exp = (*value - max_value).exp();
            exps.push(exp);
            sum += exp;
        }

        if sum <= 0.0 || !sum.is_finite() {
            return Err(CtError::InvalidProbability("softmax"));
        }

        Distribution::new(exps.into_iter().map(|value| value / sum).collect())
    }
}

/// Negative log likelihood for `(distribution, target_token)`.
#[derive(Debug, Clone)]
pub struct CrossEntropy;

impl Morphism<Product<Distribution, TokenId>, Loss> for CrossEntropy {
    fn name(&self) -> &'static str {
        "cross_entropy"
    }

    fn apply(&self, input: Product<Distribution, TokenId>) -> CtResult<Loss> {
        let (distribution, target) = input.into_parts();

        let Some(probability) = distribution.as_slice().get(target.index()).copied() else {
            return Err(CtError::OutOfRange {
                kind: "target",
                index: target.index(),
                limit: distribution.as_slice().len(),
            });
        };

        Loss::new(-probability.max(1e-9).ln())
    }
}

/// Direct path used for a commutative-diagram check.
#[derive(Debug, Clone)]
pub struct DirectPredict {
    params: Parameters,
}

impl DirectPredict {
    pub fn new(params: Parameters) -> Self {
        Self { params }
    }
}

impl Morphism<TokenId, Distribution> for DirectPredict {
    fn name(&self) -> &'static str {
        "direct_predict"
    }

    fn apply(&self, token: TokenId) -> CtResult<Distribution> {
        let embedding = Embedding::from_parameters(&self.params).apply(token)?;
        let logits = LinearToLogits::from_parameters(&self.params).apply(embedding)?;
        Softmax.apply(logits)
    }
}

/// Average cross-entropy over a training set.
pub fn average_loss(params: &Parameters, dataset: &TrainingSet) -> CtResult<Loss> {
    let embedding = Embedding::from_parameters(params);
    let linear = LinearToLogits::from_parameters(params);
    let predict = Compose::<_, _, Vector>::new(embedding, linear);
    let predict = Compose::<_, _, Logits>::new(predict, Softmax);
    let loss_fn = CrossEntropy;

    let mut total = 0.0;

    for example in dataset.examples() {
        let distribution = predict.apply(*example.first())?;
        let loss = loss_fn.apply(Product::new(distribution, *example.second()))?;
        total += loss.value();
    }

    Loss::new(total / dataset.len() as f32)
}

/// Verifies that the composed path and direct path produce the same result.
pub fn composed_prediction_matches_direct_prediction(params: &Parameters) -> CtResult<bool> {
    let token = TokenId::new(1);

    let composed = Compose::<_, _, Vector>::new(
        Embedding::from_parameters(params),
        LinearToLogits::from_parameters(params),
    );
    let composed = Compose::<_, _, Logits>::new(composed, Softmax);
    let direct = DirectPredict::new(params.clone());

    let left_path = composed.apply(token)?;
    let right_path = direct.apply(token)?;

    Ok(left_path
        .as_slice()
        .iter()
        .zip(right_path.as_slice().iter())
        .all(|(a, b)| approx_eq(*a, *b, 1e-6)))
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::domain::{ModelDimension, VocabSize};

    #[test]
    fn dataset_windowing_builds_adjacent_pairs() -> CtResult<()> {
        let tokens = TokenSequence::from_indices([1, 2, 3])?;
        let dataset = DatasetWindowing.apply(tokens)?;

        assert_eq!(dataset.len(), 2);
        assert_eq!(dataset.examples()[0].first().index(), 1);
        assert_eq!(dataset.examples()[0].second().index(), 2);
        Ok(())
    }

    #[test]
    fn composed_and_direct_prediction_match() -> CtResult<()> {
        let params = Parameters::init(VocabSize::new(5)?, ModelDimension::new(4)?);

        assert!(composed_prediction_matches_direct_prediction(&params)?);
        Ok(())
    }

    #[test]
    fn softmax_normalizes_logits_into_distribution() -> CtResult<()> {
        let distribution = Softmax.apply(Logits::new(vec![1.0, 2.0, 3.0]))?;
        let probabilities = distribution.as_slice();
        let sum: f32 = probabilities.iter().sum();

        assert!(approx_eq(sum, 1.0, 1e-6));
        assert!(probabilities[2] > probabilities[1]);
        assert!(probabilities[1] > probabilities[0]);
        Ok(())
    }

    #[test]
    fn cross_entropy_is_lower_for_more_confident_target_probability() -> CtResult<()> {
        let confident = Distribution::new(vec![0.9, 0.1])?;
        let surprised = Distribution::new(vec![0.1, 0.9])?;

        let confident_loss = CrossEntropy.apply(Product::new(confident, TokenId::new(0)))?;
        let surprised_loss = CrossEntropy.apply(Product::new(surprised, TokenId::new(0)))?;

        assert!(confident_loss.value() < surprised_loss.value());
        Ok(())
    }
}

The Whole File

src/ml.rs defines:

DatasetWindowing
Embedding
LinearToLogits
Softmax
CrossEntropy
DirectPredict
average_loss
composed_prediction_matches_direct_prediction

The chapter reads them in pipeline order.

Read each block through the same three lenses:

Rust syntax:
what struct, trait implementation, loop, or error branch does the code use?

ML concept:
which prediction, loss, or data-preparation step does the block implement?

Category theory concept:
which object, product, morphism, composition, or commutative check appears?

Worked Example: Normalizing Scores

The smallest first-principles version of “normalize scores into probabilities” does not need a model yet:

#![allow(unused)]
fn main() {
let scores = [1.0_f32, 2.0, 3.0];
let total: f32 = scores.iter().sum();
let probabilities: Vec<f32> = scores.iter().map(|score| score / total).collect();

let probability_sum: f32 = probabilities.iter().sum();
assert!((probability_sum - 1.0).abs() < 1e-6);
}

The real Softmax implementation is more careful than this toy normalization: it uses exponentials, subtracts the maximum score for numerical stability, and validates the result through Distribution::new.

Self-Check

Why is it useful for the probability-validation boundary to live in Distribution::new instead of in every caller that uses probabilities?

Scores, Probabilities, And Loss

This chapter becomes easier if you keep three numbers separate.

Logits are raw scores. They can be negative, larger than one, and they do not need to sum to one. A logit says “how strongly the model scores this token before normalization.”

Distribution values are probabilities. They must be finite, non-negative, and sum to one. A distribution says “how much probability the model assigns to each possible next token.”

That is still a local model probability, not a promise that the model’s confidence is calibrated in the outside world. Calibration asks whether events predicted with about 0.90 confidence really happen about ninety percent of the time over a population of predictions. This tiny chapter only builds and validates the normalized probability object.

Loss is a scalar penalty. Cross entropy makes the penalty small when the model assigns high probability to the correct token and large when it assigns low probability to the correct token.

The concrete path is:

raw scores
  -> probabilities
  -> surprise about the target

Here is one small numeric trace:

target token index: 0
probability assigned to target: 0.90
loss = -ln(0.90) = 0.105

target token index: 0
probability assigned to target: 0.10
loss = -ln(0.10) = 2.303

Nothing mysterious happened. The loss only looked at the probability assigned to the correct target token. A confident correct prediction receives a small penalty. A surprised prediction receives a larger penalty.

Worked Example: Do Not Use The Largest Probability

A common mistake is to compute loss from the largest probability in the distribution.

That is wrong.

Cross entropy uses the probability assigned to the actual target token, even when the model assigned a larger probability to some other token.

Consider this prediction:

probabilities over next tokens:
index 0: 0.60
index 1: 0.30
index 2: 0.10

target token index: 1

The largest probability is 0.60, but it belongs to token index 0.

The target probability is 0.30, because the correct next token is index 1.

So the loss is:

loss = -ln(0.30) = 1.204

The incorrect shortcut would be:

loss = -ln(0.60) = 0.511

That shortcut would make the prediction look better than it is. It rewards the model for being confident about the wrong token.

The Rust code prevents that confusion by pairing the distribution with the target:

Product<Distribution, TokenId>

Then CrossEntropy indexes into the distribution with target.index(). The target decides which probability becomes the loss.

The Rust path is:

Logits -> Distribution
Distribution x TokenId -> Loss

Framework Shortcut, Teaching Boundary

PyTorch’s CrossEntropyLoss accepts unnormalized logits and a target class index or target probabilities. That production API is efficient and ergonomic: the framework can combine normalization, target selection, reduction, and gradient behavior behind one call.

This book splits the same idea into smaller objects:

Logits -> Distribution -> Product<Distribution, TokenId> -> Loss

Read that as the book’s smaller Logits -> Distribution -> Product<Distribution, TokenId> -> Loss path.

That split is not a claim that production frameworks are wrong. It is a teaching boundary. It makes two questions visible before the code becomes compact:

which boundary turns scores into probabilities?
which target index selects the probability used by loss?

When moving back to frameworks, remember that the compact API still owns both roles: score normalization and target-conditioned loss.

Target-Probability Responsibility Ledger

This chapter’s most important debugging habit is to keep responsibility in the right place. Each boundary owns one job.

Use this audit card whenever the loss boundary feels slippery:

pipeline cue:
Rust handle:
ML responsibility:
category boundary:
unsafe shortcut rejected:
source-backed limit:
validation command:

Worked audit:

pipeline cue: target probability
Rust handle: distribution.as_slice().get(target.index())
ML responsibility: select the probability assigned to the correct next token
category boundary: CrossEntropy : Distribution x TokenId -> Loss
unsafe shortcut rejected: using the largest probability
source-backed limit: this checks one local supervised classification boundary,
  not calibration and not full framework equivalence
validation command:
  cargo test cross_entropy_is_lower_for_more_confident_target_probability --lib

The phrase “probability assigned to the target” should now point to one line of Rust, one ML responsibility, and one category-shaped boundary.

Source-Backed Precision Rules

This chapter uses external sources to keep the tiny prediction-and-loss path honest. Each source supports a limited claim; these citations are not proof that this crate is a production classifier, a calibrated probability model, or a framework replacement.

The transfer pattern is:

source claim -> local typed boundary -> validation command or test

For this chapter, that means reading cargo test ml::tests and the src/ml.rs morphisms as evidence for the tiny Logits -> Distribution -> Product<Distribution, TokenId> -> Loss boundary, not as evidence for every production classification stack.

The tests in src/ml.rs protect those claims: softmax normalizes logits into a distribution, and cross entropy is lower when the target token receives higher probability.

Here is the chapter’s full data-preparation and prediction diagram:

TokenSequence
     |
     | DatasetWindowing
     v
TrainingSet = [
  Product<TokenId, TokenId>,
  Product<TokenId, TokenId>,
  ...
]

For each TrainingExample:

input TokenId -------------------------------+
     |                                       |
     | Embedding                             |
     v                                       |
Vector                                      target TokenId
     |                                       |
     | LinearToLogits                        |
     v                                       |
Logits                                      |
     |                                       |
     | Softmax                               |
     v                                       |
Distribution ---------------- Product -------+
     |
     | CrossEntropy
     v
Loss

The left side is the prediction path. The right side carries the target token. CrossEntropy is the first stage that needs both, so the chapter uses Product<Distribution, TokenId> at that boundary.

The loss boundary as a rendered math view:

[ \begin{array}{ccccc} \mathrm{TokenId} & \xrightarrow{\mathrm{Embedding}} \mathrm{Vector} & \xrightarrow{\mathrm{LinearToLogits}} \mathrm{Logits} & \xrightarrow{\mathrm{Softmax}} \mathrm{Distribution} \ &&&& \downarrow \mathrm{Product(-, target)} \ &&&& \mathrm{Product}\langle \mathrm{Distribution}, \mathrm{TokenId}\rangle \xrightarrow{\mathrm{CrossEntropy}} \mathrm{Loss} \end{array} ]

How to read this diagram:

• the prediction path produces a Distribution,
• the target token does not become a prediction; it selects which probability becomes the loss,
• CrossEntropy is the first arrow that needs the product input,
• redrawing the diagram should make the target side visible, not hidden inside the word “loss”.

DatasetWindowing

The problem this block solves is:

A token sequence must become input-target pairs before supervised next-token training can happen.

The block:

/// Turns adjacent tokens into next-token training examples.
#[derive(Debug, Clone)]
pub struct DatasetWindowing;

impl Morphism<TokenSequence, TrainingSet> for DatasetWindowing {
    fn name(&self) -> &'static str {
        "dataset_windowing"
    }

    fn apply(&self, tokens: TokenSequence) -> CtResult<TrainingSet> {
        if tokens.as_slice().len() < 2 {
            return Err(CtError::EmptyInput(
                "dataset windowing requires at least 2 tokens",
            ));
        }

        TrainingSet::new(
            tokens
                .as_slice()
                .windows(2)
                .map(|pair| Product::new(pair[0], pair[1])),
        )
    }
}

Rust Syntax: Unit Struct

pub struct DatasetWindowing;

This is a unit struct.

It stores no fields because the operation has no configuration.

The value itself represents the transformation.

Rust Syntax: Morphism Shape

impl Morphism<TokenSequence, TrainingSet> for DatasetWindowing

This says:

DatasetWindowing : TokenSequence -> TrainingSet

So it consumes the raw sequence stage and produces the training-example stage.

Rust Syntax: Why It Requires At Least Two Tokens

if tokens.as_slice().len() < 2 {
    return Err(CtError::EmptyInput(
        "dataset windowing requires at least 2 tokens",
    ));
}

TokenSequence only guarantees at least one token.

But next-token training requires at least one adjacent pair.

One token:

[7]

produces zero pairs.

Two tokens:

[7, 8]

produce one pair:

7 -> 8

So this morphism owns the stronger validation rule.

Rust Syntax: windows(2)

tokens.as_slice().windows(2)

This walks adjacent pairs:

[1, 2, 3, 4]

becomes:

[1, 2]
[2, 3]
[3, 4]

Each pair becomes:

Product::new(pair[0], pair[1])

That is a TrainingExample.

ML Concept

This is the data-preparation step for next-token prediction.

Category Theory Concept

This is a morphism between two structured objects:

non-empty token list -> non-empty product list

The output examples are product objects:

TokenId x TokenId

Embedding

The problem this block solves is:

A discrete token ID needs to become a dense vector before the model can use linear algebra.

The core block:

#[derive(Debug, Clone)]
pub struct Embedding {
    table: Vec<Vec<f32>>,
}

impl Embedding {
    pub fn from_parameters(params: &Parameters) -> Self {
        Self {
            table: params.embedding_table().to_vec(),
        }
    }
}

impl Morphism<TokenId, Vector> for Embedding {
    fn name(&self) -> &'static str {
        "embedding"
    }

    fn apply(&self, token: TokenId) -> CtResult<Vector> {
        let Some(row) = self.table.get(token.index()) else {
            return Err(CtError::OutOfRange {
                kind: "token",
                index: token.index(),
                limit: self.table.len(),
            });
        };

        Ok(Vector::new(row.clone()))
    }
}

Rust Syntax: Stored Table

table: Vec<Vec<f32>>

The embedding table has shape:

vocab_size x model_dimension

Each row is the vector for one token.

Rust Syntax: Constructor From Parameters

pub fn from_parameters(params: &Parameters) -> Self

The embedding morphism is built from model parameters.

It copies the table out of Parameters:

params.embedding_table().to_vec()

This keeps the morphism simple and owned for the tiny tutorial.

Rust Syntax: Morphism Shape

impl Morphism<TokenId, Vector> for Embedding

This says:

Embedding : TokenId -> Vector

Rust Syntax: Bounds Check

let Some(row) = self.table.get(token.index()) else {
    return Err(CtError::OutOfRange { ... });
};

The code does not assume every TokenId is valid for every embedding table.

It checks the row lookup at the boundary where the table is used.

Rust Syntax: Why Clone The Row

Ok(Vector::new(row.clone()))

The morphism returns an owned Vector.

The row inside the table is borrowed, so the code clones it into the output object.

This is a deliberate ownership boundary.

ML Concept

An embedding converts a symbolic token into numerical features.

Category Theory Concept

It is an arrow:

TokenId -> Vector

LinearToLogits

The problem this block solves is:

A hidden vector must be projected into one raw score per vocabulary item.

The shape is:

Vector -> Logits

The core implementation stores:

pub struct LinearToLogits {
    weight: Vec<Vec<f32>>,
    bias: Vec<f32>,
}

The dimensions are:

weight: d_model x vocab_size
bias: vocab_size
input: d_model
output: vocab_size

Rust Syntax: Shape Validation

Inside apply, the code checks:

if self.weight.len() != d_model {
    return Err(CtError::ShapeMismatch { ... });
}

This catches a matrix whose row count does not match the input vector length.

Then each row checks:

if self.weight[feature].len() != vocab_size {
    return Err(CtError::ShapeMismatch { ... });
}

This catches rows whose column count does not match the output vocabulary size.

Rust Syntax: Linear Computation

The output begins as the bias:

let mut out = self.bias.clone();

Then each input feature contributes to every vocabulary score:

for (feature, input_value) in input.as_slice().iter().enumerate() {
    for (vocab_id, output_value) in out.iter_mut().enumerate() {
        *output_value += input_value * self.weight[feature][vocab_id];
    }
}

Mathematically:

logits = input x weight + bias

ML Concept

This is the language-model head.

It scores each possible next token.

Category Theory Concept

It is a morphism:

Vector -> Logits

It can compose after Embedding because Embedding returns Vector.

Softmax

The problem this block solves is:

Raw scores are not probabilities. They must be normalized into a valid distribution.

The block:

#[derive(Debug, Clone)]
pub struct Softmax;

impl Morphism<Logits, Distribution> for Softmax {
    fn name(&self) -> &'static str {
        "softmax"
    }

    fn apply(&self, logits: Logits) -> CtResult<Distribution> {
        if logits.as_slice().is_empty() {
            return Err(CtError::EmptyInput("softmax"));
        }

        let max_value = logits
            .as_slice()
            .iter()
            .copied()
            .fold(f32::NEG_INFINITY, f32::max);
        let mut exps = Vec::with_capacity(logits.as_slice().len());
        let mut sum = 0.0;

        for value in logits.as_slice() {
            let exp = (*value - max_value).exp();
            exps.push(exp);
            sum += exp;
        }

        if sum <= 0.0 || !sum.is_finite() {
            return Err(CtError::InvalidProbability("softmax"));
        }

        Distribution::new(exps.into_iter().map(|value| value / sum).collect())
    }
}

Rust Syntax: Unit Struct

Softmax stores no state.

It is the operation itself.

Rust Syntax: Morphism Shape

impl Morphism<Logits, Distribution> for Softmax

This says:

Softmax : Logits -> Distribution

Rust Syntax: Empty Check

Softmax over no scores is meaningless.

So the code rejects empty logits.

Rust Syntax: Numerical Stability

let max_value = ...
let exp = (*value - max_value).exp();

Subtracting the maximum value keeps exponentials smaller and more stable.

It does not change the final probabilities because softmax is invariant under adding or subtracting the same constant from every logit.

Rust Syntax: Normalization

Distribution::new(exps.into_iter().map(|value| value / sum).collect())

The raw exponentials are divided by their sum.

Then the Distribution constructor validates the probability invariant.

This is good boundary design: softmax computes, and Distribution::new enforces the distribution contract.

ML Concept

Softmax turns raw model scores into probabilities. In softmax regression and classification models, this is the step that makes one score per class interpretable as a probability distribution.

High logits become high probabilities.

Low logits become low probabilities.

The output can be interpreted as:

P(next token | current token)

Category Theory Concept

Softmax is a morphism-like transformation:

Logits -> Distribution

It changes the object from an unconstrained score vector into a probability simplex-like object.

CrossEntropy

The problem this block solves is:

A model prediction must be compared to the actual target token to produce a scalar loss.

The block:

#[derive(Debug, Clone)]
pub struct CrossEntropy;

impl Morphism<Product<Distribution, TokenId>, Loss> for CrossEntropy {
    fn name(&self) -> &'static str {
        "cross_entropy"
    }

    fn apply(&self, input: Product<Distribution, TokenId>) -> CtResult<Loss> {
        let (distribution, target) = input.into_parts();

        let Some(probability) = distribution.as_slice().get(target.index()).copied() else {
            return Err(CtError::OutOfRange {
                kind: "target",
                index: target.index(),
                limit: distribution.as_slice().len(),
            });
        };

        Loss::new(-probability.max(1e-9).ln())
    }
}

Rust Syntax: Input Type

Product<Distribution, TokenId>

Cross entropy needs both:

• the predicted distribution
• the correct target token

That pair is a product object.

Rust Syntax: Splitting The Product

let (distribution, target) = input.into_parts();

This consumes the product and extracts both values.

Rust Syntax: Target Bounds Check

distribution.as_slice().get(target.index())

The target token must be inside the probability vector.

If the distribution has 5 entries, target index 7 is invalid.

This error belongs here because this is the first place the target is used as an index into the predicted distribution.

Rust Syntax: Negative Log Likelihood

Loss::new(-probability.max(1e-9).ln())

The loss is:

-ln(probability assigned to the correct token)

The max(1e-9) avoids taking the log of zero.

Then Loss::new validates the loss scalar.

ML Concept

Cross entropy measures how surprised the model was by the true target.

If the model assigns high probability to the target, the loss is small.

If the model assigns low probability to the target, the loss is large.

This is why the chapter says loss is a training signal. It turns a probability assigned to the correct token into a number the optimizer can try to reduce.

Category Theory Concept

Cross entropy consumes a product object:

Distribution x TokenId

and maps it into:

Loss

So its shape is:

Product<Distribution, TokenId> -> Loss

DirectPredict

The problem this block solves is:

The course needs a direct implementation to compare against the composed prediction path.

DirectPredict stores parameters and implements:

TokenId -> Distribution

Internally, it still performs:

Embedding
LinearToLogits
Softmax

but it writes the steps directly.

This allows the code to test:

composed path == direct path

That is the program’s tiny commutative diagram check.

Rust Syntax

DirectPredict is a struct that owns Parameters.

Its apply method calls the prediction steps directly instead of using Compose.

ML Concept

This is the direct prediction implementation.

It exists so the composed path can be checked against a straightforward path.

Category Theory Concept

It provides the second path in a commutative diagram:

composed path
direct path

average_loss

The problem this function solves is:

Training needs one scalar loss over the whole training set.

The function builds the composed prediction path:

let embedding = Embedding::from_parameters(params);
let linear = LinearToLogits::from_parameters(params);
let predict = Compose::<_, _, Vector>::new(embedding, linear);
let predict = Compose::<_, _, Logits>::new(predict, Softmax);

The resulting shape is:

TokenId -> Distribution

Then each training example is evaluated:

let distribution = predict.apply(*example.first())?;
let loss = loss_fn.apply(Product::new(distribution, *example.second()))?;

Finally, the average is wrapped in Loss::new.

The function does not return a raw f32.

It returns a validated Loss.

Rust Syntax

The function takes borrowed parameters and a borrowed dataset:

pub fn average_loss(params: &Parameters, dataset: &TrainingSet) -> CtResult<Loss>

It does not consume either one.

The function loops through examples, accumulates scalar losses, and divides by the dataset length.

ML Concept

Average loss summarizes model performance over the full training set.

Category Theory Concept

It folds many example-level loss morphism results into one scalar object.

composed_prediction_matches_direct_prediction

The problem this function solves is:

The code should prove that the composed prediction pipeline and the direct implementation agree.

The composed path is:

TokenId -> Vector -> Logits -> Distribution

The direct path is:

TokenId -> Distribution

The function runs both on the same token and compares every probability with a small floating-point tolerance.

Category-theoretically, this is a commutative diagram test:

          composed
TokenId ------------> Distribution
   \                      ^
    \ direct              |
     ---------------------

The exact drawing is less important than the idea:

Two paths through the system should produce the same meaning.

Rust Syntax

The function builds one composed path with Compose and one direct path with DirectPredict.

It compares probabilities pairwise with approx_eq.

ML Concept

This verifies that refactoring the prediction path into smaller stages did not change the predicted probabilities.

Category Theory Concept

This is a commutative-diagram check in code.

Run The Demo

Run:

cargo run --bin category_ml

Look at sections 2 through 5 in the output.

You should see:

TokenSequence -> TrainingSet
prediction probabilities
loss for a target token

Demo Output Transfer Checklist

Sections 2 through 5 of the demo are the smallest complete ML story in the book. Read them as a boundary report.

This checklist compresses the chapter into one reader habit:

visible output -> typed boundary -> invalid shortcut rejected

The ML idea is that a training example is not just an input. It is an input paired with the answer the model should have predicted. The category-theory idea is that the answer enters through a product boundary:

Distribution x TokenId -> Loss

The Rust idea is that the boundary is not only prose. It appears as a concrete type:

Product<Distribution, TokenId>

Why This Matters

This chapter is where the course stops being abstract.

The code implements a real, tiny version of the common language-model training path:

context token -> hidden vector -> next-token probabilities -> loss

The implementation is small, but the boundaries are real. Invalid token lookup returns OutOfRange, invalid matrix shape returns ShapeMismatch, empty logits return EmptyInput, invalid probabilities return InvalidProbability, and invalid loss returns InvalidLoss.

Errors are caught where the invalid data first becomes meaningful.

Core Mental Model

In Rust terms:

each ML operation implements Morphism<Input, Output>

In ML terms:

prediction is embedding + linear projection + softmax
loss is negative log probability of the target

In category-theory terms:

prediction is composition of arrows
loss consumes a product object
the direct and composed paths should commute

Checkpoint

Where should an out-of-range target token be caught?

Correct answer:

Inside CrossEntropy, because that is where the target is used to index the predicted distribution.

Where This Leaves Us

This chapter assembled the first complete tiny ML path. A token sequence becomes training examples, a token becomes a vector, a vector becomes logits, logits become probabilities, and a probability distribution plus a target token becomes loss.

The next chapter, Training as an Endomorphism, changes the question from “how do we evaluate one prediction?” to “how do repeated updates change the model state?” That is where training enters as an endomorphism.

Further Reading

The problem this section solves is transfer. If you only read the tiny Rust implementation, larger framework APIs may still look unrelated. If you only read a framework reference, the explicit typed boundaries in this chapter may feel unnecessarily small. Use the references to connect the two views without collapsing them.

Start from the local Rust evidence:

DatasetWindowing.apply : TokenSequence -> TrainingSet
Embedding.apply        : TokenId -> Vector
LinearToLogits.apply   : Vector -> Logits
Softmax.apply        : Logits -> Distribution
CrossEntropy.apply   : Distribution x TokenId -> Loss
average_loss          : Parameters x TrainingSet -> Loss

Then read the sources in this order:

The ML bridge is:

framework call
  -> raw scores plus target index
  -> probability assigned to the target
  -> loss

The category-theory bridge is:

Logits -> Distribution
Distribution x TokenId -> Loss

The first arrow is an ordinary morphism. The second is a product-input morphism because loss needs both the model’s prediction and the correct target token.

After reading one source, answer four questions:

1. Which local boundary did it clarify?
2. Which value is raw score, probability, target, or loss?
3. Which shortcut did the source use that the tiny Rust path expands?
4. Which command or test shows the local evidence?

For this chapter, the commands are:

cargo run --bin category_ml
cargo test ml::tests --lib

Checkpoint:

When reading an external loss API, can you name which part corresponds to
Logits -> Distribution and which part corresponds to Distribution x TokenId
-> Loss?

For terminology recovery, use:

• Glossary: logits, softmax, probability distribution, cross entropy
• References: softmax regression and linear classifiers

If a source does not help you point to one local boundary and one output or test signal, it has not transferred back into this chapter yet.

Practice After This Chapter

Use Exercise 3 to trace adjacent training pairs and Exercise 9 to connect this tiny implementation to a larger ML reference. The pair checks both local code understanding and source-backed transfer.

Retrieval Practice

Recall

Recover the path before explaining the calculations.

1. What morphism turns a TokenSequence into a TrainingSet?
2. Which three arrows turn a TokenId into a Distribution?
3. Which two objects are paired before CrossEntropy can produce Loss?

Explain

Use the target token to explain why the loss boundary needs a product object.

1. Why are Logits not the same object as Distribution?
2. Why does CrossEntropy use the probability at target.index() instead of the largest probability in the distribution?
3. Why does an out-of-range target error belong inside CrossEntropy?

Apply

Use the demo output and the numeric examples in this chapter.

1. Given TokenId -> Vector -> Logits -> Distribution, write the Rust type that must appear between Embedding and Softmax.
2. A distribution is [0.70, 0.20, 0.10] and the target token index is 1. Which probability should cross entropy use?
3. A token sequence is [4, 9, 2]. Which adjacent training pairs should DatasetWindowing produce?

Debug

For each invalid shortcut, name the missing boundary or wrong object:

Logits -> Loss
Distribution -> Loss
using the maximum probability instead of the target probability

A strong answer should mention the exact typed path:

Logits -> Distribution
Distribution x TokenId -> Loss

The point is not to memorize the formula. The point is to know which object owns the probability invariant and which object selects the target probability.

Training as an Endomorphism

The problem this chapter solves is:

A model is not only used for prediction. It must also be updated by training, and one update should produce the same kind of object it consumed.

The key shape is:

Parameters -> Parameters

This is an endomorphism.

In ordinary ML terms:

old parameters
  -> compute predictions
  -> compute loss gradients
  -> subtract learning-rate-scaled gradients
  -> new parameters

In category-theory terms:

A -> A

Because the input and output type are the same, the step can be repeated.

Reader orientation: Do not read this chapter as a full backpropagation engine. It is a small, explicit training step whose purpose is to make the shape Parameters -> Parameters visible and runnable.

Chapter Outcomes

By the end of this chapter, you should be able to:

• explain why one training step is modeled as Parameters -> Parameters,
• separate loss measurement from parameter update,
• compare the tiny TrainStep(dataset, learning_rate) boundary with a production optimizer loop that calls zero_grad, backward, and step.

What You Already Know

If you have seen gradient descent, you already know the informal movement: parameters are adjusted and then used again. If you know Rust, you already know that a function can return the same type it receives. This chapter names that shape precisely: a training step is an endomorphism on Parameters.

Update Trace Before Source

Before reading src/training.rs, keep this one-step trace in view. It separates loss measurement, gradient accumulation, the parameter update, and repetition.

One optimizer update has this shape:

Parameters
  -> predictions on TrainingSet
  -> gradients
  -> Parameters

Repeated optimization is not a different kind of arrow. It is the same arrow used again:

Parameters0 -> Parameters1 -> Parameters2 -> ... -> ParametersN

That is the chapter’s main separation. Parameters -> Loss measures the model. Parameters -> Parameters updates the model. The first is diagnostic. The second is repeatable training.

The local update rule in this chapter is the same first-order shape used in standard gradient descent:

parameter = parameter - learning_rate * average_gradient

In the Rust source, that appears as:

*value -= learning_rate * grad * batch_scale;

The chapter uses a full-batch step, so one call to TrainStep::apply reads all examples in the TrainingSet, averages their gradients with batch_scale, and returns a new Parameters value. The tests repeat that one endomorphism with apply_endomorphism_n_times.

Training Debugging Checklist

When training output looks surprising, separate four questions before changing the update rule:

The example prints both roles:

TrainStep : Parameters -> Parameters
Parameters x TrainingSet -> Loss

Those lines are deliberately different. Loss tells you how the current parameters perform on the dataset. It is evidence, not the next model state. TrainStep returns the next model state. That is what makes repetition legal:

Parameters0 -> Parameters1 -> ... -> Parameters80

Use this diagnostic when changing StepCount:

The category-theory lesson is stable even when the numeric loss changes in different ways:

the update remains Parameters -> Parameters
the measurement remains Parameters x TrainingSet -> Loss

Source Snapshot

This file implements one full-batch optimizer update.

use crate::category::Morphism;
use crate::domain::{LearningRate, Parameters, TrainingSet, Vector};
use crate::error::{CtError, CtResult};
use crate::ml::{LinearToLogits, Softmax};

/// One full-batch optimizer update.
///
/// Categorically, this is an endomorphism:
///
/// `Parameters -> Parameters`
#[derive(Debug, Clone)]
pub struct TrainStep {
    dataset: TrainingSet,
    learning_rate: LearningRate,
}

impl TrainStep {
    pub fn new(dataset: TrainingSet, learning_rate: LearningRate) -> Self {
        Self {
            dataset,
            learning_rate,
        }
    }
}

impl Morphism<Parameters, Parameters> for TrainStep {
    fn name(&self) -> &'static str {
        "train_step_endomorphism"
    }

    fn apply(&self, params: Parameters) -> CtResult<Parameters> {
        let vocab_size = params.vocab_size();
        let d_model = params.d_model();

        if vocab_size == 0 || d_model == 0 {
            return Err(CtError::EmptyInput("parameters"));
        }

        let mut grad_embedding = vec![vec![0.0; d_model]; params.embedding.len()];
        let mut grad_lm_head = vec![vec![0.0; vocab_size]; d_model];
        let mut grad_bias = vec![0.0; vocab_size];

        for example in self.dataset.examples() {
            let input_id = example.first().index();
            let target_id = example.second().index();

            if input_id >= params.embedding.len() {
                return Err(CtError::OutOfRange {
                    kind: "input token",
                    index: input_id,
                    limit: params.embedding.len(),
                });
            }

            if target_id >= vocab_size {
                return Err(CtError::OutOfRange {
                    kind: "target token",
                    index: target_id,
                    limit: vocab_size,
                });
            }

            let x = &params.embedding[input_id];
            let logits = LinearToLogits::from_parts(params.lm_head.clone(), params.bias.clone())
                .apply(Vector::new(x.clone()))?;
            let probs = Softmax.apply(logits)?;

            let mut dlogits = probs.as_slice().to_vec();
            dlogits[target_id] -= 1.0;

            for (vocab_id, dlogit) in dlogits.iter().copied().enumerate() {
                grad_bias[vocab_id] += dlogit;

                for (feature, x_feature) in x.iter().copied().enumerate() {
                    grad_lm_head[feature][vocab_id] += x_feature * dlogit;
                }
            }

            for (feature, grad_feature) in grad_embedding[input_id].iter_mut().enumerate() {
                let dx = params.lm_head[feature]
                    .iter()
                    .zip(dlogits.iter())
                    .map(|(weight, dlogit)| weight * dlogit)
                    .sum::<f32>();

                *grad_feature += dx;
            }
        }

        let batch_scale = 1.0 / self.dataset.len() as f32;
        let learning_rate = self.learning_rate.value();
        let mut updated = params.clone();

        for (row, grad_row) in updated.embedding.iter_mut().zip(grad_embedding.iter()) {
            for (value, grad) in row.iter_mut().zip(grad_row.iter()) {
                *value -= learning_rate * grad * batch_scale;
            }
        }

        for (row, grad_row) in updated.lm_head.iter_mut().zip(grad_lm_head.iter()) {
            for (value, grad) in row.iter_mut().zip(grad_row.iter()) {
                *value -= learning_rate * grad * batch_scale;
            }
        }

        for (bias, grad) in updated.bias.iter_mut().zip(grad_bias.iter()) {
            *bias -= learning_rate * grad * batch_scale;
        }

        Ok(updated)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::category::{StepCount, apply_endomorphism_n_times};
    use crate::domain::{ModelDimension, Product, TokenId, TokenSequence, TrainingSet, VocabSize};
    use crate::ml::{DatasetWindowing, average_loss};

    #[test]
    fn repeated_training_step_reduces_loss() -> CtResult<()> {
        let tokens = TokenSequence::from_indices([1, 2, 3, 4, 1, 2, 3, 4])?;
        let dataset = DatasetWindowing.apply(tokens)?;
        let params = Parameters::init(VocabSize::new(5)?, ModelDimension::new(4)?);
        let before = average_loss(&params, &dataset)?;
        let train_step = TrainStep::new(dataset.clone(), LearningRate::new(1.0)?);
        let trained = apply_endomorphism_n_times(&train_step, params, StepCount::new(80))?;
        let after = average_loss(&trained, &dataset)?;

        assert!(after.value() < before.value());
        Ok(())
    }

    #[test]
    fn one_training_step_preserves_parameter_shape() -> CtResult<()> {
        let tokens = TokenSequence::from_indices([1, 2, 3, 4])?;
        let dataset = DatasetWindowing.apply(tokens)?;
        let params = Parameters::init(VocabSize::new(5)?, ModelDimension::new(4)?);
        let train_step = TrainStep::new(dataset, LearningRate::new(0.1)?);

        let trained = train_step.apply(params.clone())?;

        assert_eq!(trained.vocab_size(), params.vocab_size());
        assert_eq!(trained.d_model(), params.d_model());
        Ok(())
    }

    #[test]
    fn training_rejects_target_outside_vocabulary() -> CtResult<()> {
        let dataset = TrainingSet::new([Product::new(TokenId::new(0), TokenId::new(9))])?;
        let params = Parameters::init(VocabSize::new(2)?, ModelDimension::new(2)?);
        let train_step = TrainStep::new(dataset, LearningRate::new(0.1)?);

        assert!(matches!(
            train_step.apply(params),
            Err(CtError::OutOfRange {
                kind: "target token",
                index: 9,
                limit: 2,
            })
        ));

        Ok(())
    }
}

The Whole File

src/training.rs defines:

TrainStep
TrainStep::new
impl Morphism<Parameters, Parameters> for TrainStep
unit test proving repeated training reduces loss

The whole file is about one idea:

training is a repeatable typed transformation of model state

Source Reading Bridge: One Step Has Four Responsibilities

The short list above names the file’s pieces, but it does not yet tell you how to read the main function. The central method is TrainStep::apply in src/training.rs. Read it as four responsibilities in order:

validate the current Parameters
run the current model on each training example
accumulate gradients for embedding, output weights, and bias
subtract a learning-rate-scaled average gradient to create new Parameters

The ML intuition is gradient descent. A loss signal does not replace the model. It tells each parameter which direction would reduce the current error on the training set. The code makes that visible by separating the diagnostic value from the state update:

average_loss(&params, &dataset) -> Loss
TrainStep::apply(params)       -> Parameters

That difference matters. If TrainStep::apply returned Loss, it could tell you how bad the current model is, but it could not be composed with itself for the next update.

The category-theory connection is the same boundary in a shorter form:

TrainStep(dataset, learning_rate) : Parameters -> Parameters

The dataset and learning rate configure which update arrow you have. The gradient buffers are internal machinery used while building the output object; they are not the object being returned by the morphism.

Checkpoint:

If `TrainStep::apply` returned `Loss` instead of `Parameters`, what ability
would `apply_endomorphism_n_times` lose?

Production Optimizer Boundary

Production frameworks usually split the training loop across model parameters, stored gradients, an optimizer object, and an optimizer step. PyTorch’s torch.optim documentation describes optimizers as objects that hold current state and update parameters from computed gradients. The common loop shape is:

optimizer.zero_grad()
output = model(input)
loss = loss_fn(output, target)
loss.backward()
optimizer.step()

This book compresses the same teaching shape into one explicit Rust morphism:

TrainStep(dataset, learning_rate) : Parameters -> Parameters

The same training boundary as a rendered math view:

[ \begin{array}{ccccc} \mathrm{Parameters}_t & \xrightarrow{\mathrm{average_loss}(-,\mathrm{TrainingSet})} & \mathrm{Loss}t & \xrightarrow{\mathrm{local\ gradients}} & \nabla_t \ &&&& \downarrow \mathrm{apply\ learning\ rate} \ \mathrm{Parameters}{t+1} & \xleftarrow{\mathrm{TrainStep(dataset, learning_rate)}} & \mathrm{Parameters}_t && \end{array} ]

How to read this diagram:

• the upper path measures how wrong the current parameters are,
• the gradient path explains what should change,
• the bottom arrow is the typed update that returns the next full Parameters object,
• only the bottom arrow has the endomorphism shape Parameters -> Parameters.

The tiny Rust boundary is smaller than a production optimizer. It does not model momentum, parameter groups, optimizer state dictionaries, closures, schedulers, mixed precision, or distributed training. It keeps one full-batch gradient update inspectable.

When you return to a framework, the useful transfer question is:

which object owns the update state, and which call turns current parameters
into next parameters?

Framework-To-Rust Responsibility Ledger

If you already know the framework loop, use this ledger before reading TrainStep::apply. It prevents two common mistakes: treating the tiny Rust code as a hidden framework clone, or treating framework calls as unrelated magic.

The useful habit is to translate a framework call into a responsibility, then ask where that responsibility appears in the local Rust code. If no local handle exists, say so explicitly.

Framework-to-Rust audit card:

framework cue:
responsibility:
local Rust handle:
returned object:
category boundary:
safe non-claim:

Example:

framework cue: optimizer.step()
responsibility: apply gradients to parameters
local Rust handle: *value -= learning_rate * grad * batch_scale;
returned object: Parameters
category boundary: TrainStep(dataset, learning_rate) : Parameters -> Parameters
safe non-claim: this is one full-batch teaching update, not a production optimizer

Source-Backed Precision Rules

This chapter uses external sources to keep the tiny update honest. Each source supports a limited claim; these citations are not proof that this crate is a production optimizer or a full automatic-differentiation engine.

The transfer pattern is:

source claim -> local typed boundary -> validation command or test

For this chapter, that means reading cargo run --example 03_training_endomorphism and the src/training.rs tests as evidence for the tiny Parameters -> Parameters boundary, not as evidence for every production training system.

Worked Example: Repeating One Update

The smallest first-principles version of a repeated update is a number being moved a little at a time:

#![allow(unused)]
fn main() {
fn step_toward_zero(value: f32, learning_rate: f32) -> f32 {
    value - learning_rate * value
}

let once = step_toward_zero(10.0, 0.1);
let twice = step_toward_zero(once, 0.1);

assert!(twice < once);
}

The real training code applies the same repeatable-update idea to Parameters, not to one scalar. The output stays the same kind of object as the input, so the update can be run again.

Self-Check

Before reading the full training step, explain why Parameters -> Parameters is repeatable but Parameters -> Loss is not.

One Step Before Many Steps

Training becomes easier to reason about if you separate two ideas.

One training step has the shape:

Parameters -> Parameters

It reads the dataset, computes predictions, accumulates gradients, subtracts a learning-rate-scaled average gradient, and returns updated model state.

Repeated training is just iteration of that same shape:

Parameters0 -> Parameters1 -> Parameters2 -> ... -> ParametersN

The chapter’s category-theory word for the one-step shape is endomorphism. The ML word for the update rule is gradient descent. The Rust evidence is the trait implementation:

impl Morphism<Parameters, Parameters> for TrainStep

The tests in src/training.rs protect the learner-visible claims: one training step preserves the parameter shape, out-of-range targets fail with a typed error, and repeated steps reduce loss on the tiny dataset.

TrainStep

The problem this block solves is:

A training update needs a dataset and a learning rate, and those values should travel together as one configured operation.

The block:

/// One full-batch optimizer update.
///
/// Categorically, this is an endomorphism:
///
/// `Parameters -> Parameters`
#[derive(Debug, Clone)]
pub struct TrainStep {
    dataset: TrainingSet,
    learning_rate: LearningRate,
}

Rust Syntax

This is a named-field struct.

It stores:

dataset: TrainingSet
learning_rate: LearningRate

Both fields are private.

That means callers cannot directly replace the dataset or learning rate after construction.

The derived traits mean:

Debug -> can be printed for debugging
Clone -> can be explicitly duplicated

TrainingSet is already non-empty.

LearningRate is already finite and positive.

So TrainStep stores validated inputs.

ML Concept

A training step needs:

• examples to learn from
• a step size for parameter updates

The dataset gives the input-target pairs.

The learning rate controls how far the update moves.

Category-Theory Concept

TrainStep is the value that will implement:

Parameters -> Parameters

That makes it an endomorphism on the object Parameters.

TrainStep::new

The problem this block solves is:

Construct a configured training step from already validated pieces.

The block:

impl TrainStep {
    pub fn new(dataset: TrainingSet, learning_rate: LearningRate) -> Self {
        Self {
            dataset,
            learning_rate,
        }
    }
}

Rust Syntax

impl TrainStep defines methods for TrainStep.

The constructor takes ownership of:

dataset
learning_rate

and stores them.

It returns Self, not CtResult<Self>, because the inputs are already validated domain objects.

No extra validation is needed here.

ML Concept

This is like configuring an optimizer step:

use this dataset
use this learning rate

The actual update happens later in apply.

Category-Theory Concept

The constructor chooses one specific endomorphism from a family:

TrainStep(dataset, learning_rate) : Parameters -> Parameters

Different datasets or learning rates create different update morphisms.

Morphism Implementation

The problem this block solves is:

Make TrainStep a real typed arrow from model parameters back to model parameters.

The header:

impl Morphism<Parameters, Parameters> for TrainStep {

Rust Syntax

This says:

TrainStep implements Morphism<Input = Parameters, Output = Parameters>

So the apply method must have this effective shape:

Parameters -> CtResult<Parameters>

The name method:

fn name(&self) -> &'static str {
    "train_step_endomorphism"
}

returns a static label for the transformation.

ML Concept

The input Parameters are the current model weights.

The output Parameters are the updated weights after one full-batch step.

Category-Theory Concept

Because the input and output object are the same, TrainStep is an endomorphism.

That is what lets this work:

Parameters0 -> Parameters1 -> Parameters2 -> ... -> ParametersN

apply: Shape Checks

The problem this block solves is:

Before computing gradients, verify that the parameter object has usable dimensions.

The block:

let vocab_size = params.vocab_size();
let d_model = params.d_model();

if vocab_size == 0 || d_model == 0 {
    return Err(CtError::EmptyInput("parameters"));
}

Rust Syntax

The code asks the parameter object for two dimensions.

Then it rejects zero-sized parameters.

This uses an explicit error instead of panicking.

ML Concept

Training cannot run if:

• there are zero possible vocabulary outputs
• hidden vectors have zero width

Those shapes would make the gradient arrays meaningless.

Category-Theory Concept

The endomorphism is only defined on valid Parameters.

Invalid parameter state is rejected before the morphism performs the update.

Gradient Buffers

The problem this block solves is:

Accumulate gradients for every trainable parameter before applying the update.

The block:

let mut grad_embedding = vec![vec![0.0; d_model]; params.embedding.len()];
let mut grad_lm_head = vec![vec![0.0; vocab_size]; d_model];
let mut grad_bias = vec![0.0; vocab_size];

Rust Syntax

These are mutable matrices and vectors initialized to zero.

Their shapes mirror the trainable parameters:

grad_embedding: same row count as embedding, d_model columns
grad_lm_head:   d_model x vocab_size
grad_bias:      vocab_size

ML Concept

Gradients accumulate how each parameter should change to reduce loss.

The code uses full-batch training: it processes every example, accumulates all gradients, averages them, then updates once.

Category-Theory Concept

The gradient buffers are not the endomorphism itself.

They are internal machinery used to construct the output object in:

Parameters -> Parameters

Example Loop

The problem this block solves is:

For each training example, compute the local contribution to the parameter gradients.

The loop begins:

for example in self.dataset.examples() {
    let input_id = example.first().index();
    let target_id = example.second().index();
    ...
}

Rust Syntax

self.dataset.examples() returns a slice of TrainingExample.

Each example is a Product<TokenId, TokenId>.

So:

example.first()

is the input token.

example.second()

is the target token.

The code extracts raw indices because matrix indexing needs usize.

ML Concept

Each example says:

given input token, predict target token

The training loop calculates how wrong the current model is for that example.

Category-Theory Concept

The example is an element of:

TokenId x TokenId

The training morphism consumes many such product values while building the parameter update.

Token Bounds Checks

The problem this block solves is:

Training examples must refer to tokens that exist in the current parameter shapes.

The checks:

if input_id >= params.embedding.len() {
    return Err(CtError::OutOfRange { ... });
}

if target_id >= vocab_size {
    return Err(CtError::OutOfRange { ... });
}

Rust Syntax

These are ordinary bounds checks with typed errors.

They prevent invalid indexing into:

• the embedding table
• the vocabulary-sized output vector

ML Concept

An input token must have an embedding row.

A target token must be one of the possible prediction classes.

If either token is outside the model vocabulary, training cannot continue.

Category-Theory Concept

The example must belong to the finite token object that the parameters are currently modeling.

This check keeps the training morphism inside the intended domain.

Forward Pass Inside Training

The problem this block solves is:

To compute gradients, the training step first needs the current prediction.

The block:

let x = &params.embedding[input_id];
let logits = LinearToLogits::from_parts(params.lm_head.clone(), params.bias.clone())
    .apply(Vector::new(x.clone()))?;
let probs = Softmax.apply(logits)?;

Rust Syntax

x borrows the embedding row for the input token.

LinearToLogits::from_parts(...) builds a linear projection from the current weights.

Vector::new(x.clone()) wraps the embedding row as a Vector.

Then:

Vector -> Logits -> Distribution

runs through the same morphism interface as prediction.

ML Concept

This computes the model’s current predicted distribution for one input token.

The gradient depends on the difference between that distribution and the true target.

Category-Theory Concept

Even inside training, prediction is still a composed path:

TokenId -> Vector -> Logits -> Distribution

Training uses that path as part of a larger endomorphism:

Parameters -> Parameters

Logit Gradient

The problem this block solves is:

For softmax plus cross entropy, the gradient with respect to logits is predicted probability minus one-hot target.

The block:

let mut dlogits = probs.as_slice().to_vec();
dlogits[target_id] -= 1.0;

Rust Syntax

The probabilities are copied into a mutable vector.

Then the target class is adjusted by subtracting 1.0.

If:

probs = [0.70, 0.20, 0.10]
target = 1

then:

dlogits = [0.70, -0.80, 0.10]

ML Concept

This is the standard simplified gradient for softmax cross entropy.

It says:

• decrease the scores that are too high
• increase the target score if it was too low

Category-Theory Concept

This is local derivative information for one part of the composed prediction path.

The next loops compose that local derivative back into parameter gradients.

Worked Example: Why Subtracting A Negative Gradient Increases The Target

The update rule can feel backwards the first time you see it. The code subtracts gradients:

parameter = parameter - learning_rate * gradient

So how can training increase the target score?

Use the same three-class example:

probs  = [0.70, 0.20, 0.10]
target = 1

After the target correction:

dlogits = [0.70, -0.80, 0.10]

Now look only at the bias update with learning rate 0.1 and one example:

bias[0] = 0.0 - 0.1 *  0.70 = -0.07
bias[1] = 0.0 - 0.1 * -0.80 =  0.08
bias[2] = 0.0 - 0.1 *  0.10 = -0.01

The non-target classes had positive gradients, so subtracting them lowers their biases. The target class had a negative gradient, so subtracting it raises the target bias.

That is the local version of gradient descent: move parameters in the direction that lowers loss. In this tiny classifier, the direction says “make the target logit larger and make the overconfident non-target logits smaller.”

The Rust path is:

dlogits
  -> grad_bias
  -> bias -= learning_rate * grad * batch_scale

For output weights, the same sign passes through x_feature * dlogit. For the embedding row, the sign passes backward through the output weights. The full training step is bigger, but the sign logic starts here.

Output-Head And Bias Gradients

The problem this block solves is:

Convert the logit gradient into gradients for the output matrix and bias.

The core loop:

for (vocab_id, dlogit) in dlogits.iter().copied().enumerate() {
    grad_bias[vocab_id] += dlogit;

    for (feature, x_feature) in x.iter().copied().enumerate() {
        grad_lm_head[feature][vocab_id] += x_feature * dlogit;
    }
}

Rust Syntax

The outer loop visits every vocabulary output.

The inner loop visits every feature of the input vector.

The bias gradient is just the logit gradient.

The weight gradient is:

input feature * output gradient

ML Concept

For a linear layer:

logits = xW + b

the gradient of a weight is:

input activation * output gradient

This is the same pattern used in larger neural networks.

Category-Theory Concept

This is the local backward map for the affine projection stage.

It translates changes needed at the output object Logits into changes in the parameter object.

Embedding Gradient

The problem this block solves is:

Move the output error backward through the language-model head to the input embedding row.

The block:

for (feature, grad_feature) in grad_embedding[input_id].iter_mut().enumerate() {
    let dx = params.lm_head[feature]
        .iter()
        .zip(dlogits.iter())
        .map(|(weight, dlogit)| weight * dlogit)
        .sum::<f32>();

    *grad_feature += dx;
}

Rust Syntax

The loop mutates the gradient row for the input token.

For each feature, it pairs:

weights from that feature to every vocab output
dlogits for every vocab output

Then it sums:

weight * dlogit

ML Concept

This is backpropagation through the linear head.

It tells the embedding row how it should change so the future logits improve.

Only the row for the current input token receives an embedding gradient.

Category-Theory Concept

This is another local backward map.

The training endomorphism is built by composing local derivative information from output back toward parameters.

Parameter Update

The problem this block solves is:

Turn accumulated gradients into new parameters.

The update can be read as a loop around the same object:

Parameters_t
    |
    | prediction on TrainingSet
    v
Average Loss
    |
    | local gradients
    v
Gradient Accumulators
    |
    | subtract learning_rate * average_gradient
    v
Parameters_{t+1}

The diagram has one important boundary: the first and last objects are both Parameters. Everything in the middle explains how one state becomes the next state.

The code computes:

let batch_scale = 1.0 / self.dataset.len() as f32;
let learning_rate = self.learning_rate.value();
let mut updated = params.clone();

Then it subtracts scaled gradients from every parameter.

Rust Syntax

batch_scale averages the accumulated gradients.

learning_rate extracts the raw scalar.

updated = params.clone() creates the output parameter object.

The following loops mutate updated, not the original params.

Finally:

Ok(updated)

returns the new model state.

ML Concept

The update rule is:

parameter_new = parameter_old - learning_rate * average_gradient

This is gradient descent.

Category-Theory Concept

The final result has the same object type as the input:

Parameters -> Parameters

That completes the endomorphism.

Regression Test

The problem this block solves is:

Prove the learner-visible promise that repeated training reduces loss on the tiny dataset.

The test:

#[test]
fn repeated_training_step_reduces_loss() -> CtResult<()> {
    let tokens = TokenSequence::from_indices([1, 2, 3, 4, 1, 2, 3, 4])?;
    let dataset = DatasetWindowing.apply(tokens)?;
    let params = Parameters::init(VocabSize::new(5)?, ModelDimension::new(4)?);
    let before = average_loss(&params, &dataset)?;
    let train_step = TrainStep::new(dataset.clone(), LearningRate::new(1.0)?);
    let trained = apply_endomorphism_n_times(&train_step, params, StepCount::new(80))?;
    let after = average_loss(&trained, &dataset)?;

    assert!(after.value() < before.value());
    Ok(())
}

Rust Syntax

The test returns CtResult<()>, so it can use ?.

It builds a token sequence, turns it into a training set, initializes parameters, and configures a training step.

Then it applies the endomorphism 80 times and checks the loss decreased.

ML Concept

This is not a benchmark.

It is a sanity check:

training should make the tiny model better on the tiny data

Category-Theory Concept

The test exercises repeated endomorphism application:

Parameters0 -> Parameters1 -> ... -> Parameters80

The companion tests check the one-step contract too. One update keeps the same vocabulary size and model dimension, and invalid targets are rejected before an unsafe index can enter gradient accumulation.

Run The Example

use category_theory_transformer_rs::{
    CtResult, DatasetWindowing, LearningRate, ModelDimension, Morphism, Parameters, StepCount,
    TokenSequence, TrainStep, VocabSize, apply_endomorphism_n_times, average_loss,
};

fn main() -> CtResult<()> {
    let tokens = TokenSequence::from_indices([1, 2, 3, 4, 1, 2, 3, 4])?;
    let dataset = DatasetWindowing.apply(tokens)?;
    let params = Parameters::init(VocabSize::new(5)?, ModelDimension::new(4)?);

    let before = average_loss(&params, &dataset)?;
    let train_step = TrainStep::new(dataset.clone(), LearningRate::new(1.0)?);
    let trained = apply_endomorphism_n_times(&train_step, params, StepCount::new(80))?;
    let after = average_loss(&trained, &dataset)?;

    println!("loss before: {:.6}", before.value());
    println!("loss after:  {:.6}", after.value());
    println!();
    println!("Typed transformation:");
    println!("TrainStep : Parameters -> Parameters");
    println!("Repeated endomorphism:");
    println!("Parameters0 -> Parameters1 -> ... -> Parameters80");
    println!("Measurement:");
    println!("Parameters x TrainingSet -> Loss");

    Ok(())
}

Run:

cargo run --example 03_training_endomorphism

Expected pattern:

loss before: ...
loss after:  ...
Typed transformation:
TrainStep : Parameters -> Parameters
Repeated endomorphism:
Parameters0 -> Parameters1 -> ... -> Parameters80
Measurement:
Parameters x TrainingSet -> Loss

The second number should be smaller.

Example Output Transfer Checklist

The example output is deliberately small. It gives you two measurements and then names the update shape that produced the second measurement.

Use the printed lines this way:

This is the same separation used in standard gradient-descent explanations: compute a loss and its gradient, then update the parameters in the negative gradient direction. The measurement tells you whether the model improved. The endomorphism is the repeatable state transition that makes training possible.

If you only remember one distinction from this chapter, remember this:

Parameters -> Loss        measures
Parameters -> Parameters  trains

Core Mental Model

In Rust terms:

TrainStep implements Morphism<Parameters, Parameters>

In ML terms:

one full-batch gradient descent update

In category-theory terms:

an endomorphism that can be iterated

Checkpoint

Why is it useful that training returns Parameters instead of a raw matrix?

A strong answer:

Because the output can immediately be used as the input to the next TrainStep, preserving the Parameters -> Parameters endomorphism shape.

Where This Leaves Us

This chapter turned training into a repeatable typed transformation. The model state enters as Parameters, the training step computes gradients from the tiny dataset, and the updated model state leaves as Parameters again.

The next chapter, Functors, Naturality, Monoids, and Chain Rule, steps back from the training loop and names reusable structures that appear across the whole course: mapping inside wrappers, changing wrapper shapes consistently, combining traces, and composing local derivative rules.

Further Reading

The problem this section solves is transfer. A framework training loop compresses several responsibilities into familiar calls. This chapter expands those responsibilities so the reader can see which object is measured, which object is updated, and why the update can repeat.

Start from the local Rust evidence:

average_loss(&params, &dataset) -> Loss
TrainStep::apply(params)       -> Parameters
apply_endomorphism_n_times     -> Parameters

Then compare that with a framework loop:

optimizer.zero_grad()
loss = loss_fn(model(input), target)
loss.backward()
optimizer.step()

The framework loop is compact because the model, gradient buffers, optimizer state, parameter groups, and update rule live behind framework objects. The teaching path is expanded because the reader needs to separate four ideas:

Read the sources in this order:

1. D2L Gradient Descent: use it for the update direction and learning-rate intuition.
2. D2L Backpropagation and Computational Graphs: use it for the forward-then-reverse gradient story.
3. Automatic differentiation in machine learning: a survey: use it to keep “automatic differentiation”, “backpropagation”, “symbolic differentiation”, and “finite differences” separate.
4. PyTorch torch.optim: use it to recognize zero_grad, backward, and step as production boundaries.
5. PyTorch Autograd mechanics: use it to contrast graph-recording autograd with this chapter’s hand-written gradient path.
6. Backprop as Functor: use it only as advanced context for compositional update rules.

The transfer bridge is:

production loop
  -> measure current model
  -> compute gradients
  -> update optimizer/model state
  -> repeat

The category-theory bridge is smaller and stricter:

Parameters x TrainingSet -> Loss
TrainStep(dataset, learning_rate) : Parameters -> Parameters

The first boundary measures. The second boundary updates. Only the second one is the endomorphism that can be repeated by apply_endomorphism_n_times.

Draw the distinction like this:

[ \begin{array}{rcl} \mathrm{measure} &:& \mathrm{Parameters} \times \mathrm{TrainingSet} \to \mathrm{Loss} \ \mathrm{update} &:& \mathrm{Parameters} \to \mathrm{Parameters} \end{array} ]

If a diagram makes the measurement arrow return Parameters, or makes the update arrow return only Loss, the training story has changed meaning.

Checkpoint:

When reading an external optimizer or autograd reference, can you name which
part corresponds to Parameters x TrainingSet -> Loss and which part
corresponds to TrainStep(dataset, learning_rate) : Parameters -> Parameters?

These pages connect the tiny update to the surrounding vocabulary and source material:

• Glossary: endomorphism, parameters, learning rate, gradient
• References: gradient descent, computational graphs, backpropagation, and compositional learning

Practice After This Chapter

Use Exercise 5 to change the number of repeated training steps. The goal is not to tune a real model. The goal is to see why a Parameters -> Parameters update can be applied again and again.

Retrieval Practice

Recall

Recover the update shape before explaining the gradient.

1. What makes TrainStep an endomorphism?
2. Which line changes the probability vector into the logit gradient for the target class?
3. Which helper repeats the same Parameters -> Parameters step many times?

Explain

Separate measurement from update.

1. Why is Parameters -> Loss useful for evaluation but not itself a training endomorphism?
2. Why does the training code validate input and target token bounds before accumulating gradients?
3. Why does subtracting a negative target gradient increase the target bias or target weight?

Apply

Use the sign trace from this chapter.

1. Suppose:

   probs  = [0.65, 0.25, 0.10]
   target = 2
   learning_rate = 0.1
   batch_scale = 1.0
   bias starts at [0.0, 0.0, 0.0]
   

   What is dlogits, and what is the updated bias?

2. If you changed StepCount::new(80) to StepCount::new(1), what would you expect to happen to the loss, and why?

3. If the dataset has four examples, why does the code multiply each accumulated gradient by batch_scale = 0.25 before updating parameters?

Debug

For each invalid shortcut, name the broken shape or missing state:

returning Loss from TrainStep.apply
updating only lm_head and discarding embedding and bias
repeating Parameters -> Loss as if it were Parameters -> Parameters
skipping token bounds checks before indexing gradient buffers

A strong answer should mention the outer loop shape:

Parameters_t -> Parameters_{t+1}

The loss and gradients explain how the update is computed. They are not the object that must be returned from the training step.

Functors, Naturality, Monoids, and Chain Rule

The problem this chapter solves is:

After seeing individual ML arrows, you need names for common structures that appear across many systems: mapping inside containers, converting containers consistently, combining traces, and composing local derivatives.

The previous chapters were mostly about one pipeline. This chapter zooms out from that pipeline and asks which shapes keep appearing even when the concrete data changes. Once you can see those repeated shapes, the category-theory vocabulary stops feeling like a separate subject. It becomes a set of names for ordinary engineering moves.

This chapter covers four patterns:

Functor
NaturalTransformation
Monoid
Chain rule

They are not separate from the ML pipeline.

They explain patterns you already saw:

• mapping over many examples
• converting one wrapper shape to another
• combining pipeline traces
• composing gradients through layers

Reader orientation: This chapter is more abstract than the previous ones. Read each section in this order: first the Rust mechanism, then the ML use, then the category-theory name. The names are not decoration; they are compression for patterns that appear repeatedly in real model code.

Chapter Outcomes

By the end of this chapter, you should be able to:

• trace a functor, natural transformation, monoid, and chain-rule example through concrete Rust code,
• explain which two paths must agree in the naturality and monoid examples,
• distinguish the tiny MulOp::backward local derivative boundary from a full automatic-differentiation engine.

What You Already Know

If you have mapped over a Vec, handled an Option, appended logs, or applied the chain rule in calculus, you already know informal versions of this chapter. The new work is to name those repeated shapes and connect them to the same typed pipeline discipline used earlier.

Trace Both Paths Before The Names

Before reading src/structure.rs or src/calculus.rs, trace the paths first. The formal names in this chapter should arrive after the reader can see what has to agree.

The first two-path check is about converting a vector into an optional first item.

Path 1:
Vec<A> --map f--> Vec<B> --first--> Option<B>

Path 2:
Vec<A> --first--> Option<A> --map f--> Option<B>

Both paths should return the same Option<B>. The code names that agreement with naturality_square_holds_for_first_option.

The second two-path check is about grouping a trace.

Path 1:
(embedding <> linear) <> softmax

Path 2:
embedding <> (linear <> softmax)

Both paths should produce the same PipelineTrace. The empty trace should also leave any real trace unchanged.

The chain-rule check has a different shape: one path goes forward through the computation, and one path carries derivative information backward.

Forward:
x, y -> z = x * y -> L

Backward:
dL/dz -> dL/dx and dL/dy

For z = x * y, the local derivatives are:

dz/dx = y
dz/dy = x

So the backward path scales those local derivatives by the upstream gradient:

dL/dx = dL/dz * y
dL/dy = dL/dz * x

That is the useful mental model before any abstraction:

same result by two structural paths
or
same gradient signal carried through local paths

Now the names can be useful. A functor preserves wrapper shape while mapping inside it. A natural transformation makes the two wrapper-conversion paths agree. A monoid lets trace grouping stop mattering. The chain rule composes local derivative information.

Source Snapshots

src/structure.rs covers functors, natural transformations, and monoids.

/// A minimal functor interface for this tutorial.
pub trait Functor<A, B> {
    type WrappedA;
    type WrappedB;

    fn fmap<F>(wrapped: Self::WrappedA, f: F) -> Self::WrappedB
    where
        F: Fn(A) -> B;
}

pub struct VecFunctor;

impl<A, B> Functor<A, B> for VecFunctor {
    type WrappedA = Vec<A>;
    type WrappedB = Vec<B>;

    fn fmap<F>(wrapped: Vec<A>, f: F) -> Vec<B>
    where
        F: Fn(A) -> B,
    {
        wrapped.into_iter().map(f).collect()
    }
}

pub struct OptionFunctor;

impl<A, B> Functor<A, B> for OptionFunctor {
    type WrappedA = Option<A>;
    type WrappedB = Option<B>;

    fn fmap<F>(wrapped: Option<A>, f: F) -> Option<B>
    where
        F: Fn(A) -> B,
    {
        wrapped.map(f)
    }
}

/// A structure-preserving conversion between wrappers.
pub trait NaturalTransformation<A> {
    type From;
    type To;

    fn transform(from: Self::From) -> Self::To;
}

/// Natural transformation from `Vec<A>` to `Option<A>` by taking the first item.
pub struct VecToFirstOption;

impl<A> NaturalTransformation<A> for VecToFirstOption {
    type From = Vec<A>;
    type To = Option<A>;

    fn transform(from: Vec<A>) -> Option<A> {
        from.into_iter().next()
    }
}

pub fn naturality_square_holds_for_first_option() -> bool {
    let xs = vec![1, 2, 3];
    let f = |x| x * 10;

    let path_top_then_right = VecToFirstOption::transform(VecFunctor::fmap(xs.clone(), f));
    let path_left_then_bottom = OptionFunctor::fmap(VecToFirstOption::transform(xs), f);

    path_top_then_right == path_left_then_bottom
}

pub trait Monoid: Sized {
    fn empty() -> Self;
    fn combine(&self, other: &Self) -> Self;
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct TraceStep(&'static str);

impl TraceStep {
    pub fn new(name: &'static str) -> Self {
        Self(name)
    }

    pub fn name(&self) -> &'static str {
        self.0
    }
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PipelineTrace(Vec<TraceStep>);

impl PipelineTrace {
    pub fn from_steps(steps: impl IntoIterator<Item = TraceStep>) -> Self {
        Self(steps.into_iter().collect())
    }

    pub fn names(&self) -> Vec<&'static str> {
        self.0.iter().map(TraceStep::name).collect()
    }
}

impl Monoid for PipelineTrace {
    fn empty() -> Self {
        PipelineTrace(vec![])
    }

    fn combine(&self, other: &Self) -> Self {
        let mut combined = self.0.clone();
        combined.extend_from_slice(&other.0);
        PipelineTrace(combined)
    }
}

pub fn monoid_laws_hold_for_pipeline_trace() -> bool {
    let a = PipelineTrace::from_steps(vec![TraceStep::new("embedding")]);
    let b = PipelineTrace::from_steps(vec![TraceStep::new("linear")]);
    let c = PipelineTrace::from_steps(vec![TraceStep::new("softmax")]);
    let identity = PipelineTrace::empty();

    let left_identity = identity.combine(&a) == a;
    let right_identity = a.combine(&identity) == a;
    let associativity = a.combine(&b).combine(&c) == a.combine(&b.combine(&c));

    left_identity && right_identity && associativity
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn vec_functor_preserves_identity_for_values() {
        let values = vec![1, 2, 3];

        assert_eq!(VecFunctor::fmap(values.clone(), |value| value), values);
    }

    #[test]
    fn vec_functor_preserves_composition_for_values() {
        let values = vec![1, 2, 3];
        let add_one = |value| value + 1;
        let double = |value| value * 2;

        let map_then_map = VecFunctor::fmap(VecFunctor::fmap(values.clone(), add_one), double);
        let map_composed = VecFunctor::fmap(values, |value| double(add_one(value)));

        assert_eq!(map_then_map, map_composed);
    }

    #[test]
    fn option_functor_preserves_absence() {
        let missing = OptionFunctor::fmap(None::<i32>, |value| value * 10);

        assert_eq!(missing, None);
    }

    #[test]
    fn naturality_square_commutes() {
        assert!(naturality_square_holds_for_first_option());
    }

    #[test]
    fn pipeline_trace_obeys_monoid_laws() {
        assert!(monoid_laws_hold_for_pipeline_trace());
    }
}

src/calculus.rs keeps the chain-rule example deliberately small.

use crate::error::{CtError, CtResult};

#[derive(Debug, Clone, Copy, PartialEq)]
pub struct Scalar(f32);

impl Scalar {
    pub fn new(value: f32) -> CtResult<Self> {
        if !value.is_finite() {
            return Err(CtError::InvalidLoss(value));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> f32 {
        self.0
    }
}

#[derive(Debug, Clone, Copy, PartialEq)]
pub struct LocalGradient(f32);

impl LocalGradient {
    pub fn new(value: f32) -> CtResult<Self> {
        if !value.is_finite() {
            return Err(CtError::InvalidLoss(value));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> f32 {
        self.0
    }
}

#[derive(Debug, Clone, Copy)]
pub struct MulOp;

impl MulOp {
    pub fn forward(&self, x: Scalar, y: Scalar) -> CtResult<Scalar> {
        Scalar::new(x.value() * y.value())
    }

    /// Given upstream gradient dL/dz, return `(dL/dx, dL/dy)` for `z = x * y`.
    pub fn backward(
        &self,
        x: Scalar,
        y: Scalar,
        upstream: LocalGradient,
    ) -> CtResult<(LocalGradient, LocalGradient)> {
        let dz_dx = y.value();
        let dz_dy = x.value();

        Ok((
            LocalGradient::new(upstream.value() * dz_dx)?,
            LocalGradient::new(upstream.value() * dz_dy)?,
        ))
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn multiply_backward_returns_local_chain_rule_gradients() -> CtResult<()> {
        let mul = MulOp;
        let x = Scalar::new(2.0)?;
        let y = Scalar::new(3.0)?;
        let upstream = LocalGradient::new(1.0)?;
        let (dl_dx, dl_dy) = mul.backward(x, y, upstream)?;

        assert_eq!(dl_dx.value(), 3.0);
        assert_eq!(dl_dy.value(), 2.0);
        Ok(())
    }

    #[test]
    fn multiply_backward_scales_with_upstream_gradient() -> CtResult<()> {
        let mul = MulOp;
        let x = Scalar::new(2.0)?;
        let y = Scalar::new(3.0)?;
        let upstream = LocalGradient::new(4.0)?;
        let (dl_dx, dl_dy) = mul.backward(x, y, upstream)?;

        assert_eq!(dl_dx.value(), 12.0);
        assert_eq!(dl_dy.value(), 8.0);
        Ok(())
    }
}

The Whole Structure File

src/structure.rs defines:

Functor<A, B>
VecFunctor
OptionFunctor
NaturalTransformation<A>
VecToFirstOption
Monoid
TraceStep
PipelineTrace

Each block gives a Rust handle to one abstract pattern.

Worked Example: Mapping Over A Vector

Before reading the traits, start with the plain Rust operation that motivates them. Mapping over a vector means taking each item out, applying a function, and collecting the new values into another vector:

#![allow(unused)]
fn main() {
let values = vec![1, 2, 3];
let doubled: Vec<i32> = values.into_iter().map(|x| x * 2).collect();

assert_eq!(doubled, vec![2, 4, 6]);
}

There is no category theory hidden in that snippet. It is just ordinary Rust. The category-theory word functor appears when we notice the reusable shape: the code changes the contents while preserving the surrounding container.

Self-Check

Before reading the Functor trait, explain what stayed the same and what changed in the Vec mapping example.

Four Patterns, Four Questions

This chapter is easier if you do not try to memorize four category-theory words at once. Treat each word as an answer to one engineering question.

The chapter’s tests mirror that progression. They check small functor-law examples, a naturality square, monoid laws for traces, and the local chain-rule gradient for multiplication. The tests are not a full mathematical proof of all possible cases. They are executable anchors for the patterns the book is teaching.

The law and boundary map is:

Use this table as the chapter’s visual index. If a later section feels abstract, return to the row and ask which Rust test makes the law visible.

What Is A Law, A Test, Or An Analogy?

This chapter uses mathematical names, executable tests, and engineering analogies. Those are related, but they are not the same kind of evidence.

The rule for this book is conservative: a law word should point to a concrete Rust test, and an analogy should be named as an analogy. That keeps the chapter useful without pretending that a few tests prove all of category theory.

Source-Backed Precision Rules

This chapter uses external sources to keep the structure vocabulary precise. Each source supports a limited claim; these citations are not proof that this crate implements a full category-theory library or a production automatic-differentiation engine.

The transfer pattern is:

source claim -> local typed boundary -> validation command or test

For this chapter, that means reading cargo run --example 04_structure_and_calculus, cargo test structure::tests, and cargo test calculus::tests as evidence for these small law-shaped examples, not as evidence that every functor, natural transformation, monoid, or differentiable program has been modeled.

Functor<A, B>

The problem this block solves is:

The code needs a name for “apply a function inside a wrapper while keeping the wrapper shape.”

First principle: a trait is a contract. It says, “any type that implements this trait must provide these associated types and this method.” Here the contract is small on purpose. It does not try to model every possible functor in mathematics; it gives this tutorial one precise place to talk about fmap.

The block:

/// A minimal functor interface for this tutorial.
pub trait Functor<A, B> {
    type WrappedA;
    type WrappedB;

    fn fmap<F>(wrapped: Self::WrappedA, f: F) -> Self::WrappedB
    where
        F: Fn(A) -> B;
}

Rust Syntax

Functor<A, B> is a trait. A trait is Rust’s way to name behavior that many types can implement.

Here, the behavior is:

map a function through some wrapper shape

A and B are generic type parameters:

A = input item type
B = output item type

Generic means the trait is not tied to one concrete type like i32 or String. The same trait can describe Vec<i32> -> Vec<String>, Option<TokenId> -> Option<Embedding>, or any other pair of item types.

It has associated types:

type WrappedA;
type WrappedB;

Associated types are type names chosen by each implementation of the trait. They let the trait say:

every implementer must tell us what wrapped input and wrapped output mean

For VecFunctor, those associated types become Vec<A> and Vec<B>.

For OptionFunctor, they become Option<A> and Option<B>.

The method:

fn fmap<F>(wrapped: Self::WrappedA, f: F) -> Self::WrappedB
where
    F: Fn(A) -> B;

means:

Given a wrapped A and a function A -> B, produce a wrapped B.

The where clause is a readable place to put a bound. The bound:

F: Fn(A) -> B

means F must be callable like a function that consumes an A and returns a B.

Here is the real crate API in the smallest useful form:

use category_theory_transformer_rs::{Functor, VecFunctor};

let lengths = VecFunctor::fmap(vec!["cat", "rust"], |word| word.len());

assert_eq!(lengths, vec![3, 4]);

What to notice: The call names the structure once: VecFunctor::fmap. The closure only describes the item-level operation: &str -> usize. The vector shape is handled by the functor implementation.

ML Concept

In ML, you often apply the same transformation across a structure:

map preprocessing over a batch
map token conversion over a sequence
map loss computation over examples

The wrapper might be:

Vec
Option
Result
batch tensor

The idea is:

transform the contents, preserve the surrounding structure

Category-Theory Concept

A functor maps:

objects -> objects
morphisms -> morphisms

while preserving identity and composition.

This tutorial’s trait is deliberately small. It focuses on the practical fmap operation.

The tests in src/structure.rs check the two law-shaped habits this chapter uses: mapping identity leaves values unchanged, and mapping two functions in sequence matches mapping their composition.

VecFunctor

The problem this block solves is:

Demonstrate fmap for lists.

The block:

pub struct VecFunctor;

impl<A, B> Functor<A, B> for VecFunctor {
    type WrappedA = Vec<A>;
    type WrappedB = Vec<B>;

    fn fmap<F>(wrapped: Vec<A>, f: F) -> Vec<B>
    where
        F: Fn(A) -> B,
    {
        wrapped.into_iter().map(f).collect()
    }
}

Rust Syntax

VecFunctor is a unit struct. It stores no state.

The impl block is a trait implementation:

impl<A, B> Functor<A, B> for VecFunctor

Read it as:

for every A and B, VecFunctor knows how to behave as Functor<A, B>

The implementation chooses the associated types:

WrappedA = Vec<A>
WrappedB = Vec<B>

The method consumes the vector:

wrapped.into_iter()

maps the function over every item:

.map(f)

and collects the result:

.collect()

The runnable companion example uses the same real crate API:

use category_theory_transformer_rs::{Functor, VecFunctor};

let token_ids = vec![1, 2, 3];
let shifted = VecFunctor::fmap(token_ids, |id| id + 100);

assert_eq!(shifted, vec![101, 102, 103]);

ML Concept

If you have a batch of token IDs:

[TokenId(1), TokenId(2), TokenId(3)]

and a function:

TokenId -> Vector

mapping over the batch gives:

[Vector, Vector, Vector]

That is the same shape as VecFunctor.

Category-Theory Concept

Vec is list-like structure.

Mapping preserves the list shape:

List A -> List B

The length and order remain structurally meaningful.

OptionFunctor

The problem this block solves is:

Demonstrate the same functor idea for optional values.

The block:

pub struct OptionFunctor;

impl<A, B> Functor<A, B> for OptionFunctor {
    type WrappedA = Option<A>;
    type WrappedB = Option<B>;

    fn fmap<F>(wrapped: Option<A>, f: F) -> Option<B>
    where
        F: Fn(A) -> B,
    {
        wrapped.map(f)
    }
}

Rust Syntax

The wrapper types are:

Option<A>
Option<B>

The implementation delegates to Rust’s built-in:

wrapped.map(f)

If the value is Some(a), it becomes Some(f(a)).

If the value is None, it stays None.

The important beginner point is that Option makes absence explicit in the type. You cannot accidentally treat a missing value as a real value without handling the None case.

use category_theory_transformer_rs::{Functor, OptionFunctor};

let present = OptionFunctor::fmap(Some(7), |value| value * 2);
let missing = OptionFunctor::fmap(None::<i32>, |value| value * 2);

assert_eq!(present, Some(14));
assert_eq!(missing, None);

ML Concept

Optional values appear when data may be absent:

maybe first token
maybe cached embedding
maybe resolved department

Mapping over Option lets you transform present values without inventing a fake value for missing ones.

Category-Theory Concept

Option is a context representing possible absence.

fmap lifts:

A -> B

to:

Option<A> -> Option<B>

Conceptual Extension: Distribution<T>::map

The problem this block solves is:

A probabilistic value may contain many possible outcomes. Sometimes you want to transform every possible outcome while keeping its probability attached.

The current crate’s Distribution in src/domain.rs is a concrete validated probability vector:

pub struct Distribution(Vec<f32>);

The block below is a conceptual generic version that explains the functor idea for probabilistic outcomes:

#![allow(unused)]
fn main() {
pub struct Probability(f32);

pub struct Distribution<T> {
    outcomes: Vec<(T, Probability)>,
}

impl<T> Distribution<T> {
    pub fn map<U>(
        self,
        f: impl Fn(T) -> U,
    ) -> Distribution<U> {
        let outcomes = self
            .outcomes
            .into_iter()
            .map(|(value, probability)| {
                (f(value), probability)
            })
            .collect();

        Distribution { outcomes }
    }
}
}

The core idea is:

map changes the values inside the distribution,
but keeps the probabilities attached to them.

Rust Syntax

Start with the generic struct:

#![allow(unused)]
fn main() {
pub struct Probability(f32);

pub struct Distribution<T> {
    outcomes: Vec<(T, Probability)>,
}
}

This means:

Distribution<T> = many possible T values, each paired with a probability

If T is TokenId, then the type is:

Distribution<TokenId>

If T is String, then the type is:

Distribution<String>

The method introduces a second generic type:

pub fn map<U>(...)

T is the old outcome type.

U is the new outcome type.

So the method has this shape:

Distribution<T> -> Distribution<U>

The first parameter is:

self

That means the method consumes the old distribution.

After calling:

let text_dist = token_dist.map(decode);

the old token_dist has been moved and cannot be used again.

That is why the implementation can call:

self.outcomes.into_iter()

into_iter() consumes the vector and yields owned pairs:

(T, Probability)

The function parameter is:

f: impl Fn(T) -> U

This means:

give this method a function or closure that takes T and returns U

For example, a decoder might have this shape:

TokenId -> String

Then:

Distribution<TokenId> -> Distribution<String>

The inner mapping line is:

.map(|(value, probability)| {
    (f(value), probability)
})

For every pair:

(value, probability)

the code applies f to the value and leaves the probability unchanged.

So:

(TokenId(2), Probability(0.70))

can become:

("Rust", Probability(0.70))

Finally:

.collect()

collects the transformed pairs back into a vector, and:

Distribution { outcomes }

wraps them in the new distribution.

ML Concept

Imagine a model returns possible next tokens:

TokenId(2) -> 0.70
TokenId(4) -> 0.20
TokenId(3) -> 0.10

Those token IDs are useful to the model, but a learner or UI might need text:

TokenId(2) -> "Rust"
TokenId(4) -> "Python"
TokenId(3) -> "."

map changes the representation:

Distribution<TokenId> -> Distribution<String>

The values change:

TokenId(2) becomes "Rust"
TokenId(4) becomes "Python"
TokenId(3) becomes "."

The probabilities do not change:

0.70 stays 0.70
0.20 stays 0.20
0.10 stays 0.10

So map is for changing the meaning or representation of each possible outcome, not for changing the probability mass.

Category Theory Concept

This is the functor pattern for probabilistic context.

Given a normal deterministic function:

f : T -> U

map lifts it into the distribution context:

Distribution<T> -> Distribution<U>

In functional-programming notation:

fmap : (T -> U) -> Distribution<T> -> Distribution<U>

The outer structure is preserved:

same number of possible outcomes
same probabilities
same probabilistic context

Only the inner values are transformed.

That is the same pattern as:

Option<T> -> Option<U>
Vec<T> -> Vec<U>
Distribution<T> -> Distribution<U>

Different context, same functor idea.

Concrete Example

Here is a complete conceptual example:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy)]
pub struct TokenId(pub usize);

#[derive(Debug, Clone, Copy, PartialEq)]
pub struct Probability(pub f32);

#[derive(Debug, Clone)]
pub struct Distribution<T> {
    outcomes: Vec<(T, Probability)>,
}

impl<T> Distribution<T> {
    pub fn new(outcomes: Vec<(T, Probability)>) -> Self {
        Self { outcomes }
    }

    pub fn map<U, F>(self, mut f: F) -> Distribution<U>
    where
        F: FnMut(T) -> U,
    {
        let outcomes = self
            .outcomes
            .into_iter()
            .map(|(value, probability)| {
                (f(value), probability)
            })
            .collect();

        Distribution { outcomes }
    }
}

let vocab = ["I", "love", "Rust", "."];

let token_dist = Distribution::new(vec![
    (TokenId(2), Probability(0.70)),
    (TokenId(3), Probability(0.30)),
]);

let text_dist = token_dist.map(|token| {
    vocab[token.0].to_string()
});

assert_eq!(
    text_dist.outcomes,
    vec![
        ("Rust".to_string(), Probability(0.70)),
        (".".to_string(), Probability(0.30)),
    ],
);
}

Conceptually, the result is:

"Rust" -> 0.70
"."    -> 0.30

Why Fn(T) -> U

The signature:

f: impl Fn(T) -> U

accepts functions and closures.

It is more flexible than:

fn(T) -> U

because closures can capture values from the surrounding scope:

let vocab = ["I", "love", "Rust", "."];

let text_dist = token_dist.map(|token| {
    vocab[token.0].to_string()
});

The closure uses vocab from outside the closure body.

Why A Library Might Use FnMut

The pedagogical signature:

f: impl Fn(T) -> U

is easy to read.

A more flexible library signature is often:

pub fn map<U, F>(self, mut f: F) -> Distribution<U>
where
    F: FnMut(T) -> U,

FnMut allows the closure to mutate captured state.

For example:

let mut counter = 0;

let numbered = token_dist.map(|token| {
    counter += 1;
    (counter, token)
});

The key ownership rule is unchanged:

the method consumes the old distribution and moves each T into f

map Versus flat_map

Use map when each possible value becomes one transformed value:

T -> U

So:

Distribution<T> -> Distribution<U>

Example:

TokenId -> String

Use flat_map when each possible value produces another distribution:

T -> Distribution<U>

Without flattening, the result would be:

Distribution<Distribution<U>>

The simple distinction is:

map:
  one possible value becomes one transformed value

flat_map:
  one possible value becomes many possible future values

In language modeling:

map decodes possible tokens into text
flat_map chains uncertainty across another prediction step

Algebra Version

If:

D<T> = [(t1, p1), (t2, p2), ..., (tn, pn)]

and:

f : T -> U

then:

map(f, D<T>) = [(f(t1), p1), (f(t2), p2), ..., (f(tn), pn)]

The probabilities are untouched.

Only the values move through f.

Core Mental Model

In Rust terms:

consume self, move each T into f, preserve each Probability, collect into
Distribution<U>

In ML terms:

decode or transform every possible outcome without changing model confidence

In category-theory terms:

lift a deterministic morphism T -> U into the probabilistic context
Distribution<T> -> Distribution<U>

NaturalTransformation<A>

The problem this block solves is:

Sometimes you need to convert one wrapper shape into another without caring about the specific item type.

The beginner trap is to think a natural transformation is just any conversion. It is more disciplined than that. The conversion must be compatible with mapping. If you transform the wrapper first and then map the item, you should get the same result as mapping first and then transforming the wrapper.

The block:

/// A structure-preserving conversion between wrappers.
pub trait NaturalTransformation<A> {
    type From;
    type To;

    fn transform(from: Self::From) -> Self::To;
}

Rust Syntax

The associated types are:

From
To

The method:

fn transform(from: Self::From) -> Self::To;

converts the wrapper.

The type parameter A represents the item type inside the wrapper.

This trait has the same shape as Functor: the abstract contract is public, but each implementation chooses the concrete wrapper types through associated types.

That is why the implementation can be generic over A without knowing whether A is a token, a sentence, a loss value, or a trace step.

ML Concept

Data pipelines often convert shapes:

list of candidates -> optional selected candidate
batch -> first example
many diagnostics -> maybe first failure

The conversion should not depend on whether the item is a token, vector, or loss.

Category-Theory Concept

A natural transformation converts one functor into another in a way that is uniform over the inner type.

The important word is uniform.

It should not inspect special details of A.

VecToFirstOption

The problem this block solves is:

Convert a list into an optional first item in a type-uniform way.

The block:

/// Natural transformation from `Vec<A>` to `Option<A>` by taking the first item.
pub struct VecToFirstOption;

impl<A> NaturalTransformation<A> for VecToFirstOption {
    type From = Vec<A>;
    type To = Option<A>;

    fn transform(from: Vec<A>) -> Option<A> {
        from.into_iter().next()
    }
}

Rust Syntax

The implementation works for every A.

That is the role of:

impl<A> NaturalTransformation<A> for VecToFirstOption

The implementation does not ask for any bound on A. There is no A: Clone, no A: Debug, and no A: PartialEq. It does not need those capabilities because it never looks inside the item. It only changes the outer shape.

It consumes a vector:

from.into_iter()

then takes the first item:

.next()

If the vector is empty, the result is None.

If it has at least one item, the result is Some(first_item).

use category_theory_transformer_rs::{NaturalTransformation, VecToFirstOption};

let first = VecToFirstOption::transform(vec!["embed", "linear", "softmax"]);
let empty = VecToFirstOption::transform(Vec::<&str>::new());

assert_eq!(first, Some("embed"));
assert_eq!(empty, None);

ML Concept

This is like selecting the first candidate from a batch while preserving the possibility that the batch was empty.

It does not care what the candidate type is.

Category-Theory Concept

This is the example transformation:

Vec<A> -> Option<A>

It is natural because it is uniform over A.

Naturality Check

The problem this block solves is:

Show that mapping first, then converting gives the same result as converting first, then mapping.

The function:

pub fn naturality_square_holds_for_first_option() -> bool {
    let xs = vec![1, 2, 3];
    let f = |x| x * 10;

    let path_top_then_right = VecToFirstOption::transform(VecFunctor::fmap(xs.clone(), f));
    let path_left_then_bottom = OptionFunctor::fmap(VecToFirstOption::transform(xs), f);

    path_top_then_right == path_left_then_bottom
}

Rust Syntax

There are two paths.

Path one:

Vec<i32> -> Vec<i32> -> Option<i32>

Path two:

Vec<i32> -> Option<i32> -> Option<i32>

Both should produce the same value.

The crate exposes this as a small law check:

use category_theory_transformer_rs::naturality_square_holds_for_first_option;

assert!(naturality_square_holds_for_first_option());

ML Concept

This is a consistency check for pipeline shape conversions.

It says:

transform values, then select

matches:

select, then transform the selected value

for this operation.

Category-Theory Concept

This is the naturality square:

Vec<A> ----fmap f----> Vec<B>
  |                     |
  | transform           | transform
  v                     v
Option<A> --fmap f--> Option<B>

The square commutes when both paths agree.

The same square as a data-flow diagram:

flowchart LR
    VA["Vec<i32>"] -->|VecFunctor::fmap x10| VB["Vec<i32>"]
    VA -->|VecToFirstOption::transform| OA["Option<i32>"]
    VB -->|VecToFirstOption::transform| OB["Option<i32>"]
    OA -->|OptionFunctor::fmap x10| OB

The same square as a rendered math view:

[ \begin{array}{ccc} \mathrm{Vec}\langle A\rangle & \xrightarrow{\mathrm{VecFunctor::fmap}(f)} & \mathrm{Vec}\langle B\rangle \ \downarrow \mathrm{VecToFirstOption} && \downarrow \mathrm{VecToFirstOption} \ \mathrm{Option}\langle A\rangle & \xrightarrow{\mathrm{OptionFunctor::fmap}(f)} & \mathrm{Option}\langle B\rangle \end{array} ]

How to read this diagram:

• the top path maps inside the vector, then selects the first item,
• the left-bottom path selects the first item, then maps inside the option,
• the square commutes when both paths produce the same Option<B>,
• the Rust handle is naturality_square_holds_for_first_option.

If you had to redraw this by hand, that is a useful learning signal. Redrawing forces you to decide which objects sit at the corners and which arrows are responsible for each conversion.

Read it as two executable paths:

top then right:
Vec<i32> -> Vec<i32> -> Option<i32>

left then bottom:
Vec<i32> -> Option<i32> -> Option<i32>

The test passes only when both paths produce the same optional value.

Monoid

The problem this block solves is:

Some values can be combined repeatedly, and there should be an empty value that changes nothing.

The first-principles version is string concatenation. The empty string changes nothing, and grouping does not change the final text:

#![allow(unused)]
fn main() {
let empty = String::new();
let a = String::from("embed");
let b = String::from(" -> linear");
let c = String::from(" -> softmax");

assert_eq!(format!("{empty}{a}"), "embed");
assert_eq!(format!("{}{}", format!("{a}{b}"), c), format!("{a}{}", format!("{b}{c}")));
}

PipelineTrace uses the same idea, but with named pipeline steps instead of raw text.

The trait:

pub trait Monoid: Sized {
    fn empty() -> Self;
    fn combine(&self, other: &Self) -> Self;
}

Rust Syntax

Sized means values of this type have a known size at compile time. In this trait, it keeps the return type Self straightforward:

fn empty() -> Self;

Self means “the type implementing this trait.”

The trait requires:

empty() -> Self
combine(&self, other: &Self) -> Self

So a monoid can produce an identity value and combine two values into one.

The combine method borrows both traces:

fn combine(&self, other: &Self) -> Self;

&self and &Self are references. They let the method read the two existing values without taking ownership of them. The method returns a new combined value.

ML Concept

Common monoid-like values in ML systems include logs, traces, metrics, batches, and accumulated gradients.

You often need to combine many small values into one larger value.

Category-Theory Concept

A monoid has:

identity element
associative binary operation

The laws are:

empty combine a = a
a combine empty = a
(a combine b) combine c = a combine (b combine c)

TraceStep and PipelineTrace

The problem this block solves is:

Pipeline execution steps should be combinable into a larger trace.

The key types:

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct TraceStep(&'static str);

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PipelineTrace(Vec<TraceStep>);

Rust Syntax

TraceStep wraps one static string.

PipelineTrace wraps a vector of steps.

PipelineTrace::from_steps collects any iterable of steps.

names() returns the raw names for display:

self.0.iter().map(TraceStep::name).collect()

These wrappers matter because two values may both be strings but mean different things. A TraceStep is not a model name, a token, or a user-facing sentence. The type keeps that meaning attached to the value.

ML Concept

A trace can record:

embedding
linear
softmax
cross_entropy

This is useful for understanding which stages ran.

Category-Theory Concept

A pipeline trace is a sequence-like monoid.

The empty trace is the identity.

Combining traces is concatenation.

PipelineTrace as Monoid

The problem this block solves is:

Make traces obey the monoid interface.

The implementation:

impl Monoid for PipelineTrace {
    fn empty() -> Self {
        PipelineTrace(vec![])
    }

    fn combine(&self, other: &Self) -> Self {
        let mut combined = self.0.clone();
        combined.extend_from_slice(&other.0);
        PipelineTrace(combined)
    }
}

Rust Syntax

empty returns an empty vector.

combine clones the first trace, appends the second trace, and wraps the result again as PipelineTrace.

The clone is intentional here: combine borrows both inputs, so it cannot move steps out of either trace. Cloning the small TraceStep values lets the method produce a fresh trace while leaving the inputs usable.

use category_theory_transformer_rs::{Monoid, PipelineTrace, TraceStep};

let encoder = PipelineTrace::from_steps([TraceStep::new("embedding")]);
let head = PipelineTrace::from_steps([TraceStep::new("softmax")]);
let trace = encoder.combine(&head);

assert_eq!(trace.names(), vec!["embedding", "softmax"]);

ML Concept

This is how many execution logs work:

trace_a + trace_b = longer trace

Category-Theory Concept

This is the list monoid specialized to trace steps.

Monoid Law Check

The problem this block solves is:

Verify the identity and associativity laws for the trace type.

The function:

pub fn monoid_laws_hold_for_pipeline_trace() -> bool {
    let a = PipelineTrace::from_steps(vec![TraceStep::new("embedding")]);
    let b = PipelineTrace::from_steps(vec![TraceStep::new("linear")]);
    let c = PipelineTrace::from_steps(vec![TraceStep::new("softmax")]);
    let identity = PipelineTrace::empty();

    let left_identity = identity.combine(&a) == a;
    let right_identity = a.combine(&identity) == a;
    let associativity = a.combine(&b).combine(&c) == a.combine(&b.combine(&c));

    left_identity && right_identity && associativity
}

Rust Syntax

The function constructs three traces and checks three booleans.

It returns true only if all monoid laws hold for those examples.

use category_theory_transformer_rs::monoid_laws_hold_for_pipeline_trace;

assert!(monoid_laws_hold_for_pipeline_trace());

ML Concept

Grouping trace combination should not change the final trace.

This matters when systems combine logs from nested pipelines.

Category-Theory Concept

This directly checks the monoid laws:

identity
associativity

The associativity check can be read as a grouping diagram:

flowchart LR
    A["embedding"] --> AB["embedding + linear"]
    B["linear"] --> AB
    AB --> ABC1["(embedding + linear) + softmax"]
    C["softmax"] --> ABC1
    B --> BC["linear + softmax"]
    C --> BC
    A --> ABC2["embedding + (linear + softmax)"]
    BC --> ABC2

The same law as a rendered math view:

[ \begin{array}{ccc} (\mathrm{embedding} \diamond \mathrm{linear}) \diamond \mathrm{softmax} & = & \mathrm{embedding} \diamond (\mathrm{linear} \diamond \mathrm{softmax}) \end{array} ]

Here \(\diamond\) means PipelineTrace::combine. The equality is not about string formatting. It says the final trace meaning should not depend on where the parentheses were placed.

How to read this diagram:

• the objects are trace values,
• the arrow-like operation is combine,
• the identity object is PipelineTrace::empty(),
• the Rust handle is monoid_laws_hold_for_pipeline_trace.

The two final traces should contain the same step names in the same order. The law is not about performance or formatting. It says grouping nested trace combinations should not change what the trace means.

The Calculus File

The problem src/calculus.rs solves is:

Backpropagation is easier to understand if you first see one local derivative rule in isolation.

What to notice: Backpropagation does not require every operation to know the whole model. Each operation only needs to know how its own output changes when its own inputs change. Composition does the rest.

The file defines:

Scalar
LocalGradient
MulOp

Scalar

The block:

#[derive(Debug, Clone, Copy, PartialEq)]
pub struct Scalar(f32);

Rust Syntax

Scalar wraps one f32.

Scalar::new rejects non-finite values.

value() returns the raw float.

This repeats a pattern from the earlier domain-object chapter:

private field
validating constructor
accessor

The raw f32 is private, so callers cannot construct Scalar(f32::NAN) directly. They must use Scalar::new, which returns a Result.

use category_theory_transformer_rs::Scalar;

let scalar = Scalar::new(2.5)?;

assert_eq!(scalar.value(), 2.5);

Ok::<(), category_theory_transformer_rs::CtError>(())

ML Concept

This is a single numeric value in a computation graph.

Examples:

activation
loss component
weight

Category-Theory Concept

It is the simple numeric object used in the local derivative example.

LocalGradient

The block:

#[derive(Debug, Clone, Copy, PartialEq)]
pub struct LocalGradient(f32);

Rust Syntax

This is another f32 wrapper.

It has the same finite-value validation as Scalar.

The semantic difference is important:

Scalar = forward value
LocalGradient = derivative signal

This is the same newtype move used throughout the crate. Both wrappers store an f32, but the types prevent accidental mixing at function boundaries.

ML Concept

A gradient tells how a loss changes when an intermediate value changes.

For example:

dL/dz

Category-Theory Concept

This is information flowing backward through a composed computation.

MulOp

The problem this block solves is:

Show a forward operation and its local backward rule.

The important methods:

pub fn forward(&self, x: Scalar, y: Scalar) -> CtResult<Scalar> {
    Scalar::new(x.value() * y.value())
}

pub fn backward(
    &self,
    x: Scalar,
    y: Scalar,
    upstream: LocalGradient,
) -> CtResult<(LocalGradient, LocalGradient)> {
    let dz_dx = y.value();
    let dz_dy = x.value();

    Ok((
        LocalGradient::new(upstream.value() * dz_dx)?,
        LocalGradient::new(upstream.value() * dz_dy)?,
    ))
}

Rust Syntax

forward multiplies two scalars and validates the result.

backward takes:

x
y
upstream gradient dL/dz

and returns:

(dL/dx, dL/dy)

The return type is a Rust tuple.

The ? operator appears twice in backward:

LocalGradient::new(upstream.value() * dz_dx)?

It means:

if construction succeeded, keep the value
if construction failed, return the error from backward immediately

That keeps invalid numeric states at the boundary where they are created.

use category_theory_transformer_rs::{LocalGradient, MulOp, Scalar};

let mul = MulOp;
let x = Scalar::new(2.0)?;
let y = Scalar::new(3.0)?;
let z = mul.forward(x, y)?;
let (dl_dx, dl_dy) = mul.backward(x, y, LocalGradient::new(1.0)?)?;

assert_eq!(z.value(), 6.0);
assert_eq!(dl_dx.value(), 3.0);
assert_eq!(dl_dy.value(), 2.0);

Ok::<(), category_theory_transformer_rs::CtError>(())

ML Concept

For:

z = x * y

the local derivatives are:

dz/dx = y
dz/dy = x

By the chain rule:

dL/dx = dL/dz * dz/dx = dL/dz * y
dL/dy = dL/dz * dz/dy = dL/dz * x

The companion tests use two upstream gradients. With dL/dz = 1, the gradients are 3 and 2. With dL/dz = 4, the same local rule scales them to 12 and 8.

Category-Theory Concept

The chain rule is composition of local derivative maps.

A big neural network is many small maps composed forward, then many local gradient rules composed backward.

The local multiplication example as a rendered math view:

[ \begin{array}{ccccc} x,y & \xrightarrow{\mathrm{MulOp::forward}} & z = x \cdot y & \xrightarrow{\mathrm{loss}} & L \ && \uparrow \mathrm{d}L/\mathrm{d}z && \ \mathrm{d}L/\mathrm{d}x = (\mathrm{d}L/\mathrm{d}z),y && \mathrm{d}L/\mathrm{d}y = (\mathrm{d}L/\mathrm{d}z),x \end{array} ]

How to read this diagram:

• the top row is the forward computation,
• the bottom row names the local backward results,
• the upstream gradient dL/dz is the signal being carried backward,
• the Rust handle is MulOp::backward.

Production Autograd Boundary

Production frameworks do not ask the user to manually call one backward method for every operation. PyTorch’s autograd documentation describes a reverse automatic-differentiation system: the forward pass records the operations that produced tensors, and the backward pass traces that recorded graph with the chain rule.

The tiny Rust example keeps only one local rule:

MulOp::forward  : Scalar x Scalar -> Scalar
MulOp::backward : Scalar x Scalar x LocalGradient -> (LocalGradient, LocalGradient)

Read those as MulOp::forward : Scalar x Scalar -> Scalar and MulOp::backward : Scalar x Scalar x LocalGradient -> (LocalGradient, LocalGradient).

That is not a replacement for autograd. It is a microscope for one boundary. It answers three questions before the full graph machinery appears:

which forward value must be remembered?
which local derivative is used?
which upstream gradient is being carried backward?

When you return to a framework, the useful question is not “where did the chain rule go?” It is:

which graph edges and saved values let the framework compose the same local
rules automatically?

Run The Example

use category_theory_transformer_rs::{
    CtResult, Functor, LocalGradient, Monoid, MulOp, OptionFunctor, PipelineTrace, Scalar,
    TraceStep, VecFunctor, monoid_laws_hold_for_pipeline_trace,
    naturality_square_holds_for_first_option,
};

fn main() -> CtResult<()> {
    let squared = VecFunctor::fmap(vec![1, 2, 3], |x| x * x);
    let shifted = OptionFunctor::fmap(Some(7), |x| x + 1);

    println!("Vec fmap square: {squared:?}");
    println!("Option fmap +1: {shifted:?}");
    println!(
        "naturality square holds: {}",
        naturality_square_holds_for_first_option()
    );

    let trace = PipelineTrace::from_steps(vec![TraceStep::new("embedding")])
        .combine(&PipelineTrace::from_steps(vec![TraceStep::new("linear")]))
        .combine(&PipelineTrace::from_steps(vec![TraceStep::new("softmax")]));

    println!("trace: {:?}", trace.names());
    println!(
        "monoid laws hold: {}",
        monoid_laws_hold_for_pipeline_trace()
    );

    let mul = MulOp;
    let x = Scalar::new(2.0)?;
    let y = Scalar::new(3.0)?;
    let upstream = LocalGradient::new(1.0)?;
    let (dl_dx, dl_dy) = mul.backward(x, y, upstream)?;

    println!("dL/dx: {}", dl_dx.value());
    println!("dL/dy: {}", dl_dy.value());
    println!();
    println!("Typed transformation:");
    println!("VecFunctor::fmap : Vec<A> x (A -> B) -> Vec<B>");
    println!("OptionFunctor::fmap : Option<A> x (A -> B) -> Option<B>");
    println!("Naturality square:");
    println!("Vec<A> -> Vec<B> -> Option<B>");
    println!("Vec<A> -> Option<A> -> Option<B>");
    println!("Monoid:");
    println!("PipelineTrace x PipelineTrace -> PipelineTrace");
    println!("Chain rule:");
    println!("Scalar x Scalar -> Scalar");
    println!("dL/dz -> (dL/dx, dL/dy)");

    Ok(())
}

Run:

cargo run --example 04_structure_and_calculus

You should see mapping over Vec, mapping over Option, a naturality check, a combined trace, a monoid law check, and local gradients for multiplication. The example also prints the typed boundaries behind those values:

Typed transformation:
VecFunctor::fmap : Vec<A> x (A -> B) -> Vec<B>
OptionFunctor::fmap : Option<A> x (A -> B) -> Option<B>
Naturality square:
Vec<A> -> Vec<B> -> Option<B>
Vec<A> -> Option<A> -> Option<B>
Monoid:
PipelineTrace x PipelineTrace -> PipelineTrace
Chain rule:
Scalar x Scalar -> Scalar
dL/dz -> (dL/dx, dL/dy)

Example Output Transfer Checklist

This example is a compact law lab. Each printed line should make one consistency condition visible.

Use the output this way:

The four pattern names are useful only if they protect these boundaries. A functor protects wrapper-preserving mapping. A natural transformation protects agreement between two paths. A monoid protects repeated combination. The chain rule protects local-to-global gradient composition.

Output-To-Law Audit

When the example prints a result, do not stop at “it worked.” Turn the output line into a law-shaped claim and then narrow the claim back to the local Rust evidence.

Use this audit card:

output line:
Rust handle:
law or boundary:
source support:
safe non-claim:
validation command:

Worked audit:

output line: naturality square holds: true
Rust handle: naturality_square_holds_for_first_option
law or boundary: mapping before first-or-none matches first-or-none before mapping
source support: formal naturality vocabulary; programming-shaped wrapper conversion
safe non-claim: this checks one concrete square, not every natural transformation
validation command: cargo test structure::tests::naturality_square_commutes --lib

Second worked audit:

output line: dL/dx: 3 and dL/dy: 2
Rust handle: MulOp::backward
law or boundary: upstream gradient is multiplied by the local derivatives of x * y
source support: chain rule and reverse traversal through a computation graph
safe non-claim: this is one local derivative rule, not a production autograd engine
validation command: cargo test calculus::tests::multiply_backward_returns_local_chain_rule_gradients --lib

The pattern is the same as the rest of the book:

visible output -> Rust handle -> law-shaped claim -> source-backed limit

That last step matters. A passing law-shaped test is good evidence for the teaching example. It is not permission to claim the repository proves all functor laws, all naturality squares, every monoid, or every differentiable program.

Core Mental Model

In Rust terms:

traits name reusable operation shapes
unit structs demonstrate stateless operations
tests check laws

In ML terms:

map over structures, combine traces, compose local gradients

In category-theory terms:

functors preserve structure
natural transformations commute with mapping
monoids combine associatively with identity
chain rule composes derivative information

Checkpoint

Why is “local rule plus composition” the core idea behind backpropagation?

A strong answer:

Each operation only needs its local derivative; the chain rule composes those local derivatives into the gradient for the whole computation.

Where This Leaves Us

This chapter named the repeated structures that sit underneath the small ML pipeline. A functor explains mapping inside a wrapper. A natural transformation explains changing wrappers consistently. A monoid explains safe accumulation. A local gradient explains why a large training computation can be assembled from small derivative rules.

The next chapter, Seven Sketches Through Rust, uses the same engineering habit on a wider set of ideas from applied category theory. Instead of adding a larger ML model, it shows how the same typed-Rust style can model orders, resources, database instances, design relations, signal flow, circuits, and local-to-global behavior.

Further Reading

Do not read these links as a bibliography to admire. Read them as a transfer path from the small laws in this chapter to larger systems.

Start from the local Rust evidence:

VecFunctor::fmap : Vec<A> x (A -> B) -> Vec<B>
naturality_square_holds_for_first_option() -> bool
PipelineTrace x PipelineTrace -> PipelineTrace
MulOp::backward : Scalar x Scalar x LocalGradient -> (LocalGradient, LocalGradient)

Then read the sources in this order:

Use Glossary when a word becomes slippery. Use References when you want the full source list.

After reading one external source, ask four questions:

1. Which exact Rust type or function did it clarify?
2. Which law, local derivative, or path agreement did it support?
3. Which claim did it not license this chapter to make?
4. Which command would you run to inspect the local evidence?

For this chapter, the commands are:

cargo run --example 04_structure_and_calculus
cargo test structure::tests --lib
cargo test calculus::tests --lib

If you can answer those questions, the external sources have transferred back into the code.

Practice After This Chapter

Use Exercise 14 to trace the naturality square and monoid laws back to the exact tests. This is the chapter’s main transfer check: a term such as “natural transformation” or “monoid” should point to a runnable law check, not only to a definition.

Retrieval Practice

Recall

Recover the four reusable structures before using their names.

First, state what a functor does to values inside a wrapper.

Then name the two paths that must agree in the Vec<A> -> Option<A> naturality square.

Next, name the two laws that make PipelineTrace a monoid-like trace type in this chapter.

Finally, for MulOp::backward, name the local derivatives used for z = x * y.

Explain

Separate the law, the test, and the analogy.

Explain why OptionFunctor::fmap(None::<i32>, |value| value * 10) is expected to return None.

Explain why VecToFirstOption::transform does not need a bound such as A: Clone or A: Debug.

Explain why a pipeline trace is a good example of a monoid.

Then explain why the functor, naturality, and monoid tests are executable anchors rather than full mathematical proofs.

Apply

Use the runnable example and the law checks.

1. For xs = vec![1, 2, 3] and f = |x| x * 10, compute both naturality paths:

   Vec<i32> --fmap f--> Vec<i32> --first--> Option<i32>
   Vec<i32> --first--> Option<i32> --fmap f--> Option<i32>
   

2. For trace steps embedding, linear, and softmax, write both groupings checked by associativity:

   (embedding <> linear) <> softmax
   embedding <> (linear <> softmax)
   

   What list of names should both produce?

3. For x = 2, y = 3, and upstream gradient dL/dz = 4, what should MulOp::backward return for dL/dx and dL/dy?

4. Write a small example of a value in your own codebase that has an empty value and a combine operation. State the identity law it should satisfy.

Debug

For each broken explanation, name the missing law or wrong boundary:

mapping a Vec and changing its length without saying why
transforming Vec<A> to Option<A> by inspecting special details of A
combining traces where empty <> trace adds a visible step
claiming backpropagation is one giant derivative instead of composed local rules

A strong answer should point back to the concrete Rust checks:

VecFunctor identity and composition tests
naturality_square_commutes
pipeline_trace_obeys_monoid_laws
multiply_backward_scales_with_upstream_gradient

The goal is not to recite category-theory vocabulary. The goal is to recognize which consistency condition the code is protecting.

Seven Sketches Through Rust

The problem this chapter solves is:

Applied category theory can feel too large to connect to code. This chapter turns the seven major themes of Seven Sketches in Compositionality into small Rust blocks with tests.

The previous chapter introduced reusable structures such as functors, natural transformations, monoids, and local derivative rules. This chapter keeps the same reading style but changes scale. Instead of following the tiny ML pipeline, it takes seven broader ideas and asks how each one can be made concrete enough to inspect in Rust.

This chapter does not reproduce the paper. It gives executable handles for the ideas.

The repeated pattern is:

mathematical structure
  -> Rust type
  -> constructor or method
  -> law check

The Rust lesson is:

newtypes + private fields + validation + explicit composition

The category-theory lesson is:

objects + relationships + composition + laws

Reader orientation: Treat each sketch as a small modeling exercise. You are not expected to know the full mathematical theory before reading the Rust. Start from the type, then read the constructor, then read the law check.

Chapter Outcomes

By the end of this chapter, you should be able to:

• name the one law, relation, or boundary each Rust sketch preserves,
• explain which part of the source text each sketch deliberately does not implement,
• transfer one sketch to a small software or ML design problem without overclaiming the mathematics.

What You Already Know

If you have modeled business rules, resource limits, database relationships, or state machines in Rust, you already know the software version of this chapter. The new idea is that applied category theory gives names to those compositional structures and asks which laws make them trustworthy.

Source Snapshots

The main module:

//! Rust models for the seven applied-category-theory sketches.
//!
//! This module is a study companion for *Seven Sketches in Compositionality*.
//! It does not try to encode all of category theory. Instead, each section
//! gives one small Rust model for the main structure of a sketch: orders,
//! resources, databases, co-design, signal flow, circuits, and logic of
//! behavior.

use crate::error::{CtError, CtResult};

/// A finite order used for the first sketch: observations can be refined into
/// features, scores, and decisions.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub enum InformationLevel {
    Observation,
    Feature,
    Score,
    Decision,
}

impl InformationLevel {
    /// Returns true when information at this level can flow into `target`.
    pub fn can_flow_to(self, target: Self) -> bool {
        self <= target
    }

    /// Least upper bound in this small total order.
    pub fn join(self, other: Self) -> Self {
        self.max(other)
    }
}

const INFORMATION_LEVELS: [InformationLevel; 4] = [
    InformationLevel::Observation,
    InformationLevel::Feature,
    InformationLevel::Score,
    InformationLevel::Decision,
];

/// Checks reflexivity and transitivity for the finite information order.
pub fn information_order_obeys_preorder_laws() -> bool {
    for level in INFORMATION_LEVELS {
        if !level.can_flow_to(level) {
            return false;
        }
    }

    for first in INFORMATION_LEVELS {
        for second in INFORMATION_LEVELS {
            for third in INFORMATION_LEVELS {
                let premise = first.can_flow_to(second) && second.can_flow_to(third);
                if premise && !first.can_flow_to(third) {
                    return false;
                }
            }
        }
    }

    true
}

/// Number of concrete features in a tiny model.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct FeatureCount(usize);

impl FeatureCount {
    pub fn new(value: usize) -> CtResult<Self> {
        if value == 0 {
            return Err(CtError::EmptyInput("feature count"));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Number of abstract layers used to summarize feature capacity.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct LayerBudget(usize);

impl LayerBudget {
    pub fn new(value: usize) -> CtResult<Self> {
        if value == 0 {
            return Err(CtError::EmptyInput("layer budget"));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

const FEATURES_PER_LAYER: usize = 4;

/// Abstracts concrete features to the minimum layer budget that can hold them.
pub fn abstract_to_layer_budget(features: FeatureCount) -> CtResult<LayerBudget> {
    LayerBudget::new(features.value().div_ceil(FEATURES_PER_LAYER))
}

/// Concretizes an abstract layer budget back to feature capacity.
pub fn concretize_layer_budget(layers: LayerBudget) -> FeatureCount {
    FeatureCount(layers.value() * FEATURES_PER_LAYER)
}

/// Checks the Galois-connection law for the feature/layer abstraction.
pub fn feature_layer_galois_law_holds(
    features: FeatureCount,
    layers: LayerBudget,
) -> CtResult<bool> {
    let abstracted_fits = abstract_to_layer_budget(features)?.value() <= layers.value();
    let concrete_fits = features.value() <= concretize_layer_budget(layers).value();

    Ok(abstracted_fits == concrete_fits)
}

/// A non-negative amount in a resource bundle.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct ResourceAmount(usize);

impl ResourceAmount {
    pub fn new(value: usize) -> Self {
        Self(value)
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Compute and memory resources composed by component-wise addition.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ResourceBundle {
    compute: ResourceAmount,
    memory: ResourceAmount,
}

impl ResourceBundle {
    pub fn new(compute: ResourceAmount, memory: ResourceAmount) -> Self {
        Self { compute, memory }
    }

    pub fn compute(&self) -> ResourceAmount {
        self.compute
    }

    pub fn memory(&self) -> ResourceAmount {
        self.memory
    }

    /// Monoidal product for independent resources.
    pub fn tensor(&self, other: &Self) -> Self {
        Self {
            compute: ResourceAmount::new(self.compute.value() + other.compute.value()),
            memory: ResourceAmount::new(self.memory.value() + other.memory.value()),
        }
    }

    /// Resource preorder: supply can cover demand when every component is large
    /// enough.
    pub fn can_supply(&self, demand: &Self) -> bool {
        self.compute >= demand.compute && self.memory >= demand.memory
    }
}

/// Demonstrates monotonicity of the monoidal resource operation.
pub fn resource_tensor_is_monotone() -> bool {
    let small_supply = ResourceBundle::new(ResourceAmount::new(2), ResourceAmount::new(8));
    let large_supply = ResourceBundle::new(ResourceAmount::new(4), ResourceAmount::new(16));
    let fixed = ResourceBundle::new(ResourceAmount::new(1), ResourceAmount::new(2));

    large_supply.can_supply(&small_supply)
        && large_supply
            .tensor(&fixed)
            .can_supply(&small_supply.tensor(&fixed))
}

/// Identifier for a department row.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct DepartmentId(usize);

impl DepartmentId {
    pub fn new(value: usize) -> Self {
        Self(value)
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Identifier for an employee row.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct EmployeeId(usize);

impl EmployeeId {
    pub fn new(value: usize) -> Self {
        Self(value)
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// A row in the employee table. The department field is the schema arrow
/// `Employee -> Department`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct EmployeeRecord {
    id: EmployeeId,
    department: DepartmentId,
}

impl EmployeeRecord {
    pub fn new(id: EmployeeId, department: DepartmentId) -> Self {
        Self { id, department }
    }

    pub fn id(&self) -> EmployeeId {
        self.id
    }

    pub fn department(&self) -> DepartmentId {
        self.department
    }
}

/// A tiny database instance: schema objects become sets of ids, and schema
/// arrows become functions between those sets.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CompanyInstance {
    departments: Vec<DepartmentId>,
    employees: Vec<EmployeeRecord>,
}

impl CompanyInstance {
    pub fn new(
        departments: impl IntoIterator<Item = DepartmentId>,
        employees: impl IntoIterator<Item = EmployeeRecord>,
    ) -> CtResult<Self> {
        let departments = departments.into_iter().collect::<Vec<_>>();
        let employees = employees.into_iter().collect::<Vec<_>>();

        for employee in &employees {
            if !departments.contains(&employee.department()) {
                return Err(CtError::ShapeMismatch {
                    op: "database instance",
                    expected: String::from("employee departments exist in Department"),
                    got: format!("missing department {}", employee.department().value()),
                });
            }
        }

        Ok(Self {
            departments,
            employees,
        })
    }

    pub fn departments(&self) -> &[DepartmentId] {
        &self.departments
    }

    pub fn employees(&self) -> &[EmployeeRecord] {
        &self.employees
    }

    pub fn department_of(&self, employee_id: EmployeeId) -> Option<DepartmentId> {
        self.employees
            .iter()
            .find(|employee| employee.id() == employee_id)
            .map(EmployeeRecord::department)
    }

    pub fn foreign_keys_resolve(&self) -> bool {
        self.employees
            .iter()
            .all(|employee| self.departments.contains(&employee.department()))
    }
}

/// Requests per second needed by a design.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct Throughput(usize);

impl Throughput {
    pub fn new(value: usize) -> CtResult<Self> {
        if value == 0 {
            return Err(CtError::EmptyInput("throughput"));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Millisecond latency boundary.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct LatencyMs(usize);

impl LatencyMs {
    pub fn new(value: usize) -> CtResult<Self> {
        if value == 0 {
            return Err(CtError::EmptyInput("latency"));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Functional need in a co-design problem.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct DesignRequirement {
    minimum_throughput: Throughput,
    maximum_latency: LatencyMs,
}

impl DesignRequirement {
    pub fn new(minimum_throughput: Throughput, maximum_latency: LatencyMs) -> Self {
        Self {
            minimum_throughput,
            maximum_latency,
        }
    }
}

/// Implementation offered by a candidate component.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ImplementationOffer {
    throughput: Throughput,
    latency: LatencyMs,
}

impl ImplementationOffer {
    pub fn new(throughput: Throughput, latency: LatencyMs) -> Self {
        Self {
            throughput,
            latency,
        }
    }
}

/// A Bool-valued feasibility relation between requirements and offers.
#[derive(Debug, Clone, Copy)]
pub struct FeasibilityRelation;

impl FeasibilityRelation {
    pub fn relates(requirement: DesignRequirement, offer: ImplementationOffer) -> bool {
        offer.throughput >= requirement.minimum_throughput
            && offer.latency <= requirement.maximum_latency
    }
}

/// A scalar coefficient in a signal-flow matrix.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct SignalCoefficient(i32);

impl SignalCoefficient {
    pub fn new(value: i32) -> Self {
        Self(value)
    }

    pub fn zero() -> Self {
        Self(0)
    }

    pub fn value(&self) -> i32 {
        self.0
    }

    fn add(self, other: Self) -> Self {
        Self(self.value() + other.value())
    }

    fn multiply(self, other: Self) -> Self {
        Self(self.value() * other.value())
    }
}

/// Number of rows in a signal-flow matrix.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct MatrixRows(usize);

impl MatrixRows {
    pub fn new(value: usize) -> CtResult<Self> {
        if value == 0 {
            return Err(CtError::EmptyInput("matrix rows"));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Number of columns in a signal-flow matrix.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct MatrixCols(usize);

impl MatrixCols {
    pub fn new(value: usize) -> CtResult<Self> {
        if value == 0 {
            return Err(CtError::EmptyInput("matrix columns"));
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Matrix semantics for a signal-flow graph.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SignalMatrix {
    rows: MatrixRows,
    cols: MatrixCols,
    coefficients: Vec<Vec<SignalCoefficient>>,
}

impl SignalMatrix {
    pub fn new(
        rows: MatrixRows,
        cols: MatrixCols,
        coefficients: Vec<Vec<SignalCoefficient>>,
    ) -> CtResult<Self> {
        if coefficients.len() != rows.value() {
            return Err(CtError::ShapeMismatch {
                op: "signal matrix",
                expected: format!("{} rows", rows.value()),
                got: format!("{} rows", coefficients.len()),
            });
        }

        for row in &coefficients {
            if row.len() != cols.value() {
                return Err(CtError::ShapeMismatch {
                    op: "signal matrix",
                    expected: format!("{} columns", cols.value()),
                    got: format!("{} columns", row.len()),
                });
            }
        }

        Ok(Self {
            rows,
            cols,
            coefficients,
        })
    }

    pub fn rows(&self) -> MatrixRows {
        self.rows
    }

    pub fn cols(&self) -> MatrixCols {
        self.cols
    }

    pub fn coefficients(&self) -> &[Vec<SignalCoefficient>] {
        &self.coefficients
    }

    /// Matrix composition. If `previous` is `A -> B` and `self` is `B -> C`,
    /// the result is `A -> C`.
    pub fn compose_after(&self, previous: &Self) -> CtResult<Self> {
        if previous.rows.value() != self.cols.value() {
            return Err(CtError::ShapeMismatch {
                op: "signal matrix composition",
                expected: format!("middle dimension {}", self.cols.value()),
                got: format!("middle dimension {}", previous.rows.value()),
            });
        }

        let mut coefficients =
            vec![vec![SignalCoefficient::zero(); previous.cols.value()]; self.rows.value()];

        for (output_row, output_coefficients) in coefficients.iter_mut().enumerate() {
            for (input_col, coefficient) in output_coefficients.iter_mut().enumerate() {
                let mut total = SignalCoefficient::zero();

                for middle in 0..self.cols.value() {
                    total = total.add(
                        self.coefficients[output_row][middle]
                            .multiply(previous.coefficients[middle][input_col]),
                    );
                }

                *coefficient = total;
            }
        }

        Self::new(self.rows, previous.cols, coefficients)
    }
}

/// A named boundary port in an open circuit.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct PortName(&'static str);

impl PortName {
    pub fn new(value: &'static str) -> CtResult<Self> {
        if value.trim().is_empty() {
            return Err(CtError::EmptyInput("port name"));
        }

        Ok(Self(value))
    }

    pub fn as_str(&self) -> &'static str {
        self.0
    }
}

/// Positive resistance value.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ResistanceOhms(usize);

impl ResistanceOhms {
    pub fn new(value: usize) -> CtResult<Self> {
        if value == 0 {
            return Err(CtError::InvalidQuantity {
                kind: "resistance",
                value: value as i64,
            });
        }

        Ok(Self(value))
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// A simple resistor component between two ports.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct CircuitComponent {
    from: PortName,
    to: PortName,
    resistance: ResistanceOhms,
}

impl CircuitComponent {
    pub fn resistor(from: PortName, to: PortName, resistance: ResistanceOhms) -> Self {
        Self {
            from,
            to,
            resistance,
        }
    }

    pub fn from(&self) -> PortName {
        self.from
    }

    pub fn to(&self) -> PortName {
        self.to
    }

    pub fn resistance(&self) -> ResistanceOhms {
        self.resistance
    }
}

/// An open circuit with input ports, output ports, and internal components.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OpenCircuit {
    inputs: Vec<PortName>,
    outputs: Vec<PortName>,
    components: Vec<CircuitComponent>,
}

impl OpenCircuit {
    pub fn new(
        inputs: impl IntoIterator<Item = PortName>,
        outputs: impl IntoIterator<Item = PortName>,
        components: impl IntoIterator<Item = CircuitComponent>,
    ) -> CtResult<Self> {
        let inputs = inputs.into_iter().collect::<Vec<_>>();
        let outputs = outputs.into_iter().collect::<Vec<_>>();
        let components = components.into_iter().collect::<Vec<_>>();

        if inputs.is_empty() {
            return Err(CtError::EmptyInput("circuit inputs"));
        }

        if outputs.is_empty() {
            return Err(CtError::EmptyInput("circuit outputs"));
        }

        Ok(Self {
            inputs,
            outputs,
            components,
        })
    }

    pub fn input_count(&self) -> usize {
        self.inputs.len()
    }

    pub fn output_count(&self) -> usize {
        self.outputs.len()
    }

    pub fn component_count(&self) -> usize {
        self.components.len()
    }

    /// Serial composition wires this circuit's outputs into the next circuit's
    /// inputs.
    pub fn then(&self, next: &Self) -> CtResult<Self> {
        if self.output_count() != next.input_count() {
            return Err(CtError::ShapeMismatch {
                op: "open circuit serial composition",
                expected: format!("{} next inputs", self.output_count()),
                got: format!("{} next inputs", next.input_count()),
            });
        }

        let mut components = self.components.clone();
        components.extend_from_slice(&next.components);

        Self::new(self.inputs.clone(), next.outputs.clone(), components)
    }

    /// Parallel composition keeps the two open interfaces side by side.
    pub fn parallel(&self, other: &Self) -> CtResult<Self> {
        let mut inputs = self.inputs.clone();
        inputs.extend_from_slice(&other.inputs);

        let mut outputs = self.outputs.clone();
        outputs.extend_from_slice(&other.outputs);

        let mut components = self.components.clone();
        components.extend_from_slice(&other.components);

        Self::new(inputs, outputs, components)
    }
}

/// Truth values used for behavior classification.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TruthValue {
    False,
    True,
}

impl TruthValue {
    pub fn and(self, other: Self) -> Self {
        match (self, other) {
            (TruthValue::True, TruthValue::True) => TruthValue::True,
            _ => TruthValue::False,
        }
    }

    pub fn implies(self, other: Self) -> Self {
        match (self, other) {
            (TruthValue::True, TruthValue::False) => TruthValue::False,
            _ => TruthValue::True,
        }
    }
}

/// Discrete time point in a behavior trace.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct TimeTick(usize);

impl TimeTick {
    pub fn new(value: usize) -> Self {
        Self(value)
    }

    pub fn value(&self) -> usize {
        self.0
    }
}

/// Closed interval of time ticks.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct TimeInterval {
    start: TimeTick,
    end: TimeTick,
}

impl TimeInterval {
    pub fn new(start: TimeTick, end: TimeTick) -> CtResult<Self> {
        if start > end {
            return Err(CtError::InvalidInterval {
                start: start.value(),
                end: end.value(),
            });
        }

        Ok(Self { start, end })
    }

    pub fn start(&self) -> TimeTick {
        self.start
    }

    pub fn end(&self) -> TimeTick {
        self.end
    }
}

/// Local safety result on one interval.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct LocalSafetyCheck {
    interval: TimeInterval,
    truth: TruthValue,
}

impl LocalSafetyCheck {
    pub fn new(interval: TimeInterval, truth: TruthValue) -> Self {
        Self { interval, truth }
    }

    pub fn interval(&self) -> TimeInterval {
        self.interval
    }

    pub fn truth(&self) -> TruthValue {
        self.truth
    }
}

/// A cover of local behavior checks. The global result is true only when all
/// local checks are true.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SafetyCover(Vec<LocalSafetyCheck>);

impl SafetyCover {
    pub fn new(checks: impl IntoIterator<Item = LocalSafetyCheck>) -> CtResult<Self> {
        let checks = checks.into_iter().collect::<Vec<_>>();

        if checks.is_empty() {
            return Err(CtError::EmptyInput("safety cover"));
        }

        Ok(Self(checks))
    }

    pub fn global_truth(&self) -> TruthValue {
        self.0
            .iter()
            .fold(TruthValue::True, |truth, check| truth.and(check.truth()))
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn information_order_has_preorder_laws() {
        assert!(information_order_obeys_preorder_laws());
        assert_eq!(
            InformationLevel::Feature.join(InformationLevel::Decision),
            InformationLevel::Decision
        );
    }

    #[test]
    fn feature_layer_abstraction_obeys_galois_law() -> CtResult<()> {
        let features = FeatureCount::new(9)?;
        let layers = LayerBudget::new(3)?;

        assert!(feature_layer_galois_law_holds(features, layers)?);
        assert_eq!(abstract_to_layer_budget(features)?.value(), 3);
        assert_eq!(concretize_layer_budget(layers).value(), 12);
        Ok(())
    }

    #[test]
    fn resource_tensor_is_order_preserving() {
        assert!(resource_tensor_is_monotone());
    }

    #[test]
    fn database_instance_resolves_schema_arrow() -> CtResult<()> {
        let research = DepartmentId::new(1);
        let platform = DepartmentId::new(2);
        let ada = EmployeeId::new(7);
        let instance =
            CompanyInstance::new([research, platform], [EmployeeRecord::new(ada, research)])?;

        assert!(instance.foreign_keys_resolve());
        assert_eq!(instance.department_of(ada), Some(research));
        Ok(())
    }

    #[test]
    fn database_instance_rejects_missing_department_reference() {
        let research = DepartmentId::new(1);
        let missing_department = DepartmentId::new(99);
        let ada = EmployeeId::new(7);

        assert!(matches!(
            CompanyInstance::new([research], [EmployeeRecord::new(ada, missing_department)]),
            Err(CtError::ShapeMismatch {
                op: "database instance",
                ..
            })
        ));
    }

    #[test]
    fn feasibility_relation_matches_requirement_to_offer() -> CtResult<()> {
        let requirement = DesignRequirement::new(Throughput::new(100)?, LatencyMs::new(80)?);
        let offer = ImplementationOffer::new(Throughput::new(120)?, LatencyMs::new(50)?);

        assert!(FeasibilityRelation::relates(requirement, offer));
        Ok(())
    }

    #[test]
    fn signal_matrices_compose_like_flow_graph_semantics() -> CtResult<()> {
        let duplicate = SignalMatrix::new(
            MatrixRows::new(2)?,
            MatrixCols::new(1)?,
            vec![
                vec![SignalCoefficient::new(1)],
                vec![SignalCoefficient::new(1)],
            ],
        )?;
        let add_weighted = SignalMatrix::new(
            MatrixRows::new(1)?,
            MatrixCols::new(2)?,
            vec![vec![SignalCoefficient::new(2), SignalCoefficient::new(3)]],
        )?;

        let composed = add_weighted.compose_after(&duplicate)?;

        assert_eq!(composed.coefficients(), &[vec![SignalCoefficient::new(5)]]);
        Ok(())
    }

    #[test]
    fn signal_matrix_composition_rejects_mismatched_middle_dimension() -> CtResult<()> {
        let previous = SignalMatrix::new(
            MatrixRows::new(2)?,
            MatrixCols::new(1)?,
            vec![
                vec![SignalCoefficient::new(1)],
                vec![SignalCoefficient::new(1)],
            ],
        )?;
        let next = SignalMatrix::new(
            MatrixRows::new(1)?,
            MatrixCols::new(3)?,
            vec![vec![
                SignalCoefficient::new(1),
                SignalCoefficient::new(1),
                SignalCoefficient::new(1),
            ]],
        )?;

        assert!(matches!(
            next.compose_after(&previous),
            Err(CtError::ShapeMismatch {
                op: "signal matrix composition",
                ..
            })
        ));
        Ok(())
    }

    #[test]
    fn open_circuits_compose_in_series_and_parallel() -> CtResult<()> {
        let input = PortName::new("input")?;
        let middle = PortName::new("middle")?;
        let output = PortName::new("output")?;
        let first = OpenCircuit::new(
            [input],
            [middle],
            [CircuitComponent::resistor(
                input,
                middle,
                ResistanceOhms::new(10)?,
            )],
        )?;
        let second = OpenCircuit::new(
            [middle],
            [output],
            [CircuitComponent::resistor(
                middle,
                output,
                ResistanceOhms::new(20)?,
            )],
        )?;

        let serial = first.then(&second)?;
        let parallel = first.parallel(&second)?;

        assert_eq!(serial.input_count(), 1);
        assert_eq!(serial.output_count(), 1);
        assert_eq!(serial.component_count(), 2);
        assert_eq!(parallel.input_count(), 2);
        assert_eq!(parallel.output_count(), 2);
        Ok(())
    }

    #[test]
    fn open_circuit_serial_composition_rejects_boundary_mismatch() -> CtResult<()> {
        let input = PortName::new("input")?;
        let output = PortName::new("output")?;
        let left = OpenCircuit::new([input], [output], [])?;
        let right = OpenCircuit::new([input, output], [output], [])?;

        assert!(matches!(
            left.then(&right),
            Err(CtError::ShapeMismatch {
                op: "open circuit serial composition",
                ..
            })
        ));
        Ok(())
    }

    #[test]
    fn local_behavior_truth_glues_to_global_truth() -> CtResult<()> {
        let first = LocalSafetyCheck::new(
            TimeInterval::new(TimeTick::new(0), TimeTick::new(5))?,
            TruthValue::True,
        );
        let second = LocalSafetyCheck::new(
            TimeInterval::new(TimeTick::new(5), TimeTick::new(10))?,
            TruthValue::True,
        );
        let cover = SafetyCover::new([first, second])?;

        assert_eq!(cover.global_truth(), TruthValue::True);
        assert_eq!(
            TruthValue::True.implies(TruthValue::False),
            TruthValue::False
        );
        Ok(())
    }
}

The runnable companion:

use category_theory_transformer_rs::{
    CircuitComponent, CompanyInstance, CtResult, DepartmentId, DesignRequirement, EmployeeId,
    EmployeeRecord, FeasibilityRelation, FeatureCount, ImplementationOffer, InformationLevel,
    LatencyMs, LayerBudget, LocalSafetyCheck, MatrixCols, MatrixRows, OpenCircuit, PortName,
    ResistanceOhms, ResourceAmount, ResourceBundle, SafetyCover, SignalCoefficient, SignalMatrix,
    Throughput, TimeInterval, TimeTick, TruthValue, abstract_to_layer_budget,
    concretize_layer_budget, feature_layer_galois_law_holds, information_order_obeys_preorder_laws,
    resource_tensor_is_monotone,
};

fn main() -> CtResult<()> {
    println!(
        "orders obey preorder laws: {}",
        information_order_obeys_preorder_laws()
    );
    println!(
        "join(feature, decision): {:?}",
        InformationLevel::Feature.join(InformationLevel::Decision)
    );

    let features = FeatureCount::new(9)?;
    let layers = LayerBudget::new(3)?;
    println!(
        "feature/layer Galois law: {}",
        feature_layer_galois_law_holds(features, layers)?
    );
    println!(
        "abstract 9 features to {} layers; concretize 3 layers to {} features",
        abstract_to_layer_budget(features)?.value(),
        concretize_layer_budget(layers).value()
    );

    let encoder = ResourceBundle::new(ResourceAmount::new(2), ResourceAmount::new(8));
    let decoder = ResourceBundle::new(ResourceAmount::new(3), ResourceAmount::new(10));
    println!("combined resource bundle: {:?}", encoder.tensor(&decoder));
    println!(
        "resource tensor monotone: {}",
        resource_tensor_is_monotone()
    );

    let research = DepartmentId::new(1);
    let platform = DepartmentId::new(2);
    let ada = EmployeeId::new(7);
    let instance =
        CompanyInstance::new([research, platform], [EmployeeRecord::new(ada, research)])?;
    println!(
        "employee {:?} belongs to department {:?}",
        ada,
        instance.department_of(ada)
    );

    let requirement = DesignRequirement::new(Throughput::new(100)?, LatencyMs::new(80)?);
    let offer = ImplementationOffer::new(Throughput::new(120)?, LatencyMs::new(50)?);
    println!(
        "co-design offer feasible: {}",
        FeasibilityRelation::relates(requirement, offer)
    );

    let duplicate = SignalMatrix::new(
        MatrixRows::new(2)?,
        MatrixCols::new(1)?,
        vec![
            vec![SignalCoefficient::new(1)],
            vec![SignalCoefficient::new(1)],
        ],
    )?;
    let add_weighted = SignalMatrix::new(
        MatrixRows::new(1)?,
        MatrixCols::new(2)?,
        vec![vec![SignalCoefficient::new(2), SignalCoefficient::new(3)]],
    )?;
    let composed = add_weighted.compose_after(&duplicate)?;
    println!(
        "signal-flow matrix semantics: {:?}",
        composed.coefficients()
    );

    let input = PortName::new("input")?;
    let middle = PortName::new("middle")?;
    let output = PortName::new("output")?;
    let first_circuit = OpenCircuit::new(
        [input],
        [middle],
        [CircuitComponent::resistor(
            input,
            middle,
            ResistanceOhms::new(10)?,
        )],
    )?;
    let second_circuit = OpenCircuit::new(
        [middle],
        [output],
        [CircuitComponent::resistor(
            middle,
            output,
            ResistanceOhms::new(20)?,
        )],
    )?;
    println!(
        "serial circuit component count: {}",
        first_circuit.then(&second_circuit)?.component_count()
    );

    let safety = SafetyCover::new([
        LocalSafetyCheck::new(
            TimeInterval::new(TimeTick::new(0), TimeTick::new(5))?,
            TruthValue::True,
        ),
        LocalSafetyCheck::new(
            TimeInterval::new(TimeTick::new(5), TimeTick::new(10))?,
            TruthValue::True,
        ),
    ])?;
    println!("global behavior truth: {:?}", safety.global_truth());
    println!();
    println!("Typed transformation:");
    println!("InformationLevel <= InformationLevel checks preorder");
    println!("FeatureCount <-> LayerBudget checks Galois law");
    println!("ResourceBundle x ResourceBundle -> ResourceBundle");
    println!("EmployeeRecord -> DepartmentId must resolve in CompanyInstance");
    println!("DesignRequirement x ImplementationOffer -> bool");
    println!("SignalMatrix x SignalMatrix -> SignalMatrix when dimensions match");
    println!("OpenCircuit x OpenCircuit -> OpenCircuit when ports match");
    println!("SafetyCover -> TruthValue");

    Ok(())
}

Paper Map To Rust

Use this table as the navigation layer.

Each section below uses the same three-part lens:

Rust syntax
ML or software concept
Category theory concept

Source Scope Contract

This chapter is a study companion, not a replacement for the source text. Each Rust model preserves one inspectable idea and deliberately leaves the larger mathematical development outside the tiny example.

Use the table as a precision guard. When the Rust code checks one law, say which law it checks. When the source paper develops a larger theory, do not pretend the small Rust model has implemented all of it.

PDF-To-Rust Reading Contract

The arXiv record describes Seven Sketches in Compositionality as a long invitation to applied category theory, built around concrete examples and seven major sketches. That matters for how to use this chapter. The goal is not to compress every page into a smaller page. The goal is to give every major sketch a Rust handle that a reader can run, inspect, and test.

Complete coverage in this companion chapter means:

every major sketch area has a Rust handle;
every Rust handle names one protected law, relation, or boundary;
every protected claim says what the source text still develops beyond the code;
every reader can run one command before arguing about the abstraction.

Use this ledger while reading the PDF beside the Rust:

Page-To-Rust Decision Ladder

Use this ladder when you are reading the PDF page by page and do not yet know what to implement.

The ladder keeps the reading order concrete:

source paragraph shape
  -> first Rust move
  -> local evidence
  -> safe non-claim

Example decisions:

This is how the chapter can cover the whole source at the level promised by the book: not every theorem becomes a library feature, but every major reading shape has a disciplined path toward a Rust handle, a test, and a non-claim.

This is also the limit of the chapter. If a PDF section develops a richer construction than the Rust handle, record the richer construction as context, not as something the code has proved. The safe sentence is:

The source develops a larger theory here.
This Rust handle checks one executable boundary from that theory.

Source-Backed Precision Rules

This chapter uses external sources as scope guards. Each source supports a limited teaching claim, and each claim is tied to one local Rust boundary or test. The chapter does not claim that src/sketches.rs implements the full source text, a general categorical semantics library, or a production ML architecture theory.

The Bridge Back To Tiny ML section below is part of this source contract. Its check sentence is:

This sketch helps me reject this ML shortcut.

The transfer pattern is:

source idea -> local Rust model -> law or boundary check

For this chapter, that means reading cargo run --example 05_seven_sketches and cargo test sketches::tests as evidence for the tiny models above, not as evidence that the full seven-sketch source text, categorical deep-learning literature, or every applied-category construction has been implemented.

Choose A Sketch Without Losing The Tiny ML Thread

The source paper deliberately tours many application areas. This companion chapter keeps that breadth, but the book still has one main learning path: small typed systems that make ML structure inspectable.

Use this table to decide how deeply to read each sketch on a first pass.

On a first reading, focus on the rows marked core transfer. They connect most directly to the tiny ML pipeline. The optional rows are not less important; they are just farther from the first runnable model.

The chapter is successful if you can leave each sketch with one sentence:

This Rust model prevents this invalid composition.

Worked Example: Ordering Information Levels

The smallest first-principles version of this chapter is an ordered enum. Rust can derive an order for enum variants, and that gives the code a concrete way to ask whether one information level can safely flow into another:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
enum Level {
    Observation,
    Feature,
    Decision,
}

assert!(Level::Observation <= Level::Decision);
assert!(Level::Feature <= Level::Feature);
}

The real src/sketches.rs module uses the same idea with named domain types, validation, and law checks.

Self-Check

Before reading the first sketch, explain why Observation <= Decision is a modeling rule, not just a comparison between enum variants.

One Method Across Seven Sketches

Do not read the chapter as seven unrelated theory notes. Each sketch follows the same modeling method:

engineering problem
  -> named Rust values
  -> validated construction
  -> composition operation or relation
  -> law or boundary test

That method is the same one used in the tiny ML pipeline. The difference is the domain. Instead of tokens, logits, and parameters, this chapter uses resources, database rows, signal matrices, open circuits, and local behavior checks.

When a sketch feels abstract, ask one question:

What invalid composition should this model prevent?

The answer usually points to the real engineering value of the categorical shape.

The chapter’s applied models can be scanned by the structure they protect:

This table is the chapter’s law-and-boundary index. The point is not to memorize eight rows. The point is to see that each sketch earns its abstraction by protecting one concrete relationship.

Bridge Back To Tiny ML

The source text and the MIT course both use applied examples to make category theory portable across domains. This chapter uses the same move in a smaller way: each sketch should transfer back to one tiny ML design pressure.

Use the bridge below when the chapter starts to feel like a separate category theory tour.

Read each row as a transfer sentence:

This sketch helps me reject this ML shortcut.

That sentence is the practical reason to keep the sketch in the book. The formal category-theory vocabulary matters only after the engineering shortcut is visible.

Transfer Triage Card

Use this card when a source idea feels too large to turn into code. The goal is not to shrink the source. The goal is to choose one local boundary that can be inspected.

The completed transfer card has seven fields:

source idea:
local Rust handle:
protected law, relation, or boundary:
invalid shortcut rejected:
tiny ML transfer:
larger claim not implemented:
local evidence command or test:

Example:

source idea: schemas and instances
local Rust handle: CompanyInstance::new
protected law, relation, or boundary: EmployeeRecord -> DepartmentId must resolve
invalid shortcut rejected: letting a missing department reach feature extraction
tiny ML transfer: validate structured training rows before training
larger claim not implemented: a general categorical database semantics
local evidence command or test: cargo test sketches::tests --lib

Second example:

source idea: open systems have boundaries
local Rust handle: OpenCircuit::then
protected law, relation, or boundary: previous outputs must match next inputs
invalid shortcut rejected: wiring components by label while ignoring shape
tiny ML transfer: Tokenizer -> Embedder is legal only when the output object
matches the next input object
larger claim not implemented: decorated cospans, hypergraph categories, and
operads for general circuit composition
local evidence command or test: cargo test sketches::tests --lib

This follows the source discipline used above. Seven Sketches gives a broad tour through concrete examples. MIT’s course frames category theory as a way to organize formal systems and transfer knowledge between them. Categorical deep-learning work distinguishes architecture constraints from implementations. The local job here is smaller: choose one implementable handle, name one protected relationship, and state the non-claim.

Sketch 1: Information Order

The problem this block solves is:

Some concepts are ordered by refinement. An observation can be refined into a feature, a feature into a score, and a score into a decision.

The block begins:

#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub enum InformationLevel {
    Observation,
    Feature,
    Score,
    Decision,
}

Rust Syntax

This is an enum.

The variants are ordered because the enum derives:

PartialOrd, Ord

That means Rust can compare:

InformationLevel::Observation <= InformationLevel::Decision

The methods:

pub fn can_flow_to(self, target: Self) -> bool {
    self <= target
}

pub fn join(self, other: Self) -> Self {
    self.max(other)
}

reuse that ordering.

can_flow_to checks whether information can move upward.

join returns the more informative of two levels in this total order.

ML Or Software Concept

ML systems often move through levels of processed information:

raw observation
  -> extracted feature
  -> model score
  -> final decision

The order prevents treating a low-level observation as if it were already a decision.

Category Theory Concept

This is a preorder-shaped example.

A preorder needs:

reflexivity: a <= a
transitivity: if a <= b and b <= c, then a <= c

The law check:

information_order_obeys_preorder_laws()

iterates over the finite set and verifies those rules.

Transfer Task: Ordered States

Model a workflow from your own codebase as an ordered enum, such as:

enum ReviewState {
    Draft,
    Reviewed,
    Published,
}

Then write the Rust sentence you would want to be true:

Draft can flow to Published
Published cannot flow to Draft
state can always flow to itself

The transfer is complete when you can name the invalid flow your order prevents.

Sketch 1 Continued: Feature And Layer Galois Law

The problem this block solves is:

A concrete feature count and an abstract layer budget are different worlds, but they can be coordinated by a law.

The key types:

pub struct FeatureCount(usize);
pub struct LayerBudget(usize);

The conversion functions:

pub fn abstract_to_layer_budget(features: FeatureCount) -> CtResult<LayerBudget>

pub fn concretize_layer_budget(layers: LayerBudget) -> FeatureCount

Rust Syntax

Both FeatureCount and LayerBudget are newtypes around usize.

Their constructors reject zero because neither a zero feature count nor a zero layer budget is useful in this model.

abstract_to_layer_budget divides feature count by FEATURES_PER_LAYER and rounds up:

features.value().div_ceil(FEATURES_PER_LAYER)

concretize_layer_budget multiplies layers by features per layer.

ML Or Software Concept

This models capacity planning.

Concrete features might be:

9 measured feature channels

The abstract layer budget might be:

3 layers

The law says the two planning views agree about what fits.

Category Theory Concept

The checked law is:

abstract(features) <= layers
if and only if
features <= concretize(layers)

That is the shape of a Galois connection.

The two directions are not inverses.

They are coordinated by an order law.

Transfer Task: Concrete And Abstract Capacity

Pick one concrete measure and one abstract budget from software work:

concrete: batch size, feature count, request rate
abstract: GPU count, layer budget, service tier

Write the two functions:

abstract(concrete) -> abstract budget
concretize(abstract budget) -> concrete capacity

The transfer is complete when you can state the law in both directions:

abstract(concrete) fits budget
if and only if
concrete fits concretize(budget)

Sketch 2: Resources

The problem this block solves is:

Independent resources need a way to combine, and resource supply must respect demand ordering.

The key block:

pub struct ResourceBundle {
    compute: ResourceAmount,
    memory: ResourceAmount,
}

Rust Syntax

ResourceAmount wraps a usize.

ResourceBundle stores two resource dimensions:

compute
memory

The monoidal operation is:

pub fn tensor(&self, other: &Self) -> Self

It adds compute to compute and memory to memory.

The preorder check is:

pub fn can_supply(&self, demand: &Self) -> bool

It returns true only when every resource component is large enough.

ML Or Software Concept

This is the shape of deployment capacity:

encoder resources + decoder resources = combined resources

A machine can satisfy a demand only when it has enough compute and memory.

Category Theory Concept

This is a monoidal preorder.

The preorder is can_supply.

The monoidal product is tensor.

The law check:

resource_tensor_is_monotone()

shows that adding the same fixed resource bundle to both sides preserves the order.

Transfer Task: Resource Bundle

Extend the idea to three resource dimensions:

compute
memory
disk

Name the constructor boundary and the law check:

ResourceBundle::new(compute, memory, disk)
resource_tensor_is_monotone

The transfer is complete when you can explain why componentwise addition should not turn an adequate supply into an inadequate supply.

Sketch 3: Database Instance

The problem this block solves is:

A database row with a foreign key should not point at a missing row.

The key types:

pub struct DepartmentId(usize);
pub struct EmployeeId(usize);

pub struct EmployeeRecord {
    id: EmployeeId,
    department: DepartmentId,
}

pub struct CompanyInstance {
    departments: Vec<DepartmentId>,
    employees: Vec<EmployeeRecord>,
}

Rust Syntax

DepartmentId and EmployeeId are distinct newtypes.

That prevents mixing department IDs and employee IDs.

EmployeeRecord contains:

employee id
department id

CompanyInstance::new collects departments and employees, then checks every employee department exists:

if !departments.contains(&employee.department()) {
    return Err(CtError::ShapeMismatch { ... });
}

ML Or Software Concept

Many ML systems depend on structured data.

If a row references missing data, downstream feature extraction or training can fail later in a confusing place.

This code rejects invalid relational structure at construction time.

Category Theory Concept

The schema can be read as:

Employee -> Department

An instance assigns sets of rows to schema objects.

The foreign key is a function from employees to departments.

CompanyInstance::new checks that the function is defined for every employee.

Transfer Task: Foreign Key Boundary

Translate the pattern to another schema arrow:

Task -> Project
Order -> Customer
Comment -> Post

Name two newtypes and one record:

ProjectId
TaskId
TaskRecord { id: TaskId, project: ProjectId }

The transfer is complete when you can write the constructor failure in plain English: “reject a task whose project id is missing from the project table.”

Sketch 4: Co-Design Feasibility

The problem this block solves is:

Some relationships are not functions. A requirement and an implementation offer are related only when constraints are satisfied.

The key types:

pub struct DesignRequirement {
    minimum_throughput: Throughput,
    maximum_latency: LatencyMs,
}

pub struct ImplementationOffer {
    throughput: Throughput,
    latency: LatencyMs,
}

pub struct FeasibilityRelation;

Rust Syntax

Throughput and LatencyMs are validated newtypes.

DesignRequirement stores the minimum acceptable throughput and maximum acceptable latency.

ImplementationOffer stores what an implementation actually provides.

The relation is:

pub fn relates(requirement: DesignRequirement, offer: ImplementationOffer) -> bool {
    offer.throughput >= requirement.minimum_throughput
        && offer.latency <= requirement.maximum_latency
}

ML Or Software Concept

This models feasibility:

Can this model/service/deployment satisfy this requirement?

For example:

required throughput: at least 100 requests/sec
required latency: at most 80 ms
offer: 120 requests/sec and 50 ms

The offer is feasible.

Category Theory Concept

This is relation-shaped rather than function-shaped.

It is the small Bool-valued version of profunctor-like reasoning:

Requirement x Offer -> Bool

Not every design problem should be forced into:

A -> B

Some should be modeled as constraints or relations.

This also gives a small handle for reading newer categorical deep-learning work. At architecture scale, a model can be discussed in terms of constraints it should satisfy and implementations that realize those constraints. This Rust sketch keeps only the first tiny version of that idea:

DesignRequirement x ImplementationOffer -> bool

The important distinction is:

That is not a full theory of neural architectures. It is the learner-sized boundary that prevents a common overclaim: one implementation example is not the same thing as the whole constraint space.

Transfer Task: Feasibility Relation

Model a relation that is not a function:

Requirement x Offer -> Bool

For example, use:

minimum accuracy
maximum memory
maximum latency

Then write the architecture version:

ArchitectureConstraint x CandidateImplementation -> Bool

The transfer is complete when you can explain why many offers may satisfy one requirement, one offer may satisfy many requirements, and one passing offer is not proof that every future implementation satisfies the architecture constraint.

Sketch 5: Signal Matrices

The problem this block solves is:

Signal-flow diagrams need executable semantics. In this companion, matrices provide that meaning.

The key types:

pub struct SignalCoefficient(i32);
pub struct MatrixRows(usize);
pub struct MatrixCols(usize);

pub struct SignalMatrix {
    rows: MatrixRows,
    cols: MatrixCols,
    coefficients: Vec<Vec<SignalCoefficient>>,
}

Rust Syntax

MatrixRows and MatrixCols reject zero.

SignalMatrix::new validates that the coefficient matrix has the promised shape:

number of rows matches MatrixRows
number of columns in every row matches MatrixCols

The composition method:

pub fn compose_after(&self, previous: &Self) -> CtResult<Self>

requires compatible middle dimensions.

Then it performs matrix multiplication using:

add
multiply
sum over the middle dimension

ML Or Software Concept

This is the same shape as composing linear layers or signal-processing stages.

If one stage maps:

A -> B

and another maps:

B -> C

then the composite maps:

A -> C

The dimensions must line up.

Category Theory Concept

Signal-flow syntax gets matrix semantics.

The important principle is functorial semantics:

meaning(composed diagram)
=
composition of meanings

The code enforces the same middle-dimension law that ordinary morphism composition enforces.

Transfer Task: Shape-Safe Composition

Write the shape of two stages before writing any coefficients:

A -> B
B -> C

Then write one invalid pair:

A -> B
D -> C

The transfer is complete when you can name the exact middle dimension that must match for matrix composition to make sense.

Sketch 6: Open Circuits

The problem this block solves is:

A circuit is not only internal components. It also has a boundary where it can connect to other circuits.

The key types:

pub struct PortName(&'static str);
pub struct ResistanceOhms(usize);

pub struct CircuitComponent {
    from: PortName,
    to: PortName,
    resistance: ResistanceOhms,
}

pub struct OpenCircuit {
    inputs: Vec<PortName>,
    outputs: Vec<PortName>,
    components: Vec<CircuitComponent>,
}

Rust Syntax

PortName::new rejects empty names.

ResistanceOhms::new rejects zero resistance.

OpenCircuit::new rejects circuits with no inputs or no outputs.

Serial composition:

pub fn then(&self, next: &Self) -> CtResult<Self>

checks:

self output count == next input count

Parallel composition:

pub fn parallel(&self, other: &Self) -> CtResult<Self>

puts the two boundaries side by side.

ML Or Software Concept

This looks like component architecture:

input interface
internal implementation
output interface

Composition should fail when interfaces do not match.

That rule applies to services, data pipelines, neural layers, and circuit-like systems.

Category Theory Concept

This is the open-system idea.

The boundary is part of the object.

Composition is controlled by boundary compatibility.

The paper develops this with cospans, hypergraph categories, decorated cospans, and operads. The Rust code gives a small typed analogue.

Transfer Task: Interface Boundary

Model two components by boundary only:

Tokenizer: Text -> TokenSequence
Embedder: TokenSequence -> HiddenSequence

Then write one invalid serial composition:

Tokenizer: Text -> TokenSequence
Classifier: Logits -> Label

The transfer is complete when you can say which output boundary failed to match which input boundary.

Sketch 7: Logic Of Behavior

The problem this block solves is:

A system may be safe on local time intervals. The code needs a way to combine local safety checks into one global result.

The key types:

pub enum TruthValue {
    False,
    True,
}

pub struct TimeTick(usize);

pub struct TimeInterval {
    start: TimeTick,
    end: TimeTick,
}

pub struct LocalSafetyCheck {
    interval: TimeInterval,
    truth: TruthValue,
}

pub struct SafetyCover(Vec<LocalSafetyCheck>);

Rust Syntax

TruthValue implements Boolean-style operations:

and
implies

TimeInterval::new rejects intervals where start is after end.

SafetyCover::new rejects an empty list of checks.

global_truth folds all local truths with and:

self.0
    .iter()
    .fold(TruthValue::True, |truth, check| truth.and(check.truth()))

ML Or Software Concept

This models safety or behavior validation over time:

check interval 0..5
check interval 5..10
combine into global result

If every local check is true, global truth is true.

If any local check is false, global truth is false.

Category Theory Concept

This is a small analogue of local-to-global reasoning.

The sheaf-like idea is:

local facts can determine a global fact when they glue coherently

The code uses a simple conjunction model, not full sheaf theory.

The important lesson is that proof-like information becomes explicit data, not an informal comment.

Transfer Task: Local Checks To Global Claim

Pick one invariant over time:

loss is finite
latency stays below the budget
no batch has an empty token sequence

Split it into local intervals and assign a truth value to each interval.

The transfer is complete when you can explain why one false local check must make the global claim false.

Tests As Exercise Solutions

The problem this block solves is:

The laws should be runnable, not only described in prose.

The test module checks preorder laws, the feature/layer Galois law, resource tensor monotonicity, database foreign-key resolution, feasibility relation behavior, signal matrix composition, open circuit serial and parallel composition, and local-to-global truth.

It also includes negative boundary tests. A database instance with a missing department reference is rejected. Signal matrices with incompatible middle dimensions cannot compose. Open circuits with mismatched serial boundaries do not wire together. Those failures are part of the teaching point: the model is useful because it rejects incoherent structure near the boundary.

Rust Syntax

Every law is a normal Rust test marked with:

#[test]

Tests that may fail through constructors return:

CtResult<()>

so they can use ?.

ML Or Software Concept

The tests act as executable learning checks.

If a future change breaks a law, the project should fail quickly.

Category Theory Concept

The tests are small law checks.

They are not formal proofs, but they keep the implementation aligned with the claimed structure.

Run The Companion

Run:

cargo run --example 05_seven_sketches

The output gives one executable handle per sketch:

orders obey preorder laws: true
feature/layer Galois law: true
resource tensor monotone: true
employee EmployeeId(7) belongs to department Some(DepartmentId(1))
co-design offer feasible: true
signal-flow matrix semantics: [[SignalCoefficient(5)]]
serial circuit component count: 2
global behavior truth: True
Typed transformation:
InformationLevel <= InformationLevel checks preorder
FeatureCount <-> LayerBudget checks Galois law
ResourceBundle x ResourceBundle -> ResourceBundle
EmployeeRecord -> DepartmentId must resolve in CompanyInstance
DesignRequirement x ImplementationOffer -> bool
SignalMatrix x SignalMatrix -> SignalMatrix when dimensions match
OpenCircuit x OpenCircuit -> OpenCircuit when ports match
SafetyCover -> TruthValue

Example Output Transfer Checklist

Use the companion output as a boundary map. Each line should tell you which structure is being protected.

The typed lines at the bottom of the output are not extra decoration. They name the form each sketch protects: order, Galois law, monoidal resource composition, database instance, feasibility relation, matrix composition, open system composition, and local-to-global truth.

For the full validation gate:

cargo test --all-targets --all-features

Core Mental Model

In Rust terms:

each sketch becomes concrete types, constructors, methods, and tests

In ML or software terms:

orders, resources, schemas, feasibility, signal flow, interfaces, and safety
are all engineering structures

In category-theory terms:

the useful part is compositionality: make structure visible, then make
composition obey laws

What To Remember

The seven sketches are not seven disconnected topics.

They repeat one engineering discipline:

name the objects
name the relationships
control construction
define composition
check the law

That is also the discipline used by the tiny ML pipeline in the rest of the course.

Where This Leaves Us

This chapter showed that the book’s main discipline is not limited to language modeling. Orders, resources, database instances, feasibility relations, signal matrices, open circuits, and local behavior checks can all be read in the same way: name the values, constrain construction, define composition, and test the law that makes the composition trustworthy.

The remaining practice material in Exercises asks you to use that reading method yourself. The exercises are not meant to test memorized definitions. They are meant to train the habit of translating one Rust block into its software role and its categorical shape.

Further Reading

Do not treat these sources as a separate theory shelf. Use them to improve one local Rust sentence at a time.

Start from this local evidence:

cargo run --example 05_seven_sketches
cargo test sketches::tests --lib
src/sketches.rs
examples/05_seven_sketches.rs

Then read the sources in this order:

After reading one source, answer four questions:

1. Which Rust handle did it clarify?
2. Which law, relation, or boundary did it support?
3. Which larger claim did the source not license?
4. Which command or test shows the local evidence?