MongoDB — modeling, indexing, aggregation, transactions, ops

Engine-level MongoDB 8.2 guidance: model documents for the queries you actually run, pick the index the planner will use, write aggregation pipelines that stay index-eligible, run multi-document transactions with correct retry, and operate/secure a deployment. Driver-agnostic — every example is mongosh shell syntax that maps 1:1 to the official drivers (Node, Python, Go, Java, Rust). This skill owns the server's query and the index it picks, not any ODM's API.

When to use / When NOT to use

When to use:

• Document modeling: embed vs reference, the 16 MB cap, one-to-many/many-to-many, the subset/extended-reference/bucket/computed/outlier patterns, taming unbounded array growth.
• Index decisions: single-field, compound (the ESR ordering rule), multikey, partial, TTL, text, wildcard, 2dsphere; and when an index is NOT worth it.
• Any query that is slow or scans too much; reading explain("executionStats").
• Aggregation pipelines: stage order so $match/$sort hit an index, $lookup cost, $unwind explosion, $group/$sort memory limits and allowDiskUse, $merge/$out, faceting.
• Multi-document transactions: sessions, withTransaction retry semantics, read/write concern.
• Operating/securing: replica set, read preference, write concern, Atlas tier choice, Atlas Search & Vector Search, Queryable Encryption, role-based access, connection-pool knobs.

When NOT to use:

• Relational schema / SQL / EXPLAIN ANALYZE → postgresdb. Different engine, planner, and concurrency model.
• ODM/driver API ergonomics (Mongoose pre-save hooks, the Node driver's bulkWrite return shape, updateMany's result object) → that tool's own docs. This skill owns the server query and the index the server picks, not the JS object the driver hands back.
• App-layer caching as a product (Redis-in-front-of-reads).
• Cloud-console click-paths — we give the shell command / connection string, not the Atlas UI tour.
• Picking a vector store across engines (Pinecone vs Weaviate). Atlas Vector Search inside Mongo is in scope; cross-engine selection is not.

Deep dives: data-modeling (embed/reference tree, all six patterns, 16 MB math, polymorphic & schema versioning) · aggregation (per-stage index eligibility, $lookup variants, $facet, window fns, $merge/$out, reading pipeline explain) · transactions-and-ops (retry wrappers, concern semantics, Atlas tiers, Vector Search, Queryable Encryption, RBAC, pooling, change streams).

Non-negotiables

1. Design for the queries you run, not the shape of your data. The schema is the set of documents that make your common reads single-document and index-eligible.
2. Never let an array grow unbounded inside a document. It walks toward the 16 MB cap, bloats every read of the parent, and kills update performance — reference or bucket it.
3. The hard ceiling is 16 MB per document. If a one-to-many can exceed it, you reference; there is no TOAST-style overflow here.
4. A single-document write is already atomic. Reach for a multi-document transaction only when two or more documents must change together — otherwise you are paying for nothing.
5. Every transaction retries on the TransientTransactionError label (and commit retries on UnknownTransactionCommitResult). withTransaction does both for you; a hand-rolled loop must.
6. Index by ESR: Equality fields, then the Sort field, then Range fields. This order lets one compound index serve the filter, the sort, and the range without an in-memory sort.
7. Read explain("executionStats") before and after adding an index — confirm IXSCAN, not COLLSCAN, and totalKeysExamined ≈ nReturned. Or it didn't happen.
8. w:"majority" for money and state transitions, read concern "majority"/"snapshot" when a read must reflect a durable write. w:1 can be rolled back on a primary failover.
9. Money is Decimal128 (NumberDecimal("...")), never a JS double. Binary floats drift; 0.1 + 0.2 !== 0.3 in your ledger.
10. Never store a secret in plaintext. Use Queryable Encryption / client-side field-level encryption; never commit a mongodb://user:pass@ literal.

Decision rules

Embed or reference

Relationship	Choose	Why
Read together, small, bounded (address on a user)	embed	one read, no $lookup, atomic update
One-to-few, bounded (≤ a few dozen, won't grow)	embed	stays well under 16 MB
One-to-many, growth not bounded (comments on a post)	reference	array would chase the 16 MB cap
Many-to-many (students↔courses)	reference (array of ids on the lighter side)	shared, independently mutated
Child shared across parents	reference	one source of truth, no duplication drift
Child independently and frequently mutated	reference	avoid rewriting a big parent per child edit
High-cardinality / huge child set	reference (+ optional subset embed)	keep the hot read small

Which schema pattern

Symptom	Pattern	What it does
List view reads 3 fields of a heavy doc	subset	embed only the hot fields, reference the rest
$lookup on every read just to show a name/price	extended reference	copy the few joined fields you display
Unbounded time-ordered events (readings, logs)	bucket	group N events per doc by time window
Same count/sum recomputed on every read	computed	store the rollup, update it on write
1% of docs break the shape (a few mega-children)	outlier	flag them, overflow into linked docs
One collection holds several entity shapes	polymorphic	a type discriminator + shared _id space

Full Bad→Good documents for each in data-modeling.

Which index type

Access pattern	Index	Note
= on one field	single-field	also covers the field's sort
filter + sort + range together	compound, ordered ESR	one index serves all three
query into an array field	multikey (automatic on an array key)	one multikey field per compound index
query only a subset of docs (status:"active")	partial (partialFilterExpression)	smaller, cheaper to maintain
auto-expire docs after a time	TTL (expireAfterSeconds on a Date)	single-field only; deletes in background
language-aware text search	text or Atlas Search	Atlas Search is far richer; text is legacy
unpredictable / many query shapes on subdocs	wildcard ("$**")	last resort; never beats a targeted index
geospatial proximity / within	2dsphere	GeoJSON Point/Polygon
vector similarity (8.2, Community+)	Atlas/Vector Search index	see transactions-and-ops ref

When NOT to add an index

• Low-cardinality field (a boolean, a 3-value status) — the planner skips it; COLLSCAN wins.
• Tiny collection — a collection scan reads one or two pages; the index is pure write tax.
• A field already the left prefix of an existing compound index — redundant.
• Write-heavy field rarely filtered — every index is paid on every insert/update.
• "Just in case" indexes — an unused index costs writes and RAM, returns nothing.

Copy-paste patterns

Every fence is mongosh syntax.

Model the document for the read (Bad → Good)

// BAD: comments embedded in the post — array grows without bound toward 16 MB,
// every post read drags the entire comment history, money is a float.
db.posts.insertOne({
  _id: ObjectId(),
  title: "Indexing 101",
  authorId: ObjectId(),
  price: 9.99,                       // double — drifts in arithmetic
  comments: [ /* ...unbounded... */ ] // chases the 16 MB cap
})

// GOOD: post stays small; comments referenced; money is Decimal128;
// the few fields the feed needs are duplicated (extended reference).
db.posts.insertOne({
  _id: ObjectId(),
  title: "Indexing 101",
  author: { _id: ObjectId(), name: "Ada" }, // extended ref: name shown without a $lookup
  price: NumberDecimal("9.99"),
  commentCount: 0,                            // computed rollup, bumped on write
  createdAt: new Date()
})
db.comments.insertOne({ _id: ObjectId(), postId: ObjectId(), body: "…", createdAt: new Date() })

Compound index in ESR order + the query that uses it

// Feed query: filter by author (equality), sort by date (sort), bound by a date (range).
// ESR => author first, then the sort/range key.
db.posts.createIndex({ "author._id": 1, createdAt: -1 })

db.posts.find({ "author._id": authorId, createdAt: { $gte: since } })
        .sort({ createdAt: -1 })
        .limit(20)
// Confirm the plan: IXSCAN on the index above, no in-memory SORT stage.

Partial + TTL indexes

// Partial: index only the rows you actually query (active orders), not the archive.
db.orders.createIndex(
  { customerId: 1, createdAt: -1 },
  { partialFilterExpression: { status: "active" } }
)

// TTL: expire sessions 30 minutes after lastSeen. Field MUST be a Date.
db.sessions.createIndex({ lastSeen: 1 }, { expireAfterSeconds: 1800 })

Aggregation: $match first, $lookup, $group with allowDiskUse

db.orders.aggregate([
  // $match FIRST so it uses the compound index and shrinks the working set early.
  { $match: { status: "paid", createdAt: { $gte: since } } },
  { $sort:  { createdAt: -1 } },                  // index-eligible here, before any $group/$project
  { $lookup: {
      from: "customers",
      localField: "customerId",
      foreignField: "_id",
      as: "customer",
      pipeline: [ { $project: { name: 1 } } ]     // project inside $lookup: pull only what you need
  }},
  { $group: { _id: "$customerId", total: { $sum: "$amount" } } }
], { allowDiskUse: true })  // $group/$sort spill past 100 MB/stage; this lets large groups complete,
                            // it is NOT a substitute for a missing $match index — see anti-patterns.

Read explain("executionStats") — the four numbers

db.posts.find({ "author._id": authorId }).sort({ createdAt: -1 })
        .explain("executionStats")

Read these before declaring a fix:

1. winningPlan.stage — must be IXSCAN (or FETCH→IXSCAN), not COLLSCAN.
2. totalKeysExamined vs nReturned — close means the index is selective; a huge ratio means the index scans far more than it returns (wrong key order, low selectivity).
3. A SORT stage — an in-memory sort the index should have satisfied; reorder by ESR to remove it.
4. rejectedPlans — what the planner considered and dropped; a near-miss hints at a better index.

Multi-document transaction with full retry

// Use withTransaction — it retries the body on TransientTransactionError and retries the
// commit on UnknownTransactionCommitResult for you. Requires a replica set / sharded cluster.
const session = db.getMongo().startSession();
try {
  session.withTransaction(() => {
    const orders  = session.getDatabase("shop").orders;
    const ledger  = session.getDatabase("shop").ledger;
    orders.updateOne({ _id: orderId, status: "pending" }, { $set: { status: "paid" } }, { session });
    ledger.insertOne({ orderId, amount: NumberDecimal("9.99"), at: new Date() }, { session });
  }, { readConcern: { level: "snapshot" }, writeConcern: { w: "majority" } });
} finally {
  session.endSession();
}
// If both writes target ONE document, drop the transaction — that write is already atomic.

bulkWrite upsert

db.inventory.bulkWrite([
  { updateOne: {
      filter: { sku: "ABC-1" },
      update: { $inc: { qty: 5 }, $setOnInsert: { createdAt: new Date() } },
      upsert: true
  }}
], { ordered: false })  // ordered:false keeps going past one failed op and parallelizes

Change stream (resumable tail)

// Watch only the events you care about; persist resumeToken to restart without gaps.
const cs = db.orders.watch([{ $match: { operationType: { $in: ["insert", "update"] } } }]);
while (cs.hasNext()) { const change = cs.next(); /* process; save change._id as resume token */ }

More variants ($facet, window functions, $merge/$out, vector search) live in the references.

Anti-patterns / rationalizations → STOP

Rationalization	Reality → STOP
"Embed all the comments, it's one read"	Array grows unbounded toward 16 MB and bloats every post read. Reference or bucket.
"$lookup is just a JOIN, use it everywhere"	Mongo is not relational; per-document $lookup is expensive. Prefer modeling (extended reference) so the read needs no join.
"Wrap this single-document update in a transaction to be safe"	A single-doc write is already atomic. The transaction adds latency and a replica-set requirement for zero gain.
"COLLSCAN is fine, it's fast on my 100 docs"	It is O(n); at 4M docs it is a full table read. Add the index now and prove IXSCAN.
"Set allowDiskUse:true and the slow pipeline is fixed"	That masks a missing $match index by spilling to disk. Fix stage order / add the index first.
"Store the price as a number, round on display"	JS doubles drift across $sum/$inc. Use NumberDecimal (Decimal128).
"Group the whole collection, no $match"	A blocking $group over everything blows the 100 MB/stage limit. $match first to shrink it.
"One collection for users, orders, logs — fewer to manage"	Mixed shapes kill index selectivity and balloon working set. Split by access pattern.
"$where lets me run a quick JS predicate"	Runs JS per document, no index, a server-side injection surface. Use query operators / $expr.
"Index every field just in case"	Each index is a write tax and RAM cost; unused indexes return nothing. Index for real query shapes only.

Quick reference

Read/write concern matrix

Need	Write concern	Read concern	Note
Money / state transition	w:"majority"	"majority"	survives a primary failover
Read your own durable write	w:"majority"	"majority" (+ causal session)	no rollback window
Transaction default	w:"majority"	"snapshot"	consistent point-in-time
Logs / fire-and-forget	w:1	"local"	fast, may be rolled back

Atlas tier chooser

Tier	Use it for	Limits
M0	learning, tiny prototypes	free forever, up to 5 GB, shared, no SLA
Flex (GA Feb 2025)	small prod / variable load	$8 base capped at $30/mo, 100 ops/sec (burst 500), 5 GB; supports Atlas Search, Vector Search, Change Streams, Triggers
M10+ (dedicated)	production, isolation, scale-up	from $0.08/hr ($57/mo); dedicated resources, full features

M0 does not run Vector Search well for real workloads — move to Flex or dedicated. Legacy Serverless / M2 / M5 were auto-migrated to Flex.

Aggregation memory

Each blocking stage ($group, $sort without an index, $bucket) is capped at 100 MB. Past it the stage errors unless allowDiskUse:true lets it spill. Spilling is a correctness fallback for genuinely large groups, not a performance fix for a missing index.

Verify

Run scripts/verify.sh from your project root. It is read-only, never connects to a database, and never writes. It scans discovered .js/.mongodb.js files and flags foot-guns: a committed plaintext mongodb://user:pass@ credential (the only hard failure), createIndex calls with no options, redundant compound-index prefixes, $where predicates, unbounded $lookup, allowDiskUse:true that may be masking a missing index, and money stored as a JS number in seed scripts. If node is present it runs node --check for a syntax pass; otherwise that step is [skip]. Everything except a committed credential is advisory [warn]/[skip]. It runs on stock macOS bash 3.2 and exits 0 on a clean or empty target.

Project grounding (02-DOCS + CLAUDE.md)

When this skill runs in a project with a 02-DOCS/ layer (the harness Karpathy wiki), record this project's MongoDB decisions there and index them from the root CLAUDE.md, so the next agent inherits the conventions instead of re-deriving them.

1. Find the article 02-DOCS/wiki/stack/mongodb.md, indexed in 02-DOCS/wiki/index.md (the Knowledge map index; root CLAUDE.md points to it).
2. If missing or stale, create/update it with the project's real choices — collection layout and embed/reference decisions, the index set and its ESR rationale, read/write concern policy, the Atlas tier, and any encryption/RBAC setup — then index it in 02-DOCS/wiki/index.md (the Knowledge map; root CLAUDE.md keeps only a short pointer to it).
3. Read it first on every use and stay consistent; when a convention changes, update the article (bump its Updated date) in the same change.

No 02-DOCS/ layer? Skip silently (optionally suggest harness). Technical conventions are recorded, not gated — never block the task on this.

See Also

• references/data-modeling.md — embed/reference tree, the six patterns with worked documents, 16 MB math, polymorphic & schema versioning.
• references/aggregation.md — per-stage index eligibility, $lookup variants, $facet, window functions, $merge/$out, hybrid $scoreFusion, reading pipeline explain.
• references/transactions-and-ops.md — retry wrappers, concern semantics, replica-set requirement, Atlas tiers, Search/Vector Search, Queryable Encryption, RBAC, pooling, change streams.
• Sibling skills: harness (scaffolds the 01-TOOLS/MONGODB operational tool) and secure-coding (auth, encryption, least-privilege).
• For relational work — SQL, foreign keys, EXPLAIN ANALYZE, MVCC — use postgresdb, not this skill. Different engine and planner.
• Out of scope here — external tools with their own docs: ODM/driver API surface (Mongoose hooks, the Node driver's bulkWrite/updateMany return shapes) and cross-engine vector-store selection. This skill owns the server query and the index the server picks.