Roll up: database design

17 Nov 2026. Design only, nothing built. Source: Roll up design doc section 3 (verdicts/ROLLUP-DESIGN-2026-11-17.md), checked against our table design skill, db-architecture-planning, and the PostgreSQL schema conventions it loads.

In short

A Roll up is a named group of Contexts that you can share. It holds Contexts, not copies. Its issues, actions and decisions are the live items of its Contexts. Ticking one in a Roll up ticks the same row in its Context.

Why its own table and not context.kind

If Roll up were a kind of ContextWhy that breaks our rules
At least nine tables point at context and assume it has sources or people: context_assignment, context_item, context_pipeline_job, context_pipeline_reservation, context_source_rule, context_link, context_email_receiving, context_tag and context_member.A foreign key can't say "only Contexts of kind = context". Each of those tables would then accept a Roll up id. Only a trigger on every table could stop that, and one missed trigger is a data leak. The skill's rule is that the foreign key carries the meaning: no kind-plus-id pointers, and the specific table points at the shared one.
A Roll up has no sources, no pipeline, no rules and no email in.Most of context's columns would be empty for every Roll up row. Two different things would share one table.
With a separate table……every Roll up foreign key is exact. A Roll up can never receive an item, an assignment or a source rule, because the database itself refuses it. A Roll up of Roll ups is impossible too, because rollup_context.context_id points only at context.

The skill's rules, and where each one shows up

#Rule (db-architecture-planning)Where it shapes this design
R1Every reference is a foreign key. No ids inside JSONB, arrays or text. A relationship is stored once.Every _id column below has a real FK. Pattern evidence is a row per item in rollup_pattern_item, not an id array on the pattern. A pattern's Context chips are worked out from its items when read, not stored a second time.
R2No kind-plus-id pointers. The specific side points at the shared side.A separate rollup table, not context.kind. rollup_pattern_job is the Roll-up-specific side of the shared job table, the same shape as context_pipeline_job.
R3Write a row once. Update it only on a real change. History goes in an append-only log. Data that changes on its own schedule goes in a narrow side table.A pattern's title and explanation are written once. A re-run that confirms a pattern changes only last_confirmed_job_id. Feedback is insert-only, and the latest row per person wins.
R4Anything a person edits, or that grants access, keeps whole-row history (L.7): a trigger-captured <table>_log. Runtime may only INSERT and SELECT the log.Four tables get one: rollup_log, rollup_context_log, rollup_member_log and rollup_tag_log. Who is in a Roll up decides who can read what, so every add and remove is kept.
R5Tenant in every key; UUIDv7 ids from the writer; hash-partitioned by tenant like context.Every primary key is (tenant_id, <table>_id) and every unique index starts with tenant_id. The tables partition the same way context does.
R6Exclusivity is a unique index plus a plain INSERT. Never a lock or a check-then-write. A violation (code 23505) becomes a typed error.A Context can be in a Roll up once: UNIQUE (tenant_id, rollup_id, context_id). A person has one role: UNIQUE (tenant_id, rollup_id, user_id). A Patterns run happens once per input version: UNIQUE (tenant_id, rollup_id, input_version).
R7RLS forced on every tenant table. The runtime role has no BYPASSRLS and never switches role.Each table gets the same four tenant policies as context_member, plus the purge-role policies so a workspace delete still cleans up. Each table also gets a tests/rls/ test proving a second tenant sees and writes nothing.
R8One owner per table and one set-based write routine. Every table has a writer and a reader.Each table has exactly one owning crate (listed in the next section), with a fail-closed boundary test from its first commit. Adding 5 Contexts is one INSERT … SELECT unnest($ids), not 5 inserts.
R9Lock the narrowest entity, never the tenant.Roll up writes lock the one rollup row only.
R10Text, never varchar(n). Access columns are enums, not booleans.Names and descriptions are text, with the same length checks as Contexts so one form limit serves both. The share switch is an enum (none or viewer) so a later level doesn't need a migration of a boolean.
R11Classify every table (K-0 class) and register it: sor_table_classification.txt and db_write_path_register.txt.The class is in each table's card below. The register rows are listed in the design doc section 3.3.

Relationship diagram

newnew table existingexisting table loghistory log (no FKs, so it survives a delete). Arrows point from the referencing table to the one it references.

contextexisting context_itemexisting · one Context each tenant_memberexisting · people jobexisting rollupowner · switch · name rollup_contextwhich Contexts rollup_memberpeople + role rollup_hidden"hide for me" rollup_tag rollup_patternwritten once rollup_pattern_itemevidence rollup_pattern_feedbackinsert only rollup_pattern_jobone run per input rollup_log+ rollup_context_log+ rollup_member_log, rollup_tag_log items are read live from the Roll up's Contexts, never copied

Tables, one by one

rollup newmutable_configowner: rollup-db

tenant_id, rollup_id, name, icon, description, owner_user_id, created_by_user_id, created_at, modified_at, linked_context_access (none | viewer, default none), general_access (restricted)

Why it's built this way: the Owner is a column, not a member row, the same as Contexts, so there is exactly one Owner. The share switch lives on the Roll up because only the Owner changes it. It is an enum so a third level later is one new value. R10 The name and description checks are copied from context so the same form limits apply.

rollup_context newmutable_configowner: rollup-db

tenant_id, rollup_context_id, rollup_id, context_id, added_by_user_id, created_at

Why: this is the link table. It is the only place a Roll up names a Context. added_by_user_id matters: if the share switch is on, it is how we check, live, that the person who added the Context still owns or edits it (see the share-switch section). The Context side is restrict on purpose. A Context delete must remove these rows itself, in its purge order, so a Roll up never silently loses a Context through a cascade. The index on context_id answers "which Roll ups include this Context?" and serves the access check.

rollup_member newmutable_configowner: context-access

tenant_id, rollup_member_id, rollup_id, user_id, role (editor | viewer), grant_kind (created | share | reassignment), granted_by_user_id, created_at

Why: this is the same shape as context_member, so the one access owner (context-access) decides both. It is a separate table, not context_member with a kind column, because a foreign key has one target. R2 The Owner is never a row here.

rollup_hidden newmutable_configowner: context-access

tenant_id, rollup_hidden_id, rollup_id, user_id, created_at

Why: "Hide for me" is one person's preference, not a change to the Roll up. It is a row per person, so hiding never touches the shared row. It has no log because it grants nothing. Its shape must match whatever Contexts' own "Hide for me" (U5) builds. If U5 picks a different shape, this follows it.

rollup_tag dropped 17 Nov: tags are an aggregate of the Contexts' tagsnewmutable_configowner: rollup-db

tenant_id, rollup_tag_id, rollup_id, tag, tag_key (generated lower(tag)), created_at, modified_at

Why: this copies context_tag. Tags are case-preserving but unique case-insensitively, via UNIQUE (tenant_id, rollup_id, tag_key). A row per tag rather than a text array keeps tags filterable through the filter engine. R1 It has a log because a person edits tags. R4

rollup_pattern newmutable_configowner: rollup-pattern-db

tenant_id, rollup_pattern_id, rollup_id, kind (issue | action | decision), title, explanation, first_detected_at, status (open | retired | dismissed), status_changed_at, job_id, last_confirmed_job_id

Why: a pattern is an LLM finding across the Roll up's Contexts. Its title and explanation are written once, so they are never rewritten in place. R3 A later run that sees the same pattern changes only last_confirmed_job_id, and a gone pattern changes only its status. Both job ids are FKs to job, so every pattern traces to the run that made it. R1 It lives in its own crate because the pattern job and the Roll up editor change for different reasons.

rollup_pattern_item newmutable_configowner: rollup-pattern-db

rollup_pattern_item_id, rollup_pattern_id, context_item_id, ordinal

Why: the evidence for a pattern is a row per item, not an array of ids on the pattern. R1 The Context chips on a pattern card come from these items' own Contexts when read, so there is no second pattern-to-Context table that could drift. If an item is deleted, its evidence row goes with it (cascade).

rollup_pattern_feedback newledgerowner: rollup-pattern-db

rollup_pattern_feedback_id, rollup_pattern_id, user_id, value (useful | dismissed), created_at

Why: the 👍 / dismiss buttons. Feedback is insert-only, and the latest row per person wins. UPDATE and DELETE are revoked. That keeps every vote as a fact for tuning the prompt later, and nobody's click ever rewrites a row. R3

rollup_pattern_job newmutable_configowner: pipeline-job-db

job_id (PK), rollup_id, input_version, outcome (published | stale | deleted), published_at

Why: this is the Roll-up side of the shared job table, so the specific side points at the shared side and job gets no Roll up column. R2 UNIQUE (tenant_id, rollup_id, input_version) is what makes "run Patterns" safe to trigger many times. A second trigger with the same inputs hits the unique index and reuses the run, with no lock and no pre-check. R6 If the Roll up is deleted mid-run, the job row stays for its history, and the run ends with outcome deleted.

context_pattern_kind new · globalseeded, read-only at runtime

scope (context | rollup), prompt, prompt_version, item_window_note

Why: our rule is that prompts live in database config, never in code. Context Patterns (U8) creates this table, and Roll up adds its row. It's global because the prompt is the same for every workspace. Its one change is that a migration seeds it.

History logs ledger

rollup_log, rollup_context_log, rollup_member_log, rollup_tag_log — each: _log_id, the row id, version_no, operation, row_json, changed_fields, actor_user_id, change_reason, valid_from, recorded_at

Why: a trigger writes them, never app code, so no write path can forget. Each stores the whole row, not a diff, so any past version can be read back on its own. R4 They have no FKs on purpose, so history survives a Roll up delete. Runtime can only insert and read.

Views (read-only, run as the caller so RLS applies)

ViewWhat it's for
rollup_list_vRoll up list rows with counts (Contexts, sources), last source added, owner name and tags.
rollup_context_vThe Contexts tab inside a Roll up. Archived Contexts are left out.
directory_entry_vContexts and Roll ups in one list, for the Contexts page's Type filter. This depends on open question Q1.
rollup_context_passthrough_vThe only thing the access crate reads from the Roll up tables: which Roll ups have the switch on. A gate test pins that.

The share switch: "Viewers can open the linked Contexts"

Stored: one enum column, rollup.linked_context_access. It defaults to none and only the Owner changes it. Every change is logged with who made it.

Enforced: the one access owner (context-access) gets one extra branch in each Context read rule (Neuma reads, source reads and the two AI-tool reads). The branch lets a Roll up member read a Context only when all three of these hold:

  1. the Roll up has the switch on;
  2. the person is a member of that Roll up;
  3. the person who added the Context still owns or edits it, checked live.

Why no extra context_member rows are written: copying access into rows would need a sync on every share, switch flip and role change, and one missed sync means someone keeps access they shouldn't have. Checking live has nothing to sync. A derived reader is always a Viewer, even if they are an Editor of the Roll up. The Context page can show how they got in.

Effect: a Roll up can never share more than the people who built it could share themselves.

Write path: the skill's six questions

The skill requires these answers before any table is created. An unknown answer blocks the design.

QuestionAnswer
Writes per unit of workAdd or remove N Contexts: one set-based insert or delete, plus one log row each by trigger. Share: one member row and one log row. Patterns run: one job row, then K pattern rows and their item rows in one publish transaction.
Can updates be HOT?to confirm Yes, if rollup.modified_at has no index. The design doesn't index it. Confirm the list sort uses last_source_added_at from the view, not modified_at.
Is each row written once?Patterns: yes, and only status and the last-confirmed job change. Feedback and logs: insert-only. The Roll up row changes only on a real edit.
What makes writers wait, and for how long?Only a lock on the one rollup row, held for the length of a single request transaction. Nothing tenant-wide.
Derived or search data kept inside the write?None. Counts and chips are computed in views at read time.
Largest burstto confirm A Patterns publish for a Roll up with many Contexts. That's bounded by its item window, but the design doesn't size it yet. Unit R6 must state it.

Scrolling 10,000 items, sorted by importance and age (revised 17 Nov)

Operator challenge: reading every Context live costs more as Contexts are added. Industry practice for this is fan-out on write: a materialized feed per reader, like a Twitter or Instagram home timeline or a Stream "hot feed cache", with merge-on-read kept only as the fallback. A Roll up is a feed over its Contexts, so it gets one.

PartDesign
1. rollup_item, the Roll up's feed indexOne row per open item per Roll up: (tenant_id, rollup_id, context_item_id, context_id, kind, sort_key, rank_version). It stores order, not content: title, status and assignee are joined from context_item when read, the way a timeline cache holds tweet ids. FKs go to rollup (cascade) and context_item (cascade). Owner: rollup-db. Class: derived projection, rebuildable from source at any time.
2. One index, one range scan per page(tenant_id, rollup_id, kind, sort_key DESC, context_item_id). A page is one index range scan after the cursor. Its cost is the same for 2 Contexts or 500, and for 10,000 items or 1,000,000. The lone high item in Context 101 sorts above 100 Contexts' worth of low items, because the order is global and stored, not assembled at read time.
3. The sort keysort_key = priority class first, then age (newest first), then id as the tie-breaker. If a Roll up ranking job is chosen (Q7 option b), its score replaces the priority class in the key. One column, one index, one cursor.
4. Kept up to date by jobs, never inside the item writeThe item job already publishes items as one set. Its publish step enqueues one rollup_feed_sync job per affected Roll up, and that job does one set-based upsert. Adding a Context to a Roll up runs one INSERT … SELECT of that Context's open items. Removing it is one DELETE … WHERE context_id. Closing an item or changing its priority goes through the same sync. The feed lags its Contexts by seconds, and the change is pushed over /ws.
5. Stable scrolling while it reranksAs with Stream's hot feed cache and Elasticsearch's point-in-time with search_after: a scroll reads one pinned ranking version (rank_version in the cursor). A rerank writes the next version beside the current one, then switches over. The old version is kept until open cursors expire (15 minutes), then pruned. Readers on the old version get a "List updated, refresh" pill instead of a reshuffle.
6. CursorAn opaque token holding (rank_version, sort_key, context_item_id). No offset. A unique tie-breaker means no repeated or skipped rows.
7. Write costAn item is written once for each Roll up that includes its Context. That's usually a few, so the cost is small and bounded. The large fan-out case, one Context in hundreds of Roll ups (the "celebrity" problem), is measured in R5. Only if it becomes real does that Context fall back to merge-on-read (the hybrid Twitter uses). Nothing is built for it up front.
8. CountsCounts per kind come from the feed index: an index-only count, or a per-Roll up counter row that the sync job writes in the same transaction.
9. Proof (R5)Seed 100 Contexts with low items and 1 Context with a high item. Assert the high item is row 1. Assert page 1 and page 200 of 10,000 items take the same time, and that a sync after a priority change moves the item. Repeat at 100,000 items.

Sources: Stream, fan-out and hot feed cache; GitLab pagination guidelines; Elastic, search_after with point-in-time.

Keeping a Roll up up to date: the feed sync

Operator, 17 Nov: the sort order can be decided later. The hard part of any roll-up is keeping it current. This section is the design for that. It follows the transactional outbox pattern with idempotent consumers, plus a scheduled reconciliation sweep, which is how read models stay correct in practice.

PartDesign
1. No change can be missedA trigger on context_item (insert, status, priority, supersede, edit) and on rollup_context (add, remove) appends one row to rollup_feed_change in the same transaction as the change. If the change commits, the intent to sync commits with it. If it rolls back, nothing is queued. Being a trigger, it covers every write path: the item job, a person ticking an item done, an edit, a Context delete. The row is insert-only and narrow, so the item write stays HOT and gains one small insert.
2. Record the Context, not every Roll upThe change row names the Context and item, never the Roll ups. One item change is one row whether its Context is in 1 Roll up or 300. The sync works out the Roll ups from rollup_context (indexed on context_id). That keeps the write cheap and moves the fan-out into the background job.
3. Re-read the source, never replay the eventThe sync never applies "priority became high". It re-reads the current context_item rows for the changed ids and upserts or deletes the matching rollup_item rows. So duplicates, retries and out-of-order delivery are all harmless: running it twice gives the same result. This is the idempotent-consumer rule from the outbox pattern, and the reason the sync can't drift because of a missed or repeated event.
4. Bursts coalesceOne Temporal workflow per tenant, started with signal-with-start under a fixed id. The first change starts it and later changes just signal it, so 2,000 items published by one job become a handful of batches, not 2,000 jobs. Each batch takes changes in sequence order, groups them by Roll up, and runs one activity per Roll up in parallel (parallel by default). Each activity is one set-based upsert and one set-based delete. The workflow drains until the queue is empty, then ends. It is a tracked job with a job row, and pushes its state like every job.
5. A watermark per Roll uprollup_feed_state (rollup_id, applied_change_seq, last_synced_at). Lag = newest change for that Roll up's Contexts minus applied_change_seq. It's measured, logged and shown. The Roll up shows "Updating…" while it's behind, so a stale Roll up is never silent. The Patterns job uses the same number as its input_version, so Patterns rerun only when the feed actually changed.
6. Adding or removing a ContextAdd: a backfill activity records the current change sequence, then copies that Context's open items in one INSERT … SELECT. Any change after the recorded sequence is re-read by the normal sync, so nothing that lands mid-backfill is lost. Remove: one DELETE … WHERE rollup_id AND context_id. Context delete (U3) removes the link rows first, and the cascade does the rest.
7. Reconciliation: catching our own bugsA Temporal Schedule runs nightly, and can also be run by hand. For each Roll up it compares the feed with a live query over its Contexts (count plus a checksum of item ids and sort keys) and repairs any difference. Repairs should always be 0. Each one is logged and counted as a defect to fix, not quietly absorbed. The feed is disposable: ./run job can rebuild one Roll up's feed from source at any time.
8. Sorting decided latersort_key is just a column on the feed row. Changing how Roll ups sort is a re-sort job that rewrites sort_key from source. It never changes which items are in the feed and never touches the Contexts.
9. Cleaning up the change logThe change table is partitioned by day. A partition is dropped once every Roll up's watermark is past it and it is older than 7 days. Dropping a partition is cheap and leaves no bloat. The retention is recorded as a purge rule.
10. FailureTemporal retries each activity with backoff. Change rows stay until they are applied, so a worker outage only delays the sync; it never loses a change. Every failure is classified at the job seam. A Roll up whose sync keeps failing shows "Couldn't update" with the time of its last good sync.
Rejected: CDC through Kafka/DebeziumIt's the same idea, reading the database log, but it adds a log pipeline we don't run. A change table in the same Postgres gives the same guarantee with what we already have.
Proof (R5)Five tests:
  • Publish 2,000 items to 3 Contexts in 2 Roll ups: the feed matches a live query, and the batch count is small.
  • Change one priority, close one item and add one Context mid-backfill: all three are reflected.
  • Kill the worker mid-batch, restart: the result is the same.
  • Deliver the same changes twice: the result is the same.
  • Reconciliation on a clean feed reports 0 repairs. On a deliberately corrupted row it repairs 1 and logs it.

New tables for the sync

TableShape and reason
rollup_feed_change(tenant_id, change_seq bigint identity, context_id, context_item_id null, change_kind, recorded_at). Append-only, partitioned by day. FKs to context, but not to context_item, so a deleted item can still be recorded as removed. Written only by the trigger and read only by the sync.
rollup_feed_state(tenant_id, rollup_id PK, applied_change_seq, last_synced_at, last_error_at, last_reconciled_at, repairs_last_run). One row per Roll up, updated only by the sync. It is the lag and health source for the UI and for alerts.
rollup_itemThe feed itself, from the previous section.

Sources: transactional outbox and idempotent consumers (microservices.io pattern, Conduktor, duplicate delivery); change-buffer coalescing (pg_trickle); debounced maintenance (activerecord-materialized).

Changes from review (17 Nov)

ChangeEffect
Roll up tags are an aggregaterollup_tag and rollup_tag_log are dropped. A Roll up's tags are the distinct tags of its Contexts (context_tag), computed in rollup_list_v. Filtering Roll ups by tag joins through rollup_context to the existing context_tag index. A Roll up can't carry a tag of its own. New tables: 8 plus 3 history logs, down from 9 plus 4.
Patterns are recomputedEach rollup_pattern_job run writes its own full set of patterns, each linked to that job and written once. The current set is the latest published run. A stable pattern key carries first_detected_at from run to run, so "Detected 3 weeks ago" survives a recompute. This replaces updating last_confirmed_job_id in place. Feedback links to the pattern key, so a 👍 carries over.

Open items that change the tables

QQuestionEffect on the tables
Q1One list for Contexts and Roll ups: new /directory, or a kind on /context?Recommended: /directory, which keeps directory_entry_v. The other option puts Roll ups into the /context response but still needs no table change.
Q4Can a Roll up Editor share and add Contexts?No table change either way. It decides which roles the write routines accept.
Q5One item raised in two Contexts: one row with "+N", or two rows?Recommended: two rows, which needs no new table. "+N" needs a cross-Context match table and a job, as a later unit.
NamingThe review called the Roll up-to-Context link rollup_member. This design uses rollup_context for that link, and rollup_member for people, matching context_member.Name only.
Depends"Hide for me" and Context Patterns are not built yet (U5, U8).rollup_hidden and context_pattern_kind follow whatever shape those units choose.