chore: establish learning module baseline
This commit is contained in:
inman committed
commit
2d04197f3f
2409 files changed
+533050
No files matched your search
@@ -0,0 +1,191 @@
|
||||
# @openmaic/storage
|
||||
|
||||
The MAIC pluggable persistence layer: small, swappable-backend primitives for
|
||||
persisting app state, depending only on [`@openmaic/dsl`](../dsl).
|
||||
|
||||
The DSL owns _what_ persists (document / runtime shape + validation + migration +
|
||||
the asset `StorageProvider` interface). This package owns _where / how_ it
|
||||
persists — the primitives and their backends. The pluggable seam is the
|
||||
**backend**, not the database driver: browser backends (the zero-server
|
||||
`clone-and-run` default), HTTP clients plus a reference server, and PostgreSQL
|
||||
server backends.
|
||||
|
||||
## Dependency arrow (acyclic)
|
||||
|
||||
```
|
||||
@openmaic/storage -> @openmaic/dsl
|
||||
```
|
||||
|
||||
No dependency on React, zustand, or any host app. Backends take their `Storage`
|
||||
/ `IDBFactory` by injection, so the package is app-agnostic and testable without
|
||||
a browser.
|
||||
|
||||
## What's in here
|
||||
|
||||
| Export | Role | Browser backend |
|
||||
| --- | --- | --- |
|
||||
| `KVStore` | small `device` / `account`-scoped values not owned by the DSL | `BrowserKVStore` over `localStorage` |
|
||||
| `StorageProvider` (from `@openmaic/dsl`) | the asset seam: `put(blob) → ref`, `resolve(ref) → url`, `remove(ref)` | `BrowserAssetStore` over IndexedDB (`assets` registry + `blobs`) + object URLs |
|
||||
| `kvPersistStorage` | adapt a `KVStore` into a zustand `persist` storage | — |
|
||||
| `DocumentStore` | persist the DSL `document` aggregate (stage + scenes + embedded agents / quiz / actions + an outline snapshot) | `BrowserDocumentStore` over IndexedDB (normalized `stages` / `scenes` / `outlines`) |
|
||||
| `RuntimeStore` | persist what a learner produces while taking a course — sessions + append-only records (chat, quiz attempts, playback facts) | `BrowserRuntimeStore` over IndexedDB (`sessions` / `records`) |
|
||||
|
||||
- **Scopes.** `account` values are user data a server-backed deployment syncs
|
||||
across devices; `device` values (theme, locale, layout) never leave the
|
||||
device — every backend honours that, so the scope is part of the primitive,
|
||||
not the backend choice. The [KV HTTP contract](./docs/kv-http-contract.md) is
|
||||
`account`-only and carries no scope on the wire at all, so `HttpKVStore`
|
||||
routes `device` to a `LocalKVStore` it requires at construction — a *branded*
|
||||
local backend, because a networked store satisfies plain `KVStore`
|
||||
structurally and would otherwise be accepted as the place device values live.
|
||||
- **The asset pool.** `BrowserAssetStore` is a global asset pool (#1007): an
|
||||
**allocated** `AssetId` (`ast_` + 128 random bits) names a registry entry
|
||||
(`contentHash`, `mime`, `meta`), and the registry names content-addressed
|
||||
bytes. A document embeds only the id and the store resolves it to a URL at
|
||||
render time (a raw URL would bake in a provider + expiry and break
|
||||
portability). Two levels of indirection buy three things at once: an id
|
||||
survives the bytes behind it being regenerated through
|
||||
`BrowserAssetStore.replace`; identical bytes are stored once however many ids
|
||||
name them; and the content hash never leaves the package, so the "whoever
|
||||
knows the hash can reach the bytes" threat that pure content-addressing must
|
||||
defend against does not arise. Images, audio and video share one id space —
|
||||
the medium is a `mime` column, not a partition.
|
||||
`put` **always allocates a new id**, so its successful return values and
|
||||
branches do not reveal whether the bytes were already present. The browser
|
||||
registry embeds its `blobs` table in the same database because reference
|
||||
counting, byte writes, and reclamation must share one transaction. This is
|
||||
not a replaceable blob backend *in the browser*, because that store reclaims
|
||||
inline. The server backend collects offline instead, so no request deletes
|
||||
bytes and its byte layer is pluggable — a column of the transactional store,
|
||||
or an object store keyed by content hash. Resource-accounting channels remain:
|
||||
quota errors, storage estimates, and server billing or metering can disclose
|
||||
existence, so server deployments must budget them per principal — the
|
||||
[asset registry HTTP contract](./docs/asset-http-contract.md) requires quota
|
||||
to be accounted on a principal's logical bytes for exactly that reason. Object URLs
|
||||
are minted per id, not shared per `contentHash`: sharing would let a holder of
|
||||
two ids learn that their bytes match by comparing URL strings. Each
|
||||
`replace(id, ...)` followed by `resolve(id)` adds one retired snapshot that
|
||||
only `release(id)` or `close()` reclaims; each ref retains at most one current
|
||||
snapshot plus that retired history. A returned URL is an immutable snapshot:
|
||||
mutations affect future resolutions but never revoke a URL already issued by
|
||||
this or another store instance. Application code that constructs a concrete
|
||||
`BrowserAssetStore` owns that lifecycle (the narrower DSL `StorageProvider`
|
||||
seam exposes neither method), and media-heavy applications should reclaim
|
||||
snapshots explicitly. `release` is an owner-level escape hatch for a caller
|
||||
that owns every use of every URL returned for that id in the instance;
|
||||
`close` reclaims the whole instance.
|
||||
Cross-instance correctness comes from comparing the registry identity on
|
||||
every resolve, so a remove yields `null` and a replacement yields a fresh
|
||||
URL on the next call without reclaiming older snapshots.
|
||||
The id domain is opaque and unvalidated (the KV key-domain lesson, applied
|
||||
forward): an unrecognized id is a miss, never an error. The server backend is
|
||||
still to come.
|
||||
- **Document normalization.** The DSL `document` is a portable embedded
|
||||
aggregate; `DocumentStore` normalizes it into per-entity rows so scene-level
|
||||
writes (`putScene`) stay cheap, and reassembles it on read. Each document is
|
||||
stamped with a `dslVersion`; reads run the DSL
|
||||
migration ladder forward, and writes are validated against the DSL gate
|
||||
(`validateStage` / `validateScene`) so schema drift fails loud. The outline is
|
||||
an opaque, app-owned snapshot carried alongside — persisted verbatim, neither
|
||||
validated nor migrated.
|
||||
- **Generic over scene type.** `DocumentStore<TScene>` defaults to the DSL
|
||||
`Scene` (universal `slide` / `quiz`). An app that widens `Scene` with its own
|
||||
kinds (`interactive` / `pbl`, content the DSL does not own) parameterizes the
|
||||
store over its scene union and injects a matching `validateScene`, so those
|
||||
scenes persist and the gate stays fail-loud for the app's shapes.
|
||||
- **Runtime layer.** `RuntimeStore` is partitioned by `(stageId, learnerKey)`:
|
||||
a stage has many sessions — one or more per learner — so every listing is
|
||||
partition-scoped (there is deliberately no global listing; single-session
|
||||
operations are id-keyed, and `mergeLearner` is the one deliberate
|
||||
cross-stage sweep). Sessions are **born stamped**: the store
|
||||
writes `runtimeDslVersion` itself at `createSession`, and the runtime line
|
||||
has no unversioned epoch, so an unstamped row fails loud instead of being
|
||||
lifted like a legacy document. Records are **append-only** ordered facts
|
||||
under an **active** session; the store assigns the per-session monotonic
|
||||
`seq` on append — the sole replay ordering key, never timestamps. Record
|
||||
payloads are gated per kind by injectable validators, defaulting to the DSL
|
||||
skeleton guards for `chat` / `quizAttempt` (`playback` and app-defined kinds
|
||||
carry app-owned payloads). `mergeLearner` re-keys an anonymous learner's
|
||||
sessions to a signed-in key across all stages; `deleteLearnerRuntime`
|
||||
cascades one learner's sessions + records on one stage, and
|
||||
`deleteStageRuntime` clears a whole stage — the hook a document deletion
|
||||
cascades through.
|
||||
- `deleteAllRuntime` clears every runtime session and record for explicit
|
||||
whole-cache reset flows.
|
||||
|
||||
## Upgrading from 0.1.x
|
||||
|
||||
Version 0.2.0 removes `BrowserAssetProvider` outright; it no longer ships. The
|
||||
asset API is now `BrowserAssetStore`, whose refs are allocated ids and whose
|
||||
data lives in the new `maic-asset-pool` database.
|
||||
|
||||
Reusing a custom `dbName` created by a 0.1.x provider raises an explicit
|
||||
legacy-schema error rather than corrupting data or operating only partially.
|
||||
|
||||
- `BrowserAssetStore` deliberately does not read data written by the 0.1.x
|
||||
provider in `maic-assets`. Its content-addressed `sha256-` refs are no longer
|
||||
outward references, and the contract suite pins sha256-shaped refs as misses;
|
||||
a silent read-through would restore the reference model this release removes.
|
||||
- If persisted 0.1.x data must be carried forward, open an issue. The supported
|
||||
shape is an explicit one-time import helper that enumerates old rows,
|
||||
allocates an id per blob, and returns an old-ref-to-new-id mapping for the
|
||||
caller to apply to its documents.
|
||||
|
||||
## Backend equivalence
|
||||
|
||||
Each primitive has one implementation-agnostic contract suite
|
||||
(`test/kv-contract.ts`, `test/asset-contract.ts`, `test/document-contract.ts`,
|
||||
`test/runtime-contract.ts`).
|
||||
Every backend is proven by running the same suite against it, so browser, HTTP,
|
||||
and PostgreSQL implementations cannot silently diverge from a primitive's
|
||||
semantics. Assets use the single `test/asset-contract.ts` suite for the
|
||||
allocated-id store: identical bytes never share a caller-visible id. Asset
|
||||
backends must let the suite temporarily instrument the production allocation
|
||||
source while constructing the store through the same factory used by every
|
||||
other contract test. This proves that every successful `put` consumes exactly
|
||||
one allocator output, independent of whether the bytes already existed, without
|
||||
adding a caller-configurable allocation path.
|
||||
|
||||
## Roadmap
|
||||
|
||||
- [x] `KVStore` + browser backend; zustand `persist` adapter
|
||||
- [x] `StorageProvider` (in `@openmaic/dsl`) + browser asset registry: allocated
|
||||
`AssetId` over an embedded byte table in `BrowserAssetStore` (#1007)
|
||||
- [x] implementation-agnostic contract suites
|
||||
- [x] `DocumentStore` (aggregate ↔ normalized adapter, migrate-on-read via the
|
||||
DSL migration registry, validation gate) + browser backend
|
||||
- [x] `RuntimeStore` (sessions + append-only records, runtime version line,
|
||||
per-kind payload gate) + browser backend
|
||||
- [x] wire the app's settings + user-profile `persist` stores through `KVStore`
|
||||
(both `account` scope). No automatic migration of pre-cutover data: new
|
||||
data persists through `KVStore`, legacy `localStorage` keys are ignored
|
||||
(not migrated) and best-effort purged, and a user reconfigures once on
|
||||
upgrade
|
||||
- [ ] wire the app's third `persist` store (`agent-registry-storage`), still on
|
||||
zustand's default `localStorage`
|
||||
- [ ] wire the app's remaining ad-hoc `localStorage` keys through `KVStore`
|
||||
- [ ] a hydration gate the app actually consumes — **required before an
|
||||
`account` scope can be served remotely**. With the browser backend,
|
||||
hydration resolves within microtasks of module evaluation and nothing
|
||||
observes it; a network round trip makes the gap visible, and the one-shot
|
||||
decisions taken against a not-yet-hydrated store (classroom agent-selection
|
||||
restore, media orchestration, scene-generator retry, server-provider
|
||||
reconcile) decide wrongly and then have their corrective writes refused
|
||||
- [x] RuntimeStore HTTP backend + reference server + HTTP contract
|
||||
- [x] RuntimeStore PostgreSQL backend
|
||||
- [x] DocumentStore HTTP backend + reference-server routes + HTTP contract
|
||||
- [x] DocumentStore PostgreSQL backend
|
||||
- [x] `KVStore` (`account`) HTTP backend + HTTP contract
|
||||
- [ ] `KVStore` server-side reference backend and reference-server route
|
||||
- [x] asset registry HTTP contract (#1007)
|
||||
- [ ] asset server backend — registry (principal column, server-derived) over a
|
||||
pluggable byte layer, with transactional-store and object-store
|
||||
implementations and an offline byte collector (#1007). It must allowlist
|
||||
content types before serving bytes, and it must account quota on a
|
||||
principal's logical bytes rather than on bytes physically written
|
||||
- [ ] asset manifest: the one enumeration of "which `AssetId`s does this course
|
||||
reference?" the export paths converge on (#1007)
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Reference in new issue
Block a user