v1.0 #1
Loading…
Reference in a new issue
No description provided.
Delete branch "dev"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Monorepo skeleton with packages/{shared,proxy,web}, root pnpm workspace and single lockfile, strict NodeNext TypeScript base config, ESLint 10 flat config + Prettier, and Node 24.19.0 engine pin. Workspace cross-imports verified; pnpm -r build, typecheck, and lint are green. Framework-specific dependencies are intentionally deferred to their own tasks (V1/V5/F2/F5). Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>Logger-config factory redactLoggerOptions() spread into Fastify({ logger }) at boot — not a runtime plugin, because pino redact is a constructor option that cannot be retrofitted. Redact paths are derived from CREDENTIAL_HEADER_NAMES imported from @s3-vedrfolnir/shared (9 bracket-notation paths per header: req/err.req/res/err.res bodies+headers, bare headers, top-level, and req.query), so the contract and the redaction cannot drift. Adversarial negative suite (15 tests, including a non-vacuous control, per-header coverage, and a drift-guard). Verified empirically that F4 is defense-in-depth: Fastify 5.11.2's default req serializer omits headers and its default err serializer does not embed req, settled by reading the installed source plus positive-assertion tests that fail loudly if an upgrade flips either. Harness exported for H3 reuse against real endpoints. COMPONENTS.md still calls the interface a Fastify plugin; that doc update is architect-owned and tracked separately. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>createProxyApp factory constructs the Fastify instance with F4's redact config assigned to logger.redact at boot (the assign form, not the spread — pino reads options.redact, so spreading would silently disable redaction; the misleading sample in redact.ts is corrected here too). Redact is a security invariant: callers cannot disable it. The factory accepts an optional { level, stream } logger override so H3 can point F4's CaptureStream harness at the real app's routes. Registers /health and the /session, /s3, /config route-group plugins (handlers stubbed 501, populated by V2/V3/F6/T-series). Graceful shutdown via manual process.on (no new dependency). 9 tests cover boot, the logger-override seam, redact-active-on-real-app, the route stubs, and shutdown. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>Doc-sync from the V9 first-slice milestone. No decision changes — these reflect how the components were actually built. COMPONENTS.md: - Session Manager: drop 'connection profiles (nice-to-have)' from the responsibility (moved to v1.1 per ADR scope cut); add getBuckets() to the interface (V9 captures buckets[] from /session/probe so the bucket list doesn't re-fetch on mount). - Address-Style Detector: expand the interface to the real shape — {ok:true, buckets[], addressStyle} | {ok:false, error} carrying the path-style attempt's raw error per ADR #09. - Secret-Redaction Middleware: correct 'Fastify plugin' to 'logger-config factory' (pino redact is a constructor option, not a runtime plugin); note the shared-constants import that prevents drift; note Fastify 5's default serializers omit headers so the layer is defense-in-depth, not the sole barrier. ARCHITECTURE.md: - Security posture: align the 'secret never logged' wording with the factory mechanism + shared-constants discipline. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>The H1 integration test (real MinIO) found that every healthy multipart upload failed mid-flight with 502 upstream_error / AbortError. Root cause: streamUploadToS3 wired source.on('close', onAbort), but Node fires 'close' AFTER 'end' on normal stream completion too (the SDK reads the body to the wire, the Readable ends, then closes as Node tears down the finished stream — all before the S3 response arrives), so the in-flight send was aborted the instant the body finished flowing. The T1 unit mock resolved send without reading the body, so the race window never surfaced. Fix: track 'end' via a sourceEnded flag and treat a post-end 'close' as the normal completion it is (no abort); a pre-end 'close' remains the genuine mid-stream disconnect. The 'aborted' (IncomingMessage premature-close) and 'error' events stay UNCONDITIONAL abort triggers regardless of sourceEnded, so a real client disconnect cannot masquerade as normal end-of-stream — defense-in-depth holds. A regression test reproduces the race (the mock holds send pending across the source end+close and rejects on abort like the real SDK); demonstrated to fail on the old code. Real-disconnect still aborts (the existing test, whose source never ends, still passes). Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>The four multipart routes replace the V1 501 stubs. POST /s3/multipart/init runs CreateMultipartUpload and returns {uploadId}. PUT /s3/multipart/part streams the request body straight to S3 as an UploadPart via T1's streamUploadToS3 — a passthrough content-type parser scoped to application/octet-stream makes request.body the live Readable (not request.raw, which Fastify drains), and the handler forwards the inbound Content-Length onto the command so the aws-chunked streaming signer can derive x-amz-decoded-content-length (Bug A, found by H1 against real MinIO — the unit mocks never run the real signing stack). POST /s3/multipart/complete runs CompleteMultipartUpload and returns an ack. DELETE /s3/multipart/abort runs AbortMultipartUpload — the cancel-cleanup path the no-orphaned-parts guarantee depends on — and NEVER swallows a failure (a swallowed abort = orphaned billable parts). Per-request client, mapS3Error on every path, incoming+outgoing schema validation. The F4 redact negative-test sweep is extended to the multipart endpoints' error paths, including the architect-flagged abort-failure path, plus a non-vacuous control. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>The most load-bearing acceptance criterion in v1.0, proven end-to-end. A docker-compose brings up a real MinIO and the F3 proxy image; the integration suite drives the proxy's multipart endpoints directly (node-side HTTP with the four credential headers) and asserts four scenarios, all green: init→abort clears the upload on S3; init→(parts uploaded via the proxy part endpoint)→abort clears the upload AND its parts (ListMultipartUploads empty, ListParts -> NoSuchUpload — the strongest no-orphaned signal); a credential failure during abort surfaces a non-2xx relay error rather than {ok:true}; and a full multi-part upload (5 parts, 21 MiB) through the proxy completes and re-downloads byte-identical (SHA-256 match via /s3/download, the real proxy path). The >5 GB capability uses the identical code path — part-size auto-sizing and the 10,000-part ceiling are unit-tested in T5-web, and H1 exercises the real-backend mechanics at multi-part size. The suite is hermetic (random bucket, full cleanup in afterAll including defensive abort of any in-progress uploads) and gated off the unit suite via a separate vitest integration config (pnpm test:integration) so the standard pnpm -r test gate is unaffected. H1 earned its keep: it caught Bug A (the part PUT not forwarding Content-Length, which the aws-sdk-client-mock unit tests cannot see because they stub send above the sigv4/aws-chunked stack) and Bug B (streamUploadToS3 aborting healthy uploads on normal end-of-stream) — both now fixed and regression-pinned. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>Implements the browser-side 'download' and 'bulk-download' TransferExecutors (the proxy endpoints T4/T6 shipped; only 'upload' was registered until now). - download-fetcher.ts: fetchDownload (GET /s3/download) + fetchBulkDownloadZip (POST /s3/download/zip). Creds in the 4 CREDENTIAL_HEADERS only, never in query/body; signal-threaded; non-2xx -> RelayHttpError. Returns the live Response (executor pipes the body, never .json()). - download-executor.ts: createDownloadExecutor + createBulkDownloadExecutor + register* boot helpers, mirroring upload-executor.ts. Streaming-to-disk via the File System Access API is the primary path (body.pipeThrough(progress) .pipeTo(writer, {signal}) - zero full-body buffering, ADR #01/#02 mirrored browser-side). Blob fallback (<a download>) only when FSA is unavailable AND total <= FALLBACK_MAX_BYTES (256 MiB); over-cap throws a clear error rather than OOM the tab. Cancel: signal aborts fetch + writer + source. No 'completed on truncated body' path - mid-stream read errors reject verbatim -> T3 records 'failed'. - fsa.d.ts: ambient window.showSaveFilePicker? (TS 6.0 lib.dom.d.ts ships the handle types but not the picker entry point). - main.tsx: registerDownloadExecutor() + registerBulkDownloadExecutor() in boot(). - transfer/index.ts: barrel re-exports mirroring the upload block. Not a high-risk surface (proxy streaming done + reviewed; this is browser plumbing). 46 tests (35 dev + 11 tester-added adversarial). Reviewed: all load-bearing checks PASS, no ADR deviation. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>Completes the transfer UI surface over the T3/T5/T8a orchestration. T7 - Upload UI (fe): - components/transfer/TransferPanel.tsx: reusable, descriptor-kind-agnostic progress/cancel panel (subscribes to activeJob; kind-aware label + percent bar + cancel + warnings + terminal error by discriminated kind). Reused verbatim by T8b downloads. - components/transfer/upload-target-store.ts: Zustand cross-slot store (mirrors selection-store): writer in BucketPage, reader in AppLayout. - UploadDrawer.tsx (V5 placeholder) -> real slide-in dialog: drop zone + file picker (multiple), advanced-options collapsible (content-type, storage-class <datalist>, metadata key/value rows), role=dialog + aria-modal, Escape/scrim dismiss. Upload button in SectionHeader (reachable from root + nested prefixes). Key = prefix + file.name (empty prefix at root -> bare name, no leading slash). exactOptionalPropertyTypes honored (empty options -> {}, never {contentType: undefined}). T8b - Download UI (fe): - components/transfer/DownloadDialog.tsx + download-dialog-store.ts: modal disclosure dialog (switches on bulk vs folder; the latter mounts useS3ListAll with a 'Counting objects...' state, effect-driven not render-driven, cancellable on unmount). - selection-store.ts: extended ADDITIVELY with selectedKeys Map + toggleSelected/clearSelected; the V9 single-select selection/select/clear seam is byte-for-byte unchanged (M3 unaffected, 5 original V9 tests preserved). Multi-select = separate gestures (click=inspector, checkbox=bulk). - BucketPage.tsx: per-row checkbox + download icon, per-folder download-as-zip icon, bulk-action bar in SectionHeader. Folder-zip filters zero-byte / folder markers from the archive. - ADR #02 disclosure copy (load-bearing, Q2): rendered text asserts 'in memory', 'bounded by file count', 'never writes anything to disk' and is canary-locked to NOT contain the substring 'temp' (catches drift to 'temp storage'/'temp file'). Pre-flight refusal rendered BEFORE any fetch (e2e test asserts fetch never invoked on oversize). - styles.css: @keyframes slidein (the drawer's motion-safe animation was a no-op without it). Tests: 30 (T7) + 37 (T8b) + 8 tester-added adversarial. Reviewer-approved (APPROVED WITH NITS); nits deferred to U2/U3 (dialog focus-trap/restore, bulk-action-bar responsive wrap) - noted in docs/plans/v1.0.md. docs/plans/v1.0.md: T7 + T8 flipped to DONE with implementation notes. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>Replaces the last three 501 stubs in the /s3 route group with real handlers. - POST /s3/object/delete -> DeleteObjectCommand({Bucket, Key}) - POST /s3/object/copy -> CopyObjectCommand (same-bucket only; schema structurally forbids cross-bucket; CopySource = `${bucket}/${sourceKey}` raw, no proxy-side pre-encoding) - POST /s3/folder/create -> PutObjectCommand zero-byte marker ({Bucket, Key: prefix, Body:'', ContentLength:0}, no ContentType - folder detection stays size===0 && key.endsWith('/')) Each handler mirrors V2/T5/T6: extractCredentials -> per-request client -> parseXBody (.strict() shared schema, throws bad_request) -> send -> AckResponseSchema defense-in-depth -> reply; catch -> sendRelayError (mapS3Error, cred-safe). Stateless, multi-user safe. F4 redact sweep extended (redact-object-ops.test.ts): 3 endpoints x bad-body/missing-creds/mapped-403 + non-vacuous control. Also drops the now-orphaned not-implemented.ts (sendNotImplemented had zero consumers after O1) and refreshes two stale '501 stubs' comments per the living-doc rule. Decisions (adjudicated): copy-to-self left un-rejected at the proxy (a UI guard for O4, not a wire concern); CopySource percent-encoding deferred to H5 (aws-sdk-client-mock stubs the signing middleware; only a real backend validates - same gap as T5's ContentLength). Folder-delete count enumeration reuses V7's useS3ListAll on the web (NOT a new proxy endpoint - packages/shared/src/s3-object-ops.ts is explicit). 44 route tests (40 dev + 4 tester-added). Reviewer-approved (APPROVED WITH NITS, nits fixed). Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>The user-facing surface for object housekeeping, consuming O1/O2/O3. - components/object-ops/ObjectOpsDialog.tsx + object-ops-dialog-store.ts: one unified dialog (mounted in AppLayout) discriminating on kind (delete-object/delete-folder/rename/copy/create-folder). Mirrors the T7/T8b cross-slot Zustand pattern + dialog a11y baseline. - BucketPage: per-row RowActionsMenu (Rename/Copy/Delete on objects, Delete-folder on folders) + 'New folder' button in SectionHeader, alongside T8b's existing download icon + bulk checkbox. - Destructive-op confirmation = an 'I understand...' checkbox gating the confirm button (the explicit-confirmation gesture, per object-operations.md). - Delete-folder: mounts useS3ListAll({autoStart}) -> 'Counting objects...' -> 'N objects will be permanently deleted' -> confirmable; cancellable during enumeration and during the delete loop. - Rename: drives O3 useRenameMove with phase progress; source===dest disables confirm (defense-in-depth alongside O3's data-layer guard); honest partial-rename copy for cancelled-after-copy AND failed+targetCreated:true (the latter was a load-bearing bug caught by tester+reviewer - the terminalCopy switch now branches on isFailed && targetCreated; spec NFR 'communicate honestly'). - Copy: overwrite warning when destination matches an existing key. - Create-folder: appends '/', rejects multi-segment names. - a11y baseline matches T7/T8b (role=dialog, aria-modal, Escape, scrim, focus-visible rings); focus-trap/restore deferred to U2/U3 uniform sweep. 471 web tests. Reviewer-approved (CHANGES REQUESTED on the partial-rename honesty gap -> fixed -> APPROVED). KNOWN LIMITATION (inherited, surfaced as a spec/architecture tension for the architect, NOT an O4 bug): folder-delete count + delete cover only DIRECT children because /s3/list uses a '/' delimiter (folds nested keys into common-prefixes). object-operations.md defines folder-delete as recursive; fixing the depth needs a proxy-side non-delimited LIST mode (shared S3ListRequestSchema change) or client-side recursive walking. Documented in O4 JSDoc; does not block O4's UI correctness for flat folders. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>fetchS3Range — the web client for GET /s3/range. Same-origin GET with the 4 credential headers + Accept: application/octet-stream; reads the range-supported sentinel via the shared RANGE_SUPPORTED_HEADER constant; non-2xx surfaces as the reused RelayHttpError. Returns {bytes, rangeSupported, contentRange?}. OOM defense (the security-critical property): when rangeSupported === false, a non-range backend returns the FULL object body. A multi-GB object would blow the tab. A maxBytes cap (default 2 MB) guards it — pre-buffer content-length check cancels response.body and throws the typed RangeFallbackTooLargeError; a post-buffer check catches a lying/absent content-length. Honored 206 ranges buffer fully (the cap is fallback-only; the asymmetry is pinned by tests). Reuses buildCredentialHeaders / parseRelayError / buildRelayUrl (additively exported from data/fetcher.ts) and RelayHttpError — no duplication. New packages/web/src/metadata/ module (range-fetcher + index barrel); foundation for M2/M3. 25 tests including adversarial OOM-boundary cases (cancel-then-no-buffer, lying content-length, no-content-length huge body, honored-range > cap buffers fully, header case-insensitivity, abort propagation). 959 unit tests green. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>The metadata-extraction skeleton + the three range-based format extractors, satisfying the spec success criteria (50 MB JPEG EXIF from header bytes; multi-GB Parquet schema/row-count from footer; unsupported format = clean state). inspectMetadata({bucket,key,creds,signal,deadlineMs?}) dispatches by file extension, runs the matching extractor under a wall-clock deadline (AbortSignal.any([caller, timeout]) + Promise.race), and returns a normalized MetadataResult = {kind:'loaded',metadata:{format,fields,notice?}} | {kind:'unsupported'} | {kind:'error',message}. Extractors receive an injected readRange fn (no raw creds — credential isolation) built on M1's fetchS3Range; all parsing is wrapped in try/catch and degrades to a graceful result, never throwing into the UI. Range-based: image (exifr — EXIF/IPTC/XMP + dimensions, 512 KiB header; HEIC best-effort), parquet (hyparquet — schema/row-count/row-groups from a 1 MiB footer suffix), text/code (jschardet charset + line count + size-capped preview, 128 KiB prefix). PDF + MS Office are M2b (capped full fetch). New deps (exact pins): exifr@7.1.3, hyparquet@1.26.2, jschardet@3.1.4. Supply-chain gate green; minimumReleaseAge policy intact; duckdb-pin gate unaffected. Dedicated adversarial review (F4/T1/T3 class) caught: a real bug — jschardet processed up to a 2 MB fallback body synchronously (~700 ms main-thread block on non-range backends, affecting normal large text files), fixed by capping the processed sample to 128 KiB (~50 ms now). Documented limitation: the deadline is event-loop-bound and cannot preempt sync parsers — hyparquet can spin ~300 ms on a crafted LIST<BOOL> footer (bounded, graceful, normal Parquets fast). Web-Worker decision deferred to the U3 responsiveness audit (re-open if M2b's pdf-lib shows non-adversarial cost). No ADR #01/#02 deviation (bytes only via the Range Gateway; in-memory buffers, no temp file). 1019 unit tests green including 30 adversarial cases (sync-parse timing budgets, deadline-cannot-interrupt-sync-spin proof, malformed EXIF/Parquet, shape invariants). Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>Completes the metadata extractor set: PDF (pdf-lib) and MS Office OPC core-properties (fflate + DOMParser), wired into the M2a dispatch skeleton. Both use the capped full-fetch pattern (bytes=0-<CAP-1>, maxBytes=CAP) so M1's fallback defense applies on non-range backends. PDF: PDFDocument.load({updateMetadata:false, ignoreEncryption, throwOnInvalidObject:false}) on a 1 MiB cap -> page count + Info fields (title/author/subject/creator/producer/created/modified). Real PDFs parse in ~3 ms (incl. 4800-page specimens). MS Office (docx/docm/xlsx/xlsm/pptx/pptm): fflate unzipSync with a filter decompressing ONLY docProps/core.xml (decoy entries never inflated — proven: ~0.3 MiB heap delta on a 128 MiB decoy), pre-inflate originalSize guard, reactive post-inflate check; DOMParser 'application/xml' (XXE blocked via parsererror). Core properties -> title/author/created/modified/etc. Dedicated adversarial review (F4/T1/T3 class): office zip-bomb defense verified 3-layer genuine; XXE safe. PDF decision — ship @1 MiB for v1.0 (reviewer + tester concur, consistent with the hyparquet precedent): adversarial %PDF-1.7+garbage freezes ~2.4-3.1 s (bounded, graceful, no zombie at the default 5 s deadline — no-zombie rests on CAP sizing, not the deadline, which is proven a no-op for pdf-lib's sync chunks). pdf.ts + office.ts docstrings corrected to state the limitations honestly. Two carry-forwards to the U3 responsiveness audit: (1) move extractors (pdf-lib + hyparquet) to a Web Worker so the deadline can actually interrupt; (2) streaming-inflate hardening for core.xml (a forged central directory can cause a transient inflate OOM — narrow threat, requires bucket write access). New deps (exact pins): pdf-lib@1.17.1, fflate@0.8.3. Supply-chain gate green; minimumReleaseAge policy intact; duckdb-pin gate unaffected. No ADR #01/#02 deviation (bytes only via the Range Gateway; in-memory buffers bounded by caps, no temp file). 1067 unit tests green incl. adversarial timing pins + zip-bomb probes. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>Moves pdf-lib + hyparquet extraction off the main thread so worker.terminate() can hard-preempt a mid-chunk parse — the structural fix for the ~2.6s pdf-lib main-thread freeze (event-loop-bound Promise.race/setTimeout could not interrupt it). Range I/O stays on the main thread (credentials never reach the worker); the fetched bytes are transferred (zero-copy) to the worker, which runs the parser and posts back a MetadataResult. - worker-protocol.ts: request/response message types. - worker-coordinator.ts: main-thread seam (injectable WorkerFactory); registers the abort listener before postMessage; transfers bytes.buffer; a single 'resolved' flag serializes every settle path (onmessage/onerror/ abort) so no double-resolve, no missed cleanup; terminate() on every path. - metadata-worker.ts: the worker script (Vite bundles it as a separate chunk). - worker-parsers.ts: the pure parsePdf/parseParquet logic shared by BOTH the worker script and the in-process test factory (no duplication / no drift footgun). Tests therefore exercise the real worker parsers. office.ts hardens the OPC core.xml path: replaces fflate unzipSync with a streaming Unzip + UnzipInflate + UnzipPassThrough path and a HARD decompressed-byte cap (CORE_XML_MAX_BYTES = 1 MiB) checked after each 32 KiB chunk — closes the forged-central-directory transient-OOM hole (a lying originalSize is aborted mid-stream; fflate@0.8.3 fires ondata synchronously from push, confirmed against installed source). EOCD pre-validation keeps garbage/non-ZIP distinguishable from a valid ZIP without core.xml. inspect.ts: deadline Promise.race keeps the promise-settle guarantee; a new guard maps a worker-path result to 'Cancelled' when the CALLER's signal aborted mid-parse (so worker + non-worker paths are consistent). Dedicated adversarial review pass complete (T3 class): coordination logic is race-free (terminate-vs-resolve, late-message-after-abort, listener cleanup, signal-already-aborted, no credential leak to the worker); streaming cap verified against fflate source; forged-central-directory bomb test (§11.4) proves Layer 3 fires to the calm notice without OOM/crash. Stale 'deferred Worker retrofit' comments removed (now 'shipped'). Web suite: 713 tests green. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>Verifies the ADR #03/#04 cross-tab contracts manifest at the React COMPONENT level, not just the store level (which the existing session-cross-tab.test.ts / transfer-cross-tab.test.ts already cover). 5 tests across 3 scenarios: (a) session-changed broadcast -> requireSession loader re-evaluates -> loser tab redirects to /login (asserted on rendered LOGIN_MARKER UI) (b) transfer-acquired broadcast -> real BucketPage DownloadErrorBanner surfaces role=alert 'Another tab is busy' (executor never called) (c) storage-event fallback: same UI behavior with BroadcastChannel stubbed to undefined (unsetBroadcastChannel + dispatchStorageEvent) Renders real production components (requireSession loader, BucketPage) through createMemoryRouter + RouterProvider inside the full provider stack. Web unit count: 736 -> 741 (+5 tests). Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>Folds the PM's execution-tracker status into v1.0.md (H3/H4/H5 ✅ DONE markers + outcome notes, matching the H1/H2 pattern). Removes the temporary epic-6-hardening.md execution plan — its content is now authoritative in v1.0.md. All seven Epic 6 tasks (H1–H5) are closed. Final test totals: 1331 unit + 7 integration + 14 E2E (5 AWS skipped) = 1352 active tests, all green. Co-Authored-By: Xavier's assistant <assistant@gijoe88.com>