Second Brain

R-Same M3 Async Worker + Query Editor UI

2 min read

R-Same M3 Async Worker + Query Editor UI

Decision

M3 pass 2 shipped in commit b0e4f55 (2026-04-17). 16 files. Backend async execution via RQ worker with Redis dedup (job_id=f”q:{ds}:{hash}”). worker/db.py async session_scope. worker/jobs/execute_query.py loads QueryJob, validates DataSource not-paused, decrypts options, runs driver via asyncio.to_thread, persists QueryResult, links Query.latest_result_id, idempotent terminal-state short-circuit. api/query_engine/queue.py Redis+RQ wrapper with 600s job_timeout + 1h result_ttl. QueryService.execute: cache-hit returns (result, True, success-job); cache-miss enqueues on ds.queue_name + returns (None, False, pending-job). ExecuteResponse discriminated union {cached:true, result} | {cached:false, job}. Frontend: Monaco via CDN, /queries list + /queries/$id editor (vs-dark Geist Mono + 500ms job polling + ResultsTable sticky-header grid tabular-nums). routeTree updated. worker pyproject depends on r-same-api + r-same-query-runner via uv workspace.

Rationale

Async worker decouples request lifecycle from driver lifecycle. Inline sync (pass 1) blocked API workers for full Snowflake execution time — 60s statement timeout worst-case would cause head-of-line blocking. RQ job_id dedup collapses concurrent identical queries to one driver call (cost + rate-limit friendly). Per-data-source queue prevents slow sources from starving fast ones. 500ms polling matches Redash proven pattern, simpler than SSE for v1. Monaco via CDN avoids 3MB bundle for editor-only surface. Committing BEFORE enqueue prevents race where worker picks up job before row is visible. TypeScript discriminated union gives exhaustive-check at client call sites.

Alternatives Rejected

Inline-only exec (rejected: head-of-line blocking). SSE streaming (deferred to Wave 2 dashboard live-refresh). Monaco bundled (rejected: 3MB cost). Redis Lua lock script (rejected: RQ job_id sufficient). Streaming large result sets (deferred). Worker sync SQLAlchemy via psycopg2 (rejected: async consistency wins). Separate /execute + /enqueue endpoints (rejected: union keeps client simple). Job cancellation UI (deferred to Wave 2; statement timeout is safety valve). CodeMirror 6 (rejected: Monaco better SQL LSP). Commit-after-enqueue (rejected: race).

Outcome

Pending

Graph View

Backlinks