Second Brain

cost-observer-fivetran-sync-cost-observability-runwal-local

2 min read

Cost Observer (Fivetran sync cost observability) — Runwal-local, pull-based

Cost Observer — Runwal Fivetran cost visibility (2026-04-23)

What shipped: read-only observability over Fivetran sync costs. Queries RNWL_RAW.FIVETRAN_METADATA + FIVETRAN_METADATA_FIVETRAN_PLATFORM in Snowflake via key-pair auth, projects USD cost of would-be resync/unpause/frequency-change operations, classifies severity (routine/elevated/critical) for alerting-only. Never blocks a Fivetran API call.

Components:

  • Snowflake role FIVETRAN_COSTGUARD_RO + service user SVC_FIVETRAN_MCP_COSTGUARD (key-pair auth, no password) + warehouse WH_FIVETRAN_COSTGUARD (XS, 60s autosuspend, ~3 credits/month)
  • Audit table RNWL_RAW.FIVETRAN_MCP_AUDIT.COST_OBSERVATIONS (90d retention) — every projection logged
  • Python module fivetran_server.core.cost_observer (models, policy, client, queries.sql) — 100% test coverage on policy
  • 2 MCP tools in fivetran-mcp: observe_fivetran_connector_cost(connection_id, operation_type, new_sync_frequency_minutes?) + get_cost_observer_status(). Bumps tool count 175 → 177
  • Snowflake skill § “Fivetran Cost Intelligence” with 5 canonical SQL queries (top connectors by 30d MAR, current billing rate, projected resync cost, 12-month trend, observation audit log)

Access model: PULL-ONLY. No push notifications. No gating. Operator queries on-demand via snowflake MCP. Future push need (if ever) → reuse existing hostinger-alerts Telegram bot.

For future sessions: to query Fivetran cost intelligence, ONLY Snowflake MCP is needed. Fivetran MCP is optional (only for the 2 programmatic tools or actual Fivetran management).

Live data as of 2026-04-23: SAP HANA heaviest at 3.3M MAR/30d, Salesforce 1.4M, GA4 0.6M (cold-start). April MTD spend 379.61/million-MAR. Projected full resync cost: SAP ~26,357, GA4 ~$229.

Key architectural decision: originally scoped as blocking guardrail (G2 Cost Pre-Flight) that would intercept resync_connector and raise JSON-RPC -32006 on threshold exceed. AJ rejected mid-session — Runwal pipelines are live production, cannot be gated on threshold guess. Full same-session pivot to observability. See session-learnings.md entry “Enforcement-on-live-production-pipelines is a red flag”.

ST decisions: b92ece2e (enforcement scope — superseded), ed1b5fb6 (bias-corrections — partial, 3 corrections shipped under observability model), 90641bb8 (pivot + delivery, confidence 0.92).

  • Runwal Fivetran Platform Connector
  • Snowflake RNWL_RAW schemas
  • Enforcement-on-live-production-pipelines red flag

Graph View

Backlinks