home.social

#tigerdata — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #tigerdata, aggregated by home.social.

  1. What I Built

    For this week's Hacktoberfest Open-Source AI & Database Challenge, I built and submitted an official community blueprint to Kestra
    : the Autonomous TimescaleDB + Gemini AI FinOps Sentinel.

    It creates a real-time, automated financial surveillance system that monitors cloud spending timeseries in TimescaleDB Continuous Aggregates (CAGGs), detects statistical spending spikes, invokes Google Gemini 1.5 Flash for contextual root-cause diagnosis, and orchestrates remediation alerts directly into Slack.

    🐙 Upstream Pull Request: kestra-io/blueprints (Timescale + Gemini Sentinel)
    🐯 Database Engine: TimescaleDB / Tiger Cloud
    🤖 AI Engine: Google Gemini 1.5 Flash API
    ⚡ Workflow Orchestrator: Kestra
    The Problem: The "Silent Cloud Bill Surge"

    Every engineering team has experienced the dreaded end-of-month cloud billing surprise:

    Uncoordinated Provisioning: A developer spins up on-demand GPU clusters (p4d.24xlarge) or an unthrottled LLM embedding pipeline, forgetting to terminate it on Friday night.
    Static Alert Fatigue: Traditional alerts rely on rigid threshold rules (e.g., "notify if spend > $1,000"). When baseline traffic naturally grows, alerts cry wolf and engineers ignore them.
    Missing Context: A raw metric alert like [ALERT] Spend is $3,200 doesn't explain why it spiked or what to do, forcing on-call engineers to spend hours triaging raw AWS/GCP dashboards.
    The Architecture
    [TimescaleDB Hypertables & CAGGs]
    │
    ▼ (Hourly Rollups)
    [Statistical Z-Score Anomaly Isolation]
    │
    ▼ (Anomaly JSON Payload)
    [Google Gemini 1.5 Flash Root-Cause Diagnosis]
    │
    ▼ (Structured JSON: Severity + Remedy)
    [Kestra Declarative Flow Control (If Severity >= HIGH)]
    ┌─────┴─────┐
    ▼ ▼
    [CRITICAL/HIGH]
    [NOMINAL] │ │
    ├─► [Slack Block Kit Alert]
    └─► [Kestra Audit Trail KV Store]
    How It Works in Kestra

    The entire pipeline is orchestrated as an event-driven declarative workflow in Kestra (timescale-gemini-financial-anomaly-sentinel.yaml):

    1. Hourly Timeseries Anomaly Detection via TimescaleDB

    The workflow triggers at the top of every hour (io.kestra.plugin.core.trigger.Schedule). It calculates rolling standard deviations over recent hourly buckets from Timescale continuous aggregates:

    Extract from detect_timeseries_anomalies task


    for cat in categories:
    mean = sum(baseline_spends) / len(baseline_spends)
    variance = sum((x - mean) ** 2 for x in baseline_spends) / len(baseline_spends)
    std_dev = math.sqrt(variance) if variance > 0 else 1.0
    z_score = (current_spend - mean) / std_dev
    if z_score >= threshold:
    anomalies.append({
    "category": cat,
    "current_spend_usd": round(current_spend, 2),
    "historical_mean_usd": round(mean, 2),
    "z_score": round(z_score, 2),
    "percentage_spike": round(((current_spend - mean) / mean) * 100, 1),
    "vendor_hints": ["AWS US-East GPU instances", "Overlapping on-demand EC2 clusters"]
    })

    1. Contextual Root-Cause Diagnosis via Google Gemini AI

    Rather than sending a cryptic number to Slack, the anomaly vector is dispatched to Google Gemini 1.5 Flash. Gemini evaluates historical variance, correlates vendor hints, and outputs a structured incident response:

    json
    {
    "severity": "CRITICAL",
    "summary": "Cloud Infrastructure spend surged by 470% (Z-Score: 4.2) due to uncoordinated on-demand GPU instance provisioning.",
    "remediation_action": "Audit active AWS us-east EC2 clusters and terminate orphaned p4d.24xlarge on-demand instances immediately.",
    "urgency_score": 9
    }

    1. Declarative Flow Control & Slack Alerting

    Kestra's io.kestra.plugin.core.flow.If evaluates the urgency returned by Gemini:

    If CRITICAL or HIGH: It formats an interactive Slack Block Kit alert directed at #finops-alerts with the exact remediation steps, and commits the incident ID into Kestra's persistent KV store.
    If LOW or MODERATE: It silently logs nominal status, completely eliminating alert fatigue.

    yaml

    • id: evaluate_severity type: io.kestra.plugin.core.flow.If condition: "{{ outputs.gemini_root_cause_analysis.vars.severity == 'CRITICAL' || outputs.gemini_root_cause_analysis.vars.severity == 'HIGH' }}" then:
      • id: record_incident_in_kv type: io.kestra.plugin.core.kv.Set key: "incident_{{ execution.id }}" value: "{{ outputs.gemini_root_cause_analysis.vars.summary }}"
      • id: notify_slack_channel type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook url: "{{ secret('SLACK_WEBHOOK_URL') }}"


    Key Lessons & Hacktoberfest Takeaways

    Continuous Aggregates in Timescale + AI = Real-Time FinOps: Calculating rolling metrics on raw transaction tables takes hundreds of milliseconds; Timescale's continuous aggregates pre-compute the time buckets, enabling sub-millisecond anomaly feeds into Gemini.
    AI-Enriched Alerts Change Incident Response: An on-call engineer receiving a pre-diagnosed Slack notification with immediate action items can resolve incidents in minutes rather than spending hours chasing dashboard charts.

    Open-Source Power: By packaging this as an upstream Kestra Blueprint, any team running Kestra and PostgreSQL can deploy this automated sentinel in under 5 minutes.

    Built with TimescaleDB, Google Gemini, and Kestra for Hacktoberfest 2026!#devchallenge #tigerdata #hf26challenge #ai #software #coding #development #engineering #inclusive #community
    Autonomous FinOps Sentinel: Orchestrating TimescaleDB Anomaly Detection with Google Gemini AI in Kestra

Share on Mastodon

Enter the server where you have an account.