dwh-auditor — DWH Cost Audit & Governance Tool

dwh-auditor is an open-source CLI tool that parses BigQuery’s INFORMATION_SCHEMA to instantly perform cost optimization, security auditing, and governance enforcement for your cloud data warehouse.

Tip

It never accesses actual table data. Since it only reads metadata (INFORMATION_SCHEMA), it can be deployed instantly even in enterprise environments with strict security policies.

Key Features

#	Feature	Description
💸	Ad-hoc High-Cost Query Detection	Displays the Top-N ranking of single queries with the highest billed bytes over the past N days.
🔄	Recurring Execution Alert (Periodic High-Cost Queries)	Detects queries executed periodically from batches or dashboards that have high accumulated costs.
🚨	Full Scan Detection	Warns of inefficient full table scans caused by missing partition filters in the WHERE clause.
🧟	Zombie Table Detection	Identifies tables that have not been referenced for a long time to visualize unnecessary storage costs.
📊	Multi-format Output (Markdown / JSON)	Integrate into CI/CD to save as GitHub Actions Artifacts or output jq-parsable results.

Quickstart

pip install dwh-auditor

# Generate a configuration file
dwh-auditor init

# Audit BigQuery project (Console output)
dwh-auditor analyze --project my-gcp-project --days 30

# Generate Markdown report
dwh-auditor analyze --project my-gcp-project --output markdown

Documentation Table of Contents

Introduction

• Quickstart — Installation and Initial Audit
  • Prerequisites
  • Installation
  • GCP Authentication Settings
  • Generating Configuration File (init command)
  • Executing Audit (analyze command)
  • Command Reference
  • Next Steps
• Configuration File (config.yaml)
  • Generating the Configuration File
  • Full Configuration Schema
  • Configuration Details
  • Pydantic Schema

Architecture

• Architecture — 3-Tier DWH Audit Architecture
  • Overall Processing Flow
  • Responsibilities of Each Tier
  • Internal Data Model
  • Testing Strategy — Why Fast Testing is Possible
  • Extensibility — Supporting other DWHs

API Reference

• API Reference
  • dwh_auditor.config — Load configuration files and manage DWH settings
  • dwh_auditor.models — Internal data model for DWH auditing (Pydantic)
  • dwh_auditor.extractor — BigQuery metadata extraction layer
  • dwh_auditor.analyzer — DWH cost analysis/security diagnosis logic
  • dwh_auditor.reporter — Audit result output/report generation layer

Deployment

• Contributing Guide

Required IAM Permissions

dwh-auditor only reads metadata, requiring minimal permissions.

IAM Role	Usage
roles/bigquery.metadataViewer	View dataset and table metadata
roles/bigquery.resourceViewer	View job history (INFORMATION_SCHEMA.JOBS)

Warning

Permissions beyond roles/bigquery.dataViewer are NOT required. It does not access the actual data (records) in the tables.

Index and Search

• Index

• Module Index

• Search Page