lmiranda
89f0354ccc
Add new data-platform plugin for data engineering workflows with: MCP Server (32 tools): - pandas operations (14 tools): read_csv, read_parquet, read_json, to_csv, to_parquet, describe, head, tail, filter, select, groupby, join, list_data, drop_data - PostgreSQL/PostGIS (10 tools): pg_connect, pg_query, pg_execute, pg_tables, pg_columns, pg_schemas, st_tables, st_geometry_type, st_srid, st_extent - dbt integration (8 tools): dbt_parse, dbt_run, dbt_test, dbt_build, dbt_compile, dbt_ls, dbt_docs_generate, dbt_lineage Plugin Features: - Arrow IPC data_ref system for DataFrame persistence across tool calls - Pre-execution validation for dbt with `dbt parse` - SessionStart hook for PostgreSQL connectivity check (non-blocking) - Hybrid configuration (system ~/.config/claude/postgres.env + project .env) - Memory management with 100k row limit and chunking support Commands: /initial-setup, /ingest, /profile, /schema, /explain, /lineage, /run Agents: data-ingestion, data-analysis Test suite: 71 tests covering config, data store, pandas, postgres, dbt tools Addresses data workflow issues from personal-portfolio project: - Lost data after multiple interactions (solved by Arrow IPC data_ref) - dbt 1.9+ syntax deprecation (solved by pre-execution validation) - Ungraceful PostgreSQL error handling (solved by SessionStart hook) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
45 lines
1009 B
Markdown
45 lines
1009 B
Markdown
# /ingest - Data Ingestion
|
|
|
|
Load data from files or database into the data platform.
|
|
|
|
## Usage
|
|
|
|
```
|
|
/ingest [source]
|
|
```
|
|
|
|
## Workflow
|
|
|
|
1. **Identify data source**:
|
|
- If source is a file path, determine format (CSV, Parquet, JSON)
|
|
- If source is "db" or a table name, query PostgreSQL
|
|
|
|
2. **Load data**:
|
|
- For files: Use `read_csv`, `read_parquet`, or `read_json`
|
|
- For database: Use `pg_query` with appropriate SELECT
|
|
|
|
3. **Validate**:
|
|
- Check row count against limits
|
|
- If exceeds 100k rows, suggest chunking or filtering
|
|
|
|
4. **Report**:
|
|
- Show data_ref, row count, columns, and memory usage
|
|
- Preview first few rows
|
|
|
|
## Examples
|
|
|
|
```
|
|
/ingest data/sales.csv
|
|
/ingest data/customers.parquet
|
|
/ingest "SELECT * FROM orders WHERE created_at > '2024-01-01'"
|
|
```
|
|
|
|
## Available Tools
|
|
|
|
Use these MCP tools:
|
|
- `read_csv` - Load CSV files
|
|
- `read_parquet` - Load Parquet files
|
|
- `read_json` - Load JSON/JSONL files
|
|
- `pg_query` - Query PostgreSQL database
|
|
- `list_data` - List loaded DataFrames
|