Merge pull request 'development' (#99) from development into main #102
@@ -208,9 +208,9 @@ The codebase is structured to support multiple dashboard projects:
|
|||||||
|--------|---------|
|
|--------|---------|
|
||||||
| `public` | Shared dimensions (dim_time) |
|
| `public` | Shared dimensions (dim_time) |
|
||||||
| `raw_toronto` | Toronto-specific raw/dimension tables |
|
| `raw_toronto` | Toronto-specific raw/dimension tables |
|
||||||
| `staging` | dbt staging views |
|
| `stg_toronto` | Toronto dbt staging views |
|
||||||
| `intermediate` | dbt intermediate views |
|
| `int_toronto` | Toronto dbt intermediate views |
|
||||||
| `marts` | dbt mart tables |
|
| `mart_toronto` | Toronto dbt mart tables |
|
||||||
|
|
||||||
### Geographic Reality (Toronto Housing)
|
### Geographic Reality (Toronto Housing)
|
||||||
|
|
||||||
|
|||||||
@@ -22,12 +22,12 @@ models:
|
|||||||
staging:
|
staging:
|
||||||
toronto:
|
toronto:
|
||||||
+materialized: view
|
+materialized: view
|
||||||
+schema: staging
|
+schema: stg_toronto
|
||||||
intermediate:
|
intermediate:
|
||||||
toronto:
|
toronto:
|
||||||
+materialized: view
|
+materialized: view
|
||||||
+schema: intermediate
|
+schema: int_toronto
|
||||||
marts:
|
marts:
|
||||||
toronto:
|
toronto:
|
||||||
+materialized: table
|
+materialized: table
|
||||||
+schema: marts
|
+schema: mart_toronto
|
||||||
|
|||||||
11
dbt/macros/generate_schema_name.sql
Normal file
11
dbt/macros/generate_schema_name.sql
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
-- Override dbt default schema name generation.
|
||||||
|
-- Use the custom schema name directly instead of
|
||||||
|
-- concatenating with the target schema.
|
||||||
|
-- See: https://docs.getdbt.com/docs/build/custom-schemas
|
||||||
|
{% macro generate_schema_name(custom_schema_name, node) %}
|
||||||
|
{%- if custom_schema_name is none -%}
|
||||||
|
{{ target.schema }}
|
||||||
|
{%- else -%}
|
||||||
|
{{ custom_schema_name | trim }}
|
||||||
|
{%- endif -%}
|
||||||
|
{% endmacro %}
|
||||||
0
dbt/macros/toronto/.gitkeep
Normal file
0
dbt/macros/toronto/.gitkeep
Normal file
@@ -122,9 +122,9 @@ erDiagram
|
|||||||
|--------|---------|------------|
|
|--------|---------|------------|
|
||||||
| `public` | Shared dimensions (dim_time) | SQLAlchemy |
|
| `public` | Shared dimensions (dim_time) | SQLAlchemy |
|
||||||
| `raw_toronto` | Toronto dimension and fact tables | SQLAlchemy |
|
| `raw_toronto` | Toronto dimension and fact tables | SQLAlchemy |
|
||||||
| `staging` | Staging models | dbt |
|
| `stg_toronto` | Toronto staging models | dbt |
|
||||||
| `intermediate` | Intermediate models | dbt |
|
| `int_toronto` | Toronto intermediate models | dbt |
|
||||||
| `marts` | Analytical tables | dbt |
|
| `mart_toronto` | Toronto analytical tables | dbt |
|
||||||
|
|
||||||
### Raw Toronto Schema (raw_toronto)
|
### Raw Toronto Schema (raw_toronto)
|
||||||
|
|
||||||
@@ -149,7 +149,7 @@ Shared dimensions used across all projects:
|
|||||||
|-------|-------------|
|
|-------|-------------|
|
||||||
| `dim_time` | Time dimension (monthly grain) |
|
| `dim_time` | Time dimension (monthly grain) |
|
||||||
|
|
||||||
### Staging Schema (dbt)
|
### Staging Schema - stg_toronto (dbt)
|
||||||
|
|
||||||
Staging models provide 1:1 cleaned representations of source data:
|
Staging models provide 1:1 cleaned representations of source data:
|
||||||
|
|
||||||
@@ -164,7 +164,7 @@ Staging models provide 1:1 cleaned representations of source data:
|
|||||||
| `stg_dimensions__cmhc_zones` | raw.cmhc_zones | CMHC zone boundaries |
|
| `stg_dimensions__cmhc_zones` | raw.cmhc_zones | CMHC zone boundaries |
|
||||||
| `stg_cmhc__zone_crosswalk` | raw.crosswalk | Zone-neighbourhood mapping |
|
| `stg_cmhc__zone_crosswalk` | raw.crosswalk | Zone-neighbourhood mapping |
|
||||||
|
|
||||||
### Marts Schema (dbt)
|
### Marts Schema - mart_toronto (dbt)
|
||||||
|
|
||||||
Analytical tables ready for dashboard consumption:
|
Analytical tables ready for dashboard consumption:
|
||||||
|
|
||||||
|
|||||||
@@ -269,7 +269,7 @@ LOG_LEVEL=INFO
|
|||||||
| `db-init` | Initialize database schema |
|
| `db-init` | Initialize database schema |
|
||||||
| `db-reset` | Drop and recreate database (DESTRUCTIVE) |
|
| `db-reset` | Drop and recreate database (DESTRUCTIVE) |
|
||||||
| `load-data` | Load Toronto data from APIs, seed dev data |
|
| `load-data` | Load Toronto data from APIs, seed dev data |
|
||||||
| `load-data-only` | Load Toronto data without dbt or seeding |
|
| `load-toronto-only` | Load Toronto data without dbt or seeding |
|
||||||
| `seed-data` | Seed sample development data |
|
| `seed-data` | Seed sample development data |
|
||||||
| `run` | Start Dash dev server |
|
| `run` | Start Dash dev server |
|
||||||
| `test` | Run pytest |
|
| `test` | Run pytest |
|
||||||
|
|||||||
@@ -103,7 +103,15 @@ models:
|
|||||||
staging:
|
staging:
|
||||||
{dashboard_name}:
|
{dashboard_name}:
|
||||||
+materialized: view
|
+materialized: view
|
||||||
+schema: staging
|
+schema: stg_{dashboard_name}
|
||||||
|
intermediate:
|
||||||
|
{dashboard_name}:
|
||||||
|
+materialized: view
|
||||||
|
+schema: int_{dashboard_name}
|
||||||
|
marts:
|
||||||
|
{dashboard_name}:
|
||||||
|
+materialized: table
|
||||||
|
+schema: mart_{dashboard_name}
|
||||||
```
|
```
|
||||||
|
|
||||||
Follow naming conventions:
|
Follow naming conventions:
|
||||||
|
|||||||
Reference in New Issue
Block a user