Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,9 @@ service-account.json
*.temp
*.tmp

# dbt union test worktrees
worktrees/

# Claude instructions
CLAUDE.md

Expand Down
14 changes: 11 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
# dbt_hubspot v1.8.0

[PR #201](https://github.com/fivetran/dbt_hubspot/pull/201) includes the following updates:

## Under the Hood
- Migrates the `union_connections`, `apply_source_relation`, and `partition_by_source_relation` macros to the `dbt_fivetran_utils` package.
- Adds the `fivetran_using_source_casing` variable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the [Additional Configurations](https://github.com/fivetran/dbt_hubspot/#source-casing-for-case-sensitive-destinations) section of the README for details.

# dbt_hubspot v1.7.2

[PR #198](https://github.com/fivetran/dbt_hubspot/pull/198) includes the following update:
Expand Down Expand Up @@ -213,9 +221,9 @@ This release includes the following updates.
**6 total changes • 6 possible breaking changes**
| **Data Model/Column** | **Change type** | **Old** | **New** | **Notes** |
|-----------|----------------|--------------|--------------|---------|
| [`hubspot__deal_stages`](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__deal_stages) | Column Renamed | `is_pipeline_stage_closed_won` | `is_pipeline_stage_closed` | Column renamed to more accurately match the source data. The column is generated via a coalesce between the previous `closed_won` source column and the newly added `is_closed` column. This renamed column represents whether a deal is closed, regardless of its label as Closed Won or Closed Lost. You can still use the `pipeline_stage_label` column to differentiate between "Closed Won" and "Closed Lost" stages. ([PR #161](https://github.com/fivetran/dbt_hubspot/pull/161)) |
| [`hubspot__deal_stages`](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__deal_stages) | Column Renamed | `is_pipeline_stage_closed_won` | `is_pipeline_stage_closed` | Column renamed to more accurately match the source data. The column is generated via a coalesce between the previous `closed_won` source column and the newly added `is_closed` column. This renamed column represents whether a deal is closed, regardless of its label as "Closed Won" or "Closed Lost." You can still use the `pipeline_stage_label` column to differentiate between "Closed Won" and "Closed Lost" stages. ([PR #161](https://github.com/fivetran/dbt_hubspot/pull/161)) |
| [`hubspot__tickets`](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot.hubspot__tickets)<br> - `ticket_pipeline_id` <br> - `ticket_pipeline_stage_id` | Column datatype | `int` or `string` | `string` | Explicitly cast ticket pipeline identifiers to string format to prevent datatype errors in compilation. (Upstream changes from [source PR #151](https://github.com/fivetran/dbt_hubspot_source/pull/151)) |
| [`stg_hubspot__deal_pipeline_stage`](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot_source.stg_hubspot__deal_pipeline_stage) | Column Renamed | `is_closed_won` | `is_closed` | Column renamed to more accurately match the source data. The renamed `is_closed` column is generated via a coalesce between the previous `closed_won` source column and the newly added `is_closed` column. This renamed column represents whether a deal is closed, regardless of its label as Closed Won or Closed Lost. You can still use the `pipeline_stage_label` column to differentiate between "Closed Won" and "Closed Lost" stages. ([PR #161](https://github.com/fivetran/dbt_hubspot/pull/161)) |
| [`stg_hubspot__deal_pipeline_stage`](https://fivetran.github.io/dbt_hubspot/#!/model/model.hubspot_source.stg_hubspot__deal_pipeline_stage) | Column Renamed | `is_closed_won` | `is_closed` | Column renamed to more accurately match the source data. The renamed `is_closed` column is generated via a coalesce between the previous `closed_won` source column and the newly added `is_closed` column. This renamed column represents whether a deal is closed, regardless of its label as "Closed Won" or "Closed Lost." You can still use the `pipeline_stage_label` column to differentiate between "Closed Won" and "Closed Lost" stages. ([PR #161](https://github.com/fivetran/dbt_hubspot/pull/161)) |
| [`stg_hubspot__ticket`](https://fivetran.github.io/dbt_hubspot_source/#!/model/model.hubspot_source.stg_hubspot__ticket)<br> - `ticket_pipeline_id` <br> - `ticket_pipeline_stage_id` | Column datatype | `int` or `string` | `string` | Explicitly cast ticket pipeline identifiers to string format to prevent datatype errors in compilation. ([Source PR #151](https://github.com/fivetran/dbt_hubspot_source/pull/151)) |
| [`stg_hubspot__ticket_pipeline`](https://fivetran.github.io/dbt_hubspot_source/#!/model/model.hubspot_source.stg_hubspot__ticket_pipeline)<br> - `ticket_pipeline_id` | Column datatype | `int` | `string` | Changed cast of ticket pipeline identifier to the original string format to prevent datatype errors in compilation. ([Source PR #151](https://github.com/fivetran/dbt_hubspot_source/pull/151)) |
| [`stg_hubspot__ticket_pipeline_stage`](https://fivetran.github.io/dbt_hubspot_source/#!/model/model.hubspot_source.stg_hubspot__ticket_pipeline_stage)<br>- `ticket_pipeline_id` <br> - `ticket_pipeline_stage_id` <br> | Column datatype | `int` | `string` | Changed cast of ticket pipeline identifiers to their original string format to prevent datatype errors in compilation. ([Source PR #151](https://github.com/fivetran/dbt_hubspot_source/pull/151)) |
Expand Down Expand Up @@ -917,4 +925,4 @@ PR [#84](https://github.com/fivetran/dbt_hubspot/pull/84) incorporates the follo
- Please note, if you are installing a version of `dbt_utils` in your `packages.yml` that is not in the range above then you will encounter a package dependency error.

# dbt_hubspot v0.1.0 -> v0.4.1
Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!
Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!
89 changes: 31 additions & 58 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
<!--section="hubspot_transformation_model"-->
# Hubspot dbt Package
# HubSpot dbt Package

This dbt package transforms data from Fivetran's Hubspot connector into analytics-ready tables.
This dbt package transforms data from Fivetran's HubSpot connector into analytics-ready tables.

## Resources

- Number of materialized models¹: 147
- Connector documentation
- [Hubspot connector documentation](https://fivetran.com/docs/connectors/applications/hubspot)
- [Hubspot ERD](https://fivetran.com/docs/connectors/applications/hubspot#schemainformation)
- [HubSpot connector documentation](https://fivetran.com/docs/connectors/applications/hubspot)
- [HubSpot ERD](https://fivetran.com/docs/connectors/applications/hubspot#schemainformation)
- dbt package documentation
- [GitHub repository](https://github.com/fivetran/dbt_hubspot)
- [dbt Docs](https://fivetran.github.io/dbt_hubspot/#!/overview)
Expand Down Expand Up @@ -68,12 +68,12 @@ You can either add this dbt package in the Fivetran dashboard or import it into
<!--section-end-->

### Install the package
Include the following hubspot package version in your `packages.yml` file:
Include the following HubSpot package version in your `packages.yml` file:
> TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
```yaml
packages:
- package: fivetran/hubspot
version: [">=1.7.0", "<1.8.0"] # we recommend using ranges to capture non-breaking changes automatically
version: [">=1.8.0", "<1.9.0"] # we recommend using ranges to capture non-breaking changes automatically
```
> All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/hubspot_source` in your `packages.yml` since this package has been deprecated.

Expand All @@ -96,76 +96,41 @@ dispatch:
```

### Define database and schema variables

#### Option A: Single connection
By default, this package runs using your [destination](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile) and the `hubspot` schema. If this is not where your hubspot data is (for example, if your hubspot schema is named `hubspot_fivetran`), add the following configuration to your root `dbt_project.yml` file:
By default, this package runs using your destination and the `hubspot` schema. If this is not where your HubSpot data is (for example, if your HubSpot schema is named `hubspot_fivetran`), add the following configuration to your root `dbt_project.yml` file:

```yml
vars:
hubspot:
hubspot_database: your_database_name
hubspot_schema: your_schema_name
hubspot_database: your_destination_name
hubspot_schema: your_schema_name
```

#### Option B: Union multiple connections
If you have multiple hubspot connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. For each source table, the package will union all of the data together and pass the unioned table into the transformations. The `source_relation` column in each model indicates the origin of each record.
If you have multiple HubSpot connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. For each source table, the package will union all of the data together and pass the unioned table into the transformations. The `source_relation` column in each model indicates the origin of each record.

To use this functionality, you will need to set the `hubspot_sources` variable in your root `dbt_project.yml` file:

```yml
# dbt_project.yml

vars:
hubspot:
hubspot_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name
```

##### Recommended: Incorporate unioned sources into DAG
> *If you are running the package through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore), the below step is necessary in order to synchronize model runs with your hubspot connections. Alternatively, you may choose to run the package through Fivetran [Quickstart](https://fivetran.com/docs/transformations/quickstart), which would create separate sets of models for each hubspot source rather than one set of unioned models.*

By default, this package defines one single-connection source, called `hubspot`, which will be disabled if you are unioning multiple connections. This means that your DAG will not include your hubspot sources, though the package will run successfully.

To properly incorporate all of your hubspot connections into your project's DAG:
1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_hubspot.yml` [file](https://github.com/fivetran/dbt_hubspot/blob/main/models/staging/src_hubspot.yml).

```yml
# a .yml file in your root project

version: 2

sources:
- name: <name> # ex: Should match name in hubspot_sources
schema: <schema_name>
database: <database_name>
loader: fivetran
config:
loaded_at_field: _fivetran_synced
freshness: # feel free to adjust to your liking
warn_after: {count: 72, period: hour}
error_after: {count: 168, period: hour}

tables: # copy and paste from hubspot/models/staging/src_hubspot.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once
hubspot:
hubspot_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name
```

> **Note**: If there are source tables you do not have (see [Disable/enable models and sources](https://github.com/fivetran/dbt_hubspot?tab=readme-ov-file#disableenable-models-and-sources)), you may still include them, as long as you have set the right variables to `False`.
#### Optional: Incorporate unioned sources into DAG

2. Set the `has_defined_sources` variable (scoped to the `hubspot` package) to `True`, like such:
```yml
# dbt_project.yml
vars:
hubspot:
has_defined_sources: true
```
If you use [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore) and are unioning multiple HubSpot connections, you can define your sources in a property `.yml` file, [using this as a template](https://github.com/fivetran/dbt_hubspot/blob/main/models/staging/src_hubspot.yml). Set the variable `has_defined_sources: true` under the HubSpot namespace in your `dbt_project.yml`. Otherwise, your HubSpot connections won't appear in your DAG. See the `union_connections` macro [documentation](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#optional-union-connections-defined-sources-configuration) for full configuration details.

### Disable/enable models and sources
When setting up your Hubspot connection in Fivetran, it is possible that not every table this package expects will be synced. This can occur because you either don't use that functionality in Hubspot or have actively decided to not sync some tables. Therefore we have added enable/disable configs in the `src.yml` to allow you to disable certain sources not present. Downstream models are automatically disabled as well. In order to disable the relevant functionality in the package, you will need to add the relevant variables in your root `dbt_project.yml`.
When setting up your HubSpot connection in Fivetran, it is possible that not every table this package expects will be synced. This can occur because you either don't use that functionality in HubSpot or have actively decided to not sync some tables. Therefore we have added enable/disable configs in the `src.yml` to allow you to disable certain sources not present. Downstream models are automatically disabled as well. In order to disable the relevant functionality in the package, you will need to add the relevant variables in your root `dbt_project.yml`.

By default, all variables are assumed to be `true`, **with the exception of the below**. These default to `false` and must be explicitly enabled if needed:

Expand Down Expand Up @@ -386,6 +351,14 @@ vars:
hubspot_<default_source_table_name>_identifier: your_table_name
```

#### Source casing for case-sensitive destinations
By default, the package applies case-insensitive comparisons when resolving `source_relation` values. If your destination is case-sensitive and you want downstream transformations to respect the exact casing of your source database and schema names, set the following variable:

```yml
vars:
fivetran_using_source_casing: true
```

### (Optional) Orchestrate your models with Fivetran Transformations for dbt Core™
<details><summary>Expand for details</summary>
<br>
Expand Down Expand Up @@ -424,4 +397,4 @@ We highly encourage and welcome contributions to this package. Learn how to cont

## Are there any resources available?
- If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_hubspot/issues/new/choose) section to find the right avenue of support for you.
- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'hubspot'
version: '1.7.2'
version: '1.8.0'

config-version: 2
require-dbt-version: [">=1.3.0", "<3.0.0"]
Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

Loading
Loading