PR #201 includes the following updates:
- Migrates the
union_connections,apply_source_relation, andpartition_by_source_relationmacros to thedbt_fivetran_utilspackage. - Adds the
fivetran_using_source_casingvariable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the Additional Configurations section of the README for details.
PR #198 includes the following update:
1 total change • 0 possible breaking changes
| Data Model(s) | Change type | Old | New | Notes |
|---|---|---|---|---|
stg_hubspot__contact |
Changed field | merged_object_ids (string or integer) |
merged_object_ids (string) |
HubSpot can sync this field as an integer when no contacts have been merged. Casting to string prevents runtime errors in the downstream merge_contacts() macro where the split function requires a string. |
PR #195 includes the following updates:
- Optimizes the
merge_contacts()macro for Databricks destinations.
PR #194 includes the following updates:
4 total changes • 4 possible breaking changes
| Data Model(s) | Change type | Old | New | Notes |
|---|---|---|---|---|
hubspot__email_sends |
Removed column | was_unsubcribed |
Field deprecated in v1.5.0 release has been sunset. Use the corrected spelling was_unsubscribed existing field instead. |
|
hubspot__engagements |
Removed columns | owner_idis_activecreated_timestampoccurred_timestampactivity_typelast_updated_timestamp |
Fields deprecated for HubSpot v3 API connectors in v0.11.0 release have been sunset. The engagement source table now only contains type, id, _fivetran_synced, portal_id, and source_relation. All engagement properties (timestamps, owner, status) are available in child engagement tables (e.g., hubspot__engagement_emails, hubspot__engagement_notes). Junction table aggregations (contact_ids, deal_ids, company_ids) remain available in hubspot__engagements. |
|
stg_hubspot__contact_list |
Removed columns | is_deletableis_dynamicmetadata_errormetadata_last_processing_state_change_atmetadata_last_size_change_atmetadata_processingmetadata_sizeportal_id |
Fields deprecated for HubSpot v3 API connectors in v0.22.0 release have been sunset. | |
stg_hubspot__engagement |
Removed columns | owner_idis_activecreated_timestampoccurred_timestamp |
Fields deprecated for HubSpot v3 API connectors v0.11.0 release have been sunset. The engagement source table now only contains type, id, _fivetran_synced, and portal_id. These properties are available in child engagement staging tables instead. |
- Updates
engagements_joinedmacro to remove all coalesce logic to remove sunset columns. - Brought seed column types in
integration_tests/dbt_project.ymlup to date to resolve runtime warnings.
- Removes documentation for above sunset fields in the above models.
PR #193 includes the following updates:
- Removes duplicate column definitions in YAML schema files to avoid
dbt parsewarnings. - Corrects descriptions for several fields.
PR #192 includes the following updates:
- Updates README with standardized Fivetran formatting.
- In the
quickstart.ymlfile:- Adds
supported_varsfor Quickstart UI customization.
- Adds
3 total changes • 1 possible breaking changes
| Data Model(s) | Change type | Old | New | Notes |
|---|---|---|---|---|
stg_hubspot__contacthubspot__contacts |
Removed column | calculated_merged_vids |
Field deprecated in v1.3.0 has been sunset. Use merged_object_ids instead |
|
hubspot__email_sends |
Added column | was_unsubscribed |
Fix spelling error - replaces misspelled was_unsubcribed column |
|
hubspot__email_sends |
Deprecated column | was_unsubcribed |
[DEPRECATED] Maintained for backwards compatibility. Use was_unsubscribed instead. Will be removed in a future release. |
- Updates seeds and macros to support the schema changes.
PR #188 includes the following updates:
- Increases the required dbt version upper limit to v3.0.0
PR #186 and PR #187 include the following updates:
8 total changes • 6 possible breaking changes
| Data Model(s) | Change type | Old | New | Notes |
|---|---|---|---|---|
stg_hubspot__owner_teamstg_hubspot__formstg_hubspot__rolestg_hubspot__teamstg_hubspot__team_userstg_hubspot__users |
New column | is_deleted (aliased from _fivetran_deleted) |
Boolean indicating whether a record has been deleted in Hubspot and/or inferred deleted in Hubspot by Fivetran | |
| All models | New column | source_relation |
Identifies the source connection when using multiple hubspot connections | |
hubspot__company_history |
Updated surrogate key | id = company_id + field_name + valid_from |
id = source_relation + company_id + field_name + valid_from |
|
hubspot__contact_history |
Updated surrogate key | id = contact_id + field_name + valid_from |
id = source_relation + contact_id + field_name + valid_from |
|
hubspot__deal_history |
Updated surrogate key | id = deal_id + field_name + valid_from |
id = source_relation + deal_id + field_name + valid_from |
|
hubspot__daily_ticket_history |
Updated surrogate key | ticket_day_id = date_day + ticket_id |
ticket_day_id = source_relation + date_day + ticket_id |
|
int_hubspot__pivot_daily_ticket_historyint_hubspot__ticket_calendar_spine |
Updated surrogate key | id = date_day + ticket_id |
id = source_relation + date_day + ticket_id |
|
int_hubspot__daily_ticket_history |
Updated surrogate key | id = date_day + ticket_id + field_name |
id = source_relation + date_day + ticket_id + field_name |
If you are currently using the hubspot__company_pass_through_columns or hubspot__deal_pass_through_columns variables to persist the _fivetran_deleted field, please add an alias to avoid duplicate column errors.
Please note that
_fivetran_deletedis coalesced with theis_<company/deal>_eabledfield present in thehubspot__companies,hubspot__deals, andhubspot__deal_stagesmodels.
vars:
hubspot__company_pass_through_columns:
- name: "_fivetran_deleted"
alias: "_fivetran_company_deleted" # Can choose any alias other than _fivetran_deleted
hubspot__deal_pass_through_columns:
- name: "_fivetran_deleted"
alias: "_fivetran_deal_deleted" # Can choose any alias other than _fivetran_deleted- Union Data Functionality: This release supports running the package on multiple hubspot source connections. See the README for details on how to leverage this feature.
- Coalesces
is_<company/deal>_deletedwith_fivetran_deletedinstg_hubspot__companyandstg_hubspot__deal.
- Removes uniqueness tests. The new unioning feature requires combination-of-column tests to consider the new
source_relationcolumn in addition to the existing primary key, but this is not supported across dbt versions. - These tests will be reintroduced once a version-agnostic solution is available.
We have kept uniqueness tests on the surrogate keys listed above.
- Ensures that the
datatypeconfig of_fivetran_deleted/is_deleteduses the cross-compatibledbt.type_boolean()macro.
PR #182 includes the following updates:
5 total changes • 2 possible breaking changes
| Data Model(s) | Change type | Old | New | Notes |
|---|---|---|---|---|
| stg_hubspot__property | Primary Key Change | _fivetran_id + hubspot_object |
property_name + hubspot_object |
The composite primary key is now property_name and hubspot_object instead of _fivetran_id and hubspot_object. The _fivetran_id field is marked as deprecated. |
| stg_hubspot__property_option | Primary Key Change | property_id + property_option_label |
property_option_name + hubspot_object + property_option_label |
The composite primary key is now property_option_name, hubspot_object, and property_label instead of property_id and property_option_label. The property_id field is marked as deprecated. |
| stg_hubspot__property_option | New Column | hubspot_object |
Added to support new composite primary key and joins with the PROPERTY table. |
|
| stg_hubspot__property_option | New Column | property_option_name |
Added to support new composite primary key. | |
| int_hubspot__email_aggregate_status_change | New Column | unique_key |
Adds surrogate key generated from email_campaign_id and email_send_id to ensure uniqueness. Previously we only included email_send_id in its uniqueness test. |
- Fixes join condition in
hubspot__email_sendsmodel to include bothemail_send_idandemail_campaign_idwhen joining with unsubscribes data, preventing potential data integrity issues with duplicate email send IDs across different campaigns.
- Updates join logic in the
add_property_labelsmacro to joinPROPERTYandPROPERTY_OPTIONonhubspot_objectinstead of_fivetran_id/property_id. This is used in the following staging models:stg_hubspot__companystg_hubspot__contactstg_hubspot__dealstg_hubspot__ticket
- Updates
get_property_option_columnsmacro to include the newhubspot_objectandnamecolumns for property options. - Updates uniqueness tests in
int_hubspot__email_aggregate_status_changeto use the newunique_keyfield instead ofemail_send_id. - Updates documentation in staging models to reflect the deprecated status of
_fivetran_idandproperty_idfields and explain the new composite primary key structure. - Updates
property_optionseed data to include the newly added columns. - Adds consistency data validation tests for
hubspot__companies.
PR #171 includes the following updates:
3 total changes • 2 possible breaking changes
| Model | Change type | Old | New | Notes |
|---|---|---|---|---|
| hubspot__contacts int_hubspot__contact_merge_adjust stg_hubspot__contact |
New Column | merged_object_ids |
This column is now used to merge contacts instead of calculated_merged_vids. It is more complete in that it captures multi-step or chained contact merges, which calculated_merged_vids does not. It is aliased from property_hs_merged_object_ids in the raw source data. |
If you have the following configuration, please remove it. Otherwise, the above field additions will produce duplicate column errors.
# dbt_project.yml
vars:
hubspot__contact_pass_through_columns:
- name: property_hs_merged_object_ids
alias: merged_object_ids- Adjusted join logic in
hubspot__ticketsto avoid potential data type mismatch errors. - Updated
merge_contacts()macro to referencemerged_object_idsinstead ofcalculated_merged_vids. - Resolved a dbt Fusion error around
{% set abc = abc.append(...) %}, as this syntax is valid only in dbt Core. We have opted fordoinstead ofset.
- Updated README to clarify the default value of source-disabling/enabling variables here.
PR #167 includes the following updates:
- Removed the dependency on the
fivetran/hubspot_sourcepackage.- All functionality from the source package has been merged into this transformation package for improved maintainability and clarity.
- If you reference
fivetran/hubspot_sourcein yourpackages.yml, you must remove this dependency to avoid conflicts. - Any source overrides referencing the
fivetran/hubspot_sourcepackage will also need to be removed or updated to reference this package. - Update any hubspot_source-scoped variables to be scoped to only under this package. See the README for how to configure the build schema of staging models.
- As part of the consolidation, vars are no longer used to reference staging models, and only sources are represented by vars. Staging models are now referenced directly with
ref()in downstream models.
- Updated package to maintain compatibility with dbt-core versions both before and after v1.10.6, which introduced a breaking change to multi-argument test syntax (e.g.,
unique_combination_of_columns). - Temporarily removed unsupported tests to avoid errors and ensure smoother upgrades across different dbt-core versions. These tests will be reintroduced once a safe migration path is available.
- Removed all
dbt_utils.unique_combination_of_columnstests. - Moved
loaded_at_field: _fivetran_syncedunder theconfig:block insrc_hubspot.yml.
- Removed all
This release includes the following updates.
6 total changes • 6 possible breaking changes
| Data Model/Column | Change type | Old | New | Notes |
|---|---|---|---|---|
hubspot__deal_stages |
Column Renamed | is_pipeline_stage_closed_won |
is_pipeline_stage_closed |
Column renamed to more accurately match the source data. The column is generated via a coalesce between the previous closed_won source column and the newly added is_closed column. This renamed column represents whether a deal is closed, regardless of its label as "Closed Won" or "Closed Lost." You can still use the pipeline_stage_label column to differentiate between "Closed Won" and "Closed Lost" stages. (PR #161) |
hubspot__tickets- ticket_pipeline_id - ticket_pipeline_stage_id |
Column datatype | int or string |
string |
Explicitly cast ticket pipeline identifiers to string format to prevent datatype errors in compilation. (Upstream changes from source PR #151) |
stg_hubspot__deal_pipeline_stage |
Column Renamed | is_closed_won |
is_closed |
Column renamed to more accurately match the source data. The renamed is_closed column is generated via a coalesce between the previous closed_won source column and the newly added is_closed column. This renamed column represents whether a deal is closed, regardless of its label as "Closed Won" or "Closed Lost." You can still use the pipeline_stage_label column to differentiate between "Closed Won" and "Closed Lost" stages. (PR #161) |
stg_hubspot__ticket- ticket_pipeline_id - ticket_pipeline_stage_id |
Column datatype | int or string |
string |
Explicitly cast ticket pipeline identifiers to string format to prevent datatype errors in compilation. (Source PR #151) |
stg_hubspot__ticket_pipeline- ticket_pipeline_id |
Column datatype | int |
string |
Changed cast of ticket pipeline identifier to the original string format to prevent datatype errors in compilation. (Source PR #151) |
stg_hubspot__ticket_pipeline_stage- ticket_pipeline_id - ticket_pipeline_stage_id |
Column datatype | int |
string |
Changed cast of ticket pipeline identifiers to their original string format to prevent datatype errors in compilation. (Source PR #151) |
- Removed integer casting for
hs_pipelineandhs_pipeline_stageinhubspot__daily_ticket_history, asticket_pipeline_idandticket_pipeline_stage_idare now strings from the schema changes above. (PR #165)
- Updated seed files to validate that the new identifiers compile as expected in both the source and transform packages. (PR #165)
- Documented the column updates and deprecations in the respective yml files. (PR #161)
- Updated the
deal_pipeline_stage_dataseed to include theis_closedcolumn. (PR #161) - Introduced the generate-docs github workflow for consistent docs generation. (PR #161)
- Added the
consistency_deal_stagesvalidation test to verify accuracy of thehubspot__deal_stagesmodel on subsequent releases. (PR #161) - Updated conditions in
.github/workflows/auto-release.yml. (PR #162) - Added
+docs: show: Falsetointegration_tests/dbt_project.yml. (PR #162) - Migrated
flags(e.g.,send_anonymous_usage_stats,use_colors) fromsample.profiles.ymltointegration_tests/dbt_project.yml. (PR #162) - Updated
maintainer_pull_request_template.mdwith improved checklist. (PR #162) - Updated
.gitignoreto exclude additional DBT, Python, and system artifacts. (PR #162)
PR #160 includes the following updates:
8 total changes • 8 possible breaking changes
| Model | Change type | Old name | New name | Notes |
|---|---|---|---|---|
| hubspot__contacts | Column Renamed | first_conversion_form_id |
calculated_first_conversion_form_id |
Column renamed to avoid conflicts when passthrough columns are enabled. |
| hubspot__contacts | Column Renamed | first_conversion_date |
calculated_first_conversion_date |
Column renamed to avoid conflicts when passthrough columns are enabled. |
| hubspot__contacts | Column Renamed | first_conversion_form_name |
calculated_first_conversion_form_name |
Column renamed to avoid conflicts when passthrough columns are enabled. |
| hubspot__contacts | Column Renamed | first_conversion_form_type |
calculated_first_conversion_form_type |
Column renamed to avoid conflicts when passthrough columns are enabled. |
| hubspot__contacts | Column Renamed | most_recent_conversion_form_id |
calculated_most_recent_conversion_form_id |
Column renamed to avoid conflicts when passthrough columns are enabled. |
| hubspot__contacts | Column Renamed | most_recent_conversion_date |
calculated_most_recent_conversion_date |
Column renamed to avoid conflicts when passthrough columns are enabled. |
| hubspot__contacts | Column Renamed | most_recent_conversion_form_name |
calculated_most_recent_conversion_form_name |
Column renamed to avoid conflicts when passthrough columns are enabled. |
| hubspot__contacts | Column Renamed | most_recent_conversion_form_type |
calculated_most_recent_conversion_form_type |
Column renamed to avoid conflicts when passthrough columns are enabled. |
- Minor change to the
consistency_ticketsvalidation test to ensure it's only run when the service variable is enabled. - Updated
contact_dataseed file to confirm duplicate column potential issue is addressed.
PR #159 includes the following updates:
Note: This is not relevant to Fivetran Quickstart users.
Migrated freshness from a top-level source property to a source config in alignment with recent updates from dbt Core (HubSpot Source v0.20.0). This will resolve the following deprecation warning that users running dbt >= 1.9.6 may have received:
[WARNING]: Deprecated functionality
Found `freshness` as a top-level property of `hubspot` in file
`models/src_hubspot.yml`. The `freshness` top-level property should be moved
into the `config` of `hubspot`.
IMPORTANT: Users running dbt Core < 1.9.6 will not be able to utilize freshness tests in this release or any subsequent releases, as older versions of dbt will not recognize freshness as a source config and therefore not run the tests.
If you are using dbt Core < 1.9.6 and want to continue running HubSpot freshness tests, please elect one of the following options:
- (Recommended) Upgrade to dbt Core >= 1.9.6
- Do not upgrade your installed version of the
hubspotpackage. Pin your dependency on v0.23.0 in yourpackages.ymlfile. - Utilize a dbt override to overwrite the package's
hubspotsource and apply freshness via the previous release top-level property route. This will require you to copy and paste the entirety of the previous releasesrc_hubspot.ymlfile and add anoverrides: hubspot_sourceproperty.
- Updates to ensure integration tests use latest version of dbt.
PR #157 includes the following updates:
- These updates are considered breaking due to changes in model schemas, including the addition of new columns and joins.
int_hubspot__owners_enhanced: Enriches owner records with team and role metadataint_hubspot__form_metrics__by_contact: Adds form submission and conversion metrics at the contact levelhubspot__engagement_communication: Represents the relationship between communications and engagements
The following models have been updated with new columns sourced from the new intermediate models. Refer to the linked documentation for more details.
-
int_hubspot__owners_enhancedis joined to:hubspot__dealshubspot__tickets- New columns added to both models:
- Owner data
owner_active_user_id
- Team data:
owner_primary_team_idowner_primary_team_nameowner_all_team_idsowner_all_team_names
- Role data:
owner_role_idowner_role_name
- Owner data
-
int_hubspot__form_metrics__by_contactis joined to:hubspot__contacts- New columns added:
total_form_submissionstotal_unique_form_submissionsfirst_conversion_form_idfirst_conversion_datefirst_conversion_form_namefirst_conversion_form_typemost_recent_conversion_form_idmost_recent_conversion_datemost_recent_conversion_form_namemost_recent_conversion_form_type
The following variables control whether certain models or features are enabled based on your HubSpot connection.
Note: This dynamic enablement only applies when using Quickstart. Models will automatically be disabled if their required source tables are missing.
For dbt Core users, the default values below apply unless explicitly overridden.
hubspot_contact_form_enabled(default:true)hubspot_engagement_communication_enabled(default:false)hubspot_team_enabled(default:true)hubspot_role_enabled(default:true)hubspot_team_user_enabled(default:true)
See Step 4 of the README for more details on the variables.
Model behavior based on the above variables:
-
int_hubspot__owners_enhanced- Model enabled when
hubspot_owner_enabledistrue(existing variable) - Includes additional enrichment when the following are enabled:
hubspot_team_enabled– adds team-related columnshubspot_role_enabled– adds role-related columns
- Team and role enrichment are optional; you may enable either, both, or neither
- Model enabled when
-
hubspot__dealsandhubspot__tickets- Joins in
int_hubspot__owners_enhancedwhenhubspot_owner_enabledistrue(existing variable) - Inherit any team or role enrichment included in
int_hubspot__owners_enhanced, based on its upstream configuration
- Joins in
-
int_hubspot__form_metrics__by_contact- Enabled by
hubspot_contact_form_enabled
- Enabled by
-
hubspot__contacts- Joins in
int_hubspot__form_metrics__by_contactwhenhubspot_contact_form_enabledistrue
- Joins in
-
hubspot__engagement_communication- Enabled by
hubspot_engagement_communication_enabled
- Enabled by
- Added column-level descriptions for all new models and fields.
- Added seeds for each new source.
- Introduced new sources and their associated staging models to support expanded HubSpot data coverage:
| Source Table | Staging Model | Enablement Variable(s) - Default is true unless otherwise mentioned |
|---|---|---|
contact_form_submission |
stg_hubspot__contact_form_submission |
hubspot_contact_form_enabled hubspot_marketing_enabled |
engagement_communication |
stg_hubspot__engagement_communication |
hubspot_engagement_communication_enabled (default: false) hubspot_engagement_enabled hubspot_sales_enabled |
form |
stg_hubspot__form |
hubspot_contact_form_enabled hubspot_marketing_enabled |
owner_team |
stg_hubspot__owner_team |
hubspot_team_enabled |
role |
stg_hubspot__role |
hubspot_role_enabled |
team |
stg_hubspot__team |
hubspot_team_enabled |
team_user |
stg_hubspot__team_user |
hubspot_team_user_enabled hubspot_team_enabled |
user |
stg_hubspot__user |
hubspot_role_enabled |
- See Step 4 of the README for more details on enabling/disabling sources.
- Updated the
ownersource andstg_hubspot__ownerstaging model:- Added the
active_user_idcolumn. - Removed the requirement for
hubspot_sales_enabledto be true since this logic now applies to multiple objects (e.g. tickets and deals) downstream.
- Added the
- Added
get_*_columnsmacros for each new source.
PR #156 includes the following updates:
- Select fields have been deprecated from
hubspot__contact_listsmodel as well as the upstreamstg_hubspot__contact_listmodel (see HubSpot Source (#137) for details). The full removal date is planned for May 10th, 2025. Follow Deprecation Issue #138 for more details and updates around the planned sunsetting of the fields.- The deprecated fields include the following:
is_deletableis_dynamicmetadata_errormetadata_last_processing_state_change_atmetadata_last_size_change_atmetadata_processingmetadata_sizeportal_id
- The deprecated fields include the following:
- Select fields have been added to the
hubspot__contact_listsmodel as well as the upstreamstg_hubspot__contact_listmodel (see HubSpot Source (#137) for details).- The newly added fields include the following:
created_by_idobject_type_idprocessing_statusprocessing_typelist_versionfilters_updated_at
- The newly added fields include the following:
- All deprecated fields have been documented as notice of deprecation in the respective yml documents.
- All added fields have been documented.
- Added a deprecation tracker GitHub issue template to ensure deprecated fields are appropriately tracked for full removal in a future planned update.
- Added consistency validation tests to ensure the
hubspot__contact_listsmodel remains consistent on subsequent updates. - Included an integrity validation test to ensure the accuracy of the
hubspot__contact_listsmodel. - Updated the daily ticket integrity and consistency tests to be turned off if the
hubspot_service_enabledvariable is disabled.
This release includes the following updates:
- For Quickstart users, the models
hubspot__daily_ticket_historyandhubspot__ticketsare now available within the Quickstart UI and are enabled by default. (#154)
- Added Quickstart model counts to README. (#152)
- Corrected references to connectors and connections in the README. (#152)
- Added
hubspot__daily_ticket_historyandhubspot__ticketsmodels to thepublic_modelslist inquickstart.yml. (#154)
PR #154 includes the following updates:
- Added Quickstart model counts to README. (#152)
- Corrected references to connectors and connections in the README. (#152)
- Added
hubspot__daily_ticket_historyandhubspot__ticketsmodels to thepublic_modelslist inquickstart.yml.- This pre-release tests models that are disabled by default for compatibility with quickstart.
PR #150 includes the following updates:
-
Introduced a new
categorycolumn to the following upstream models (see dbt_hubspot_source CHANGELOG notes). This association field differentiates records by either HUBSPOT_DEFINED (default label) or USER_DEFINED (custom label) and was introduced to the Hubspot connector in October 2024. See the connector release notes for more.stg_hubspot__deal_companystg_hubspot__deal_contactstg_hubspot__engagement_companystg_hubspot__engagement_contactstg_hubspot__engagement_dealstg_hubspot__ticket_companystg_hubspot__ticket_contactstg_hubspot__ticket_dealstg_hubspot__ticket_engagement
-
Since new columns were added upstream, a
--full-refreshis needed.
- Updated the respective seed files in the integration_tests folder to property test for the new
categoryfield. - Updated seed files to make consistent with the seed files in
dbt_hubspot_source.
PR #148 includes the following updates:
- Removed
hubspot__daily_ticket_history,hubspot__ticketsfrom the quickstart.yml since these models may not always be present.
PR #147 includes the following updates:
⚠️ Since the following changes result in the table format changing, we recommend running a--full-refreshafter upgrading to this version to avoid possible incremental failures.
- We have made this a breaking change due to upstream changes that may alter your schema. While changes are made 'behind the scenes' to now allow models to successfully run with both
hubspot__pass_through_all_columnsandhubspot__<>_pass_through_columns, this may be a breaking change due to leveraging theremove_duplicate_and_prefix_from_columnsmacro. This is a breaking change because this macro can remove duplicate fields, resulting in an impact to your schema. For more information refer to the upstream dbt_hubspot_source v0.16.0 release notes.
- Updated seed data to include fields with special syntax in order to test the above changes.
PR #144 includes the following updates:
⚠️ Since the following changes result in the table format changing, we recommend running a--full-refreshafter upgrading to this version to avoid possible incremental failures.
-
For Databricks All-Purpose clusters, incremental models will now be materialized using the delta table format (previously parquet).
- Delta tables are generally more performant than parquet and are also more widely available for Databricks users. This will also prevent compilation issues on customers' managed tables.
-
For Databricks SQL Warehouses, incremental materialization will not be used due to the incompatibility of the
insert_overwritestrategy.
- The
is_incremental_compatiblemacro has been added and will returntrueif the target warehouse supports our chosen incremental strategy.- This update was applied as there have been other Databricks runtimes discovered (ie. an endpoint and external runtime) which do not support the
insert_overwriteincremental strategy used.
- This update was applied as there have been other Databricks runtimes discovered (ie. an endpoint and external runtime) which do not support the
- Added integration testing for Databricks SQL Warehouse.
- Added consistency tests for
hubspot__daily_ticket_history.
PR #142 includes the following updates:
- Fixed the
fivetran_utils.enabled_varsconditional by adding thehubspot_contact_list_member_enabledvariable in thehubspot__contact_listsmodel. This solves for compilation errors when thecontact_listsource table is not synced in the destination. Ifhubspot_contact_list_member_enabledistrue,int_hubspot__email_metrics__by_contact_listis brought in as a dependency, and ignored otherwise.
- Updated the
integration_tests/dbt_project.ymlvariables to be global to ensure more effective testing of our seed data. - Updated
property_closed_dateandproperty_createdatedatatypes inticket_datato cast as timestamp to fix datetime data type issues in BigQuery tests. - Updated the maintainer PR template to resemble the most up to date format.
- Removed the check docs GitHub Action as it is no longer necessary.
PR #140 includes the following updates:
- Included explicit datatype casts to
{{ dbt.type_string() }}within the join ofcontact_merge_audit.vid_to_mergetocontacts.contact_idin theint_hubspot__contact_merge_adjustmodel.- This update was required to address a bug where the IDs in the join would overflow to bigint or be interpreted as strings. This change ensures the join fields have matching datatypes.
PR #137 includes the following updates:
- The following are now dependent on the
hubspot_email_event_sent_enabledvariable being defined astrue: -hubspot__contact_listsemail metric fields -hubspot__contactemail metric fields -hubspot__email_campaignsmodel -int_hubspot__email_metrics__by_contact_listmodel
Note: This is a breaking change for users who have the
hubspot_email_event_sent_enabledvariable set tofalseas this update will change the above behaviors of the package. For all other users, these changes will not be breaking.
PR #135 includes the following updates:
- Added unique tests on
event_idfor event-based models,contact_idforhubspot__contacts,contact_list_idforhubspot__contact_lists,deal_idforhubspot__deals,deal_stage_idforhubspot__deal_stages, andcompany_idforhubspot__companies, on the condition the titular fields have not been deleted. We would advise running through these to ensure they work successfully.
- Updates the
hubspot__deal_stagesandhubspot__dealsmodels to remove stale deals that have been merged since into other deals. In addition we've introduced a new fieldmerged_deal_idsthat lists all historic merged deals for each deal.- Please note you must have the underlying
merged_dealstable to take advantage of the above mentioned merged deal update. This may be enabled by settinghubspot_merged_deal_enabledto True (by default this will be False). Also,hubspot_sales_enabledandhubspot_deal_enabledmust not be set to False (by default these are True). See Step 4 of the README for more details.
- Please note you must have the underlying
- The following adjustments have been made concerning the
hubspot_contact_enabledvariable:- The
email_events_joined()macro now includes conditional logic to include contact information only if thehubspot_contact_enabledvariable is defined astrue. - The
int_hubspot__email_metrics__by_contact_listmodel is now dependent on thehubspot_contact_enabledvariable being defined astrue. - The
hubspot__contact_listsmodel now takes into consideration thehubspot_contact_enabledvariable being defined astruebefore joining in email metric information to the contact lists data.
- The
- Modified the config enabled logic within the
hubspot__email_sendsmodel to be dependent on thehubspot_email_event_sent_enabledvariable beingtruein addition to thehubspot_marketing_enabledandhubspot_email_event_enabledvariables.
- Added quickstart.yml for Quickstart customers.
- Added
not_nulltests that were previously missing todeal_idforhubspot__dealsanddeal_stage_idforhubspot__deal_stages.
PR #129 includes the following updates:
- Updated model
int_hubspot__deals_enhancedso that fields from theownersource are not included whenhubspot_owner_enabledis set to false.
- Included auto-releaser GitHub Actions workflow to automate future releases.
PR #127 includes the following updates:
- Updated variables used to determine if engagements are enabled in
hubspot__contactsto also check variablehubspot_engagement_contact_enabled.
-
The following changes stem from changes made in the source package. See the source package v0.14.0 CHANGELOG entry for more details.
-
Added the following staging models, along with documentation and tests:
stg_hubspot__propertystg_hubspot__property_option- These tables can be disabled by setting
hubspot_property_enabled: Falsein your dbt_project.yml vars. See Step 4 of the README for more details.
-
When including a passthrough
property_hs_*column, you now have the option to include the corresponding, human-readable labels.- The above-mentioned
propertytables are required for this feature. If you do not have them and have to disable them, unfortunately you will not be able to use this feature. - See the Adding property label section of the README for more details on how to enable this feature!
- We recommend being selective with the label columns you add. As you add more label columns, your run time will increase due to the underlying logic requirements.
- This update applies to models:
hubspot__companieshubspot__contactshubspot__dealshubspot__tickets
- The above-mentioned
- Within the source package the
created_atandclosed_atfields in the below mentioned staging models have been renamed tocreated_dateandclosed_daterespectively to be consistent with the source data. Additionally, this will ensure there are no duplicate column errors when passing through allproperty_*columns, which could potentially conflict withproperty_created_atorproperty_closed_at. (PR #119)stg_hubspot__company- Impacts
hubspot__companies
- Impacts
stg_hubspot__contact- Impacts
hubspot__contacts
- Impacts
stg_hubspot__deal- Impacts
hubspot__deals
- Impacts
stg_hubspot__ticket- Impacts
hubspot__tickets
- Impacts
Introducing Service end models! These are disabled by default but can be enabled by setting hubspot_service_enabled to true (PR #123):
hubspot__tickets- Docshubspot__daily_ticket_history- Docs- See additional configurations for the history model in README
- Addition of the following variables to allow the disabling of the
*_property_historymodels if they are not being leveraged. All variables aretrueby default. (PR #122)hubspot_company_property_history_enabledhubspot_contact_property_history_enabledhubspot_deal_property_history_enabled
- Updates to the seed files and seed file configurations for the package integration tests to ensure updates are properly tested. (PR #122)
- This release will be a breaking change due to the removal of below dependencies.
- Removes the dependencies on dbt-expectations and dbt-date. (PR #118)
- Specifically we removed the
dbt_expectations.expect_column_values_to_be_uniquetest that was used to validate uniqueness under given conditions for the primary keys among the end models. We will be working to replace this with a similar test. (PR #118)
- This release includes breaking changes as a result of upstream changes within the
v0.12.0release of the dbt_hubspot_source package. Please see below for the relevant breaking change release notes from the source package. (PR #120) - The following models in the dbt_hubspot_source package now use a custom macro to remove the property_hs_ prefix in staging columns, while also preventing duplicates. If de-prefixed columns match existing ones (e.g.,
property_hs_meeting_outcomevs.meeting_outcome), the macro favors theproperty_hs_field, aligning with the latest HubSpot API update. (PR #115)stg_hubspot__engagement_callstg_hubspot__engagement_companystg_hubspot__engagement_contactstg_hubspot__engagement_dealstg_hubspot__engagement_emailstg_hubspot__engagement_meetingstg_hubspot__engagement_notestg_hubspot__engagement_taskstg_hubspot__ticketstg_hubspot__ticket_companystg_hubspot__ticket_contactstg_hubspot__ticket_dealstg_hubspot__ticket_engagementstg_hubspot__ticket_property_history
PR #114 includes the following updates:
This change is made breaking in part to updates applied to the upstream dbt_hubspot_source package following upgrades to ensure compatibility with the HubSpot v3 API updates. Please see below for the relevant upstream changes:
- Following the May 2023 connector update the HubSpot connector now syncs the below parent and child tables from the new v3 API. As a result the dependent fields and field names from the downstream staging models have changed depending on the fields available in your HubSpot data. Now the respective staging models will sync the required fields for the dbt_hubspot downstream transformations and all of your
property_hs_*fields. Please be aware that theproperty_hs_*will be truncated from the field name in the staging and downstream models. The impacted sources (and relevant staging models) are below:
- `ENGAGEMENT`
- `ENGAGEMENT_CALL`
- `ENGAGEMENT_COMPANY`
- `ENGAGEMENT_CONTACT`
- `ENGAGEMENT_DEAL`
- `ENGAGEMENT_EMAIL`
- `ENGAGEMENT_MEETING`
- `ENGAGEMENT_NOTE`
- `ENGAGEMENT_TASK`
- `TICKET`
- `TICKET_COMPANY`
- `TICKET_CONTACT`
- `TICKET_DEAL`
- `TICKET_ENGAGEMENT`
- `TICKET_PROPERTY_HISTORY`-
Please note that while these changes are breaking, the package has been updated to ensure backwards compatibility with the pre HubSpot v3 API updates. As a result, you may see some
nullfields which are artifacts of the pre v3 API HubSpot version. Be sure to inspect the relevant field descriptions for an understanding of which fields remain for backwards compatibility purposes. These fields will be removed once all HubSpot connectors are upgraded to the v3 API. -
The
engagements_joinedmacro has been adjusted to account for the HubSpot v3 API changes. In particular, the fields that used to only be available in theengagementssource are now available in the individualengagement_[type]sources. As such, coalesce statements were added to ensure the correct populated fields are used following the join. Further, to avoid ambiguous columns adbt-utils.starwas added to remove the explicitly declared columns within the coalesce statements.- The coalesce is only included for backwards compatibility. This will be removed in a future release when all HubSpot connectors are on the latest API version.
- The
base_modelargument used for theengagements_joinedmacro has been updated to be an explicitrefas opposed to the previously usedvarwithin the below models:hubspot__engagement_callshubspot__engagement_emailshubspot__engagement_meetingshubspot__engagement_noteshubspot__engagement_tasks- This update was applied as the
varresult was producing inconsistent results during compile and runtime when leveraging thedbt-utils.starmacro. However, the explicitrefalways provided consistent results.
- This update was applied as the
- As new fields were added in the v3 API updates, and old fields were removed, the documentation was updated to reflect the v3 API consistent fields. Please take note if you are still using the pre v3 API, you will find the following end models no longer have complete field documentation coverage:
hubspot__engagement_callshubspot__engagement_emailshubspot__engagement_meetingshubspot__engagement_noteshubspot__engagement_tasks
Explicitly casts join fields (engagement_id and deal_id) in hubspot__deals as the appropriate data types to avoid potential errors in joining. PR #113
These changes are made breaking due to changes in the source.
- Columns
updated_atandcreated_atwere added to the following sources and their corresponding staging models in the source package:DEAL_PIPELINEDEAL_PIPELINE_STAGETICKET_PIPELINETICKET_PIPELINE_STAGE
- As a result, the following columns have been added (#111):
- Model
hubspot__deals:deal_pipeline_created_atdeal_pipeline_updated_at
- Model
hubspot__deal_stages:deal_pipeline_stage_created_atdeal_pipeline_stage_updated_at
- Model
- Documentation has also been updated with these new columns. (#111)
- Updated README to include the variables
hubspot_ticket_deal_enabledandhubspot_owner_enabled. (#111)
- Modified the
unnestlogic in themerge_contactsmacro for Redshift users to reduce runtime of theint_hubspot__contact_merge_adjustmodel. (#110) - Updated seed data for testing newly added columns. (#111)
- @kcraig-ats (#110)
See upstream hubspot_source release notes here.
- A new variable was added
hubspot_using_all_email_eventsto allow package users to remove filtered email events from thestg_hubspot__email_eventstaging model as well as the relevant downstream reporting models. This is crucial for HubSpot users who greatly take advantage of marking events as filtered in order to provide accurate reporting. (#104)- The
hubspot_using_all_email_eventsvariable istrueby default. Set the variable tofalseto filter out specified email events in your staging and downstream models. (#104)
- The
- Introduced new macro
adjust_email_metricsto prevent failures that may occur when certain tables are disabled or enabled. This macro removes any metrics that are unavailable from the defaultemail_metricslist, ensuring that the models runs smoothly. It's worth noting that you can still manually set the email_metrics list as described in the README's (Optional) Step 5: Additional configurations. (#105)
- The
email_event_data.csvseed file was updated to include events that are listed astruefor filtered_events. This is to effectively test the above mentioned feature update. (#104) - Included
hubspot_using_all_email_events: falseas a variable declared in the finalrun_models.shstep to ensure our integration tests gain coverage over this new feature and variable. (#104)- See the source package CHANGELOG for updates made to the staging layer in
dbt_hubspot_source v0.9.1.
- See the source package CHANGELOG for updates made to the staging layer in
- Updated the following models to utilize the
adjust_email_metricsmacro (#105):- hubspot__contacts
- hubspot__contact_lists
- hubspot__email_campaigns
- int_hubspot__email_metrics__by_contact_list
- Incorporated the new
fivetran_utils.drop_schemas_automationmacro into the end of each Buildkite integration test job. (#103) - Updated the pull request templates. (#103)
In November 2022, the Fivetran Hubspot connector switched to v3 of the Hubspot CRM API, which deprecated the CONTACT_MERGE_AUDIT table and stored merged contacts in a field in the CONTACT table. This has not been rolled out to BigQuery warehouses yet. BigQuery connectors with the CONTACT_MERGE_AUDIT table enabled will continue to sync this table until the new CONTACT.property_hs_calculated_merged_vids field and API version becomes available to them.
This release introduces breaking changes around how contacts are merged in order to align with the above connector changes. It is, however, backwards-compatible.
PR #100 applies the following changes:
- Updates logic around the recently deprecated
CONTACT_MERGE_AUDITtable.- The package now leverages the new
property_hs_calculated_merged_vidsfield to filter out merged contacts. This is the default behavior for all destinations, including BigQuery (the package will run successfully but not actually merge any contacts). This is achieved by the package's newmerge_contacts()macro. - Backwards-compatibility: the package will only reference the old
CONTACT_MERGE_AUDITtable to merge contacts ifhubspot_contact_merge_audit_enabledis explicitly set totruein your rootdbt_project.ymlfile. - The
int_hubspot__contact_merge_adjustmodel (where the package performs contact merging) is now materialized as a table by default. This used to be ephemeral, but the reworked merge logic requires this model to be either a table or view.
- The package now leverages the new
PR #100 applies the following changes:
- Updates seed data to test new merging paradigm.
See the source package CHANGELOG for updates made to the staging layer in dbt_hubspot_source v0.9.0.
- Following the release of
v0.8.0, the end model uniqueness tests were not updated to account for the added flexibility of the inclusion of deleted records. As such the respective end model tests have been adjusted to test uniqueness only on non-deleted records. (#94)
- Addition of the dbt-expectations package to be used for more robust testing of uniqueness for end models. (#94)
PR #92 incorporates the following changes:
-
The below models/macros have new soft deleted record flags explicitly added to them:
is_contact_deletedadded inmacros/email_events_joinedwhich can affect the following downstream models found inmodels/marketing/email_events/:hubspot__email_event_bouncehubspot__email_event_clickshubspot__email_event_deferredhubspot__email_event_deliveredhubspot__email_event_droppedhubspot__email_event_forwardhubspot__email_event_openhubspot__email_event_printhubspot__email_event_senthubspot__email_event_spam_reporthubspot__email_event_status_change
is_deal_pipeline_deletedandis_deal_pipeline_stage_deletedadded inmodels/sales/intermediate/int_hubspot__deals_enhancedwhich can affect the following downstream models found inmodels/sales/:hubspot__deal_stageshubspot__deals
is_deal_pipeline_deleted,is_deal_pipeline_stage_deletedandis_deal_deletedadded inmodels/sales/hubspot__deal_stages
-
Soft deleted records are also now implicitly inherited (e.g. via a
select *) from thedbt_hubspot_sourcepackage's staging models and can affect the below final models (for more information, see dbt_hubspot_source PR #96):- Soft deleted company records (
companies.is_company_deleted) included in the below models:sales/hubspot__companies
- Soft deleted deal records (
deals.is_deal_deleted) included in the below models:sales/hubspot__deal_stagessales/hubspot__deals
- Soft deleted contact list records (
contact_lists.is_contact_list_deleted) included in the below models:marketing/hubspot__contact_lists
- Soft deleted contact records (
contacts.is_contact_deleted) included in the below models:marketing/hubspot__contacts
- Soft deleted email sends records (
sends.is_contact_deleted) included in the below models:marketing/hubspot__email_sends
- Soft deleted company records (
-
For completeness, soft deleted records are also now included for
ticket*tables, however does not currently affect this package directly and includes the following staging models:stg_hubspot__ticket(is_ticket_deleted)stg_hubspot__ticket_pipeline_stage(is_ticket_pipeline_stage_deleted)stg_hubspot__ticket_pipeline(is_ticket_pipeline_deleted)
PR #86 includes the following breaking changes:
- Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically
{{ dbt_utils.<macro> }}have been updated to{{ dbt.<macro> }}for the below macros:any_valuebool_orcast_bool_to_textconcatdate_truncdateadddatediffescape_single_quotesexcepthashintersectlast_daylengthlistaggpositionreplacerightsafe_castsplit_partstring_literaltype_biginttype_floattype_inttype_numerictype_stringtype_timestamparray_appendarray_concatarray_construct
- For
current_timestampandcurrent_timestamp_in_utcmacros, the dispatch AND the macro names have been updated to the below, respectively:dbt.current_timestamp_backcompatdbt.current_timestamp_in_utc_backcompat
dbt_utils.surrogate_keyhas also been updated todbt_utils.generate_surrogate_key. Since the method for creating surrogate keys differ, we suggest all users do afull-refreshfor the most accurate data. For more information, please refer to dbt-utils release notes for this update.- Dependencies on
fivetran/fivetran_utilshave been upgraded, previously[">=0.3.0", "<0.4.0"]now[">=0.4.0", "<0.5.0"].
- 🧱 Databricks compatibility! (PR #89)
PR #84 incorporates the following updates:
- Added column descriptions that were missing in our documentation.
- Updated the docs as there were issues with the previous deployment. (#82)
- Introduced a GitHub workflow to ensure docs are re-built prior to merges to
main. (#82)
- The README had holdover merge artifacts included within it. Those artifacts have since been removed in this release. #80
- Updated README documentation updates for easier navigation and setup of the dbt package (#67).
- Included
hubspot_[source_table_name]_identifiervariable for additional flexibility within the package when source tables are named differently (#67). - Adds
hubspot_ticket_deal_enabledvariable (default value=False) to disable modelling and testing of theticket_dealsource table. If there are no associations between tickets and deals in your Hubspot environment, this table will not exist (#79).
- The
hubspot__deal_stagesmodel has undergone two major changes (#78):- The stage and pipeline label columns now reflect where the deal was at a certain point of time (from
date_stage_enteredtodate_stage_exited). Previously, this information reflected the deal's current stage and pipeline in every record associated with the deal. - This model previously passed through all fields from the parent deals model. We removed these fields, as they are all present in
hubspot__dealsfinal model and do not change across deal stages. If you would like to joinDEALfields into thehubspot__deal_stagesmodel, joinhubspot__deal_stageswithhubspot__dealsondeal_id.
- The stage and pipeline label columns now reflect where the deal was at a certain point of time (from
- Consistently renames
property_dealname,property_closedate, andproperty_createdatetodeal_name,closed_at, andcreated_at, respectively, in thedealsmodel. Previously, ifhubspot__pass_through_all_columns = true, only the prefixproperty_was removed from the names of these fields, while they were completely renamed todeal_name,closed_at, andcreated_atifhubspot__pass_through_all_columns = false(#79).
- Typo fix and spelling correction within the README. (#70)
- Spelling correction of the variable names within the README. (#74)
- Added integration testing to support the new
stg_hubspot__deal_contactmodel to thedbt_hubspot_sourcepackage. - Updated the below models such that the respective
lead()functions also account forfield_namein partitionshubspot__contact_historyhubspot__company_historyhubspot__deal_history
- Added test cases for the above models that can be found, respectively, in:
models/marketing/history/history.ymlmodels/sales/history/history.yml
- Updated the
engagements_joinedmacro to conditionally include relevant fields if their respective variables are enabled. Previously if a variable was disabled then thehubspot__engagement_callsmodel, which depends on theengagements_joinedmacro, would error out on the missing fields that were from the disabled variables. (#65)
- Modified the join conditions within the
deal_fields_joinedcte of int_hubspot__deals_enhanced model to leverage the more appropriateleft joinrather than theusingcondition. This is to allow for correct and accurate joins across warehouses. (#60)
🎉 dbt v1.0.0 Compatibility 🎉
- Adjusts the
require-dbt-versionto now be within the range [">=1.0.0", "<2.0.0"]. Additionally, the package has been updated for dbt v1.0.0 compatibility. If you are using a dbt version <1.0.0, you will need to upgrade in order to leverage the latest version of the package.- For help upgrading your package, I recommend reviewing this GitHub repo's Release Notes on what changes have been implemented since your last upgrade.
- For help upgrading your dbt project to dbt v1.0.0, I recommend reviewing dbt-labs upgrading to 1.0.0 docs for more details on what changes must be made.
- Upgrades the package dependency to refer to the latest
dbt_hubspot_source. Additionally, the latestdbt_hubspot_sourcepackage has a dependency on the latestdbt_fivetran_utils. Further, the latestdbt_fivetran_utilspackage also has a dependency ondbt_utils[">=0.8.0", "<0.9.0"].- Please note, if you are installing a version of
dbt_utilsin yourpackages.ymlthat is not in the range above then you will encounter a package dependency error.
- Please note, if you are installing a version of
Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!