feature/add_errors_and_warnings_model (#196)

* Buildkite improvements FY27 Q2 (#192)

* Buildkite improvements

* Update dbt_project.yml

---------

Co-authored-by: Catherine Fritz <111930712+fivetran-catfritz@users.noreply.github.com>

* feature/add_errors_and_warnings_model

* update_model

* Update integration_tests/ci/test_scenarios.yml

Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com>

---------

Co-authored-by: Fivetran Maintainer Bot <129092423+fivetran-data-model-bot@users.noreply.github.com>
Co-authored-by: Catherine Fritz <111930712+fivetran-catfritz@users.noreply.github.com>

Author: 3 people

.buildkite/hooks/pre-command

@@ -10,6 +10,8 @@ table_variables:
     - user
   fivetran_platform_using_connection:
     - connection
+  fivetran_platform_using_connector_sdk_log:
+    - connector_sdk_log
 
 destination_configurations:
   databricks:
@@ -24,5 +26,6 @@ public_models: [
   "fivetran_platform__connection_status",
   "fivetran_platform__usage_history",
   "fivetran_platform__audit_table",
-  "fivetran_platform__mar_table_history"
+  "fivetran_platform__mar_table_history",
+  "fivetran_platform__errors_and_warnings"
 ]

.buildkite/pipeline.yml

@@ -1,6 +1,18 @@
 # dbt_fivetran_log v2.6.0
-[PR #XXX](https://github.com/fivetran/dbt_fivetran_log/pull/XXX) includes the following updates:
+[PR #196](https://github.com/fivetran/dbt_fivetran_log/pull/196) includes the following updates:
 
+## Schema/Data Change
+**2 total changes • 0 possible breaking changes**
+
+| Data Model(s) | Change type | Old | New | Notes |
+| ------------- | ----------- | --- | --- | ----- |
+| fivetran_platform__errors_and_warnings | New model | |  | Unions error and warning events from the `log` (standard connections) and `connector_sdk_log` (Connector SDK connections) tables, enriched with `connection_name`. |
+| stg_fivetran_platform__connector_sdk_log | New model |  |  | Staging model for the new `connector_sdk_log` source table. |
+
+## Feature Update
+- Adds the `fivetran_platform_using_connector_sdk_log` variable (default `true`) to enable or disable the `connector_sdk_log` source and its downstream logic. Set this to `false` if your destination does not contain the `connector_sdk_log` table.
+
+# dbt_fivetran_log v2.5.2
 [PR #183](https://github.com/fivetran/dbt_fivetran_log/pull/183) includes the following updates:
 
 ## Under the Hood

.buildkite/scripts/run_models.sh

@@ -25,3 +25,16 @@ vars:
 Some records in the `fivetran_platform__mar_table_history` model may lack an associated `connection_id`. This can occur for a few reasons. For example, the record may originate from a deleted connection or from HVR sources that do not populate this field.
 
 Previously, we excluded these records under the assumption that they were erroneous. However, we've found cases where these rows provide valuable context, particularly for tracking metadata activity from legacy or nonstandard sources and now include them in the model. While this change will increase the number of rows returned, the added visibility supports more complete auditing and analysis.
+
+## Severity values in `fivetran_platform__errors_and_warnings`
+The `fivetran_platform__errors_and_warnings` model unions error and warning events from two sources: the `log` table (standard connections) and the `connector_sdk_log` table (Connector SDK connections). Both sources report `WARNING` and `SEVERE`, while `ERROR` is reported only by the `connector_sdk_log` source. We pass each source's `severity_level` through as its raw value rather than normalizing or remapping it, so each event reflects exactly what its source reported. The `connector_type` column (`standard_connector` or `connector_sdk`) identifies which source a row came from.
+
+We exclude events that are not attributable to a connection (where `connection_id` is null). The `log` table records some warning- and error-level events that are not connector events — for example, the dbt run output of a transformation job — and these carry no `connection_id`. The goal is for customers to monitor the severity of warnings and errors raised by their connectors, not warnings raised by the transformation jobs recorded in the log, so we require a non-null `connection_id` in both sources.
+
+## Connector SDK log source
+Not all customers have the `connector_sdk_log` source table, as it is only populated for accounts using the Fivetran Connector SDK. This source and its downstream logic are controlled by the `fivetran_platform_using_connector_sdk_log` variable, which defaults to `true`. If your destination does not contain the `connector_sdk_log` table, set this variable to `false` in your root `dbt_project.yml` to disable the source and exclude Connector SDK events from `fivetran_platform__errors_and_warnings`:
+
+```yml
+vars:
+  fivetran_platform_using_connector_sdk_log: false ## Default is true. Set to false if the connector_sdk_log source table is not present in your destination.
+```

.quickstart/quickstart.yml

@@ -40,6 +40,7 @@ By default, this package materializes the following final tables:
 | [fivetran_platform__schema_changelog](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__schema_changelog) | Documents all schema changes made to your connections including table alterations, table creations, schema creations, and configuration changes with detailed metadata about each event to track data structure evolution and troubleshoot schema-related issues. |
 | [fivetran_platform__audit_table](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__audit_table) | Replaces the deprecated [`_fivetran_audit` table](https://fivetran.com/docs/getting-started/system-columns-and-tables#audittables) and tracks each table receiving data during connection syncs with comprehensive timestamps for connection and table-level sync progress plus detailed counts of records inserted, replaced, updated, and deleted to monitor data processing and sync performance. |
 | [fivetran_platform__audit_user_activity](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__audit_user_activity) | Records all user-triggered actions within your Fivetran account to provide a comprehensive audit trail that helps you trace user activities to specific [log events](https://fivetran.com/docs/logs#logeventlist) such as schema changes, sync frequency updates, manual syncs, connection failures, and other operational events for compliance and troubleshooting purposes. |
+| [fivetran_platform__errors_and_warnings](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__errors_and_warnings) | Consolidates error and warning events from both standard connections and Connector SDK connections into a single feed, enriched with connection details, so you can monitor and triage the severity of issues across your data pipelines. Excludes `INFO`-level events. |
 
 ¹ Each Quickstart transformation job run materializes these models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`.
 
@@ -103,12 +104,13 @@ vars:
 ```
 
 ### Disable Models for Non Existent Sources
-If you do not leverage Fivetran RBAC, then you will not have the `user` or `destination_membership` source tables. The `user` and `destination_membership` are enabled by default. Therefore in order to switch the default configurations, you must add the following variable(s) to your root `dbt_project.yml` file for the respective source tables you wish to disable:
+If you do not leverage Fivetran RBAC, then you will not have the `user` or `destination_membership` source tables. The `user` and `destination_membership` are enabled by default. Similarly, the `connector_sdk_log` table is only present if you use the Fivetran Connector SDK. Therefore in order to switch the default configurations, you must add the following variable(s) to your root `dbt_project.yml` file for the respective source tables you wish to disable:
 
 ```yml
 vars:
     fivetran_platform_using_destination_membership: false # Default is true. This will disable only the destination membership logic
     fivetran_platform_using_user: false # Default is true. This will disable only the user logic
+    fivetran_platform_using_connector_sdk_log: false # Default is true. This will disable the connector_sdk_log source and exclude Connector SDK events from fivetran_platform__errors_and_warnings
 ```
 
 #### Leveraging `CONNECTION` vs `CONNECTOR`  

CHANGELOG.md

@@ -21,6 +21,7 @@ vars:
     destination: "{{ source('fivetran_platform', 'destination') }}"
     destination_membership: "{{ source('fivetran_platform', 'destination_membership') }}"
     log: "{{ source('fivetran_platform', 'log') }}"
+    connector_sdk_log: "{{ source('fivetran_platform', 'connector_sdk_log') }}"
     user: "{{ source('fivetran_platform', 'user') }}"
     usage_cost: "{{ source('fivetran_platform', 'usage_cost') }}"
     transformation_runs: "{{ source('fivetran_platform','transformation_runs') }}"