Trying to use Signoz Terraform provider to create and manage Alerts and dashboards on Signoz ECS Fargate Deployment. From terraform perspective HCL is valid, plan and applies are generally successful, object is created in Terraform state, but it never shows up in Signoz Alerts. Have also checked on the sqlite db on the container, and is not in alerts table.
Please advise.
Steps to Reproduce
- have pre-existing Alert/Notification Channel Configured and tested working (I had a slack webhook working)
- Create Signoz_alert resource using Terraform Provider version 0.0.8 see
main.tf below, plan and apply
resource "signoz_alert" "new_alert" {
alert = "TF Test Alert"
alert_type = "METRIC_BASED_ALERT"
broadcast_to_all = false
severity = "info"
condition = jsonencode({
absentFor = 10
alertOnAbsent = true
compositeQuery = {
queryType = "builder"
panelType = "graph"
unit = "bytes"
builderQueries = {
A = {
dataSource = "metrics"
queryName = "A"
IsAnomaly = false
QueriesUsedInFormula = null
ShiftBy = 0
limit = 0
offset = 0
pageSize = 0
aggregateOperator = "avg"
aggregateAttribute = {
key = "system_memory_usage"
dataType = "float64"
type = "Gauge"
isColumn = true
isJSON = false
}
timeAggregation = "avg"
spaceAggregation = "sum"
filters = {
items = [
{
key = {
key = "state"
dataType = "string"
type = "tag"
isColumn = false
isJSON = false
}
op = "!="
value = "free"
}
]
op = "AND"
}
expression = "A"
disabled = false
stepInterval = 60
groupBy = [
{
key = "host_name"
dataType = "string"
type = "tag"
isColumn = false
isJSON = false
}
]
legend = "{{host_name}}"
reduceTo = "avg"
}
}
}
op = ">"
target = 90
targetUnit = "gibibytes"
matchType = "0"
selectedQueryName = "A"
})
description = "Alert is fired when the defined metric (current value: {{$value}}) crosses the threshold ({{$threshold}})"
eval_window = "5m0s"
frequency = "1m0s"
labels = {
"observer" = "local-test"
}
preferred_channels = [
"Slack Test"
]
rule_type = "threshold_rule"
version = "v4"
}
- Verify object is in Terraform state
terraform state list
data.signoz_alert.get_alert
signoz_alert.new_alert
- Go to Signoz Alerts Section, wait a bit, refresh, etc It never shows up.
- Verify that API Key Reports Last Used time that corresponds with Terraform Apply Time
- Also verify that no alert rule was created in sqlite db on signoz-container
ECS Exec to signoz-container
ecsta exec -c signoz --id aslkfj;sdfjsdujfiosdajfsjdf;lsdjfklsdjf;sdkjsl;dk --container signoz-container
Starting session with SessionId: ecs-execute-command-q9eqaqcs2qzt9f9te2h77aqno4
~ # sqlite3 /signoz-setup/var/lib/signoz/signoz.db
SQLite version 3.45.3 2024-04-15 13:34:05
Enter ".help" for usage hints.
Verify new alert not created
sqlite> SELECT COUNT(*) AS total_entries FROM rule;
1
sqlite> SELECT EXISTS(
SELECT 1
FROM rule
WHERE json_extract(data, '$.alert') = 'TF Test Alert'
) AS alert_exists;
0
- Also as another confirmation, can create an alert by hand, and do a TF Provider data lookup on that alert, and that works just fine.
Expected Behaviour
New Alert Shows up in Alerts Section when I go to Signoz
Actual Behaviour
Alert never shows up, and isn't in sqlite rules table, even though no errors thrown with Terraform plan/apply step
What are next steps for figuring out where this issue lies? I'm really relying on being able to provision our dashboards and alerts as code and thought upgrading Signoz to latest stable of v0.93.0 and using latest Terraform provider version would work.
For context I have tested this and confirmed working TF Provider on an earlier EC2 docker deployment of signoz, so I know it works.
Trying to use Signoz Terraform provider to create and manage Alerts and dashboards on Signoz ECS Fargate Deployment. From terraform perspective HCL is valid, plan and applies are generally successful, object is created in Terraform state, but it never shows up in Signoz Alerts. Have also checked on the sqlite db on the container, and is not in alerts table.
Please advise.
Steps to Reproduce
main.tfbelow, plan and applyECS Exec to signoz-container
Verify new alert not created
Expected Behaviour
New Alert Shows up in Alerts Section when I go to Signoz
Actual Behaviour
Alert never shows up, and isn't in sqlite rules table, even though no errors thrown with Terraform plan/apply step
What are next steps for figuring out where this issue lies? I'm really relying on being able to provision our dashboards and alerts as code and thought upgrading Signoz to latest stable of v0.93.0 and using latest Terraform provider version would work.
For context I have tested this and confirmed working TF Provider on an earlier EC2 docker deployment of signoz, so I know it works.