Skip to content

Commit 4ff489d

Browse files
author
Derrick Lee
committed
Merge branch 'derricklee/asim-update-template-validation-run-on-checkout' of https://github.com/Azure/Azure-Sentinel into derricklee/asim-update-template-validation-run-on-checkout
2 parents de3cd0d + 27bdd21 commit 4ff489d

177 files changed

Lines changed: 38860 additions & 10799 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/copilot-instructions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ When working with the Solutions Analyzer tools in `Tools/Solutions Analyzer/`, f
8888
- **`run-solution-analyzer`** — running the mapper, doc generator, ASIM browser, and `upload_to_kusto`; output locations; force-refresh and caching.
8989
- **`update-solution-analyzer`** — modifying scripts, updating `script-docs/`, README changelog rules, CSV output sync with `upload_to_kusto.py`, and static/interactive index plus markdown/HTML entity page synchronization.
9090

91-
**MANDATORY:** Before editing ANY file under `Tools/Solutions Analyzer/` — including small fixes such as renames, regex/pattern tweaks, escaping changes, override additions, or one-line behavior changes — first load `.github/skills/update-solution-analyzer/SKILL.md` and follow it. In particular, **every script change must add or extend an entry in the `## Version History` section of `Tools/Solutions Analyzer/README.md`** within the same commit. Small fixes are NOT exempt.
91+
**MANDATORY:** Before editing ANY file under `Tools/Solutions Analyzer/` — including small fixes such as renames, regex/pattern tweaks, escaping changes, override additions, or one-line behavior changes — first load `.github/skills/update-solution-analyzer/SKILL.md` and follow it. A `## Version History` changelog entry in `Tools/Solutions Analyzer/README.md` is **required for feature additions and behavior changes** (new features, changed analysis logic, new/renamed/removed CSV columns, parameter changes); pure bug fixes — such as typo corrections, crash fixes, escape/regex tweaks, or one-line fixes that restore intended behavior — may be logged at your discretion but are not required.
9292

9393
There are THREE different output scenarios - **never confuse them**:
9494

‎.github/skills/asim-parser-github-pr-packager/SKILL.md‎

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -153,15 +153,34 @@ If there is no file, then we will need to create a file.
153153
}
154154
```
155155

156-
## Step 9: Commit and push
156+
## Step 9. Generate a sample data file
157+
158+
Extract the EventVendor and EventProduct from the parameter-less parser. Create a sample data file in the directory `SampleData/ASIM` with the following name:
159+
`{EventVendor}_{EventProduct}_{ASIMSchema}_IngestedLogs.csv`
160+
161+
The headers of the CSV file should match the name of the columns from the source table. The source table columns can be derived from Step 8.
162+
Generate 10 rows of sample data for the CSV file.
163+
- Make sure that you do not use any information that could be used to identify a real person or organization.
164+
- Make sure to use all the columns that exist in the source table.
165+
- For columns that represent enumerations (dictated by the parser), ensure that the values are representative of the possible options from the parser.
166+
167+
## Step 10. Ensure all files are present
168+
169+
For new parsers, ensure that all of the following files are present or have been modified:
170+
- Addition of the new parser files specific to the product being normalized (e.g., `ASimAuditEvent{ProductName}.yaml` and `vimAuditEvent{ProductName}.yaml`)
171+
- Addition of the new changelog files for the new parser files (e.g., `ASimAuditEvent{ProductName}.md` and `vimAuditEvent{ProductName}.md`)
172+
- Modification of the unifying parsers (e.g., `ASimAuditEvent.yaml` and `imAuditEvent.yaml`)
173+
- Modification of the the changelog files for the unifying parsers (e.g., `ASimAuditEvent.md` and `imAuditEvent.md`)
174+
- The sample data file (e.g., `<EventVendor>_<EventProduct>_<ASIMSchema>_IngestedLogs.csv`)
175+
176+
## Step 11. Generate ARM templates
177+
Provided that you are in the root directory of the repository (Azure-Sentinel), run the following command to generate the ARM templates:
178+
`.script\kqlFuncYaml2Arm.ps1`
179+
This will generate the necessary ARM templates for the new parsers and update the unifying parsers.
180+
There should not be any files deleted after running this script.
181+
182+
## Step 11: Commit the changes
157183
Commit all changes with a meaningful commit message, e.g.:
158184
```
159185
Add ASIM <SchemaName> parser for <Vendor> <Product>
160186
```
161-
Push the branch to the remote repository.
162-
163-
## Step 10: Create the pull request
164-
After pushing, instruct the user to create a pull request from the pushed branch. Provide them with the following details to use when creating the PR:
165-
- **Title:** `Add ASIM <SchemaName> parser for <Vendor> <Product>`
166-
- **Base branch:** `main`
167-
- **Description:** A summary of the parser, the schema it targets, and what data source it normalizes.

‎.github/skills/update-solution-analyzer/SKILL.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: update-solution-analyzer
3-
description: "MANDATORY guidelines for ANY edit to ANY file under Tools/Solutions Analyzer/ — including the mapper (map_solutions_connectors_tables.py), doc generator (generate_connector_docs.py), interactive docs (generate_interactive_docs.py), ASIM browser, collect_table_info, collect_asim_fields, upload_to_kusto, compare_runs, solution_analyzer_overrides.csv, filter_field_resolution.yaml, or any other script/config in that folder. Use when: editing/modifying/refactoring/fixing/renaming/tweaking ANY logic in those files, even one-line fixes such as renaming a classification, escaping a character, adjusting a regex, adding an override row, suppressing a false positive, or changing a constant. Covers: keeping script-docs in sync, README Version History changelog rules (MUST be updated on EVERY script change in the same commit), CSV output sync with upload_to_kusto.py, static/interactive index synchronization, and markdown/HTML entity page synchronization."
3+
description: "MANDATORY guidelines for ANY edit to ANY file under Tools/Solutions Analyzer/ — including the mapper (map_solutions_connectors_tables.py), doc generator (generate_connector_docs.py), interactive docs (generate_interactive_docs.py), ASIM browser, collect_table_info, collect_asim_fields, upload_to_kusto, compare_runs, solution_analyzer_overrides.csv, filter_field_resolution.yaml, or any other script/config in that folder. Use when: editing/modifying/refactoring/fixing/renaming/tweaking ANY logic in those files, even one-line fixes such as renaming a classification, escaping a character, adjusting a regex, adding an override row, suppressing a false positive, or changing a constant. Covers: keeping script-docs in sync, README Version History changelog rules (required for feature/behavior changes; optional for small bug fixes), CSV output sync with upload_to_kusto.py, static/interactive index synchronization, and markdown/HTML entity page synchronization."
44
---
55

66
# Update Solution Analyzer Scripts
@@ -19,9 +19,9 @@ When updating a script, update the corresponding doc in `script-docs/` to reflec
1919

2020
## README Changelog
2121

22-
**MANDATORY for every script change, including small fixes.** Renames, escape-character tweaks, regex adjustments, override-row additions, false-positive suppressions, and one-line behavior changes all qualify and must appear in the changelog. There is no "too small to log" threshold for Solutions Analyzer scripts.
22+
**Required for feature additions and behavior changes; optional for small bug fixes.** New features, changed analysis logic, new/renamed/removed CSV columns, parameter changes, and other user-visible behavior changes must appear in the changelog. Pure bug fixes — such as correcting a typo, fixing a crash, escape-character tweaks, regex corrections, or one-line fixes that restore intended behavior without changing it — may be logged at your discretion but are not required.
2323

24-
Update the `## Version History` section in `Tools/Solutions Analyzer/README.md`:
24+
When a changelog entry is warranted, update the `## Version History` section in `Tools/Solutions Analyzer/README.md`:
2525
- Add the change under the **latest version heading** at the top of the changelog
2626
- If the latest version has not been committed yet (i.e., it already has uncommitted changes in the changelog), add to that existing version — do **not** create a new version entry
2727
- If the latest version was already committed, create a new version entry with an incremented version number and a short descriptive title

‎.script/tests/KqlvalidationsTests/CustomTables/FieldEffectAROAlerts_CL.json‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"Name": "FieldEffectAROAlerts_CL",
3+
"Properties": [
4+
{ "Name": "TimeGenerated", "Type": "Datetime" },
5+
{ "Name": "ID", "Type": "String" },
6+
{ "Name": "Severity", "Type": "String" },
7+
{ "Name": "AROType", "Type": "String" },
8+
{ "Name": "Title", "Type": "String" },
9+
{ "Name": "Hostname", "Type": "String" },
10+
{ "Name": "IPAddress", "Type": "String" },
11+
{ "Name": "LastUser", "Type": "String" },
12+
{ "Name": "DetailsMarkdown", "Type": "String" },
13+
{ "Name": "DetailsRaw", "Type": "String" },
14+
{ "Name": "PortalUrl", "Type": "String" }
15+
]
16+
}

‎.script/tests/KqlvalidationsTests/CustomTables/Mulesoft_MuleSoftCloudhubLogs_KqlValidation.json‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
{
2+
"Name": "MuleSoftCloudhubLogs_CL",
3+
"Properties": [
4+
{
5+
"Name": "TenantId",
6+
"Type": "String"
7+
},
8+
{
9+
"Name": "SourceSystem",
10+
"Type": "String"
11+
},
12+
{
13+
"Name": "TimeGenerated",
14+
"Type": "DateTime"
15+
},
16+
{
17+
"Name": "Type",
18+
"Type": "String"
19+
},
20+
{
21+
"Name": "_ResourceId",
22+
"Type": "String"
23+
},
24+
{
25+
"Name": "EventTime",
26+
"Type": "DateTime"
27+
},
28+
{
29+
"Name": "Level",
30+
"Type": "String"
31+
},
32+
{
33+
"Name": "Logger",
34+
"Type": "String"
35+
},
36+
{
37+
"Name": "Message",
38+
"Type": "String"
39+
},
40+
{
41+
"Name": "Thread",
42+
"Type": "String"
43+
},
44+
{
45+
"Name": "Exception",
46+
"Type": "String"
47+
},
48+
{
49+
"Name": "ApplicationDomainName",
50+
"Type": "String"
51+
},
52+
{
53+
"Name": "InstanceId",
54+
"Type": "String"
55+
},
56+
{
57+
"Name": "WorkerId",
58+
"Type": "String"
59+
},
60+
{
61+
"Name": "Environment",
62+
"Type": "String"
63+
},
64+
{
65+
"Name": "Region",
66+
"Type": "String"
67+
},
68+
{
69+
"Name": "ApplicationData",
70+
"Type": "Dynamic"
71+
}
72+
]
73+
}

‎.script/tests/KqlvalidationsTests/CustomTables/Mulesoft_MulesoftCloudhubAlerts_KqlValidation.json‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
{
2+
"Name": "MulesoftCloudhubAlerts_CL",
3+
"Properties": [
4+
{
5+
"Name": "TenantId",
6+
"Type": "String"
7+
},
8+
{
9+
"Name": "SourceSystem",
10+
"Type": "String"
11+
},
12+
{
13+
"Name": "TimeGenerated",
14+
"Type": "DateTime"
15+
},
16+
{
17+
"Name": "Type",
18+
"Type": "String"
19+
},
20+
{
21+
"Name": "_ResourceId",
22+
"Type": "String"
23+
},
24+
{
25+
"Name": "CreatedAt",
26+
"Type": "DateTime"
27+
},
28+
{
29+
"Name": "LastModified",
30+
"Type": "DateTime"
31+
},
32+
{
33+
"Name": "Id",
34+
"Type": "String"
35+
},
36+
{
37+
"Name": "Name",
38+
"Type": "String"
39+
},
40+
{
41+
"Name": "Severity",
42+
"Type": "String"
43+
},
44+
{
45+
"Name": "OrganizationId",
46+
"Type": "String"
47+
},
48+
{
49+
"Name": "EnvironmentId",
50+
"Type": "String"
51+
},
52+
{
53+
"Name": "ProductName",
54+
"Type": "String"
55+
},
56+
{
57+
"Name": "Enabled",
58+
"Type": "Boolean"
59+
},
60+
{
61+
"Name": "IsSystem",
62+
"Type": "Boolean"
63+
},
64+
{
65+
"Name": "Condition",
66+
"Type": "Dynamic"
67+
},
68+
{
69+
"Name": "Actions",
70+
"Type": "Dynamic"
71+
}
72+
]
73+
}

‎.script/tests/KqlvalidationsTests/CustomTables/StealthTalkAnomalousAuth_CL.json‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
{
2+
"Name": "StealthTalkAnomalousAuth_CL",
3+
"Properties": [
4+
{
5+
"Name": "TimeGenerated",
6+
"Type": "DateTime"
7+
},
8+
{
9+
"Name": "EventType",
10+
"Type": "String"
11+
},
12+
{
13+
"Name": "UserId",
14+
"Type": "String"
15+
},
16+
{
17+
"Name": "DeviceId",
18+
"Type": "String"
19+
},
20+
{
21+
"Name": "AppVersion",
22+
"Type": "String"
23+
},
24+
{
25+
"Name": "Severity",
26+
"Type": "String"
27+
},
28+
{
29+
"Name": "RawEventId",
30+
"Type": "String"
31+
},
32+
{
33+
"Name": "LoginTime",
34+
"Type": "DateTime"
35+
},
36+
{
37+
"Name": "WorkingHoursStart",
38+
"Type": "String"
39+
},
40+
{
41+
"Name": "WorkingHoursEnd",
42+
"Type": "String"
43+
},
44+
{
45+
"Name": "DeviationMinutes",
46+
"Type": "Int"
47+
},
48+
{
49+
"Name": "IsWeekend",
50+
"Type": "Boolean"
51+
},
52+
{
53+
"Name": "NewDeviceId",
54+
"Type": "String"
55+
},
56+
{
57+
"Name": "NewDeviceOS",
58+
"Type": "String"
59+
},
60+
{
61+
"Name": "LoginCountry",
62+
"Type": "String"
63+
},
64+
{
65+
"Name": "LoginCity",
66+
"Type": "String"
67+
},
68+
{
69+
"Name": "AssignedCountry",
70+
"Type": "String"
71+
},
72+
{
73+
"Name": "AssignedCity",
74+
"Type": "String"
75+
},
76+
{
77+
"Name": "IpAddress",
78+
"Type": "String"
79+
},
80+
{
81+
"Name": "PassedAttempts",
82+
"Type": "Int"
83+
},
84+
{
85+
"Name": "LoginBlockingSeconds",
86+
"Type": "Int"
87+
}
88+
]
89+
}

‎.script/tests/detectionTemplateSchemaValidation/ValidConnectorIds.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,7 @@
9494
"ExtraHopNetworks",
9595
"F5",
9696
"F5BigIp",
97+
"FieldEffectCCF",
9798
"Flare",
9899
"ForcepointCasb",
99100
"ForcepointDlp",
@@ -183,6 +184,7 @@
183184
"SonraiDataConnector",
184185
"SophosCloudOptix",
185186
"SophosXGFirewall",
187+
"StealthTalkAnomalousAuth",
186188
"SquadraTechnologiesSecRmm",
187189
"SquidProxy",
188190
"Symantec",

0 commit comments

Comments
 (0)