Add CloudFormation IaC for AWS S3 data connector#14557
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a CloudFormation IaC deployment path for the Microsoft Sentinel AWS S3 data connector, including AWS-side resource provisioning and step-by-step setup docs.
Changes:
- Added CloudFormation templates for OIDC setup and per-signal AWS resources (GuardDuty, VPC Flow Logs, CloudTrail, CloudWatch).
- Added README documentation guiding deployment via AWS Console and required Sentinel/Azure inputs.
- Added CloudWatch exporter Lambda + EventBridge schedule for CloudWatch-to-S3 export.
Reviewed changes
Copilot reviewed 11 out of 72 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| DataConnectors/AWS-S3/CloudFormation/OpenID/Template 1_ OpenID connect authentication deployment.json | Adds OIDC provider CloudFormation template. |
| DataConnectors/AWS-S3/CloudFormation/OpenID/README.md | Documents OIDC stack creation steps. |
| DataConnectors/AWS-S3/CloudFormation/GuardDuty/template 2_AWS GuardDuty resources deployment.yaml | Provisions GuardDuty integration resources (role, KMS, S3, SQS). |
| DataConnectors/AWS-S3/CloudFormation/GuardDuty/README.md | Documents GuardDuty stack creation and findings export. |
| DataConnectors/AWS-S3/CloudFormation/Flow Logs/template 2_AWS VPC Flow Logs resources deployment.yaml | Provisions VPC Flow Logs integration resources (role, S3, SQS). |
| DataConnectors/AWS-S3/CloudFormation/Flow Logs/README.md | Documents VPC Flow Logs stack creation and manual log enablement. |
| DataConnectors/AWS-S3/CloudFormation/CloudWatch/template 2_AWS CloudWatch resources deployment.yaml | Provisions CloudWatch connector base resources (role, S3, SQS). |
| DataConnectors/AWS-S3/CloudFormation/CloudWatch/template 3_AWS CloudWatch resources deployment.yaml | Adds exporter Lambda and EventBridge schedule. |
| DataConnectors/AWS-S3/CloudFormation/CloudWatch/README.md | Documents CloudWatch stack creation for templates 2 & 3. |
| DataConnectors/AWS-S3/CloudFormation/CloudTrail/template 2_AWS CloudTrail resources deployment.yaml | Provisions CloudTrail integration resources (role, optional S3, SQS). |
| DataConnectors/AWS-S3/CloudFormation/CloudTrail/README.md | Documents CloudTrail stack creation and trail configuration. |
…se to the Copilot/CI feedback, grouped by theme. **Functional fixes** - **EventBridge → Lambda invocation (CloudWatch Step 2):** the schedule targeted the Lambda with a `RoleArn`, but EventBridge authorizes Lambda targets via a resource-based permission, not an IAM role — the `RoleArn` was ignored, so the rule would have deployed cleanly but never actually invoked the exporter. Replaced the unused events role with an `AWS::Lambda::Permission` (principal `events.amazonaws.com`, scoped to the rule ARN). - **OIDC role-name prefix (CloudWatch Step 1):** the `SentinelRoleName` parameter had no pattern; added `AllowedPattern: OIDC_[-_a-zA-Z0-9]+` to match the other templates, since the connector requires the `OIDC_` prefix to fetch its auth token. **Security hardening** - **S3 → SQS notifications (all resource templates):** the `SendMessage` grant is now scoped to the `s3.amazonaws.com` service principal (was `*` in CloudTrail/GuardDuty) and conditioned on both `aws:SourceArn` and `aws:SourceAccount`, closing the confused-deputy gap. - **CloudWatch Step 1:** removed the redundant `CloudWatchReadOnlyAccess` managed policy from the Sentinel role (it reads only from S3/SQS via the resource policies; the CloudWatch API role belongs to the Step 2 Lambda); added SSE-S3 default encryption + `PublicAccessBlockConfiguration` on the bucket and SSE on the queue. - **CloudWatch Step 2:** scoped the exporter Lambda's execution-role resources (CloudWatch read limited to this account/region's log groups; self-logging pinned to the function's own log group) and removed a hardcoded account ID left in an inline code comment. **Exporter efficiency (CloudWatch Step 2)** - Added an optional `LogGroupPrefixes` allowlist plus an early-exit in the stream loop, so the Lambda no longer enumerates every log group/stream on every run (avoids throttling/timeout in large accounts). Empty = previous all-groups behavior. **VPC Flow Logs** - Removed environment-specific parameter defaults (the bucket name was a globally-unique value that would collide on deploy) so the bucket/queue names require explicit input; removed the unused `FlowLogsPrefix` parameter and `UseCustomPrefix` condition; cleaned up encoding artifacts in the description/comments. **OIDC Template 1** - Corrected the `Description`, which claimed the stack creates an assumed role; it creates only the OIDC web identity provider (the role lives in the resources template, by design — the provider is an account-level singleton meant to be deployed once). **Documentation (READMEs)** - Reconciled each README's parameter list against its template: corrected names that had drifted from sibling connectors (e.g. CloudTrail listed GuardDuty's `GuardDutyBucketName` and a non-existent `BucketName` toggle), fixed a `LambdaScheduleMinutes` typo, added the missing required `CloudWatchBucketName` to the CloudWatch doc, and corrected guidance where a template always creates the bucket (no `CreateNewBucket` toggle). **Template format (YamlFileValidation)** - Converted the CloudFormation templates from YAML to JSON. The YAML validator parses files with generic js-yaml, which rejects CloudFormation short-form intrinsic tags (`!Ref`, `!Sub`, `!GetAtt`, etc.); JSON is the existing convention for CloudFormation in this folder and is out of scope for that validator. All intrinsics were preserved and every template passes `cfn-lint` with no errors or warnings.
Contributor
Author
|
Hello @v-maheshbh and @v-atulyadav , I pushed the updates based on the Copilot and CI feedback. Summary below:
Please let me know if anything else needs to be adjusted. |
Collaborator
|
Hi @KanenasCS, |
Contributor
Author
|
Hi @v-atulyadav I've validated the connector end-to-end and attached the screenshots below:
So the full path (S3 -> SQS -> Sentinel) is confirmed working. I've redacted the workspace ID, AWS account ID, and bucket/queue names since this is a public repo. Let me know if you'd like any additional evidence (e.g. SQS queue metrics or the per-interval event counts). |
v-atulyadav
approved these changes
Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added CloudFormation Infrastructure-as-Code scripts for the Amazon Web Services S3 data connector.
Added automated deployment support for the required AWS-side resources used by the Microsoft Sentinel Amazon Web Services S3 connector.
Added configuration for the required AWS resources, including S3/SQS-related components and IAM permissions required by the connector.
Added deployment outputs required for the Microsoft Sentinel connector configuration, such as the relevant AWS resource identifiers.
Added/updated documentation to describe prerequisites, deployment steps, parameters, outputs, and validation steps.
Reason for Change(s):
The existing Amazon Web Services S3 data connector provides an automated setup experience using the Microsoft PowerShell script.
This PR adds a CloudFormation-based Infrastructure-as-Code deployment option, allowing customers to deploy the required AWS resources in a controlled, repeatable, and auditable way.
This is useful for enterprise environments where AWS resources must be deployed through approved IaC processes and reviewed before implementation.
The change does not replace the existing PowerShell setup flow. It provides an additional deployment option for the same connector scenario.
Version Updated:
Not applicable.
This PR does not update Detection or Analytic Rule templates.
Testing Completed:
Yes.
Validated that the CloudFormation template can deploy the required AWS-side resources for the Amazon Web Services S3 data connector.
Validated that the required outputs are generated and can be used during the Microsoft Sentinel connector configuration.
Confirmed that no secrets, credentials, customer-specific values, or hardcoded environment-specific identifiers are included.
Reviewed the deployment parameters and documentation for consistency with the existing Amazon Web Services S3 connector setup flow.
Confirmed that the changes are limited to the Amazon Web Services S3 connector CloudFormation/IaC scope.
Checked that the validations are passing and have addressed any issues that are present:
Yes.
Validations were reviewed, and any identified issues were addressed before submitting the PR.