source-s3: CSV newlines_in_values option silently dropped in v4 migration

[devin-ai-integration]

Summary

The newlines_in_values CSV option from source-s3 v3 was silently dropped in the v4 migration. Users who relied on this setting to handle embedded newlines in CSV values now experience sync failures when upgrading to v4, with no migration path or documentation about the change.

What changed (v3 → v4)

v3 (≤ 3.x) used PyArrow to parse CSVs and exposed a user-facing toggle newlines_in_values (default: False). When set to True, PyArrow's C-level parser allowed newline characters inside quoted CSV values.

• Defined in: airbyte-integrations/connectors/source-s3/source_s3/source_files_abstract/formats/csv_spec.py (lines 54-59)

v4 (≥ 4.0.0) switched to the file-based CDK, which uses Python's csv.DictReader instead of PyArrow. The new CsvFormat model in the CDK (airbyte_cdk/sources/file_based/config/csv_format.py) has no newlines_in_values field.

The legacy config migration drops the field

The LegacyConfigTransformer (airbyte-integrations/connectors/source-s3/source_s3/v4/legacy_config_transformer.py, lines 70-143) converts v3 configs to v4 but does not map newlines_in_values at all. This is even asserted in unit tests — the test input has newlines_in_values: True, and the expected output strips it (unit_tests/v4/test_legacy_config_transformer.py, lines 115-148).

The v4 migration guide (docs/integrations/sources/s3-migrations.md) does not mention this change.

Practical effect

1. The user-facing knob is gone. A v3 user who had newlines_in_values: True has no equivalent v4 setting. Behavior is now hard-coded to whatever Python's csv.DictReader does.

2. v4's parser is stricter about malformed rows. When a multi-line value isn't fully RFC-4180-quoted, or when field counts don't match the header due to a stray unquoted newline, the CDK raises ERROR_PARSING_RECORD_MISMATCHED_COLUMNS / ERROR_PARSING_RECORD_MISMATCHED_ROWS (see airbyte_cdk/sources/file_based/file_types/csv_parser.py, lines 84-113). In v3 with newlines_in_values: True, PyArrow was more permissive about these edge cases.

3. Potential newline="" issue. v4's stream_reader.open_file opens files via smart_open.open(..., mode="r", encoding=...) without newline="". Per Python's csv docs, this can mis-handle embedded newlines when line endings are translated — so CRLF vs LF vs lone-CR files can behave differently than the PyArrow implementation did.

Current workaround

• Use ignore_errors_on_fields_mismatch: true in the v4 stream's CSV format config to skip offending rows instead of failing the sync.
• Ensure fields with embedded newlines are fully double-quoted per RFC 4180, and confirm the file uses LF or standard CRLF (not lone CR).
• If the file is not RFC 4180-compliant, pre-process before S3 ingest.

Suggested fix

One or more of:

1. Re-add a newlines_in_values option (or equivalent) in the file-based CDK's CsvFormat that opens the file with newline="" to let csv.DictReader handle embedded newlines correctly per Python docs.
2. Update the legacy config migration (legacy_config_transformer.py) to translate newlines_in_values into whatever the v4 equivalent is.
3. Document the behavioral change in docs/integrations/sources/s3-migrations.md and mention the ignore_errors_on_fields_mismatch workaround.

Reported by

Community user via Slack.

---

Internal Tracking: https://github.com/airbytehq/oncall/issues/12046

[octavia-bot]

AI Triage Starting...

Thank you for opening this issue! I'm an AI assistant that will do an initial review and may:

• Ask clarifying questions if needed
• Suggest relevant documentation or workarounds
• Flag this for internal visibility

Note: Community issues are reviewed on a best-effort basis and do not have a guaranteed response time. For additional help:

• Join the Airbyte Community Slack to connect with other users and community members
• If you're an Airbyte Cloud customer, you can open a support ticket or contact your account representative, referencing this issue URL

View playbook

• 🔗 Session URL: https://app.devin.ai/sessions/73ca1029a24446d69b508fbd812ed435

✅ Devin AI session created successfully!

[devin-ai-integration]

Thanks for the detailed report! We've escalated this to our team for investigation: airbytehq/oncall#12046.

In the meantime, v4 users hitting ERROR_PARSING_RECORD_MISMATCHED_COLUMNS / ERROR_PARSING_RECORD_MISMATCHED_ROWS on CSVs that relied on newlines_in_values: True in v3 can work around this by:

• Setting ignore_errors_on_fields_mismatch: true in the v4 stream's CSV format config to skip offending rows.
• Ensuring fields with embedded newlines are fully double-quoted per RFC 4180, and the file uses LF or standard CRLF (not lone CR).
• Pre-processing non-compliant files before S3 ingest if they cannot be fixed at the source.

Community issues are reviewed on a best-effort basis without a guaranteed SLA.

---

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.

---

Devin session