Skip to content

fix: escape control characters in native TCP query parameters#1834

Open
Laotree wants to merge 9 commits into
ClickHouse:mainfrom
Laotree:fix-1792-query-param-control-chars
Open

fix: escape control characters in native TCP query parameters#1834
Laotree wants to merge 9 commits into
ClickHouse:mainfrom
Laotree:fix-1792-query-param-control-chars

Conversation

@Laotree

@Laotree Laotree commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

String parameters containing tab, newline, CR, backslash, or NUL were silently truncated when sent via the native TCP protocol. The server decodes parameter values through two stages — readQuoted then deserializeTextEscaped — so control characters must be double-encoded by the client. Replace the single-quote-only ReplaceAll in encodeFieldDump with a strings.Replacer that handles all special characters correctly.

Fixes #1792

Summary

Checklist

Delete items not relevant to your PR:

String parameters containing tab, newline, CR, backslash, or NUL were
silently truncated when sent via the native TCP protocol. The server
decodes parameter values through two stages — readQuoted then
deserializeTextEscaped — so control characters must be double-encoded
by the client. Replace the single-quote-only ReplaceAll in
encodeFieldDump with a strings.Replacer that handles all special
characters correctly.

Fixes ClickHouse#1792

Signed-off-by: rui cheng <chengrui0428@gmail.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Laotree

This comment was marked as outdated.

@github-actions

Copy link
Copy Markdown

Summary

This PR fixes a real bug: string query parameters containing tab, newline, CR, backslash, or NUL were silently truncated when sent over the native TCP protocol because the server runs parameter values through two decode stages (readQuoted then deserializeTextEscaped), but the client only escaped single quotes. The fix is correct — replacing the single strings.ReplaceAll with a package-level strings.Replacer that double-encodes all special characters. The encoding logic and its comments are accurate and clear, the unit tests are table-driven and well-constructed, and the README addition is a welcome improvement. The approach is sound.

Should fix

  1. No HTTP protocol integration test. The review checklist requires coverage for both TCP and HTTP. The HTTP path encodes parameters as URL query params (a separate code path untouched by this PR), so the bug never affected it — but a test confirming special characters round-trip correctly over HTTP would catch any future regression and satisfy the checklist. Consider adding an HTTP variant in tests/issues/1792_test.go using GetConnectionHTTP (or the equivalent) with the same case table.

  2. No database/sql (std API) integration test. The checklist requires coverage for both the native Open() API and the database/sql OpenDB() API. Adding a Test1792Std that exercises sql.Named parameters with the same special characters via the std shim would complete the coverage.

Nits

  • nit: t.Skip(fmt.Errorf("unsupported clickhouse version")) wraps a string in an error solely to pass it to t.Skip, which accepts ...any. Prefer t.Skipf("unsupported clickhouse version") — it reads more naturally and matches the pattern used elsewhere in the test suite.
  • nit: The return immediately after t.Skip(...) is unreachable — t.Skip calls runtime.Goexit() internally. It's harmless but can be dropped.

Verdict

Request changes — the fix and unit tests are correct, but HTTP and std API integration test coverage is missing per the project checklist.

Laotree and others added 3 commits April 20, 2026 21:56
- Replace t.Skip(fmt.Errorf(...)) with t.Skipf(...) and remove unreachable return
- Extract shared test cases into controlCharCases to avoid duplication
- Add Test1792HTTP: round-trip control characters via HTTP protocol
- Add Test1792Std: round-trip control characters via database/sql with clickhouse.Named()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
String parameters containing tab, newline, CR, backslash, or NUL were
misinterpreted when sent via the HTTP protocol. The server applies
deserializeTextEscaped (TSV format) to param_<name> values, so raw
control characters act as field/record delimiters and cause parse errors,
and a literal backslash+t is read as a tab. Apply TSV escaping via
httpQueryParamReplacer before adding values to the URL query string.

Signed-off-by: rui cheng <chengrui0428@gmail.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Summary

Fixes a real correctness bug: string parameters containing tab/newline/CR/backslash/NUL were silently mangled on the native TCP path because encodeFieldDump only escaped single quotes. The fix replaces the single strings.ReplaceAll with a strings.Replacer that double-encodes for TCP (to survive readQuoted + deserializeTextEscaped) and adds a parallel single-encoding strings.Replacer for the HTTP path. Encoding strategy is correct for both paths, comments accurately explain the two-stage server decoding, and there is good unit-test coverage of the encoder plus a real integration round-trip against ClickHouse for both protocols. The approach is sound.

Must fix

  • tests/issues/1792_test.go:142fmt.Sscanf(version, ...) discards both the count and the error return. Per project rules (.claude/CLAUDE.md: Never discard errors with _. Return, handle, or wrap — never swallow.) the error must be handled. At minimum, return false on parse failure rather than silently treating an unparsable version as 0.0.0 and then claiming the min-version check passed/failed based on garbage.
  • tests/issues/1792_test.go:7-13 — Import grouping merges external (github.com/stretchr/testify/...) and internal (github.com/ClickHouse/clickhouse-go/v2/...) packages in a single block. Project convention (and the rest of tests/issues/, e.g. 1066_test.go) requires three groups separated by blank lines: stdlib → external → internal.

Should fix

  • Coverage matrix is incomplete. The skill checklist requires both protocols × both APIs (native + std). The PR has: native API + TCP (Test1792), native API + HTTP (Test1792HTTP), std API + HTTP (Test1792Std) — but no std API + native TCP. Given the bug only manifests on the TCP path, the std-over-TCP variant is the most valuable missing case. Please add it.
  • Test1792Std uses db.QueryRow(...) without a context, while the README block this PR adds explicitly recommends db.QueryRowContext(ctx, ...). The integration test should model the documented best practice.
  • The new README section ends with See full examples: native API · database/sql, but neither example file actually demonstrates special-character handling — both only use plain ASCII parameter values. Either extend the examples with a `\t`/`\n`/backslash/quote case (the natural place for it given the PR's focus), or rewrite the link text so it doesn't promise something the examples don't deliver.
  • checkStdMinVersion in tests/issues/1792_test.go duplicates CheckMinServerVersion from tests/std/utils.go. Consider lifting a single helper into the root tests/ package (the issues package can't import tests/std without creating a cycle, which is presumably why it was inlined — but the duplication will rot). Not blocking.

Nits

  • nit: lib/proto/query.go:230 — the inline comment single quote → \': readQuoted produces ', not special in TSV would read more clearly as something like single quote → \': prevents premature string termination in readQuoted. The current wording explains the output but not the reason it has to be escaped.
  • nit: conn_http.go:42-51 and lib/proto/query.go:228-238 — the two replacers handle the same character set with deliberately different encodings (single- vs double-escaping). A one-line cross-reference comment in each (// HTTP variant: see fieldDumpReplacer in lib/proto/query.go for the double-encoding TCP equivalent) would make the symmetry obvious to future maintainers.
  • nit: tests/issues/1792_test.go — the file name is 1792_test.go, which matches existing convention in tests/issues/, but .claude/CLAUDE.md documents the naming as issue_1234_test.go. This is a project-instructions vs. directory-convention mismatch, not a problem with this PR — flagging for awareness; the maintainers may want to reconcile one with the other.
  • nit: Test1792HTTP could note in its header comment that the HTTP path still goes through deserializeTextEscaped server-side (which is why TSV escaping is needed at all over HTTP). Right now the comment only mentions URL-encoding.

Verdict

Request changes — the two Must fix items (discarded fmt.Sscanf error and import grouping) are small but should be corrected before merge. Once those plus the std+TCP coverage gap are addressed, this is a clean, well-explained fix with strong test coverage.

- Fix import grouping in tests/issues/1792_test.go: split external
  (testify) and internal (clickhouse-go) packages into separate blocks
  per project convention (stdlib → external → internal)
- Fix checkStdMinVersion: handle fmt.Sscanf error instead of discarding
  both return values; return false on parse failure
- Add Test1792StdTCP: std/database-sql coverage over native TCP,
  completing the 2×2 protocol×API coverage matrix
- Switch Test1792Std to db.QueryRowContext(ctx, ...) per documented
  best practice
- Improve single-quote comment in fieldDumpReplacer to explain why
  it must be escaped (prevents premature string termination in readQuoted)
- Add cross-reference comments between httpQueryParamReplacer (conn_http.go)
  and fieldDumpReplacer (lib/proto/query.go) to make the intentional
  encoding difference obvious to future maintainers

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Laotree

Laotree commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review. All feedback has been addressed in commit 2b7f29c:

MUST FIX — resolved

  • Import grouping (tests/issues/1792_test.go): Split the second import block into two separate groups — external (github.com/stretchr/testify/...) and internal (github.com/ClickHouse/clickhouse-go/v2/...) — with a blank line between them, matching the stdlib → external → internal convention.

  • fmt.Sscanf error handling (checkStdMinVersion): Both return values (count and error) are now captured. The function returns false on parse failure rather than silently treating the server as version 0.0.0.

SHOULD FIX — resolved

  • Missing std + TCP coverage: Added Test1792StdTCP using clickhouse.OpenDB with Protocol: clickhouse.Native. The 2×2 coverage matrix (protocol × API) is now complete:

    Protocol Native API database/sql
    TCP Test1792 Test1792StdTCP ✅ new
    HTTP Test1792HTTP Test1792Std
  • Context in Test1792Std: Changed db.QueryRow(...) to db.QueryRowContext(ctx, ...) in both Test1792Std and Test1792StdTCP.

Nits — resolved

  • Single-quote comment (lib/proto/query.go): Updated to explain why it must be escaped — "prevents premature string termination in readQuoted" — rather than just describing the output.

  • Cross-reference comments: Added a NOTE: to both httpQueryParamReplacer (conn_http.go) and fieldDumpReplacer (lib/proto/query.go) pointing to each other, making the intentional single-stage vs. double-stage encoding difference obvious to future maintainers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Doc/Bug: Query params behavior in clickhouse-go

2 participants