Skip to content

Expose struct column extraction helper for AppendStruct inserts #1874

Description

@rbroggi

Observed

  1. Existing issue Default values are ignored in AppendStruct #587 documents that AppendStruct fails when a table contains columns that are absent from the Go struct, even when those columns have server-side DEFAULT expressions.
  2. The current workaround is to explicitly list the inserted columns in PrepareBatch, for example INSERT INTO events (id, name, timestamp), so ClickHouse can apply defaults to omitted columns.
  3. That workaround is effective, but it duplicates the struct's ch tags in raw SQL. In wide or frequently evolving ingestion structs, the column list can drift from the struct definition.

Expected behaviour

The driver should expose a small helper that extracts the ordered ClickHouse column names represented by a struct, so callers can build explicit insert column lists from the same source of truth used by AppendStruct.

This does not need to change AppendStruct semantics or add an implicit DESCRIBE round trip. It would make the documented workaround for #587 easier to use correctly.

Code example

package main

import (
    "context"
    "fmt"
    "strings"

    "github.com/ClickHouse/clickhouse-go/v2"
)

type Event struct {
    ID   uint64 `ch:"id"`
    Name string `ch:"name"`
}

func insert(ctx context.Context, conn clickhouse.Conn, events []Event) error {
    columns, err := clickhouse.StructColumns(Event{})
    if err != nil {
        return err
    }

    batch, err := conn.PrepareBatch(ctx, fmt.Sprintf(
        "INSERT INTO events (%s)",
        strings.Join(columns, ", "),
    ))
    if err != nil {
        return err
    }
    defer batch.Close()

    for i := range events {
        if err := batch.AppendStruct(&events[i]); err != nil {
            return err
        }
    }
    return batch.Send()
}

With a table like this, ClickHouse can apply source server-side because the generated insert statement only names id and name:

CREATE TABLE events (
    id UInt64,
    name String,
    source String DEFAULT 'server default'
) ENGINE = MergeTree()
ORDER BY id;

Error log

Current open-ended insert behaviour still fails as described in #587:

clickhouse [AppendStruct]: missing destination name "source" in *main.Event

Details

Environment

  • clickhouse-go version: current main and released v2 versions with AppendStruct
  • Interface: ClickHouse API
  • Go version: not version-specific
  • Operating system: not OS-specific
  • ClickHouse version: any version where the table has DEFAULT columns
  • Is it a ClickHouse Cloud?
  • ClickHouse Server non-default settings, if any:
  • CREATE TABLE statements for tables involved: see code example above
  • Sample data for all these tables, use clickhouse-obfuscator if necessary

Related: #587

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions