{"id":3966,"date":"2026-04-30T16:10:26","date_gmt":"2026-04-30T16:10:26","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/04\/30\/from-security-blocked-to-prod-ready-clickhouse-on-docker-hardened-images\/"},"modified":"2026-04-30T16:10:26","modified_gmt":"2026-04-30T16:10:26","slug":"from-security-blocked-to-prod-ready-clickhouse-on-docker-hardened-images","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/04\/30\/from-security-blocked-to-prod-ready-clickhouse-on-docker-hardened-images\/","title":{"rendered":"From Security Blocked to Prod Ready: ClickHouse on Docker Hardened Images"},"content":{"rendered":"

In November 2025, a team self-hosting Langfuse<\/a>, an open-source LLM observability platform, on Kubernetes uploaded their ClickHouse image to AWS ECR as part of their production preparation. They found that the pipeline scanner had returned three critical vulnerabilities \u2013 not in ClickHouse, but in the base image. Their security team saw the findings and blocked the deployment before it ever reached production.<\/p>\n

\n
\n
\n

\u201cOur security team is not allowing us to take it to production. Please suggest alternatives.<\/em>\u201c<\/h3>\n
\n

vinaygoel586<\/p>\n

GitHub Issue #286<\/a>, November 28, 2025<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

If you\u2019ve shipped containers into an enterprise environment recently, this situation will sound familiar. A perfectly functional deployment gets blocked not because something is broken, but because a scanner found CVEs in packages the application never even touches. A day goes into investigating the findings, a risk exception gets written up, and the security team rejects it anyway, because the vulnerabilities are technically real even if they\u2019re practically irrelevant to your workload.<\/p>\n

This post is about how Docker Hardened Images (DHI) gets you unstuck, when a security team blocks the deployment of a container that has CVEs. In this case we will specifically look at the image for ClickHouse, one of the most widely pulled database images on Docker Hub.<\/p>\n

A Quick Word on ClickHouse<\/h2>\n

ClickHouse<\/a> is an open-source columnar database built for analytical workloads at scale. It is capable of querying billions of rows and returning results in milliseconds in a way that traditional row-oriented databases simply can\u2019t match. Companies such as Cloudflare, Uber, and Spotify all run it in production. With over 100 million pulls from Docker Hub<\/a>, it has become the default infrastructure choice for teams that need serious analytics throughput. The image\u2019s default security posture, though, was designed with developer ease-of-use in mind rather than the hardening that enterprise production environments demand and that gap is where the trouble starts.<\/p>\n

\n \"image1\n <\/div>\n

Figure: The layered architecture of ClickHouse<\/em><\/p>\n

How ClickHouse is Structured<\/h3>\n

ClickHouse follows a layered architecture. It is designed for analytical speed at scale. SQL queries arrive over HTTP (port 8123) or TCP (port 9000), then pass through the optimizer which parses into an abstract syntax tree and prunes it before the pipeline executor picks it up and hands the work off to parallel threads. Beneath the query layer sits the MergeTree storage engine, the heart of ClickHouse which stores data in columnar .bin<\/code> files. It uses a sparse primary index to skip irrelevant granules without reading entire columns, and runs background merge processes to compact parts and maintain query performance over time.\u00a0<\/p>\n

At the bottom, storage is pluggable: local disk, S3, HDFS, or Azure Blob, with tiered hot\/warm\/cold policies to balance cost and latency. In distributed deployments, ClickHouse Keeper (or ZooKeeper) coordinates replication across replicas, while sharding splits data horizontally across nodes allowing the cluster to scale reads and writes independently. The result is a database that processes hundreds of millions of rows per second per server, making it the default choice for teams running serious analytics workloads.<\/p>\n

The Real Problem: It\u2019s Not ClickHouse, It\u2019s the Packaging<\/h2>\n

The standard clickhouse\/clickhouse-server<\/code> image is built on a full Ubuntu 22.04 base. The base ships with a lot of things ClickHouse doesn\u2019t need such as Perl, system utilities, apt itself, and dozens of transitive dependencies that exist in the image simply because Ubuntu brought outdated package along and in many cases, Ubuntu maintainers decide to not backport fixes from upstream.<\/p>\n

ClickHouse doesn\u2019t use most of those system utilities. But the CVEs in those packages are real. They show up in Trivy, Grype, and AWS ECR has no way to distinguish a vulnerable library that\u2019s never loaded from one that\u2019s actively running in production. Your security team sees critical findings and blocks the deployment, which is the correct thing for them to do given what the scanner is telling them.<\/p>\n

The instinct at this point is to argue the case, documenting why each CVE doesn\u2019t apply to your workload, writing risk exceptions and escalating, but that\u2019s a slow process. The only real fix is to remove those unnecessary packages entirely. That\u2019s what Docker Hardened Images do.<\/p>\n

What DHI Actually Changes<\/h2>\n

Docker Hardened Images<\/a> for ClickHouse are built around a straightforward question: what does the database actually need to run? Rather than starting from a full Ubuntu base and hoping the CVE count stays manageable, DHI ships only what ClickHouse requires and leaves everything else out.<\/p>\n

The most immediate consequence of that is the absence of apt<\/code> at runtime. Without a package manager, an attacker who gains a foothold in the container has no obvious path to installing tools or establishing persistence. Network utilities like curl<\/code> and wget<\/code> are gone for the same reason, the standard clickhouse\/clickhouse-server<\/code> image has been carrying wget with CVE-2021-31879<\/code> unpatched since 2021 because there is no upstream fix as noted by the Ubuntu maintainer<\/a>, a vulnerability in a tool ClickHouse never needed in the first place. DHI doesn\u2019t patch it; it simply doesn\u2019t include wget<\/code> at all. A shell is still available for operational work, but without the package manager and network tools, there\u2019s very little an attacker can actually do with it.<\/p>\n

To make this practical across different stages of a pipeline, DHI ships two variants. The development image (dev) includes additional tooling that makes local testing and debugging more comfortable. The production image (runtime) strips that back to the absolute minimum, giving you the smallest possible attack surface for the workload that actually faces the world. The intent is that teams adopt the dev variant early in the pipeline and promote the hardened production image through to deployment, rather than discovering the differences at the point where it matters most.<\/p>\n

The image also runs as a non-root user uid=65532<\/code> out of the box, with no additional Dockerfile configuration required. On the provenance side, every DHI image ships with SLSA Level 3 attestation, which provides cryptographic proof of exactly what went into the build and how it was produced. Docker\u2019s security team actively tracks and patches CVEs, and the presence of 2026 CVE IDs in DHI\u2019s findings is evidence of that remediation happening ahead of public disclosure feeds rather than in response to them.<\/p>\n

Getting Started<\/h2>\n

Before you can pull a DHI image, you need to mirror it to your organization\u2019s namespace on Docker Hub. This is a one-time setup per image not per tag and it means all future updates flow to your namespace automatically.<\/p>\n

    \n
  1. Log in to Docker Hub and open the DHI catalog<\/li>\n
  2. Find clickhouse-server<\/code> and select Mirror to repository<\/strong><\/li>\n
  3. Follow the on-screen instructions<\/li>\n
  4. Authenticate locally: docker login dhi.io<\/code><\/li>\n<\/ol>\n

    Once that\u2019s done, you\u2019re pulling from your own namespace with the same image, same tags, same ClickHouse \u2013 just hardened.<\/p>\n

    Your first DHI ClickHouse container<\/h3>\n
    \n
    \ndocker run --name my-clickhouse-server -d \n  --ulimit nofile=262144:262144 \n dhi.io\/clickhouse-server:26.2-debian13\n<\/pre>\n<\/div>\n
    The --ulimit nofile=262144:262144<\/code> flag is a ClickHouse requirement, not a DHI one \u2013 ClickHouse needs high file descriptor limits to operate correctly. Keep it in all your run commands.<\/p>\n
    Verify it started:<\/p>\n
    \n
    \ndocker exec my-clickhouse-server clickhouse-client \n  --query \"SELECT 'Hello from DHI ClickHouse!'\"\n<\/pre>\n<\/div>\n
    Production setup with persistent storage<\/h3>\n
    For anything beyond local testing, you want volumes and a password:<\/p>\n
    \n
    \ndocker run -d \n  --name my-clickhouse-server \n  --ulimit nofile=262144:262144 \n  -e CLICKHOUSE_PASSWORD=mysecretpassword \n  -v clickhouse-data:\/var\/lib\/clickhouse \n  -v clickhouse-logs:\/var\/log\/clickhouse-server \n  -p 8123:8123 -p 9000:9000 \n  dhi.io\/clickhouse-server:26.2-debian13\n<\/pre>\n<\/div>\n
    Note that CLICKHOUSE_PASSWORD<\/code> is required if you want to access ClickHouse over the network. DHI disables unauthenticated network access by default which is the right call for any production deployment.<\/p>\n
    Test it over HTTP:<\/p>\n
    \n
    \ncurl \"http:\/\/localhost:8123\/?query=SELECT%20version()&user=default&password=mysecretpassword\"\n<\/pre>\n<\/div>\n
    Custom configuration<\/h3>\n
    If you\u2019re already running ClickHouse with custom XML config, nothing changes. Same format, same mount path:<\/p>\n
    \n
    \ncat > custom-config.xml << EOF\n<clickhouse>\n    <logger>\n        <level>information<\/level>\n        <console>true<\/console>\n    <\/logger>\n    <listen_host>0.0.0.0<\/listen_host>\n<\/clickhouse>\nEOF\n\ndocker run -d \n  --name my-clickhouse-server \n  --ulimit nofile=262144:262144 \n  -v $(pwd)\/custom-config.xml:\/etc\/clickhouse-server\/config.d\/custom.xml:ro \n  -p 8123:8123 -p 9000:9000 \n  dhi.io\/clickhouse-server:26.2-debian13\n\n<\/pre>\n<\/div>\n
    Running DHI ClickHouse on Kubernetes<\/h2>\n
    For Kubernetes, there\u2019s one important addition to your pod spec. Since DHI runs as a non-root user, you need to set fsGroup<\/code> to ensure your persistent volume data is accessible:<\/p>\n
    \n
    \nspec:\n  template:\n    spec:\n      securityContext:\n        runAsNonRoot: true\n        runAsUser: 65532     # DHI nonroot user\n        fsGroup: 65532       # makes mounted volumes accessible to the nonroot user\n      containers:\n      - name: clickhouse-server\n        image: dhi.io\/clickhouse-server:26.2-debian13\n        ports:\n        - containerPort: 8123\n        - containerPort: 9000\n        volumeMounts:\n        - name: clickhouse-data\n          mountPath: \/var\/lib\/clickhouse\n        - name: clickhouse-logs\n          mountPath: \/var\/log\/clickhouse-server\n        resources:\n          limits:\n            cpu: \"2\"\n            memory: \"4Gi\"\n\n<\/pre>\n<\/div>\n
    One thing worth mentioning: ClickHouse\u2019s default ports 8123 and 9000 are above the 1024 privileged port boundary, so running as nonroot doesn\u2019t cause any port binding issues.<\/p>\n
    The metrics exporter<\/h3>\n
    If you\u2019re running ClickHouse on Kubernetes and need Prometheus metrics, Docker also ships clickhouse-metrics-exporter<\/code> \u2013 a hardened image that works with the ClickHouse Operator to expose a \/metrics<\/code> endpoint. It\u2019s 65% smaller than the standard exporter (10.3 MB vs 29.4 MB) and has 75% fewer layers (5 vs 20). Same data, dramatically smaller surface.<\/p>\n
    \n
    \ncontainers:\n- name: metrics-exporter\n  image: dhi.io\/clickhouse-metrics-exporter:0-debian13\n  ports:\n  - name: metrics\n    containerPort: 8888\n  resources:\n    limits:\n      cpu: 100m\n      memory: 128Mi\n    requests:\n      cpu: 50m\n      memory: 64Mi\n\n<\/pre>\n<\/div>\n
    Debugging without the usual tools<\/h2>\n
    The debugging story is simpler than it might seem. docker debug<\/code> attaches an ephemeral layer to the running container that includes bash, curl, strace, vim, and anything else you need without modifying the production image itself. When you exit, the layer disappears and the container is exactly as it was. It\u2019s a cleaner approach than shelling directly into a production container, and in practice it\u2019s a single command:<\/p>\n
    \n
    \ndocker debug my-clickhouse-server\n<\/pre>\n<\/div>\n
    Or if you prefer, you can mount a debug image alongside the container:<\/p>\n
    \n
    \ndocker run --rm -it --pid container:my-clickhouse-server \n  --mount=type=image,source=<your-namespace>\/dhi-busybox,destination=\/dbg,ro \n  dhi.io\/clickhouse-server:26.2-debian13 \/dbg\/bin\/sh\n<\/pre>\n<\/div>\n
    There\u2019s also a broader security benefit that goes beyond CVE counts. If something does go wrong in production, an attacker who gets into the container finds no package manager to install tools with, no curl or wget to exfiltrate data through, and no obvious path to reach out to the network which significantly limits what a compromise can actually turn into.<\/p>\n
    ClickHouse: Non-hardened Image vs. Hardened Image Compared<\/h2>\n
    A Docker Scout<\/a> scan of both images puts the difference in plain numbers. Using ubuntu:22.04<\/code> as its base, the standard image carries 8<\/code> medium and 11<\/code> low severity vulnerabilities across 111 packages, including the wget and tar findings that are most likely to trigger a security block in an enterprise pipeline. The DHI image eliminates all medium severity findings entirely and comes in at 14<\/code> low severity items but these are in core system libraries like glibc<\/code> and openssl<\/code> where no fix exists on any distribution, not in unnecessary utilities that had no business being in the image. The 3<\/code> unconfirmed findings that Scout surfaces have already been assessed and suppressed via VEX attestation, which ships with the image as part of its SLSA Level 3 provenance<\/p>\n
    To view the difference between versions for any other image, you can run your own scan with Docker Scout for a quick comparison using this command:<\/p>\n
    \n
    \ndocker scout quickview clickhouse\/clickhouse-server:latest\n\ndocker pull dhi.io\/clickhouse-server:26.2-debian13\ndocker tag dhi.io\/clickhouse-server:26.2-debian13 clickhouse-dhi:latest\ndocker scout quickview clickhouse-dhi:latest\n\n<\/pre>\n<\/div>\n
    \n                \"image2\"\n        <\/div>\n\n
    \n                \"image3\"\n        <\/div>\n\n
    \n\n\n\n\n\n\n\n\n\n\n\n
    \n<\/th>\n\n
    Non-Hardened\u00a0 ClickHouse Image<\/span><\/p>\n<\/th>\n
    		\n
    Docker Hardened Image<\/span><\/p>\n<\/th>\n<\/tr>\n
    \n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Default user<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    root (steps down to clickhouse user at runtime via entrypoint, but Dockerfile has no USER directive overridable with CLICKHOUSE_RUN_AS_ROOT=1)<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    nonroot (enforced at image level via USER directive cannot be overridden at runtime)<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n<\/tr>\n
    \n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Shell access<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Full shell (bash\/sh) available<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    bash present, no network tools or package manager<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n<\/tr>\n
    \n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Package manager<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    apt available<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    No package manager<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n<\/tr>\n
    \n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    CVE exposure<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Ships wget (<\/span>CVE-2021-31879<\/span><\/a>, unpatched since 2021), tar (<\/span>CVE-2025-45582<\/span><\/a>)<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    No wget, no tar \u2013 unnecessary packages removed entirely<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n<\/tr>\n
    \n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    CVE patching<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Unpatched findings from 2021\u20132025 due to the lack of upstream fixes from Ubuntu base image.<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Actively tracked, 2026 CVE IDs show proactive remediation<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n<\/tr>\n
    \n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Provenance<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Standard<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    SLSA Level 3 attestation<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n<\/tr>\n
    \n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Compliance<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Manual hardening required<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    CIS, NIST, FedRAMP-aligned<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n<\/tr>\n
    \n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Debugging<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Traditional shell debugging<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n
    		\n                    <\/span>\n
                        
    \n                                                    <\/span><\/span><\/p>\n
    Use docker debug or Image Mount for troubleshooting<\/span><\/p>\n
                        
    \n                                            \n            <\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
    The Security Team Conversation<\/h2>\n
    The team that got blocked at AWS ECR in November 2025 didn\u2019t have a ClickHouse problem, they had a base image problem. Their database was fine; what the scanner was finding were CVEs in Perl, system utilities, and other packages that had come along in the Debian base and never used by the application. Nothing in the scanner output made that distinction, so the security team did exactly what they were supposed to do and blocked the deployment.<\/p>\n
    With DHI, that conversation with your security team becomes considerably more straightforward. Rather than building a case for why specific CVEs don\u2019t apply to your workload, you can point to an image built by Docker\u2019s security team from the minimum required components, with SLSA Level 3 provenance and independent validation by SRLabs. The ClickHouse runtime itself is unchanged ~ queries, ports, configuration files, and performance all carry over so the only thing you\u2019re actually changing is the answer you can give when someone asks whether this image can go to production.For teams that need stronger guarantees, DHI Enterprise adds SLA-backed CVE remediation within seven days, FIPS and STIG variants, and extended lifecycle support. For most teams, the free Enterprise trial<\/a> is the right starting point. It answers the question that actually matters before you commit to anything. Interested to learn further? Start with this blog that walks through<\/a> the trial and sets you up for success.<\/p>\n
    Migration Checklist<\/h2>\n
    \n
    \n\u2610 Mirror clickhouse-server DHI image to your Docker Hub namespace (one-time setup)\n\u2610 Update your image reference to dhi.io\/clickhouse-server:26.2-debian13\n\u2610 Set CLICKHOUSE_PASSWORD (required for network access in DHI)\n\u2610 Keep --ulimit nofile=262144:262144 on all run commands\n\u2610 In Kubernetes: add fsGroup: 65532 to your pod securityContext\n\u2610 Switch from kubectl exec to kubectl debug for troubleshooting\n\u2610 Run trivy against both images to see the difference yourself:\n     trivy image clickhouse\/clickhouse-server:latest\n     trivy image dhi.io\/clickhouse-server:26.2-debian13\n\n<\/pre>\n<\/div>\n
    The migration is narrower in scope than it might appear \u2013 your volume mounts, port mappings, and existing XML configuration files all carry over without modifications, and on Kubernetes the only structure addition is the fsGroup security context. Everything else is an image reference change.<\/p>\n
    Resources<\/h2>\n