I was in the middle of a night‑time release when the Harness delegate pod died with a **SIGKILL** right after pulling the Docker image. The pod logs were useless because the container never started, and the whole pipeline stalled for 12 minutes while the orchestrator kept retrying. When I finally forced a new pod, the image size was 1.7 GB – far larger than the 530 MB we were promised in the docs. That bloat was the real culprit: the delegate was loading an Ubuntu‑full image **plus** the entire Java, Go, and .NET toolchains even though our pipeline only needed a handful of binaries.
That night taught me three things that still hold in 2026:
- **Image size is pipeline latency** – every megabyte you shave off is a second saved on pull, start, and cache warm‑up.
- **Bloat is a security liability** – a giant base image brings in hundreds of CVEs you never use.
- **You can’t treat Harness delegates and Flutter agents as the same beast** – they have different execution models, toolchain expectations, and cost profiles.
If you’re juggling Harness CD delegates and custom Flutter build agents, this guide will give you the exact strategies, data, and code you need to stop those 12‑minute stalls and bring your CI/CD runtime down to single‑digit minutes.
- Use distroless or UBI‑minimal bases for Harness delegates to cut image size and surface vulnerabilities.
- Split Flutter SDK layers with multi‑stage builds; cache `pub get` separately.
- Leverage layer caching for plugins and toolchains; bind‑mount persistent caches where possible.
- Benchmark shows a 65 % cost reduction and 70 % pipeline speedup with optimized images.
- Handle common daemon timeouts and permission errors with exponential backoff scripts.
Before you start: Docker 26.x, Harness Delegate 23.xx.xxxxx, Flutter 3.16+, a GCP or Azure project with sufficient VM quota, and a basic `docker`‑ready CI runner (GitHub Actions or Cloud Build).
Optimizing Docker Images for Harness Delegates and Flutter Build Agents
Optimizing Docker images for Harness CI/CD Delegates and Flutter build agents requires distinct strategies. For Harness, use minimal bases like `ubi‑minimal` and layer caching for plugins. For Flutter, employ multi‑stage builds to separate the large SDK from the final image and strategically cache `pub` dependencies. Proper optimization reduces pipeline time, cloud costs, and security risks.
Why Docker Image Optimization Matters for CI/CD Performance
Impact on Pipeline Speed and Cloud Costs
A 500 MB image pulls in ~12 seconds on a 40 Mbps egress link, while a 1.8 GB image takes over a minute. Multiply that by dozens of agents running in parallel, and the difference shows up on your monthly cloud bill. In our own benchmark (see later), a trimmed Harness delegate cut average compute cost per pipeline run from $0.12 to $0.04 – a **65 %** saving.
Security Implications of Bloated Base Images
The 2025 Sysdig report warned that **58 %** of production containers contain high‑or‑critical vulnerabilities, mostly from outdated OS packages. A monolithic Ubuntu image ships with over 150 CVEs out‑of‑the‑box. Swapping to a **distroless** or **UBI‑minimal** foundation reduces the attack surface dramatically, and vulnerability scans become a trivial step.
Core Architectural Differences: Harness Delegate vs. Flutter Build Agents
Harness Agent: Delegated Execution Model & Static vs. Dynamic Images
Harness delegates run as long‑lived pods (or VMs) that accept work from the Control Plane. They download a **static** Docker image once, keep it cached, and reuse it for thousands of steps. The image must therefore include everything the delegate might need: Java for the Harness SDK, Go for plugins, and any custom CLI tools. Because the delegate lives across builds, you can afford a slightly larger image *if* you cache wisely.
Flutter Build Agent: Platform‑Specific Toolchain Requirements
Flutter agents spin up per‑build (often as GitHub Actions runners). Each build needs the Android SDK, iOS toolchain, and Web compiler—massive binaries that dwarf most other CI tools. Since the container is thrown away after the job, you want the **final** image to be as tiny as possible, pulling in only the compiled artifact and the runtime needed to execute tests. Hence multi‑stage builds are a must.
Optimization Strategies for Harness CI/CD Agent Images (2026)
Choosing the Right Slim or Distroless Base
| Base Image | Size (GB) | Glibc Compatibility | Typical Use‑Case |
|---|---|---|---|
| `harness/delegate:ubi-minimal` | 0.16 | Full | Most Java/Go plugins |
| `gcr.io/distroless/base` | 0.09 | Limited (no shell) | Pure binary workloads |
| `docker.io/library/alpine:3.20` | 0.06 | Musl only (may break Java) | Lightweight scripts |
I found `ubi‑minimal` the sweet spot for Harness because its RPM‑based package manager lets us install Java‑11 and other Red Hat tools without pulling in a full OS. The distroless option is great for custom plugins that don’t need a shell, but you lose the ability to `apt‑get` on the fly.
Layer Caching for Build Tools and Drone Plugins
# syntax=docker/dockerfile:1.5
# Docker 26.x
FROM ghcr.io/harness/delegate:ubi-minimal AS base
# Install Java 11 (required by Harness SDK)
RUN microdnf install -y java-11-openjdk && \
microdnf clean all
# Cache Drone plugins (example: drone-docker)
ARG DRONE_PLUGIN_VERSION=2.0.5
RUN curl -fSL https://github.com/drone-plugins/drone-docker/releases/download/v${DRONE_PLUGIN_VERSION}/drone-docker_linux_amd64.tar.gz \
| tar -xz -C /usr/local/bin drone-docker && \
chmod +x /usr/local/bin/drone-docker
# Copy the delegate binary (unchanged across releases)
COPY --from=delegate:23.xx.xxxxx /usr/local/bin/harness-delegate /usr/local/bin/harness-delegate
ENTRYPOINT ["/usr/local/bin/harness-delegate"]
*Why this works:* The `microdnf` layer is cached forever unless the Java major version changes. The plugin layer only invalidates when the version argument changes. That means a fresh pull on a new build pulls **only** the top three layers, shaving seconds off each start.
Managing Persistent vs. Ephemeral Data with Bind Mounts
Instead of packing caches into the image, mount a host volume:
# harness-pipeline.yaml snippet
steps:
- type: Docker
spec:
image: myorg/harness-delegate:optimized
mount:
- name: gradle-cache
path: /home/harness/.gradle
- name: maven-repo
path: /home/harness/.m2
Mounts keep large caches across pod restarts while keeping the image immutable. Just remember to give the delegate the right Linux user ID (`1001`) to avoid permission spikes.
Optimization Strategies for Flutter Build Agent Images
Multi‑Stage Builds to Strip SDK from Final Image
# syntax=docker/dockerfile:1.5
# Docker 26.x
FROM ghcr.io/flutter/flutter:3.16.0 AS builder
# Install Android SDK command‑line tools
RUN apt-get update && \
apt-get install -y wget unzip && \
wget -q https://dl.google.com/android/repository/commandlinetools-linux-9477386_latest.zip -O /tmp/cmdline.zip && \
unzip /tmp/cmdline.zip -d /opt/android && \
yes | /opt/android/cmdline-tools/bin/sdkmanager --install "platform-tools" "platforms;android-34" "build-tools;34.0.0" && \
rm -rf /tmp/cmdline.zip /var/lib/apt/lists/*
# Cache flutter pub deps
WORKDIR /app
COPY pubspec.yaml pubspec.lock ./
RUN flutter pub get
# Copy source and build
COPY . .
RUN flutter build apk --release
# ---------- FINAL IMAGE ----------
FROM gcr.io/distroless/base-debian12
COPY --from=builder /app/build/app/outputs/flutter-apk/app-release.apk /app.apk
ENTRYPOINT ["java", "-jar", "/app.apk"] # placeholder for runtime
The builder stage holds the huge Android and Flutter SDKs (≈2 GB). The final stage copies only the compiled `apk`, resulting in a **≈120 MB** container that can run tests or be used for artifact upload.
Platform‑Specific Layer Splitting
If you also need iOS builds, create separate stages for `xcode` and reuse the same `pub get` layer. The key is to **never** mix Android and iOS SDK binaries in a single final image; they bloat the image and increase attack surface.
Leveraging `flutter pub get` Caching Within Docker Layers
The trick is to copy only the `pubspec.*` files before installing dependencies. Docker caches the `RUN flutter pub get` step until those files change. In a monorepo where most packages share the same dependencies, you can even create a **shared cache image**:
FROM ghcr.io/flutter/flutter:3.16.0 AS deps
WORKDIR /shared
COPY common/pubspec.yaml .
RUN flutter pub get
Then downstream builds `FROM deps AS base` to inherit the cached `.pub-cache`.
Head‑to‑Head: Key Trade‑Offs & Decision Framework
| Dimension | Harness Delegate (single image) | Custom Flutter Agents (multi‑image) |
|---|---|---|
| **Scalability** | Delegated pods scale automatically via Harness auto‑scale groups. | Each repo may need its own runner pool, leading to fragmented resources. |
| **Complexity** | One Dockerfile to maintain; less per‑team autonomy. | More files, but each team can tune its own SDK versions. |
| **Control** | Central governance; security scans happen once per image. | Individual teams may fall behind on patches. |
| **Cold‑Start** | Warm delegate stays alive → sub‑second start. | Runner starts per build → ~5‑10 s cold start unless `–self-hosted` with pre‑pull. |
| **Cache Invalidation** | Global cache; a plugin upgrade forces a full image rebuild. | Fine‑grained caches per stage; updating Android SDK only invalidates one layer. |
**My take:** If your organization values tight security governance and you have a large number of short‑lived builds, go with a *single, well‑crafted Harness delegate*. If teams need isolated SDK versions (e.g., Google Play vs. Huawei) and you don’t mind a bit more operational overhead, the *custom Flutter agents* win on flexibility.
Production Gotchas and Real‑World Benchmark Data
Common Pitfalls with Volume Mounts and Agent Permissions
A frequent error is:
[ERROR] Failed to create directory /home/harness/.gradle: Permission denied
**Why:** The delegate runs as user `1001` but the host PVC is owned by `root`.
**Fix:** Add a `securityContext` to set `fsGroup: 1001` or `chmod -R 777` on the mount point (the latter only in dev).
securityContext:
runAsUser: 1001
fsGroup: 1001
Impact Analysis: Image Size vs. Pipeline Runtime (2024‑2026 Data)
| Image | Avg Pull (s) | Avg Start (s) | Avg Build (min) | Cost per Run ($) |
|---|---|---|---|---|
| Ubuntu‑full Flutter agent (2.1 GB) | 42 | 8 | 22 | 0.12 |
| Optimized Harness delegate (530 MB) | 12 | 2 | 7 | 0.04 |
| Multi‑stage Flutter distroless (120 MB) | 3 | 1 | 5 | 0.03 |
The data come from a medium‑sized Flutter monorepo (≈150 M LOC) run on `n2-standard-8` VMs in GCP.
Handling Multi‑Architecture Builds for IoT & Mobile
When you need `arm64` builds (e.g., for Apple Silicon CI), use Docker Buildx:
docker buildx create --use --name multiarch
docker buildx build \
--platform linux/amd64,linux/arm64 \
-t myorg/flutter-agent:latest \
--push .
Be aware that the Android SDK `commandlinetools` only ships for `x86_64`; for `arm64` you must install the `android-sdk` via `apt` on an `arm64` base. Cache those layers separately to avoid rebuilds.
Step‑by‑Step Implementation Case Study
Code Walkthrough: Sample Optimized Dockerfile for Each Agent
**Harness Delegate (optimized)**
# Dockerfile (harness-delegate-optimized.dockerfile)
# syntax=docker/dockerfile:1.5
# Docker 26.x
FROM ghcr.io/harness/delegate:ubi-minimal AS base
# Install Java 11 + essential tools
RUN microdnf install -y java-11-openjdk git && \
microdnf clean all
# Pre‑install common Harness plugins
ARG PLUGIN_VER=2.0.5
RUN curl -fsSL https://releases.harness.io/plugins/drone-docker-${PLUGIN_VER}.tar.gz \
| tar -xz -C /usr/local/bin && \
chmod +x /usr/local/bin/drone-docker
# Optional: Add a health‑check script with exponential backoff
COPY healthcheck.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/healthcheck.sh
HEALTHCHECK --interval=30s --timeout=5s \
CMD /usr/local/bin/healthcheck.sh || exit 1
ENTRYPOINT ["/usr/local/bin/harness-delegate"]
`healthcheck.sh` (exponential backoff example):
#!/usr/bin/env bash
# healthcheck.sh – retries Harness delegate status endpoint
MAX_RETRIES=5
DELAY=2
for ((i=1;i<=MAX_RETRIES;i++)); do
if curl -sf http://localhost:8000/api/status; then
exit 0
fi
sleep $DELAY
DELAY=$((DELAY*2))
done
exit 1
**Flutter Build Agent (multi‑stage)**
# Dockerfile (flutter-agent-optimized.dockerfile)
# syntax=docker/dockerfile:1.5
# Docker 26.x
FROM ghcr.io/flutter/flutter:3.16.0 AS builder
# Android SDK
RUN apt-get update && \
apt-get install -y wget unzip && \
wget -q https://dl.google.com/android/repository/commandlinetools-linux-9477386_latest.zip -O /tmp/cmdline.zip && \
unzip /tmp/cmdline.zip -d /opt/android && \
yes | /opt/android/cmdline-tools/bin/sdkmanager \
"platform-tools" "platforms;android-34" "build-tools;34.0.0" && \
rm -rf /tmp/cmdline.zip /var/lib/apt/lists/*
WORKDIR /src
COPY pubspec.yaml pubspec.lock ./
RUN flutter pub get
COPY . .
RUN flutter build apk --release
FROM gcr.io/distroless/base-debian12
COPY --from=builder /src/build/app/outputs/flutter-apk/app-release.apk /app.apk
ENTRYPOINT ["cat", "/app.apk"] # placeholder; typically you upload the artifact
Integrating into CI Workflow
**Harness Pipeline YAML** (excerpt)
pipeline:
name: Flutter CI
stages:
- name: Build
steps:
- type: Run
spec:
delegate: my-org/harness-delegate:optimized
script: |
docker run --rm \
-v /tmp/gradle-cache:/home/harness/.gradle \
-v /tmp/maven-repo:/home/harness/.m2 \
myorg/harness-delegate:optimized \
./run-flutter-build.sh
**GitHub Actions** (using the private Flutter image)
name: Flutter CI
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
container:
image: ghcr.io/myorg/flutter-agent:optimized
options: '--user 1001:1001' # match the non‑root user