Home › Guides › Zabbix APM lab

Zabbix 8.0 APM Lab: Copy-Paste Setup

The fastest way to understand Zabbix APM is to see your own traces in it. This is the lab from my Zabbix APM course, on one Ubuntu 24.04 machine: Zabbix server, a proxy with the APM collector, ClickHouse, and a small demo shop that sends real OpenTelemetry traces, metrics and logs. Copy each block in order. I ran every command on a fresh machine before publishing this.

What you are building

Demo shop4 services→ OTLP/gRPC :4317 → Zabbix proxyAPM collector→ HTTP :8123 → ClickHouseotel database← Zabbix frontendAPM pages

The proxy only accepts OTLP over gRPC on port 4317. One of the shop's services (storefront) sends OTLP over HTTP on purpose, the way browsers and many Node apps do, so it goes through an OpenTelemetry Collector that turns it into gRPC. The collector also adds CPU, memory and load metrics from the machine. Telemetry never touches the Zabbix database: the proxy writes it to ClickHouse, and the frontend reads it from there.

What you need

Lab passwords. The commands use fixed passwords (zabbix, ChAdmin-2026, zabbix-ch) so every block works as pasted. That is fine on a lab machine. Change them for anything that faces a network you don't control.

Step 1: Zabbix 8.0 repository

Minimal images don't ship wget, so install the basics first.

sudo apt update && sudo apt install -y wget curl gnupg
wget https://repo.zabbix.com/zabbix/8.0/release/ubuntu/pool/main/z/zabbix-release/zabbix-release_latest_8.0+ubuntu24.04_all.deb
sudo dpkg -i zabbix-release_latest_8.0+ubuntu24.04_all.deb
sudo apt update

Step 2: Zabbix server, frontend and PostgreSQL

A normal Zabbix install. php8.3-curl matters here: the frontend talks to ClickHouse over HTTP when you open the APM pages.

sudo apt install -y zabbix-server-pgsql zabbix-frontend-php php8.3-pgsql php8.3-curl zabbix-apache-conf zabbix-sql-scripts zabbix-agent2 postgresql
sudo -u postgres psql -c "CREATE USER zabbix WITH PASSWORD 'zabbix';"
sudo -u postgres createdb -O zabbix zabbix
zcat /usr/share/zabbix/sql-scripts/postgresql/server.sql.gz | sudo -u zabbix psql -q zabbix > /dev/null
sudo sed -i 's/^# DBPassword=.*/DBPassword=zabbix/' /etc/zabbix/zabbix_server.conf
sudo systemctl enable --now zabbix-server zabbix-agent2
sudo systemctl restart apache2
hostname -I

Open http://YOUR-IP/zabbix (the first address from hostname -I) and click through the setup wizard: database type PostgreSQL, password zabbix, everything else as it is. Then log in as Admin / zabbix.

Step 3: a proxy with the APM collector

APM data is received by a proxy, never by the server. On Ubuntu 24.04 and Debian 12 or newer, the normal proxy package already has APM built in. These sed lines make the same four edits I make in nano in the video:

sudo apt install -y zabbix-proxy-sqlite3
sudo mkdir -p /var/lib/zabbix
sudo chown zabbix:zabbix /var/lib/zabbix
sudo sed -i \
  -e 's/^Hostname=Zabbix proxy/Hostname=apm-proxy/' \
  -e 's/^# ListenPort=10051/ListenPort=10052/' \
  -e 's|^DBName=zabbix_proxy|DBName=/var/lib/zabbix/zabbix_proxy.db|' \
  -e 's/^# StartAPMCollector=0/StartAPMCollector=1/' \
  /etc/zabbix/zabbix_proxy.conf
sudo grep -E '^(Server|Hostname|ListenPort|DBName|StartAPMCollector)=' /etc/zabbix/zabbix_proxy.conf
sudo systemctl enable --now zabbix-proxy
sleep 3
sudo grep "APM support" /var/log/zabbix/zabbix_proxy.log

The last line must print this. If it says NO, stop here: nothing else will work.

APM support:           YES

Step 4: ClickHouse and the otel tables

Zabbix 8.0 supports ClickHouse 26.3 to 26.9. The newest stable release is often outside that range, so this uses ClickHouse's LTS channel (26.8 at the time of writing). The block sets the admin password, then runs otel_all.sh, the script that ships with Zabbix and creates the six telemetry tables. -t 604800 keeps data for 7 days. Last comes a zabbix user that can only read and insert in that database.

CH_ADMIN_PW='ChAdmin-2026'
CH_ZABBIX_PW='zabbix-ch'
curl -fsSL https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key | sudo gpg --batch --yes --dearmor -o /usr/share/keyrings/clickhouse-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb lts main" | sudo tee /etc/apt/sources.list.d/clickhouse.list
sudo apt update
sudo DEBIAN_FRONTEND=noninteractive apt install -y clickhouse-server clickhouse-client
echo "<clickhouse><users><default><password remove=\"1\"/><password_sha256_hex>$(echo -n "$CH_ADMIN_PW" | sha256sum | cut -d' ' -f1)</password_sha256_hex></default></users></clickhouse>" | sudo tee /etc/clickhouse-server/users.d/default-password.xml > /dev/null
sudo systemctl enable clickhouse-server
sudo systemctl restart clickhouse-server
sleep 5
clickhouse-client --password "$CH_ADMIN_PW" -q "SELECT version()"
clickhouse-client --password "$CH_ADMIN_PW" -q "CREATE DATABASE IF NOT EXISTS otel"
/usr/share/zabbix/sql-scripts/clickhouse/otel_all.sh -s http://127.0.0.1:8123 -d otel -u default -p "$CH_ADMIN_PW" -t 604800
clickhouse-client --password "$CH_ADMIN_PW" -q "CREATE USER zabbix IDENTIFIED WITH sha256_password BY '$CH_ZABBIX_PW'"
clickhouse-client --password "$CH_ADMIN_PW" -q "GRANT SELECT, INSERT ON otel.* TO zabbix"
clickhouse-client --password "$CH_ADMIN_PW" -q "SHOW TABLES FROM otel"

You should see six tables:

otel_logs
otel_metrics_exponential_histogram
otel_metrics_gauge
otel_metrics_histogram
otel_metrics_sum
otel_traces

No summary table. There is no otel_metrics_summary, so OpenTelemetry summary metrics are rejected. Most SDKs don't send them, but some Prometheus-style exporters do.

Step 5: connect Zabbix to ClickHouse and enable APM on the proxy

In the frontend

  1. Administration → Data source → APM: tick Enabled, URL http://127.0.0.1:8123, database otel, username zabbix, password zabbix-ch. Update.
  2. Administration → Proxies → Create proxy: name apm-proxy, mode Active. On the APM tab, enable data collection and set the rate limit to 100 requests per second. Add.

Or paste this instead

The same two changes through the Zabbix API, run on the Zabbix machine:

ZBX=http://127.0.0.1/zabbix/api_jsonrpc.php
api() { curl -s -H 'Content-Type: application/json-rpc' ${TOKEN:+-H "Authorization: Bearer $TOKEN"} -d "{\"jsonrpc\":\"2.0\",\"method\":\"$1\",\"params\":$2,\"id\":1}" $ZBX; echo; }
TOKEN=$(api user.login '{"username":"Admin","password":"zabbix"}' | sed 's/.*"result":"\([^"]*\)".*/\1/')
api settings.update '{"apm_global_db":{"status":1,"url":"http://127.0.0.1:8123","db":"otel","authentication_type":0,"username":"zabbix","password":"zabbix-ch"}}'
api proxy.create '{"name":"apm-proxy","operating_mode":0,"apm":{"data_collection_status":1,"max_messages_per_second":100}}'

The proxy re-reads its APM settings every second, so there's nothing to restart. Within a minute, Administration → Proxies shows apm-proxy with a fresh Last seen.

The rate limit counts requests, not spans. One request can carry hundreds of spans. When a sender goes over the limit, the OpenTelemetry Collector drops that batch (Exporting failed. Dropping data), while the Python SDK retries. Watch it with the proxy's internal item zabbix[apm,export].

Step 6: the demo shop

Four services (storefront, checkout, inventory, payment) place an order every second or two. Some payments fail and some stock queries are slow, so there is always something to find. It's four small files. This one block writes them with cat, builds the image and starts it, so paste all of it at once.

sudo apt install -y docker.io
mkdir -p ~/shop && cd ~/shop
cat > requirements.txt <<'EOF'
opentelemetry-sdk
opentelemetry-exporter-otlp-proto-grpc
opentelemetry-exporter-otlp-proto-http
EOF
cat > Dockerfile <<'EOF'
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY shop.py .
COPY outage /usr/local/bin/outage
RUN ln -s outage /usr/local/bin/slow && mkdir -p /state
CMD ["python", "-u", "shop.py"]
EOF
cat > outage <<'EOF'
#!/bin/sh
# outage on|off|next, slow on|off - break the demo shop on purpose
# outage: the payment gateway fails most requests; next: only the next order times out; slow: stock queries take 2-3 s
case "$(basename "$0") $1" in
  "outage on")   touch /state/outage; echo "payment gateway outage: ON" ;;
  "outage off")  rm -f /state/outage; echo "payment gateway outage: OFF" ;;
  "outage next") touch /state/timeout_next; echo "the next order gets a payment gateway timeout" ;;
  "slow on")     touch /state/slow; echo "slow stock queries: ON" ;;
  "slow off")    rm -f /state/slow; echo "slow stock queries: OFF" ;;
  *)             echo "usage: outage on|off|next, slow on|off"; exit 1 ;;
esac
EOF
chmod +x outage
cat > shop.py <<'EOF'
"""Demo online shop for a Zabbix APM lab: storefront -> checkout -> inventory + payment, each with its own
service.name, sending traces, metrics and logs over OTLP.

- checkout, inventory and payment send OTLP/gRPC straight to the Zabbix proxy (OTEL_EXPORTER_OTLP_ENDPOINT, :4317).
- storefront sends OTLP/HTTP (like browser or many Node apps) to STOREFRONT_OTLP_HTTP (an OpenTelemetry Collector,
  :4318), which the Zabbix proxy does not accept itself. Without that variable it sends gRPC like the others.
- One order every ~1.4 s; some payments fail (card declined, gateway timeout), some stock queries are slow.
- `outage on|off|next` and `slow on|off` (see ./outage) change that while it runs."""
import logging, os, random, time
from opentelemetry import trace, metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter as HTTPSpanExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter as HTTPMetricExporter
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter as HTTPLogExporter
from opentelemetry.trace import SpanKind, Status, StatusCode

ENDPOINT = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")
HTTP = os.environ.get("STOREFRONT_OTLP_HTTP")  # e.g. http://otelcol:4318
ERROR_RATE = float(os.environ.get("ERROR_RATE", "0.04"))
STATE = "/state"


def flag(name): return os.path.exists(f"{STATE}/{name}")
def error_rate(): return 0.7 if flag("outage") else ERROR_RATE


class Service:
    def __init__(self, name, version, http=None):
        res = Resource.create({"service.name": name, "service.version": version, "host.name": f"{name}-01"})
        if http:
            spans, mets, logs = (HTTPSpanExporter(endpoint=f"{http}/v1/traces"),
                                 HTTPMetricExporter(endpoint=f"{http}/v1/metrics"), HTTPLogExporter(endpoint=f"{http}/v1/logs"))
        else:
            spans, mets, logs = (OTLPSpanExporter(endpoint=ENDPOINT, insecure=True),
                                 OTLPMetricExporter(endpoint=ENDPOINT, insecure=True), OTLPLogExporter(endpoint=ENDPOINT, insecure=True))
        tp = TracerProvider(resource=res); tp.add_span_processor(BatchSpanProcessor(spans))
        self.tracer = tp.get_tracer("shop")
        mp = MeterProvider(resource=res, metric_readers=[PeriodicExportingMetricReader(mets, export_interval_millis=15000)])
        self.meter = mp.get_meter("shop")
        lp = LoggerProvider(resource=res); lp.add_log_record_processor(BatchLogRecordProcessor(logs))
        self.log = logging.getLogger(name); self.log.setLevel(logging.INFO); self.log.propagate = False
        self.log.addHandler(LoggingHandler(level=logging.INFO, logger_provider=lp))


web = Service("storefront", "2.3.0", HTTP)
checkout, inventory, payment = (Service(n, v) for n, v in
                                (("checkout-service", "1.4.2"), ("inventory-service", "1.1.0"), ("payment-service", "3.0.1")))

orders = checkout.meter.create_counter("shop.orders", unit="{order}", description="Orders placed, by result")
order_time = web.meter.create_histogram("shop.checkout.duration", unit="s", description="Time to place an order")
amounts = payment.meter.create_histogram("shop.payment.amount", unit="EUR", description="Charged amounts")
stock = {"sku-1001": 140, "sku-1002": 75, "sku-1003": 22}
inventory.meter.create_observable_gauge("shop.stock.level", unit="{item}", description="Items in stock per SKU",
    callbacks=[lambda o: [metrics.Observation(q, {"sku": s}) for s, q in stock.items()]])


def work(lo, hi): time.sleep(random.uniform(lo, hi) / 1000)


def order():
    t0 = time.time()
    user = random.choice(["anna", "ben", "carla", "david", "emma", "frank"])
    items, sku = random.randint(1, 4), random.choice(list(stock))
    with web.tracer.start_as_current_span("POST /checkout", kind=SpanKind.SERVER, attributes={
            "http.request.method": "POST", "url.path": "/checkout", "user.id": user, "cart.items": items}) as root:
        result = "ok"
        try:
            work(5, 15)
            with checkout.tracer.start_as_current_span("place_order", kind=SpanKind.SERVER, attributes={"order.items": items}):
                with inventory.tracer.start_as_current_span("reserve_stock", kind=SpanKind.SERVER, attributes={"sku": sku}):
                    slow = flag("slow") or (random.random() < 0.1 and not flag("timeout_next"))
                    with inventory.tracer.start_as_current_span("SELECT stock", kind=SpanKind.CLIENT, attributes={
                            "db.system.name": "postgresql", "db.query.text": "SELECT qty FROM stock WHERE sku = $1"}):
                        work(2200, 3000) if flag("slow") else work(300, 700) if slow else work(8, 30)
                    if slow: inventory.log.warning("slow stock query for %s", sku)
                    with inventory.tracer.start_as_current_span("UPDATE stock", kind=SpanKind.CLIENT, attributes={
                            "db.system.name": "postgresql"}):
                        work(5, 20)
                    stock[sku] = stock[sku] - items if stock[sku] > 10 else 150
                result = charge(root)
                if result == "ok":
                    with checkout.tracer.start_as_current_span("INSERT orders", kind=SpanKind.CLIENT, attributes={
                            "db.system.name": "postgresql"}):
                        work(5, 25)
                    checkout.log.info("order placed for %s, %d items", user, items)
        finally:
            orders.add(1, {"result": result})
            order_time.record(time.time() - t0, {"result": result})
        if result == "ok":
            root.set_attribute("http.response.status_code", 200); root.set_status(Status(StatusCode.OK))


def charge(root):
    amount = round(random.uniform(9, 240), 2)
    with payment.tracer.start_as_current_span("charge_card", kind=SpanKind.SERVER, attributes={
            "payment.provider": "stripe", "payment.amount": amount}) as pay:
        with payment.tracer.start_as_current_span("POST api.stripe.com/v1/charges", kind=SpanKind.CLIENT, attributes={
                "http.request.method": "POST", "server.address": "api.stripe.com"}) as gw:
            r, rate = random.random(), error_rate()
            if flag("timeout_next"):  # `outage next`: this order gets a gateway timeout
                os.remove(f"{STATE}/timeout_next"); r = 0
            if r < rate / 2:
                work(2500, 2900)
                gw.set_attributes({"error.type": "timeout", "http.response.status_code": 504})
                gw.set_status(Status(StatusCode.ERROR, "gateway timeout"))
                pay.set_status(Status(StatusCode.ERROR, "payment gateway timeout"))
                pay.record_exception(TimeoutError("api.stripe.com did not answer within 2.5 s"))
                payment.log.error("payment gateway timeout after 2.5 s, order failed")
                root.set_status(Status(StatusCode.ERROR, "order failed")); root.set_attribute("http.response.status_code", 504)
                return "timeout"
            if r < rate:
                work(80, 200)
                gw.set_attributes({"error.type": "card_declined", "http.response.status_code": 402})
                gw.set_status(Status(StatusCode.ERROR, "card declined"))
                pay.set_status(Status(StatusCode.ERROR, "card declined"))
                payment.log.error("payment declined: card declined by issuer")
                root.set_status(Status(StatusCode.ERROR, "order failed")); root.set_attribute("http.response.status_code", 402)
                return "declined"
            work(120, 350)
            gw.set_attribute("http.response.status_code", 200)
            gw.set_status(Status(StatusCode.OK)); pay.set_status(Status(StatusCode.OK))
            amounts.record(amount)
            return "ok"


if __name__ == "__main__":
    os.makedirs(STATE, exist_ok=True)
    print(f"shop: OTLP/gRPC to {ENDPOINT}" + (f", storefront OTLP/HTTP to {HTTP}" if HTTP else ""), flush=True)
    while True:
        try: order()
        except Exception as e: print("order:", e, flush=True)
        time.sleep(random.uniform(0.8, 2.0))
EOF
sudo docker build -t shop .
sudo docker run -d --name shop --network host --restart unless-stopped \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317 \
  -e STOREFRONT_OTLP_HTTP=http://127.0.0.1:4318 shop
cd ~

--network host lets the containers reach the proxy on 127.0.0.1. Without it, use the machine's IP: inside a container, 127.0.0.1 is the container itself. That's also why you shouldn't set APMListenIP to 127.0.0.1 when your senders run in Docker.

Step 7: the OpenTelemetry Collector

Until this runs, the shop logs Connection refused for port 4318. That's the storefront looking for the collector. The other three services already send straight to the proxy.

mkdir -p ~/otelcol
cat > ~/otelcol/config.yaml <<'YAML'
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:
      load:
exporters:
  otlp_grpc/zabbix:
    endpoint: 127.0.0.1:4317
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp_grpc/zabbix]
    metrics:
      receivers: [otlp, hostmetrics]
      exporters: [otlp_grpc/zabbix]
    logs:
      receivers: [otlp]
      exporters: [otlp_grpc/zabbix]
YAML
sudo docker run -d --name otelcol --network host --restart unless-stopped \
  -v ~/otelcol/config.yaml:/etc/otelcol-contrib/config.yaml:ro \
  otel/opentelemetry-collector-contrib:0.165.0

Use 0.165.0 or newer. Older collector releases don't know the otlp_grpc exporter name and exit with unknown type: "otlp_grpc". On an old collector, call the exporter otlp/zabbix instead.

Step 8: check that data arrives

Give it a minute, then ask ClickHouse what landed:

clickhouse-client --password 'ChAdmin-2026' -q "SELECT ServiceName, count() FROM otel.otel_traces GROUP BY ServiceName"
clickhouse-client --password 'ChAdmin-2026' -q "SELECT count() FROM otel.otel_logs"

All four services should be there, with counts that grow every time you run it:

payment-service    40
checkout-service   39
storefront         22
inventory-service  60

Now open APM → Traces in Zabbix. Every row is a POST /checkout from storefront. Click one to see the span tree across all four services. APM → Metrics has the shop's order counter, checkout duration histogram and stock gauge, plus the machine's CPU and memory from the collector. APM → Logs has the shop's log lines, each one linked to its trace.

Break it on purpose

A demo that never fails teaches nothing. The shop has switches:

sudo docker exec shop outage next   # the next order gets a payment gateway timeout
sudo docker exec shop outage on     # the payment gateway fails most orders
sudo docker exec shop outage off
sudo docker exec shop slow on       # stock queries take 2-3 seconds
sudo docker exec shop slow off

Errors show up as red counts in the trace list, and slow orders as long bars in the span tree. This is where telemetry query items come in. They turn "p95 of checkout duration" or "failed payments per minute" into normal Zabbix items that triggers can alert on. The video builds both, including the time shift and lookback settings that trip people up.

If nothing shows up

What you seeWhy
zabbix[apm,export] shows zero accepted requests The senders can't reach the proxy: wrong address or port, a firewall, or 127.0.0.1 used from inside a container.
Requests are accepted, but zero rows arrive ClickHouse is refusing the inserts: wrong password or no INSERT grant. Check the proxy log for the ClickHouse reply.
The proxy won't start DBName still points to a relative path the zabbix user can't write. Use step 3's full path.
Proxy log: APM database is down, retrying ClickHouse isn't running or the data source URL is wrong. The proxy keeps the data in RAM and retries forever, so a proxy restart loses it.
Data is in ClickHouse but the APM pages are empty The frontend's data source: URL, credentials, or missing php8.3-curl.
An app sends to port 4318 and nothing happens The proxy has no OTLP/HTTP listener. Put a collector in front, like step 7.

Watch the full course

The whole setup on camera, plus how the proxy works inside (read from the 8.0.0 source code), mutual TLS, telemetry query items and triggers, templates, permissions, what Zabbix APM still can't do, and how Grafana fills those gaps from the same ClickHouse tables:

Taking Zabbix APM to production?

A lab is one machine and one demo app. Production means sizing ClickHouse, retention, TLS between senders and proxies, permissions, and items that alert on the right things. I do Zabbix audits, template building and integrations through KorFlux. Tell me what you run and I'll tell you how I'd set it up.