Logs, metrics and traces

Everything every container writes to stdout and stderr, kept for seven days, including containers that have since restarted and an environment that is asleep. A log file inside a container is not read, so log to stdout. One line per event; a JSON line with a level field, or a level word such as ERROR, gives the line its level. A line longer than 64 KiB is cut, and says so.

When a container's lines need parsing first, such as a stack trace over several lines, the pod template carries OpenTelemetry's logs annotation with parsers:

metadata:
  annotations:
    io.opentelemetry.discovery.logs.app/config: |
      operators:
        - type: recombine
          combine_field: body
          is_first_entry: body matches "^\\d{4}-\\d{2}-\\d{2}"

Only parsers and transformers are allowed. A configuration that is refused gives way to the default parsing, and a warning line in that container's own logs says why. io.opentelemetry.discovery.logs.<container>/enabled: "false" stops collecting a container. A project that logs far more than its share is read more slowly rather than dropped; if a container's log files rotate away before they are read, a warning line says which time range was lost.

Without code

Every container's CPU, memory, network and file system, its restarts and readiness, why it is waiting (CrashLoopBackOff, ImagePullBackOff) and why it last ended (OOMKilled), and each pod's phase, every 30 seconds, kept for fifteen days. Every series is labelled with the environment, namespace, workload, pod and container, the names in your k8s/ files.

Requests

Every request that reaches one of your HTTPRoutes through the platform's gateway is counted, with no code: edge_requests by route and code_class (2xx to 5xx), and edge_request_duration, the seconds until your app answered. The editor's Metrics panel shows each route's requests per second, its server errors and its 95th percentile response time.

Your own metrics

A container that serves Prometheus's text format is scraped at its own address once its pod template opts the port in:

metadata:
  annotations:
    io.opentelemetry.discovery.metrics.8080/enabled: "true"
    io.opentelemetry.discovery.metrics.8080/config: |
      metrics_path: /metrics
      collection_interval: 30s

Only the path and the interval (10 seconds at least) can be set; the endpoint needs no credentials. Each opted-in port reports an up series, 1 or 0, with scrape_error saying why when it is 0. Or push your metrics with an OpenTelemetry SDK, as below. A project keeps up to 10,000 series.

Names starting with edge, k8s or container are the platform's, and so are labels starting with k8s or basable: a metric of your own with such a name is not kept, and such a label is removed from it.

Every container gets these variables, unless it sets OTEL_EXPORTER_OTLP_ENDPOINT in its own env:

BASABLE_NODE_IP=<its node's address>
OTEL_EXPORTER_OTLP_ENDPOINT=http://$(BASABLE_NODE_IP):4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Every request through the platform's gateway starts a trace: a span of the edge service with the method, the address without its query string, the status, the user agent and the route. It carries no visitor's IP address. The gateway passes the trace on to your app in the traceparent header, so an app with an SDK continues it and the trace shows the request from the gateway to your database.

Any OpenTelemetry SDK reads them, so traces, metrics and logs it sends reach your project with no address of your own. Basable's Rust framework (basable-app 0.8.0 and later) starts the SDK by itself and sends its spans. Traces are kept for seven days and open by their trace id. Inside your environment's cluster the variables show in kubectl exec <pod> -- env, not in the pod's own spec.

What a container sends is always filed under the project and pod that sent it, whatever the data says about itself.

Every project's logs, metrics and traces are also in Grafana, at grafana.basable.com. You sign in with your Basable account; nothing to set up. Each of your organisations is an organisation in Grafana, with its projects in it and nobody else's.

Its Basable folder holds a project's overview (CPU and memory per workload, restarts, volumes, and each route's requests, server errors and response time), its logs and its traces; pick the project and the environment at the top. Save dashboards of your own beside them: only your organisation's members see them. In a query of your own, a project is picked by its id, basable_tenant_id (in traces resource.basable.tenant_id), which the project page's address shows.

The Grafana button in the editor, and Open in Grafana in its metrics window, open the project's overview there with the project already chosen.

Grafana reads only what your Basable account may read, decided again by Basable on every query: leave an organisation and its projects disappear from Grafana within a minute. Grafana gets no email address of yours. Disconnect it under Account Settings like any other app.

Basable staff have no standing access to your projects' logs, metrics and traces. When they need to look, they open a read grant on one project, with a reason, for an hour. Every grant on your organisation's projects is listed on its Staff access page with the reason, and the project's page shows a notice while one is open.

Basable watches every live environment and mails your organisation's contact address when a container keeps crashing, when one runs out of memory and is restarted, when more than half of the requests to a route end in a server error, or when a volume is more than 90% full. One mail per environment lists what it found, at most once a day for each problem while it lasts, with a link to the project.

Mail is off until you turn it on, per environment, in .basable/config.yaml: alerts: {email: {enabled: true}} under the environment. The same block takes your own rules: PromQL over the project's metrics, each with a name, read in that environment alone, alerting for every series the query returns, so the query carries its own threshold (jobs_waiting > 1000). A rule the metrics store cannot parse fails the deploy, naming it. Your own rules work on preview too.

Alerts can also go to your app: add webhook: {path: /basable/alerts} to the environment's alerts. Basable then POSTs each alert there, on the environment's own address on Basable and never another host, signed the Standard Webhooks way with an Ed25519 key (webhook-signature: v1a,...). The public keys are at basable.com/api/alerts/webhook-keys. Check the signature, check that the body's project_id, environment and target are your own, and drop a webhook-id you have already seen: a delivery that got no answer is tried again. Your app forwards it wherever you like, to a chat or a pager. Route the path to a small service of its own, so an alert still arrives while the app itself is down.