DTRC Observability

DTRC Observability Platform

A deployable Grafana, Prometheus, Loki, Tempo and Alertmanager stack for monitoring banking applications in real time, with a BookStack wiki for its own build and configuration documentation. Identical on-prem and on AWS.

https://clientmon1.dtrc.co.inhttps://clientwiki1.dtrc.co.in
Open setup wizard Download PDF guide Next steps

Platform components

Every service in the stack, what it is responsible for, and the file that configures it.

nginx (edge proxy)

TLS termination, path-based routing, per-tenant bearer-token auth on ingest, IP allowlist plus basic auth on ops UIs.

Route / port: 443 / 80 Config: nginx/templates/clientmon1.conf.template

Grafana

Dashboards, explore, unified alerting UI. Provisioned datasources for Prometheus, Loki and Tempo.

Route / port: /grafana/ Config: grafana/dashboards/*.json

Prometheus

Metrics TSDB, scrape jobs with file-based service discovery, remote-write receiver, recording and alert rules.

Route / port: /prometheus/ Config: prometheus/prometheus.yml

Pushgateway

Accepts metrics from short-lived batch and cron jobs that cannot be scraped.

Route / port: /pushgateway/ Config: docker-compose.yml

Loki

Multi-tenant log store and query engine. Ingest at /loki/api/v1/push, queried through Grafana.

Route / port: /loki/ Config: loki/loki-config.yaml

Promtail

Log shipper for host and container logs; the reference agent for client onboarding.

Route / port: n/a Config: promtail/

Tempo

Distributed trace storage with OTLP ingest, linked from logs and metrics via exemplars.

Route / port: /tempo/ Config: tempo/

Alertmanager

Alert deduplication, grouping, silencing and notification routing to receivers.

Route / port: /alertmanager/ Config: alertmanager/alertmanager.yml

BookStack wiki

Documentation portal on its own vhost, backed by MariaDB, hosting build and onboarding pages.

Route / port: WIKI_DOMAIN Config: wiki/

Architecture

All ingress passes through nginx, which terminates TLS, routes by path and enforces per-tenant bearer tokens on ingest and IP allowlisting on operator endpoints.

                  clientmon1.dtrc.co.in            clientwiki1.dtrc.co.in
                             |                                |
                        443 (TLS)                        443 (TLS)
                  +----------v-----------+          +---------v---------+
                  |   nginx edge proxy    |          |  nginx vhost      |
                  |  TLS / routing / auth |          |  IP allowlist     |
                  +--+-----+-----+-----+--+          +---------+---------+
                     |     |     |     |                       |
            /grafana/ /prometheus/ /loki/ /alertmanager/   +----v----+
                     |     |     |     |                  |BookStack|
              +------v-+ +-v-----v-+ +-v----------+       | MariaDB |
              |Grafana | |Prometheus| |   Loki     |       +---------+
              |dash +  | | + Push-  | | multi-     |
              |alerts  | | gateway  | | tenant log |
              +---+----+ +----+-----+ +-----^------+
                  |           |             |
                  |     remote_write   /loki/api/v1/push
                  |           |             |
              +---v-----------v-------------+------+
              |  Tempo (traces)  |  Promtail / Agent |
              +-------------------------------------+
                              |
                     +--------v---------+
                     |  Alertmanager    |
                     |  routing/notify  |
                     +------------------+

Deployment modes

The same compose definition and the same .env drive all three modes.

Local evaluation

HTTP-only overrides, synthetic log/metric/trace generators and a 400-day Loki retention window for backfilled history.

Entry point: scripts/eval-up.sh Env file: .env.eval

OnPrem

Bash installer that provisions Docker, generates any CHANGE_ME BookStack secrets and brings the stack up under systemd.

Entry point: deploy/onprem/install.sh Env file: .env

AWS

Terraform provisions the instance, security groups and secrets; user-data renders the same env file and runs the installer.

Entry point: deploy/aws/ Secrets: AWS Secrets Manager

Next steps

Follow these in order to go from a fresh clone to a verified, client-onboarded platform.

  1. Create your .env from the template

    Run the setup wizard to fill in domains, TLS mode, ops CIDRs and secrets, then download the generated file into the repo root.

    cp .env.example .env # or download from /setup

  2. Validate the stack configuration

    Checks that every required file exists and that all compose, Prometheus, Loki, Alertmanager and dashboard files parse.

    npm run validate

  3. Provision certificates

    Let's Encrypt for public deployments, or self-signed certificates for internal networks.

    ./scripts/init-letsencrypt.sh # or ./scripts/generate-certs.sh

  4. Bring the stack up

    Evaluation mode runs the full stack over plain HTTP on localhost with synthetic data generators.

    ./scripts/eval-up.sh # production: docker compose up -d

  5. Onboard the first client

    Issues a tenant bearer token and writes the client's nginx and Prometheus target files.

    ./scripts/onboard-client.sh <client-name>

  6. Verify end-to-end data flow

    Confirm metrics are scraped, logs land in Loki and traces reach Tempo before handing the platform over.

    curl -s localhost/prometheus/-/healthy && curl -s localhost/loki/ready

  7. Deploy for real

    OnPrem installer or the AWS Terraform stack — both consume the same .env.

    ./deploy/onprem/install.sh # or: cd deploy/aws && terraform apply