# Building an observability platform on Hono and Bun

We had fifteen services and fifteen ways of finding out something was wrong, most of them starting with someone opening a dashboard because a customer had complained. I wanted one place that could name the failing service, point at the endpoint, and say when it had started. A full tracing stack would have answered more than that and taken a quarter to build.

## Two alerting systems that did not work

Collecting the data turned out to be the easy half. Deciding what deserved a phone call took three attempts.

Version one alerted on error rate per service, which is useless the moment a service handles two requests an hour: one bad request is a fifty percent error rate and a phone call at 2am. Version two alerted on absolute error counts instead, and slept straight through a service that had quietly started failing every third call all week.

Version three is a threshold on both, evaluated over a rolling fifteen minutes, with a floor on request volume before anything is allowed to fire. Slack gets the summary and the link opens the filtered request list, not a dashboard home page, because those ten seconds of re-applying filters at 2am are where people give up and go back to sleep.

-- version three: a rate, a count, and a floor under both
SELECT svc,
       count(*) AS total,
       count(*) FILTER (WHERE status >= 500) AS errors
FROM request
WHERE at > now() - interval '15 minutes'
GROUP BY svc
HAVING count(*) >= 20                              -- volume floor
   AND count(*) FILTER (WHERE status >= 500) >= 5  -- absolute
   AND avg((status >= 500)::int) > 0.05;               -- rate

## One middleware per service

Against three versions of the alerting logic, the collection side has never been changed. Every service ships the same twelve-line middleware: it times the request, reads the status off the response, and posts a batch to the ingest endpoint on a timer. Nothing blocks the response, and a dead ingest endpoint costs the caller nothing but a dropped batch.

# the whole client integration, copied into every service
app.use('*', async (c, next) => {
  const started = performance.now()
  await next()
  queue.push({
    svc: Bun.env.SERVICE_NAME, path: c.req.path,
    status: c.res.status, ms: performance.now() - started,
  })
})

// the other half: flushing happens on a timer, never on the request path
setInterval(async () => {
  if (queue.length === 0) return

  const batch = queue.splice(0, queue.length)

  await fetch(INGEST_URL, { method: 'POST', body: JSON.stringify(batch) })
    .catch(() => {})  // a dead ingest costs the caller one batch
}, 5000)

## What Bun changed

• ▸Ingest is cheap enough to be boring. A single Bun process handles the batch volume from every service without a queue in front of it, so there is one less moving part to page about.
• ▸The built-in SQL client removed a dependency. Batches go straight into Postgres with prepared statements, in one transaction per batch.
• ▸Hono runs the same code on the edge and on a box. The ingest endpoint and the dashboard share route definitions and validation, which is most of the reason the project stayed small.

// no pg driver, no query builder: one transaction per batch
import { sql } from 'bun'

await sql.begin(async (tx) => {
  await tx`INSERT INTO request ${tx(batch)}`
})

// one route file, served by Bun locally and by a worker at the edge
import { validator } from 'hono/validator'

app.post(
  '/ingest',
  validator('json', (v) => BatchSchema.parse(v)),
  async (c) => {
    const batch = c.req.valid('json')
    await store(batch)
    return c.json({ accepted: batch.length }, 202)
  },
)

## What I left out on purpose

No distributed tracing, no sampling strategy, no retention tiers. Request-level data and alerts we believed answered every question fifteen services actually raised in a year. The day one of those questions needs a trace id is the day to add tracing.