Building Georgia’s Statewide Analytics Dashboard Without an Application Server
Turning a Real-Time Request Into a Nightly Pipeline That Still Runs Years Later
A first-person case study on the State of Georgia's public analytics dashboard: questioning a real-time requirement, a TypeScript collector with retries and checkpoints, pre-aggregated JSON, a static Eleventy site and a nightly AWS CodeBuild job.
Executive Summary
Georgia runs dozens of state agency websites on a shared platform. Digital Services Georgia wanted a public dashboard showing how that whole platform performs, statewide and agency by agency.
The data came from Siteimprove, which reports per site. Turning that into one statewide picture meant about 700 API requests per run across 75+ agency sites, followed by aggregation into statewide and per-agency datasets.
The first request was a real-time dashboard. I questioned that requirement, showed that daily data met the actual need, and the client agreed. That one conversation shaped everything after it: a nightly batch pipeline instead of a live application, pre-aggregated JSON instead of a database, and a static site instead of a server.
At Lullabot, I owned the backend and data architecture, which Lullabot’s architect validated. I wrote the collector, picked up substantial front-end work when a teammate was injured, and then set up and tuned the AWS production pipeline.
The Requirement I Challenged
“Real time” is one of the most expensive words in a requirements document. It implies live API calls on every page view, a server to make them, caching to protect the upstream API, and a failure mode where a slow upstream response becomes a slow public page.
So I asked what real time was for. What came out of that conversation was that the client wanted the information to be public and current, and the underlying numbers only changed meaningfully day to day. Nobody needed a number from five minutes ago. They needed today’s number, every day, reliably.
I also pointed out a practical risk. With this volume of requests, a live approach would hit timeouts, and visitors could see different numbers depending on which calls succeeded. The client agreed, and appreciated that I had pushed back. Engineering Approach Note About 700 upstream requests are needed to build one consistent statewide view. Doing that during a page view is slow and fragile; doing it per visitor is wasteful. Run the collection once per night, when nobody is waiting on it, and publish a single consistent snapshot for everyone. The dashboard is up to a day old. For public, platform-level performance data that was the right trade, and it removed the need for a running application server entirely.AN-001 — Nightly Batch Over Real Time
Context
The Approach
Tradeoffs
Architecture Overview
| Layer | Role |
|---|---|
| Siteimprove API | Per-site metrics for each agency website on the platform |
| TypeScript collector | Nightly job: requests, retries, logging, checkpoints, aggregation |
| Aggregated JSON | Statewide and per-agency datasets, computed once |
| Eleventy | Static site generator that reads the JSON and renders the dashboard |
| AWS S3, CodeBuild, CloudFront | Storage, the nightly scheduled build, and delivery |
| Email and Slack | Failure alerts for the team |
Engineering Approach Note
AN-002 — A Collector Designed to Finish
- Approach
- TypeScript job with timeout handling, retries, logging and resume-from-checkpoint
- Risk
- One failed request out of hundreds spoiling the whole run
Context
Hundreds of calls to a third-party API will not all succeed on the first try every night. A collector that gives up on the first timeout produces either no data or partial data.
The Approach
The collector waits for all requests to complete before processing, retries requests that time out, logs every failure, and if it is interrupted, continues from the last item it finished instead of starting over. Only then does it aggregate.
Tradeoffs
Checkpointing and retries add code that a happy-path script would not need. That code is what made the nightly run dependable without someone watching it.
Engineering Approach Note
AN-003 — Pre-Aggregated JSON and Static Pages
- Approach
- Do all calculation at build time; ship plain HTML, CSS and JavaScript
- Client goal
- No more servers to maintain
Context
The client was explicit that they did not want another server to maintain. The aggregated data was large and varied, and it had to load quickly for the public.
The Approach
The collector writes the finished statistics into a JSON file. The Eleventy site reads that file locally at build time and renders the pages, so a visitor downloads a page whose numbers are already computed.
Tradeoffs
Every change in the data requires a rebuild, which the nightly job already does. In return there is no runtime database, no application server and almost nothing to patch, and the pages load immediately.
Engineering Approach Note
AN-004 — Failing Transparently, With a Fallback
- Approach
- Alert by email and Slack, keep serving the previous day's data
- Principle
- Transparency over silence
Context
Upstream APIs change. Even with defensive code, a removed field or a broken endpoint can stop a run. The client cared most about knowing, quickly and in Slack, when that happened.
The Approach
If the nightly run fails, the site keeps the previous day's data, and the team receives an email and a Slack message explaining what failed, short in Slack and more detailed in the email. Nothing breaks for the public, and nothing fails silently for the team.
Tradeoffs
Early on the alerts produced several false positives. I tuned them until a real upstream break produced essentially one alert, which is what makes people keep reading alerts.
Running It in Production
Once the build was done, Lullabot’s contract with the client still had time available, and I used it to take the pipeline to production on AWS: an S3 bucket, a nightly AWS CodeBuild job, the region the project required, and CloudFront in front of the site. There were other layers we could have added. I recommended against them, because each one would have been something else to maintain.
One honest detail: an early CodeBuild configuration cost more than it should have. I studied the service, tested it, and tuned the configuration until the nightly run cost what it should.
I also did a fair amount of front-end work. When my front-end colleague was injured partway through, I stepped in so the project did not stall.
Outcomes
- No server for the client to maintain. The public site is plain HTML, CSS and JavaScript built from pre-computed data.
- Fast pages. Visitors receive numbers that were calculated overnight, so nothing is computed while they wait.
- Updates without anyone involved. The nightly job refreshes the data on its own, and failures arrive as a clear alert with yesterday’s data still online.
- Longevity. When I checked in October 2026, analytics.georgia.gov was still live and showed data last updated on September 2, 2026, years after Lullabot delivered it.
Lessons Learned
Ask what a requirement is for. “Real time” turned out to mean “public and current”. That difference removed a server, a cache and a whole class of failures.
Do the work when nobody is waiting. Moving computation from request time to build time is one of the most reliable performance improvements available.
Design the failure path first. A fallback to yesterday’s data plus a clear alert is better than a perfect pipeline that fails silently.
Tune alerts until people trust them. One alert per real problem is a feature. Five alerts per problem is noise.
Approach Notes Summary
| Note | Topic | Nature |
|---|---|---|
| AN-001 | Nightly batch over real time | Requirement change, agreed with the client |
| AN-002 | A collector designed to finish | Implemented |
| AN-003 | Pre-aggregated JSON and static pages | Implemented |
| AN-004 | Failing transparently, with a fallback | Implemented and tuned |
Final Thoughts
The most valuable thing I did on this project happened before I wrote any code: I questioned one word in the requirements. Everything else, the nightly job, the JSON, the static site, the alerts, followed from taking the client’s real need seriously instead of the literal request. It is also why a dashboard built in 2021 was still quietly refreshing itself in 2026.
Explore all Engineering Case Studies or read ECS-005: Migrating Carnegie Mellon University From Cascade CMS to Drupal.
