Repository intelligence platform
Hecate
Star counts are a bad proxy for whether a project is alive. Hecate collects repository and package data from four sources, collects discussion from two more, and keeps a daily snapshot so growth can be measured rather than guessed.
v1.2.0, running unattended on a daily schedule
- dbt models in the transform layer
- 63
- upstream sources normalised to one schema
- 6
- issues closed to reach v1.2.0
- 32
The problem
A repository with 60,000 stars and no commits in three years is a different thing from one with 8,000 stars and weekly releases, but most tooling treats them as comparable. Download counts say more about real use than either, and discussion says something different again, usually earlier.
The harder problem is discovery. If you seed a tracker from cumulative popularity, which is what GitHub's most-starred and npm's most-installed give you, then anything trending before it is famous is excluded by the seeding itself. The tracker can only ever confirm what you already knew.
How it works
Four extractors pull projects from GitHub, npm, PyPI and GitLab. Two more pull conversation from Hacker News and Lobsters. Each one is rate-limit aware and retries on failure.
A transformer normalises everything to a single schema, then loads into PostgreSQL idempotently. A run that dies halfway through can just be run again, which matters more than it sounds when the whole thing is unattended.
Posts about projects Hecate does not yet track are the interesting case. They are not discarded. They are kept, ranked, and the projects behind them get fetched and added, so discussion decides what gets tracked rather than the other way round.
Every tracked project is snapshotted daily. That table is the only history in the system, and it is the only reason growth can be measured at all, because everything else describes the present and upserts in place.
dbt turns the raw tables into staging views, then facts, dimensions, growth and momentum models. Grafana reads the result, Prometheus watches the jobs, and alerts fire when a run does not land.
Where it landed
The platform reached v1.2.0 with all 32 tracked issues closed, and now runs on a daily schedule without anyone starting it. The Grafana dashboard exposes 15 panels on anonymous read, so the data is inspectable without an account.
What it taught me
A green panel can still be measuring the wrong thing
Three dashboard panels were rendering correctly, refreshing on time, and reporting a number that did not mean what the panel title claimed. Nothing in the monitoring could catch that, because the failure was in the definition rather than the pipeline. I now write down what a metric is supposed to mean before building the panel, and check the query against that sentence.
Idempotency is a scheduling feature, not a database one
Making every load idempotent looked like extra work during the build. It is the single reason the daily schedule is safe to leave alone, because recovery from a partial run is just running it again rather than working out what landed.
Scheduled jobs fail quietly in ways live services do not
A job that never starts produces no error, no log line, and no alert unless you are specifically watching for absence. Catching a container image mismatch took far longer than it should have because the run simply was not there to fail.
Built with
- Python
- PostgreSQL
- dbt
- Kubernetes
- Docker
- Prometheus
- Grafana