The pipeline: what we collect, where it goes, what we keep
Public data comes in from its publishers, is read by code on a schedule, published on this site and its API, and should be kept whole in a vault per source. This page shows the whole path and, store by store, what is kept and what is not. How each data set is shown and licensed.
The schedule
- GitHub Actions runs
.github/workflows/pages.ymlat :17 and :47 each hour (the second time added on 18 November) and on every push to dev. Each run: 9 sources read hourly, events every 6 hours, then the build, the gate, the leak scan and the deploy. - What actually fired: GitHub runs scheduled jobs when it can. On 18 November only 2 scheduled runs fired between 00:00 and 14:40 UTC (02:26 and 11:39); the rest of the data came from runs on each push. That is why a reader saw data 6 hours old. The second time each hour halves the chance of a gap; a run started from outside (workflow_dispatch from a scheduled wake) would close it.
- Discovery (Perplexity) runs once a day per place in data.b@'s session, because CI has no OpenRouter key.
What is kept, store by store
| Source | Read | Processed (latest.json) | Raw bytes per read | Processed history | Read log | Vault |
|---|---|---|---|---|---|---|
| Flights around Faro airport | hourly | 1 KB | not kept (private aircraft) | latest only (overwritten each run) | 336 runs (14 days hourly) | sd1fblzs: not written since 17 Nov |
| Public contracts (Portal BASE) | every 6 hours | 1436 KB | HTML pages, vault only | latest only (overwritten each run) | 336 runs (14 days hourly) | none |
| Public live cameras: where they are and whether they are up | hourly | 60 KB | HTML pages, vault only | latest only (overwritten each run) | 336 runs (14 days hourly) | none |
| Events in Portugal, by place | every 6 hours | 75 KB | HTML pages, vault only | latest only (overwritten each run) | 336 runs (14 days hourly) | none |
| Code from Portuguese organisations on GitHub | every 6 hours | 379 KB | HTML pages, vault only | latest only (overwritten each run) | 336 runs (14 days hourly) | none |
| IPMA daily forecasts by place | hourly | 35 KB | 36 files, about 41 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | dn8m3051: not written since 17 Nov |
| IPMA fire risk | hourly | 51 KB | 3 files, about 63 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | 94lw1wl8: not written since 17 Nov |
| IPMA station observations | hourly | 41 KB | 2 files, about 625 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | a3y1ezgg: not written since 17 Nov |
| IPMA sea forecast | hourly | 9 KB | 4 files, about 10 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | 2q3ospco: not written since 17 Nov |
| IPMA UV index | hourly | 13 KB | 2 files, about 23 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | mah4euep: not written since 17 Nov |
| IPMA weather warnings | hourly | 7 KB | 1 files, about 36 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | 4ol6ht3d: not written since 17 Nov |
| MeteoAlarm CAP feed for Portugal | hourly | 40 KB | 1 files, about 114 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | whliuye4: not written since 17 Nov |
| Social signals: organisations' public posts, by link | every 6 hours | 66 KB | HTML pages, vault only | latest only (overwritten each run) | 336 runs (14 days hourly) | none |
| Store | What | Size now | Kept how long | State |
|---|---|---|---|---|
| This site (GitHub Pages) | latest.json, reads.json, raw files of the latest read, events, claims, the register, the Storm data, the pages | about 18 MB a build | replaced on every deploy | working |
| Git (this repository) | collectors, cards, the Storm data (8.8 MB), discovery answers and claims, the register | about 14 MB | for ever (history) | working |
| GitHub Actions run | everything a run reads, raw bytes included | about 1 to 3 MB a run | deleted when the run ends | the raw bytes are lost here today |
| Vaults (sgit, one per source) | every raw file, a dated copy when the bytes change, full sha256 | about 1.5 MB an hour if kept (about 1 GB a month) | for ever, versioned | blocked: CI needs SG_SEND_ACCESS_TOKEN and DATA_VAULT_KEYS (Dinis); last written 17 Nov |
What it would take to keep everything
- Vault secrets in CI (the simplest): Dinis adds
SG_SEND_ACCESS_TOKENandDATA_VAULT_KEYSto this repository's Actions secrets.collect/vault-push.mjsalready runs in the workflow and is skipped only because they are missing. Every raw file and a dated copy when it changes would then be kept, per source, readable with each vault's public read key. - An append lane instead: CI would hold only a write-only append token (it cannot read or delete), the safer key to give GitHub. Each hourly bundle would be sealed and appended; a drain would file it into the vaults. It needs lanes large enough for about 1.5 MB an hour.
- Processed history: each hour's latest.json could also go to the vault (small: about 0.6 MB an hour for all sources), so the site can draw a day or a week, not only now.