# The pipeline: what we collect, where it goes, what we keep

Public data comes in from its publishers, is read by code on a schedule, published on this site and its API, and should be kept whole in a vault per source. This page shows the whole path and, store by store, what is kept and what is not. [How each data set is shown and licensed](/how/).

A diagram: sources, collection in GitHub Actions, the latest files and the log on this site, the raw bytes to the vaults (blocked), the readers.

## The schedule

- GitHub Actions runs `.github/workflows/pages.yml` at :17 and :47 each hour (the second time added on 18 November) and on every push to dev. Each run: 9 sources read hourly, events every 6 hours, then the build, the gate, the leak scan and the deploy.
- **What actually fired:** GitHub runs scheduled jobs when it can. On 18 November only 2 scheduled runs fired between 00:00 and 14:40 UTC (02:26 and 11:39); the rest of the data came from runs on each push. That is why a reader saw data 6 hours old. The second time each hour halves the chance of a gap; a run started from outside (workflow_dispatch from a scheduled wake) would close it.
- Discovery (Perplexity) runs once a day per place in data.b@'s session, because CI has no OpenRouter key.

## What is kept, store by store

| Source | Read | Processed (latest.json) | Raw bytes per read | Processed history | Read log | Vault |
|---|---|---|---|---|---|---|
| [Flights around Faro airport](/sources/adsb-faro/) | hourly | 1 KB | not kept (private aircraft) | latest only (overwritten each run) | 336 runs (14 days hourly) | sd1fblzs: not written since 17 Nov |
| [Public contracts (Portal BASE)](/sources/base-gov-pt/) | every 6 hours | 1436 KB | HTML pages, vault only | latest only (overwritten each run) | 336 runs (14 days hourly) | none |
| [Public live cameras: where they are and whether they are up](/sources/cams/) | hourly | 60 KB | HTML pages, vault only | latest only (overwritten each run) | 336 runs (14 days hourly) | none |
| [Events in Portugal, by place](/sources/events-pt/) | every 6 hours | 75 KB | HTML pages, vault only | latest only (overwritten each run) | 336 runs (14 days hourly) | none |
| [Code from Portuguese organisations on GitHub](/sources/github-pt/) | every 6 hours | 379 KB | HTML pages, vault only | latest only (overwritten each run) | 336 runs (14 days hourly) | none |
| [IPMA daily forecasts by place](/sources/ipma-city-forecasts/) | hourly | 35 KB | 36 files, about 41 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | dn8m3051: not written since 17 Nov |
| [IPMA fire risk](/sources/ipma-fire-risk/) | hourly | 51 KB | 3 files, about 63 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | 94lw1wl8: not written since 17 Nov |
| [IPMA station observations](/sources/ipma-observations/) | hourly | 41 KB | 2 files, about 625 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | a3y1ezgg: not written since 17 Nov |
| [IPMA sea forecast](/sources/ipma-sea/) | hourly | 9 KB | 4 files, about 10 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | 2q3ospco: not written since 17 Nov |
| [IPMA UV index](/sources/ipma-uv/) | hourly | 13 KB | 2 files, about 23 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | mah4euep: not written since 17 Nov |
| [IPMA weather warnings](/sources/ipma-warnings/) | hourly | 7 KB | 1 files, about 36 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | 4ol6ht3d: not written since 17 Nov |
| [MeteoAlarm CAP feed for Portugal](/sources/meteoalarm-pt/) | hourly | 40 KB | 1 files, about 114 KB a read | latest only (overwritten each run) | 336 runs (14 days hourly) | whliuye4: not written since 17 Nov |
| [Social signals: organisations' public posts, by link](/sources/social/) | every 6 hours | 66 KB | HTML pages, vault only | latest only (overwritten each run) | 336 runs (14 days hourly) | none |

| Store | What | Size now | Kept how long | State |
|---|---|---|---|---|
| This site (GitHub Pages) | latest.json, reads.json, raw files of the latest read, events, claims, the register, the Storm data, the pages | about 18 MB a build | replaced on every deploy | working |
| Git (this repository) | collectors, cards, the Storm data (8.8 MB), discovery answers and claims, the register | about 14 MB | for ever (history) | working |
| GitHub Actions run | everything a run reads, raw bytes included | about 1 to 3 MB a run | deleted when the run ends | the raw bytes are lost here today |
| Vaults (sgit, one per source) | every raw file, a dated copy when the bytes change, full sha256 | about 1.5 MB an hour if kept (about 1 GB a month) | for ever, versioned | **blocked:** CI needs SG_SEND_ACCESS_TOKEN and DATA_VAULT_KEYS (Dinis); last written 17 Nov |

## What it would take to keep everything

- **Vault secrets in CI (the simplest):** Dinis adds `SG_SEND_ACCESS_TOKEN` and `DATA_VAULT_KEYS` to this repository's Actions secrets. `collect/vault-push.mjs` already runs in the workflow and is skipped only because they are missing. Every raw file and a dated copy when it changes would then be kept, per source, readable with each vault's public read key.
- **An append lane instead:** CI would hold only a write-only append token (it cannot read or delete), the safer key to give GitHub. Each hourly bundle would be sealed and appended; a drain would file it into the vaults. It needs lanes large enough for about 1.5 MB an hour.
- **Processed history:** each hour's latest.json could also go to the vault (small: about 0.6 MB an hour for all sources), so the site can draw a day or a week, not only now.

---

Version v0.1.47 · data read 2026-11-18 19:18 UTC · Code Apache-2.0; our pages, records and visualisations CC BY 4.0; each source's data under its own licence or terms, linked · [Repository](https://github.com/PrompTugal/PrompTugal__News__Data) · [Pay for what you read](https://promptugal.news/account/) · https://data.promptugal.news/pipeline/
