I ported a knowledge-format (OKF) library to zero-dependency .NET — here’s what I learned

If you can cat a file, you can read the knowledge base. If you can git clone a repo, you can ship it. No vector database to stand up, no proprietary export format, no vendor lock-in — just a directory of markdown files with YAML frontmatter that a human can open in any editor and an agent can read with ReadFile. That’s the whole pitch, and it’s the reason I spent the last few weeks porting a Rust library to C# to get it onto .NET.

The format behind that pitch is Google’s Open Knowledge Format (OKF) v0.1, and the library is OKF4net (docs & project site) — a zero-dependency .NET (C#, net10.0) implementation, plus an optional layer for wiring OKF bundles straight into agents built on the Microsoft Agent Framework. This post is the launch story: what OKF actually is, why I ported it instead of writing a wrapper, and what « zero dependency » really costs and buys you.

What OKF is

OKF defines a bundle: a directory tree of UTF-8 markdown files, where each file is a concept — a YAML frontmatter block followed by a markdown body. Concepts cross-link each other with ordinary markdown links, index.md files give you progressive-disclosure directory listings, and log.md files record date-grouped change history. The only hard conformance requirement is a non-empty type field on every concept; everything else — unknown types, unknown keys, broken links — has to be tolerated by a conformant consumer. It’s deliberately boring as a format, which is the point: the format is context here, not the pitch. The pitch is what you get to do with plain files — diff them, review them in a PR, grep them, back them up with nothing but git.

The port story

This repository used to ship a Rust implementation of OKF. I removed it at commit d20343c — but only after proving, file by file and command by command, that the C# port produced byte-identical output. tests/fixtures/golden/ holds five golden captures taken directly from the Rust binary’s stdout — validate, info, graph --dot, fmt, and index — against a shared example bundle, and the C# CLI is diffed byte-for-byte against every one of them in CI. As of today the full suite passes end-to-end, including five byte-exact golden CLI comparisons against the original captures.

I’ll say the quiet part out loud: this port was AI-assisted, done largely with Claude Code driving the migration file by file, spec section by spec section, with the golden fixtures as the ground truth it had to match exactly. I think that’s worth stating plainly rather than glossing over — a byte-exact port across languages is a fairly mechanical, well-specified translation task with an unambiguous pass/fail signal (does the output match the captured bytes, yes or no), which is exactly the kind of task where an AI pair-programmer earns its keep and where you can trust the result because you can verify it byte-for-byte rather than having to take anyone’s word for it. The interesting design decisions — the YAML subset, the permissive-loading philosophy, the two-tier validation split — came from following the spec and the Python reference implementation; the AI assistance was in the grinding, get-every-byte-right execution, not the architecture.

Show, don’t tell

Here’s the library, loading a bundle and running a conformance check:

using OKF4net;

var bundle = Bundle.Load("./my_bundle");
Console.WriteLine($"{bundle.Count} concepts");

// Conformance check (§9).
var report = BundleValidator.Validate(bundle);
if (report.IsConformant)
{
    Console.WriteLine($"conformant with OKF v{OkfSpec.Version}");
}

// Traverse the cross-link graph.
var id = ConceptId.Parse("tables/orders");
foreach (var link in bundle.LinksFrom(id))
{
    Console.WriteLine($"{id} -> {link.Target} (exists: {link.Exists})");
}

Bundle.Load never aborts on a malformed concept file — it collects parse failures into bundle.ParseErrors and keeps walking the tree, because a knowledge base that one bad file can take down entirely is a bad knowledge base.

And here’s the CLI, which is the same tool the Rust binary used to be, invocation-for-invocation:

okf validate ./bundles/ga4
okf graph ./bundles/ga4 --dot | dot -Tsvg > graph.svg

okf validate exits non-zero on a non-conformant bundle, so it drops straight into a CI step. The CLI ships as a self-contained, Native AOT single-file binary — no .NET runtime install required on the machine that runs it.

The agents angle

The reason I care about this format enough to port a whole library for it is OKF4net.Agents, which turns an OKF bundle into tools and context for the Microsoft Agent Framework. OkfBundleTools wraps one bundle root and exposes nine function tools — read, browse, graph, search, write, append-log, regenerate-indexes, validate, changes-since — that an AIAgent can call directly:

var tools = new OkfBundleTools("./my_bundle");
AIAgent agent = chatClient.AsAIAgent(tools: tools.GetTools());
var response = await agent.RunAsync("Search the bundle for concepts about refunds.");

Layer OkfContextProvider onto the same tools instance and, opted in explicitly, an agent’s exchanges get captured as long-term memory — one markdown concept per UTC day, written through the same validated, lock-protected write path the tools use, plus a matching log.md entry. That’s the part I think is genuinely different from the usual answer to « give my agent memory »: instead of an opaque vector store you can’t audit, memory is a markdown file in a git-tracked directory. You can open it, diff it across commits, redact a line, or point a second agent at the exact same directory with no export step. It’s not a fit for every use case — the README is upfront that v1 memory is bundle-global and unscoped, so it’s opt-in and meant for a shared, non-sensitive bundle rather than a multi-tenant deployment — but for a single team’s shared knowledge base, « memory you can git blame » is a real capability, not a slogan.

Design choices

The whole library — OKF4net and OKF4net.Cli — has zero third-party runtime dependencies: no YAML library, no CLI-parsing package, nothing. It has its own documented YAML subset parser (frontmatter is scalars, lists, and shallow maps — no anchors, no tags, no multi-document files, and it says so with a clear error if you hand it those), its own markdown link scanner, and its own argument parsing, all on top of the .NET base class library. That constraint is what makes the CLI publishable as a single-file Native AOT binary with no runtime to install, and it’s what keeps the barrier to contributing low — there’s no framework to learn before you can read the code. OKF4net.Agents is the one exception, since talking to Microsoft.Agents.AI requires depending on it; everything else stays dependency-free by design, enforced project by project. The project also ships OKF4net.Catalog, a local multi-bundle catalog with search-by-source resolution, and OKF4net.Mcp, an MCP server that plugs a bundle straight into Claude Desktop or Claude Code — so agents and tools have a ready path to discover and query bundles without writing that plumbing themselves.

Come contribute

OKF4net is young and I’d rather it stay welcoming than gate-kept. You don’t need any prior OKF knowledge to help — the good first issue label names the files to touch and the test that should go green when you’re done, ROADMAP.md lays out where the project is headed, and Discussions is the place to ask a question before you write any code. The project is licensed LGPL-3.0-or-later, and the bar to your first PR is exactly three commands: dotnet build, dotnet test, dotnet format. If any part of « knowledge bundles you can cat and agents that remember things in files you can read » sounds useful to you, I’d love the help — and the feedback.

Migrating Matomo from Cloud to self-hosted — Part 3: Cutover, archiving and operations

Parts 1 and 2 covered the model and the full import/configuration procedure. This last part is about timing and running it in production: the day-before / day-of / stabilization timeline, scheduled archiving, backups, GDPR re-activation, and rollback.

The single most under-estimated fact of this whole migration:

The migration starts the day before, not on go-live day. The 05:00 go-live is only the tracking switch. All the heavy lifting — import, configuration, first archive — happens the evening before.

WhenWhatDuration
Day-1, afternoonImport + configuration + Tag Manager + rehearsal≈ 4 h (incl. ≈1h45 import)
Day-1, eveningFirst full archiveseveral hours on a large history
Day, before 05:00Tracking cutover≈ 30 min

Phase A — Day before (Day-1)

This phase runs everything from Part 2, in order, ending with the first full archive. The steps that specifically belong to the day-before rehearsal:

A1–A2. Sanity gates. Containers are up (podman ps), and the image version is  the dump’s version_core. If the image is older, stop here — core:update will refuse and nothing downstream matters.

A3–A8. Import and wire up. Detect the real dump format, derive the prefix, create the DB/user, verify the DB is empty, import (in tmux), verify the import and restore durability, write config.ini.php. (All detailed in Part 2.)

A9. core:update → expect « Everything is already up to date ».

A10. Tag Manager → activate, regenerate containers, confirm a 200.

A11. Inventory then disable scheduled reports — save the list first, because you’ll need it to re-enable exactly the same ones later:

$DB_EXEC="podman exec -i mariadb"
$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -e \
 "SELECT idreport, idsite, login, description, period FROM report WHERE deleted = 0;" \
 | tee /var/backups/matomo/active-reports.txt
$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -e \
 "UPDATE report SET deleted = 1 WHERE deleted = 0;"

deleted is a reversible flag, not a real delete. Keep active-reports.txt — it’s the only record of which reports to bring back.

A12. Disable the GDPR purge during migration — to rule out any deletion concurrent with cutover:

$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -e \
 "UPDATE \`option\` SET option_value='0' WHERE option_name='delete_logs_enable';"
$APP_EXEC ./console core:clear-caches

A13–A14. Configure geolocation (UI) and generate security files.

A15. Persistence test — do not skip this. This is the check that catches the silent volume trap before it costs you. Restart the app container and confirm everything survived:

podman restart matomo && sleep 25
$APP_EXEC sh -c 'head -4 /var/www/html/config/config.ini.php'         # config survived?
$APP_EXEC sh -c 'ls /var/www/html/js/container_*.js 2>/dev/null | wc -l'  # containers survived?
curl -sS -o /dev/null -w "container %{http_code}\n" \
     "https://analytics.example.com/js/container_${IDC}.js"           # still 200?
$APP_EXEC ./console plugin:list | grep -i tagmanager                  # still Activated?

If the config disappears or the container falls back to 404, the volumes are not persistent → stop and fix with your infra team before any cutover. This is the trap that breaks everything silently on the first container recreation.

A16. First full archive — the evening of Day-1. Run it manually, in tmux:

tmux new -s archive
time $APP_EXEC ./console core:archive --url=https://analytics.example.com

Expect it to end with Done archiving!. An exit code 1 alongside Done archiving! is normal here — it comes from a failed report send (no SMTP yet), not from an archiving failure. This must run the evening before; started on go-live morning it won’t finish in time and reports would be slow and incomplete when users log in.

A17. Functional rehearsal — walk the go/no-go checklist below.


The go / no-go checklist

Before cutover, confirm:

  • [ ] Image on the pinned build; core:version ≥ dump’s version_core.
  • [ ] core:update ran without error.
  • [ ] Superuser login works (Cloud credentials); 2FA works.
  • [ ] Historical data visible (a past period renders).
  • [ ] Site main_url values updated (no leftover Cloud URLs).
  • [ ] Geolocation active for new traffic.
  • [ ] Scheduled reports disabled, and the idreport list saved.
  • [ ] HTTPS workingforce_ssl = 1, security files generated.
  • [ ] Behind the proxy: visits carry the real client IP, not the proxy’s.
  • [ ] GDPR settings verified (anonymization, retention).
  • [ ] Single collation — the collation query returns exactly one row.
  • [ ] 🚨 Tag Manager active and containers served (curl → 200, not 404).
  • [ ] config.ini.php, plugins, GeoIP and js/container_*.js on persistent volumes — verified by restarting the container.
  • [ ] A test hit shows up in real time.
  • [ ] Archive timer active + first run OK.
  • [ ] Monitoring in place (timer failure, disk space).

Some items are deliberately not satisfied at cutover and that’s fine — track them, don’t tick them: SMTP not configured, scheduled reports disabled, premium plugins absent, GDPR purge disabled, restorable backup tested. They’re decisions, not failures. Never tick them « to look clean » — a successful core:test-email during rehearsal would mean SMTP is live, which means reports can go out, exactly what you’re avoiding.


Phase B — Go-live day, before 05:00 (~30 min window)

B1. Confirm the day-before archive finished.

B2. Re-check the Tag Manager container serves a 200 (rerun A10 if it’s 404).

B3. Switch the tracking — the irreversible move on the sites. First, separate your two populations, because they switch differently:

# Sites WITH a Tag Manager container -> switch the container URL
$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -e \
 "SELECT s.idsite, s.name, c.idcontainer FROM \`site\` s
    JOIN tagmanager_container c ON c.idsite = s.idsite AND c.status='active'
   ORDER BY s.idsite;"

# Sites WITHOUT a container -> switch the classic tracking code
$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -e \
 "SELECT idsite, name FROM \`site\`
   WHERE idsite NOT IN (SELECT idsite FROM tagmanager_container WHERE status='active');"

Then, on the sites:

  1. Update the classic tracking code (matomo.js / matomo.php URL) to analytics.example.com — the sites without a container first, and anywhere the snippet is hard-coded.
  2. Update the Tag Manager container URL on the sites that use one.
  3. Or switch DNS if you keep the same hostname — then no URL changes are needed.

« All sites are reporting » is not a sufficient check — it doesn’t prove the Tag-Manager-published sites are covered. Check the two populations separately in B4.

B4. Verify real-time collection, per population:

$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -e \
 "SELECT s.idsite, s.name,
         CASE WHEN c.idsite IS NULL THEN 'direct code' ELSE 'Tag Manager' END AS mode,
         COUNT(v.idvisit) AS visits_30min
    FROM \`site\` s
    LEFT JOIN (SELECT DISTINCT idsite FROM tagmanager_container WHERE status='active') c
           ON c.idsite = s.idsite
    LEFT JOIN log_visit v ON v.idsite = s.idsite
           AND v.visit_last_action_time > NOW() - INTERVAL 30 MINUTE
   GROUP BY s.idsite, s.name, mode ORDER BY visits_30min ASC;"

If all Tag Manager sites are at 0 while direct-code sites report, the container isn’t served or the URL wasn’t switched → revisit B2/B3. A single site at 0 isn’t necessarily a failure — low-traffic sites at 5 AM legitimately show zero; compare to each site’s usual volume, not to zero.

B5. Verify real IPs — the reverse-proxy trap. Don’t rely on counting distinct IPs. Test against a known IP:

# 1) From the test machine, note its public IP:
curl -s https://ifconfig.me ; echo
# 2) Generate a visit from that machine on a tracked site.
# 3) Confirm Matomo recorded THAT IP, not the proxy's:
$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -e \
 "SELECT INET6_NTOA(location_ip) AS ip, COUNT(*) AS visits FROM log_visit \
   WHERE visit_last_action_time > NOW() - INTERVAL 15 MINUTE \
   GROUP BY location_ip ORDER BY visits DESC;"

The IP from step 1 must appear. If every visit carries the proxy IP (or an internal 10.x / 172.16-31.x / 192.168.x), the forwarded-for headers aren’t being applied → fix proxy_client_headers and core:clear-cachesFix immediately — visits collected meanwhile are falsified and unrecoverable. The header name must match what your proxy actually sends (X-Forwarded-For usually, sometimes X-Real-IP).


Phase C — Right after cutover (H+0 to H+2)

C1. Watch real-time for ~1h; confirm all sites report.

C2. Enable scheduled archiving — on a container host, prefer a systemd timer over cron; it shares the container’s mode (rootful/rootless) and logging.

matomo-archive.service:

[Unit]
Description=Matomo report archiving
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/podman exec -u www-data matomo ./console core:archive --url=https://analytics.example.com

matomo-archive.timer:

[Unit]
Description=Matomo hourly archiving

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target

Enable it (rootful shown; add --user for rootless, which also needs loginctl enable-linger <user>):

systemctl daemon-reload && systemctl enable --now matomo-archive.timer
systemctl list-timers matomo-archive.timer
journalctl -u matomo-archive.service -n 50

core:archive does more than archive. At the end of each run it triggers the scheduled tasks: emailing reports and the GDPR log purge. Without this timer, neither the reports nor the purge ever run. To trigger them in isolation: ./console scheduled-tasks:run.

C3. Inform users: new URL, unchanged credentials (passwords and 2FA migrated), email reports temporarily suspended, and any premium features currently unavailable.


Phase D — Stabilization (Day+1 to Day+7)

Order is imposed — do not invert it. Reports have been disabled since A11. Wire SMTP first (D1) so you can test it empty and safe, then re-enable reports (D2) knowingly. The reverse order — reports active before a working SMTP — blasts emails on the next archive run.

D1. SMTP — configure the relay, then validate empty:

$APP_EXEC ./console core:test-email

Expect the test email to arrive. Reports are still disabled, so no mass send is possible yet. If it fails, fix the relay before D2.

D2. Re-enable scheduled reports — only the ones saved in active-reports.txt, and only after D1 passes:

$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -e \
 "UPDATE report SET deleted = 0 WHERE idreport IN ( /* ids from active-reports.txt */ );"

Never run a global UPDATE report SET deleted = 0 — it would resurrect reports that were intentionally deleted while on Cloud.

D3. Re-enable the GDPR purge:

$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -e \
 "UPDATE \`option\` SET option_value='1' WHERE option_name='delete_logs_enable';"
$APP_EXEC ./console core:clear-caches

Leaving it off makes the database grow indefinitely and steps outside your declared retention — it’s a compliance control, not an optimization.

D4. Backups — set up database + application volume, and test a restore. The application volume matters: it holds config.ini.php, plugins and the GeoIP database — restoring only the database won’t bring the service back. A reference logical dump:

podman exec -i mariadb mariadb-dump --single-transaction --quick \
  --default-character-set=utf8mb4 -u matomo -p"<APP_PWD>" matomo \
  | gzip > /backups/matomo-$(date +%F).sql.gz

Rotate on a separate target. A backup that has never been restored is not a backup.

D5. Confirm durability is restored (SELECT @@innodb_flush_log_at_trx_commit; → 1).

D6. Cancel the Cloud subscription — point of no return. Only after: a conclusive observation period, a backup successfully restored at least once, and any needed export of the data gap (below).


The data gap — a decision to make explicitly

The dump is a snapshot at time T. Between the dump and the cutover, Cloud keeps collecting. A common, defensible decision is to accept the gap: don’t replay a final dump at go-live; the visits between the dump’s last data and the tracking switch stay only in Cloud and aren’t recovered on-premise.

Consequences to keep in mind:

  • ✅ The cutover window stays short — no ~2h import to fit in, just switching tracking code / DNS.
  • ✅ The pinned image stays valid (it matches the already-imported dump) — no version re-qualification.
  • ⚠️ The gap grows over time — its size is the interval between the dump’s last data and the cutover date. The later the go-live, the longer the missing period. That’s the one argument for cutting over sooner rather than later.
  • 💡 Cloud remains readable until cancellation — if the missing period ever needs analysis, export or consult it from Cloud before cancelling (D6 is the point of no return).

Rollback

As long as Cloud is not cancelled, rolling back is quick:

  1. Restore the old tracking code and Cloud container URLs on the sites.
  2. Confirm collection resumes on Cloud.
  3. The on-premise instance can stay in place for analysis.

No on-premise data is lost — the imported database stays intact. Keep both the original dump and an on-premise backup before any destructive operation.


Troubleshooting cheat-sheet

SymptomCauseFix
Unknown collation 'utf8mb4_0900_ai_ci'MySQL 8 dump on MariaDBin-stream sed conversion (Part 2, Step 3/4)
« empty database » / no datatables_prefix ≠ dumpfix config.ini.php; re-read the prefix
core:update « more recent version »image < Cloudrebuild image on the pinned build
Unsupported hosttrusted_hosts incompleteadd the domain to trusted_hosts[]
All visits share one IPproxy headers not declaredproxy_client_headers[]
Config/plugins lost on restartwritten off a persistent volumemove to persistent volumes
Corrupted accentsimport without --default-character-set=utf8mb4re-import
MySQL server has gone awaymax_allowed_packet too lowraise it + re-import
SQL error on option/sitereserved word without backticks`option` / `site`
core:archive exit 1 but « Done archiving! »scheduled reports + no SMTPcalibrate alerts on log content, not exit code
tar returns nothing (exit 1).tar.gz that’s actually a plain gzipuse gunzip -c
podman ps shows nothingwrong mode (root vs user)run as the container-owning user

Series wrap-up

Three parts, one migration:

The recurring lesson across all three: the import is the easy part. What breaks a Matomo Cloud→self-hosted migration is everything the dump doesn’t carry — and the fact that most of it fails silently, well after the migration looks done.

Migrating Matomo from Cloud to self-hosted — Part 2: The step-by-step procedure

Part 1 explained the model and the pitfalls. This part is the hands-on walkthrough: prepare the database, inspect and import the dump, wire Matomo to it, align the schema, and do the post-migration configuration the dump never carried.

Placeholders used throughout: containers mariadb / matomo, database/user matomo / matomo, public domain analytics.example.com, web user www-data. All commands assume Podman; substitute docker if that’s your runtime. In a rootless Podman setup, run every podman command as the container-owning user — never with sudo, or you’ll drive a different set of containers.

Two shortcuts used below:

DB_EXEC="podman exec -i mariadb"            # SQL commands
APP_EXEC="podman exec -u www-data matomo"   # Matomo console

Step 1 — Prepare the MariaDB database

1a. Import-time server settings

These durable settings belong to the container configuration (ask your infra team):

max_allowed_packet      = 1G
innodb_buffer_pool_size = 4G          # ~50-70% of allocated RAM
character-set-server    = utf8mb4
collation-server        = utf8mb4_general_ci

Remember the collation coupling from Part 1: utf8mb4_general_ci must be identical here, in the CREATE DATABASE below, and as the target of the import sed.

One optimization can be applied hot, without restarting the container, to speed up the import:

$DB_EXEC mariadb -u root -p"<ROOT_PWD>" -e "SET GLOBAL innodb_flush_log_at_trx_commit=2;"

Set this back to 1 as soon as the import finishes (see Step 5b). Leaving it at 2 means a hard server crash can lose ~1s of committed transactions — fine for a replayable import, not acceptable once you’re collecting production traffic.

1b. Create the database, user and grants

$DB_EXEC mariadb -u root -p"<ROOT_PWD>" << 'SQL'
CREATE DATABASE IF NOT EXISTS matomo
  CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

-- '%' because the app container connects over the internal container network
CREATE USER IF NOT EXISTS 'matomo'@'%' IDENTIFIED BY '<APP_PWD>';

GRANT SELECT, INSERT, UPDATE, DELETE,
      CREATE, DROP, ALTER, INDEX,
      CREATE TEMPORARY TABLES, LOCK TABLES,
      CREATE VIEW, SHOW VIEW, TRIGGER, EXECUTE
  ON matomo.* TO 'matomo'@'%';
FLUSH PRIVILEGES;
SQL

The DDL grants (CREATEDROPALTERINDEX) are required because core:update modifies the schema; CREATE TEMPORARY TABLES / LOCK TABLES are required for archiving.


Step 2 — Receive and inspect the dump

These operations happen on the host — the dump never needs to enter a container (the import streams in via stdin).

2a. Detect the real format — the extension lies

Support sometimes ships a file named .sql.tar.gz that is actually a plain gzip of the .sql, not a tar archive. tar then fails silently (empty output, exit 1), and listing a real tar would need to decompress the whole 20 GB. One-second test:

file "$DUMP"
gunzip -c "$DUMP" | head -3      # "-- MySQL dump ..." => plain gzip
  • Plain gzip → use gunzip -c.
  • Real tar archive → use tar -xzOf.

2b. Read the prefix (and prepare to read the version)

gunzip -c "$DUMP" | grep -m1 -i 'CREATE TABLE'   # first table -> the prefix

matomo_log_visit → prefix matomo_log_visit or access → empty prefix. This is Invariant 1 from Part 1 — get it wrong and Matomo shows an empty database.

2c. One-pass compatibility scan (MySQL 8 → MariaDB)

A MySQL 8 dump can contain constructs MariaDB doesn’t support. This single pass lists them and grabs the version along the way:

gunzip -c "$DUMP" | awk '
  index($0,"utf8mb4_0900_ai_ci")>0 {c++}      # MySQL 8 collation (must convert)
  index($0,"DEFAULT_GENERATED")>0  {dg++}      # generated columns
  index($0," VISIBLE")>0           {vis++}     # invisible columns (MySQL 8)
  index($0,"/*!80")>0              {v80++}     # 8.0-gated syntax
  index($0,"CHECK (")>0            {chk++}     # CHECK constraints
  index($0,"CONSTRAINT")>0         {cons++}    # foreign keys
  { p=index($0,"version_core"); if (p>0 && !v) { print "version_core => " substr($0,p,60); v=1 } }
  END{ printf "0900_ai_ci=%d generated=%d invisible=%d gated80=%d check=%d fk=%d\n",
       c,dg,vis,v80,chk,cons }'

Only utf8mb4_0900_ai_ci should be > 0 (handled in Step 3). Everything else — generated / invisible / gated80 / check / fk — must be 0 for a direct import; if not, resolve them before importing. On a ~22 GB dump this scan takes ~3 minutes.


Step 3 — Convert the collation

utf8mb4_0900_ai_ci doesn’t exist in MariaDB (Invariant 3). A single substitution to utf8mb4_general_ci fixes it. The validated method is in-stream during the import — no intermediate file, no extra 20 GB on disk. You’ll see it inline in the next step.

If you do need a converted dump on disk (for multiple replays):

gunzip -c "$DUMP" | sed 's/utf8mb4_0900_ai_ci/utf8mb4_general_ci/g' | gzip > dump.maria.sql.gz

Step 4 — Import the dump

The dump stays on the host and is pushed to podman exec‘s stdin — the -i flag is mandatory. The dump already sets UNIQUE_CHECKS=0 and FOREIGN_KEY_CHECKS=0 in its header. Run this inside tmux or screen — the import takes over an hour and must survive an SSH disconnect.

First, confirm the target database is empty — CREATE DATABASE IF NOT EXISTS does not fail on a populated database, and importing over existing content can leave it incoherent:

$DB_EXEC mariadb -u root -p"<ROOT_PWD>" -N -e \
 "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='matomo';"
# Expected: 0

Then import, converting the collation in-stream:

tmux new -s matomo
time (gunzip -c "$DUMP" \
  | sed 's/utf8mb4_0900_ai_ci/utf8mb4_general_ci/g' \
  | podman exec -i mariadb \
      mariadb --default-character-set=utf8mb4 -u root -p"<ROOT_PWD>" matomo)

For a plain .sql, replace gunzip -c "$DUMP" with cat "$DUMP". The --default-character-set=utf8mb4 matters — without it, accents come back corrupted.

Track progress from another session. The table counter freezes for a long time while the big log_* tables load, so volume size is a better progress indicator than table count:

watch -n 60 "podman exec -i mariadb mariadb -u root -p'<ROOT_PWD>' -N -e \
 \"SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='matomo';\""

As an order of magnitude: ~430 tables and ~22 GB imported in about 1h45. If the import is interrupted, start over from an empty database (DROP DATABASE matomo; then recreate it) — a partial import is not reliable.


Step 5 — Verify the import and restore durability

5a. Check the data landed

$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo <<'SQL'
SELECT COUNT(*) AS nb_tables FROM information_schema.tables WHERE table_schema='matomo';
SELECT option_value AS version_core FROM `option` WHERE option_name='version_core';
SELECT COUNT(*) AS sites FROM `site`;
SELECT COUNT(*) AS log_visit FROM log_visit;
SELECT DATE(MAX(visit_last_action_time)) AS last_data FROM log_visit;
-- collation coherence: MUST return a single row
SELECT table_collation, COUNT(*) FROM information_schema.tables
 WHERE table_schema='matomo' GROUP BY table_collation;
SQL

With an empty prefix, remember the backticks on `option` and `site`. The last query is the collation check from Part 1: a single row (utf8mb4_general_ci) means healthy; multiple rows mean a mixed estate you must fix now, before future archive_* tables inherit the wrong collation and throw Illegal mix of collations weeks later.

5b. Restore durability immediately

$DB_EXEC mariadb -u root -p"<ROOT_PWD>" -e \
 "SET GLOBAL innodb_flush_log_at_trx_commit=1; SELECT @@innodb_flush_log_at_trx_commit;"
# Expected: 1

Do this now, not later — the =2 optimization was import-only, and the instance is about to start collecting real traffic. This value is volatile; have your infra team make it durable in the container config so a restart can’t reset it.


Step 6 — Verify the Matomo image version

The image is built and provided by your infra team; this is a verification, and it’s the check that decides whether core:update will succeed:

$APP_EXEC php -r 'require "/var/www/html/core/Version.php"; echo Piwik\Version::VERSION, "\n";' \
  2>/dev/null || $APP_EXEC grep -m1 "const VERSION" /var/www/html/core/Version.php

The value must be  the dump’s version_core (ideally identical, which makes core:update a no-op). If it’s lower, stop — the image must be rebuilt on the correct pinned build (Invariant 2, Part 1). Don’t proceed; core:update will refuse.


Step 7 — Wire Matomo to the imported database

Write config.ini.php into the volume mounted at /var/www/html/config/, as the container’s web user.

Derive the prefix from the dump — don’t type it from memory. Get the first table and map it to a prefix, and stop if it isn’t recognized:

FIRST_TABLE=$(gunzip -c "$DUMP" | grep -m1 -i 'CREATE TABLE' | sed -E 's/.*`([^`]+)`.*/\1/')
case "$FIRST_TABLE" in
  access)   export TABLE_PREFIX="" ;;
  *_access) export TABLE_PREFIX="${FIRST_TABLE%access}" ;;
  *)        echo "!! Unrecognized prefix -> STOP, inspect the dump" ;;
esac
echo "TABLE_PREFIX=[$TABLE_PREFIX]"
SALT=$(openssl rand -hex 16)
podman exec -i -u www-data matomo sh -c 'cat > /var/www/html/config/config.ini.php' <<EOF

[database]

host = « mariadb » username = « matomo » password = « <APP_PWD> » dbname = « matomo » tables_prefix = « $TABLE_PREFIX » charset = « utf8mb4 » adapter = « PDO\MYSQL » [General] trusted_hosts[] = « analytics.example.com » salt = « $SALT » force_ssl = 1 proxy_client_headers[] = « HTTP_X_FORWARDED_FOR » proxy_host_headers[] = « HTTP_X_FORWARDED_HOST » EOF podman exec -u root matomo chmod 640 /var/www/html/config/config.ini.php # Re-read to confirm the prefix that was actually written $APP_EXEC grep tables_prefix /var/www/html/config/config.ini.php

Four things that matter here:

  • host is the database container name on the internal network — not localhost.
  • tables_prefix uses the derived value. A wrong prefix shows an empty database while the data is right there. And re-read the file: automated generation can silently drop an empty-string prefix (Part 1).
  • The two proxy_* lines are mandatory behind the reverse proxy, or every visit carries the proxy IP. The header names must match what your proxy actually sends.
  • This file must live on a persistent volume. Written to the ephemeral container layer, it vanishes on the first restart and Matomo reruns its installer.

Step 8 — Align the schema (core:update)

$APP_EXEC ./console core:update --yes
$APP_EXEC ./console core:clear-caches
$APP_EXEC ./console core:version
  • « Everything is already up to date » = versions aligned (the expected result when your image matches the dump’s build).
  • « database created by a more recent version » = your image is older than Cloud → back to Step 6, the image must be rebuilt.

Step 9 — Post-migration configuration

Nothing below is in the dump. This is where a migration that imported cleanly becomes a migration that actually works.

9a. Tag Manager — the critical one

If your sites use Tag Manager, this step keeps them tracked (Part 1 explains why). Enable the plugin, regenerate every published container, and verify a 200:

$APP_EXEC ./console plugin:activate TagManager
$APP_EXEC ./console tagmanager:regenerate-released-containers

# Verify a container is actually served (must be HTTP 200, ~100 KB)
IDC=$($DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -N -e \
      "SELECT idcontainer FROM tagmanager_container WHERE status='active' LIMIT 1;" | tr -d '\r')
curl -sS -o /dev/null -w "container %{http_code} / %{size_download} bytes\n" \
     "https://analytics.example.com/js/container_${IDC}.js"

A 404 means the files weren’t generated or /var/www/html/js/ isn’t reachable — do not cut over, tracking would be down. Remember these files must be on a persistent volume.

9b. Geolocation (GeoIP)

The GeoIP database is not in the dump. In the UI: Administration → Geolocation → GeoIP 2, pick a provider (DB-IP Lite is free, auto-downloaded, no account), and enable automatic updates. In a container, the downloaded database must land on a persistent volume or geolocation silently reverts after the next recreation.

Historical visits keep the geolocation computed while on Cloud (it’s stored in the DB, so it migrated) — only new traffic uses the on-premise database. Re-attributing history is a heavy operation (./console usercountry:attribute over millions of visits) and usually unnecessary.

9c. Premium plugins

Inventory what was active on Cloud, and check actual volume (rows in the DB), not just activation:

SELECT REPLACE(option_name,'LastPluginActivation.','') AS plugin,
       FROM_UNIXTIME(option_value) AS last_activation
  FROM `option` WHERE option_name LIKE 'LastPluginActivation.%'
 ORDER BY option_value DESC;

Premium plugins that carry real data (FormAnalytics, ActivityLog, …) will lose their reports in the UI until licensed — but no data is deleted; the tables stay intact and re-adding the license later restores everything, history included. The only required action is to tell users which reports are temporarily unavailable. Plugins that are Cloud-only (CloudBillingCDNOAuth2, …) are meaningless self-hosted — don’t try to reinstall them. Any plugin installed outside the image must live on a persistent volume.

9d. Users, passwords and 2FA — nothing to do

Users, hashed passwords and 2FA secrets are all in the dump. Everyone logs back in with their existing Cloud credentials; no mass reset. A couple of admin commands for edge cases:

$APP_EXEC ./console twofactorauth:disable-2fa-for-user --login=<login>   # lost 2FA device
$APP_EXEC ./console login:unblock-blocked-ips                            # brute-force lockout

(Note: core:create-superuser / user:set-password do not exist in this version — create or reset users via the UI.)

9e. GDPR / privacy settings

Retention, IP anonymization and Do-Not-Track are in-database settings, so they migrated with the dump. But they engage compliance — verify them explicitly:

$DB_EXEC mariadb -u root -p"<ROOT_PWD>" matomo -e \
 "SELECT option_name, option_value FROM \`option\` \
   WHERE option_name LIKE 'PrivacyManager%' ORDER BY option_name;"

One subtlety worth understanding: the log-purge setting (delete_logs_enable) was already active and running on Cloud. Self-hosting doesn’t create the purge — it can only prevent it, because these tasks only run if you schedule archiving (Part 3). During the migration itself it’s common to disable the purge to rule out any concurrent deletion, then re-enable it after stabilization. IP anonymization and Do-Not-Track settings inherited from Cloud should be confirmed with your DPO.

9f. Security files, SMTP and scheduled reports

Generate the security files:

$APP_EXEC ./console core:create-security-files

SMTP and scheduled reports are wired last, and in a specific order — this is a cutover concern, covered in Part 3. The rule to remember now: never connect SMTP before you’ve decided what happens to the migrated scheduled reports, or the next archive run blasts emails to real recipients.


Where this leaves us

At this point the database is imported and verified, Matomo is wired to it and schema- aligned, Tag Manager serves live containers, and geolocation, plugins, users and privacy settings are handled. What’s left is timing — doing all of this in the right order around a go-live so tracking never drops and no stray email goes out.

Part 3 covers exactly that: the day-before / day-of / stabilization timeline, archiving via systemd, backups, the reverse-proxy IP check, GDPR re-activation, and rollback.