Worker: troubleshooting#

Find the line you see in the logs, then apply the fix. The logs:

journalctl -u tog-worker-signer -f     # "tog-signer listening"
journalctl -u tog-worker-executor -f   # "tog-executor serving"
journalctl -u tog-worker -f            # togd

For more detail from togd, set RUST_LOG=info,libp2p=warn,tog_worker=debug in its unit and restart it.

Start-up and permissions#

You see	Cause and fix
signer: `peer refused uid=…`; togd: `connecting to the signer`	The signer's `--uid` is not the uid togd runs as. Set it to `id -u tog`. The same goes for the executor's `togd_uid` (`no user with uid N` if that uid does not exist)
`…/worker.key: mode 644, must not be readable by group or others`	`chmod 600` the key files, `togd/network.key` too
`is not a 32-byte seed`	The key file is not exactly 32 bytes. Make it again
`runtime = "process" has no sandbox; set allow_unisolated = true for development`	Add `allow_unisolated = true` to `executor.toml`
`policy.tiers offers t1 without policy.accept_t1_risk`	Add `accept_t1_risk = true` under `[policy]`
`paths must be absolute`	Every path in `executor.toml` must start with `/`
`CAS layout …: …cas/in: owner 0 mode …` or `cas/out layout`	root created the cache. Run `sudo chown -R tog:tog /opt/tog/worker/togd` and restart
`No space left on device`	Lower `[cas] in_max_bytes` in `togd.toml` and `[images] max_bytes` in `executor.toml`, free disk, restart
`p2p.bootnodes is empty: togd needs Hub full nodes to follow`	Fill `[p2p] bootnodes` from the Networks page

Registration#

You see	Cause and fix
`RegisterWorker has nonce N, but the account expects M`	Build `register.borsh` again with nonce `M`
`RegisterWorker not included after 120s`	The Hub dropped it. Usually the owner's free balance is below the bond, more than about 10 minutes passed since the registration was built, or this worker key was registered before. Use fresh `worker.key` and `vrf.key` in the last case, then build and send again
`kernel set <hash> is not in the CAS; pass --kernel-set <file>`	Download the current kernel set and pass `--kernel-set`
togd repeats `worker <id> is not registered on the Hub`	The registration has not been included yet, or went to another network. Check `tog-cli worker <WORKER_ADDRESS>`

Network and following the Hub#

You see	Cause and fix
Port 30333 not reachable from outside	Firewall, cloud security group or NAT. Other workers then measure your bandwidth as zero and cannot fetch from you. Open or forward TCP 30333 and set `external`
`trusted state expired: newest verified header is from …` or `hub light client: operator action needed`	The trusted header is older than 14 days. Put a recent checkpoint in `trusted_height` and `trusted_block_hash` and restart
togd stops following at an era change	No peer from a second network confirmed the new validator set. List every published bootnode
Clock not synchronized	Run chrony or systemd-timesyncd. `timedatectl` should say `System clock synchronized: yes`

Registered but no jobs#

Check, in this order:

tog-cli worker <WORKER_ADDRESS> shows status Active and a bond of at least 10 000 test-TOG.
The p2p port is reachable from another machine: nc -vz <PUBLIC_IP> 30333.
Its bandwidth has been measured (it shows in tog-cli worker). That happens during the first sessions after start.
There may be no test jobs at the moment. Jobs come in bursts on the testnet.

Reporting a problem#

Include the release tag you installed, your OS, the commands you ran and the last 200 lines of each unit's log. Never include key files. See Getting help.