Node is offline (decided)

SILO Node Operators
1

Hello, the node does not start, the node is offline, port forwarding is configured, the firewall is configured, as I understand it, the problem is widespread, 3 people already have the same problem, I am attaching the node startup log in Docker

2025-01-22 13:27:12 2025-01-22 10:27:12,839 WARN received SIGTERM indicating exit request
2025-01-22 13:27:12 2025-01-22 10:27:12,839 INFO waiting for processes-exit-eventlistener, storagenode, storagenode-updater to die
2025-01-22 13:27:12 2025-01-22T10:27:12ZINFOGot a signal from the OS: “terminated”{“Process”: “storagenode-updater”}
2025-01-22 13:27:12 2025-01-22 10:27:12,841 INFO stopped: storagenode-updater (exit status 0)
2025-01-22 13:27:12 2025-01-22T10:27:12ZINFOGot a signal from the OS: “terminated”{“Process”: “storagenode”}
2025-01-22 13:27:12 2025-01-22T10:27:12ZINFOcontact:servicecontext cancelled{“Process”: “storagenode”, “Orbital ID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”}
2025-01-22 13:27:13 2025-01-22 10:27:13,408 INFO stopped: storagenode (exit status 0)
2025-01-22 13:27:13 2025-01-22 10:27:13,408 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2025-01-22 13:27:18 2025-01-22 10:27:18,761 INFO Set uid to user 0 succeeded
2025-01-22 13:27:18 2025-01-22 10:27:18,763 INFO RPC interface ‘supervisor’ initialized
2025-01-22 13:27:18 2025-01-22 10:27:18,763 INFO supervisord started with pid 1
2025-01-22 13:27:19 2025-01-22 10:27:19,767 INFO spawned: ‘processes-exit-eventlistener’ with pid 58
2025-01-22 13:27:19 2025-01-22 10:27:19,773 INFO spawned: ‘storagenode’ with pid 59
2025-01-22 13:27:19 2025-01-22 10:27:19,778 INFO spawned: ‘storagenode-updater’ with pid 60
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOConfiguration loaded{“Process”: “storagenode-updater”, “Location”: “/app/config/config.yaml”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “server.private-address”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “healthcheck.enabled”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “contact.external-address”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “healthcheck.details”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “storage.allocated-bandwidth”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “operator.wallet”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “operator.wallet-features”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “console.address”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “operator.email”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOInvalid configuration file key{“Process”: “storagenode-updater”, “Key”: “server.address”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOAnonymized tracing enabled{“Process”: “storagenode-updater”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFORunning on version{“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.0.0-P”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFODownloading versions.{“Process”: “storagenode-updater”, “Server Address”: “https://updater.silos3.com”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOConfiguration loaded{“Process”: “storagenode”, “Location”: “/app/config/config.yaml”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOAnonymized tracing enabled{“Process”: “storagenode”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOOperator email{“Process”: “storagenode”, “Address”: “[email protected]”}
2025-01-22 13:27:19 2025-01-22T10:27:19ZINFOOperator wallet{“Process”: “storagenode”, “Address”: “0x5bf65dec155a62b162734fb0049131fa95316c0a”}
2025-01-22 13:27:20 2025-01-22T10:27:20ZINFOserverkernel support for server-side tcp fast open remains disabled.{“Process”: “storagenode”}
2025-01-22 13:27:20 2025-01-22T10:27:20ZINFOserverenable with: sysctl -w net.ipv4.tcp_fastopen=3{“Process”: “storagenode”}
2025-01-22 13:27:20 2025-01-22T10:27:20ZINFOCurrent binary version{“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.0.0-P”}
2025-01-22 13:27:20 2025-01-22T10:27:20ZINFOVersion is up to date{“Process”: “storagenode-updater”, “Service”: “storagenode”}
2025-01-22 13:27:20 2025-01-22T10:27:20ZINFOCurrent binary version{“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.0.0-P”}
2025-01-22 13:27:20 2025-01-22T10:27:20ZINFOVersion is up to date{“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2025-01-22 13:27:20 2025-01-22T10:27:20ZINFOTelemetry enabled{“Process”: “storagenode”, “instance ID”: “1n1kPXrf6S7CSeaqfMQSZKmQKxgm4vorJq5AgBsooGRKsmd4KB”}
2025-01-22 13:27:20 2025-01-22T10:27:20ZINFOEvent collection enabled{“Process”: “storagenode”, “instance ID”: “1n1kPXrf6S7CSeaqfMQSZKmQKxgm4vorJq5AgBsooGRKsmd4KB”}
2025-01-22 13:27:20 2025-01-22 10:27:20,840 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-22 13:27:20 2025-01-22 10:27:20,840 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-22 13:27:20 2025-01-22 10:27:20,841 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-22 13:27:21 2025-01-22T10:27:21ZINFOdb.migrationDatabase Version{“Process”: “storagenode”, “version”: 62}
2025-01-22 13:27:22 2025-01-22T10:27:22ZINFOpreflight:localtimestart checking local system clock with trusted orbitals’ system clock.{“Process”: “storagenode”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOpreflight:localtimelocal system clock is in sync with trusted orbitals’ system clock.{“Process”: “storagenode”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOcollectorexpired pieces collection started{“Process”: “storagenode”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFONode 1n1kPXrf6S7CSeaqfMQSZKmQKxgm4vorJq5AgBsooGRKsmd4KB started{“Process”: “storagenode”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOPublic server started on [::]:28967{“Process”: “storagenode”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOPrivate server started on 127.0.0.1:7778{“Process”: “storagenode”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOfailed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See UDP Buffer Sizes · quic-go/quic-go Wiki · GitHub for details.{“Process”: “storagenode”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOtrustScheduling next refresh{“Process”: “storagenode”, “after”: “7h24m51.317581683s”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFObandwidthPersisting bandwidth usage cache to db{“Process”: “storagenode”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOpieces:trashemptying trash started{“Process”: “storagenode”, “Orbital ID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOlazyfilewalker.trash-cleanup-filewalkerstarting subprocess{“Process”: “storagenode”, “orbitalID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOcollectorexpired pieces collection completed{“Process”: “storagenode”, “count”: 0}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOlazyfilewalker.trash-cleanup-filewalkersubprocess started{“Process”: “storagenode”, “orbitalID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOlazyfilewalker.trash-cleanup-filewalker.subprocesstrash-filewalker started{“Process”: “storagenode”, “orbitalID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”, “Process”: “storagenode”, “dateBefore”: “2025-01-15T10:27:23Z”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOlazyfilewalker.trash-cleanup-filewalker.subprocessDatabase started{“Process”: “storagenode”, “orbitalID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”, “Process”: “storagenode”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOlazyfilewalker.trash-cleanup-filewalker.subprocesstrash-filewalker completed{“Process”: “storagenode”, “orbitalID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”, “Process”: “storagenode”, “bytesDeleted”: 0, “numKeysDeleted”: 0}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOlazyfilewalker.trash-cleanup-filewalkersubprocess finished successfully{“Process”: “storagenode”, “orbitalID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”}
2025-01-22 13:27:23 2025-01-22T10:27:23ZINFOpieces:trashemptying trash finished{“Process”: “storagenode”, “Orbital ID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”, “elapsed”: “211.798474ms”}
2025-01-22 13:27:25 2025-01-22T10:27:25ZERRORcontact:serviceping orbital failed {“Process”: “storagenode”, “Orbital ID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”, “attempts”: 1, “error”: “ping orbital: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID”, “errorVerbose”: “ping orbital: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID\n\tsilos3.com/silo/storagenode/contact.(*Service).pingOrbitalOnce:209\n\tsilos3.com/silo/storagenode/contact.(*Service).pingOrbital:157\n\tsilos3.com/silo/storagenode/contact.(*Chore).updateCycles.func1:87\n\tsilos3.com/common/sync2.(*Cycle).Run:102\n\tsilos3.com/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2025-01-22 13:27:27 2025-01-22T10:27:27ZERRORcontact:serviceping orbital failed {“Process”: “storagenode”, “Orbital ID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”, “attempts”: 2, “error”: “ping orbital: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID”, “errorVerbose”: “ping orbital: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID\n\tsilos3.com/silo/storagenode/contact.(*Service).pingOrbitalOnce:209\n\tsilos3.com/silo/storagenode/contact.(*Service).pingOrbital:157\n\tsilos3.com/silo/storagenode/contact.(*Chore).updateCycles.func1:87\n\tsilos3.com/common/sync2.(*Cycle).Run:102\n\tsilos3.com/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}
2025-01-22 13:27:31 2025-01-22T10:27:31ZERRORcontact:serviceping orbital failed {“Process”: “storagenode”, “Orbital ID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”, “attempts”: 3, “error”: “ping orbital: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID”, “errorVerbose”: “ping orbital: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: tls peer certificate verification: tlsopts: peer ID did not match requested ID\n\tsilos3.com/silo/storagenode/contact.(*Service).pingOrbitalOnce:209\n\tsilos3.com/silo/storagenode/contact.(*Service).pingOrbital:157\n\tsilos3.com/silo/storagenode/contact.(*Chore).updateCycles.func1:87\n\tsilos3.com/common/sync2.(*Cycle).Run:102\n\tsilos3.com/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

2

I think you missed on step try,

For windows - ./trustkey.exe authorize silonode <email:your-authorization-token>

For Linux - trustkey authorize silonode <email:your-authorization-token>

make sure you replace email:your-authorization-token with your email:token

3

no, I didn’t forget, everything was done correctly, one node works for me, the second one unfortunately doesn’t. We also set up a node yesterday at my friend’s, absolutely the same error occurs. We redid it many times, but the error is the same. the second node is on my other PC

4

make sure both of your nodes have different port number

5

are they on different PCs, or do you still need different ports?

6

you can run on same PC but port should not be same

Node 1: 28967
Node 2: 28968

Like this

7

changed ports, recreated the container, created a new port forwarding and firewall rules, the error is the same
2025-01-22 14:47:50 2025-01-22T11:47:50Z ERROR contact:service ping orbital failed {“Process”: “storagenode”, “Orbital ID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”, “attempts”: 2, “error”: “ping orbital: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: EOF”, “errorVerbose”: “ping orbital: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: EOF\n\tsilos3.com/silo/storagenode/contact.(*Service).pingOrbitalOnce:209\n\tsilos3.com/silo/storagenode/contact.(*Service).pingOrbital:157\n\tsilos3.com/silo/storagenode/contact.(*Chore).updateCycles.func1:87\n\tsilos3.com/common/sync2.(*Cycle).Run:102\n\tsilos3.com/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

used this command to create a container


docker run -d --restart unless-stopped --stop-timeout 300 -p 28968:28968/tcp -p 28968:28968/udp -p 127.0.0.1:14002:14002 -e WALLET=“0x5bf65dec155a62b162734fb0049131fa95316c0a” -e EMAIL="[email protected]" -e ADDRESS=“95.31…:28968” -e STORAGE=“6TB” --mount type=bind,source=“C:\Users\Web\AppData\Roaming\Silo\Trustkey\silonode”,destination=/app/trustkey --mount type=bind,source=“E:\SILO”,destination=/app/config --name storagenode silos3/silo-node:latest

8

Try to ping this 95.31…:28968 from any other network and check are you able to reach it, Problem looks like you ip:port is not able to access by the SILO network

Try the website from and check - https://portchecker.co/

9

checked from a remote computer, the port is open Port 28968 is open. the errors are still the same

10

Test Container Connectivity

Ping the updater server to ensure external connectivity:

ping updater.silos3.com
11

ping is ok

Обмен пакетами с updater.silos3.com [104.21.91.225] с 32 байтами данных:
Ответ от 104.21.91.225: число байт=32 время=22мс TTL=54
Ответ от 104.21.91.225: число байт=32 время=21мс TTL=54
Ответ от 104.21.91.225: число байт=32 время=21мс TTL=54
Ответ от 104.21.91.225: число байт=32 время=21мс TTL=54

12

try to Regenerate the Trustkey and restart the complete process from start on the second node

13

This is the error it gives me


PS C:\Windows\system32> ./trustkey.exe authorize silonode [email protected]:1TSogH52m2tGJ8boCESne…ZrERamKapx3MoqgwG3MmWqPdnH8cLUE2zp8z32Tf8rpJ5fTdZB8azrE
2025-01-22T16:50:52+03:00 INFO Anonymized tracing enabled
Error: certificate: authorization already claimed: [email protected]:1TSogH…

14

Please get a new token from Silo Site

15

it turns out that for each node we need to create a new token?

16

Yes, you need to new auth token for every new node

17

the error is the same, I have a question, for this command ./trustkey.exe authorize silonode email:your-authorization-token I have to use a VPN connection, which creates its own network card and as I understand it, the signal when activated goes through this network card. after activation, I turn off the VPN and the connection goes from my network card. the question is, is there a binding to the network card, if yes, then the reason for the error is clear.

18

which error? please share it

VPN is not creating any issue when you run the authorize command then it communicates with the Silo server which is not accessed by your normal internet

make sure your firewall opens Inbound & outbound ports 28967, 28968, 14002, 1111, 80, 443, 10000

19

firewall is offline, port forwarding tcp\udp 28968 to 28968

command: docker run -d --restart unless-stopped --stop-timeout 300 -p 28968:28968/tcp -p 28968:28968/udp -p 127.0.0.1:14002:14002 -e WALLET=“0x5bf65dec155a62b162734fb0049131fa95316c0a” -e EMAIL="[email protected]" -e ADDRESS=“95.31…:28968” -e STORAGE=“6TB” --mount type=bind,source=“C:\Users\Web\AppData\Roaming\Silo\Trustkey\silonode”,destination=/app/trustkey --mount type=bind,source=“E:\SILO”,destination=/app/config --name storagenode silos3/silo-node:latest

docker log error:
2025-01-22 19:35:31 2025-01-22T16:35:31ZERRORcontact:serviceping orbital failed {“Process”: “storagenode”, “Orbital ID”: “1dkRxqhU46NfjV1xJZrxnt8RwWGK2uAas1jtncvZoj4s8HVVK4”, “attempts”: 1, “error”: “ping orbital: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: EOF”, “errorVerbose”: “ping orbital: failed to ping storage node, your node indicated error code: 0, rpc: tcp connector failed: rpc: EOF\n\tsilos3.com/silo/storagenode/contact.(*Service).pingOrbitalOnce:209\n\tsilos3.com/silo/storagenode/contact.(*Service).pingOrbital:157\n\tsilos3.com/silo/storagenode/contact.(*Chore).updateCycles.func1:87\n\tsilos3.com/common/sync2.(*Cycle).Run:102\n\tsilos3.com/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78”}

20

As I See your port open:-

Screenshot 2025-01-22 at 10.27.23 pm
Screenshot 2025-01-22 at 10.27.23 pm760×912 38.8 KB

but TCP response on Browser:-

Screenshot 2025-01-22 at 10.28.43 pm
Screenshot 2025-01-22 at 10.28.43 pm842×580 9.75 KB

And Expected Response (when open http://95.31.XXXXX:28968):-

Screenshot 2025-01-22 at 10.30.56 pm
Screenshot 2025-01-22 at 10.30.56 pm1154×262 14.4 KB