Inspecting Logs
Log analysis is essential to understanding how a server is performing. In any infrastructure, reviewing logs from both the operating system and services is critical for detecting issues, verifying process health, and ensuring operational stability.
Audit Auth Logs
Review authentication logs to detect failed login attempts, sudo misuse, and other security-relevant events. Monitoring /var/log/auth.log helps identify unauthorized access and privilege escalation issues.
Show all failed sudo authentications
grep 'sudo: pam_unix(sudo:auth): authentication failure' /var/log/auth.logExample output:
2025-06-24T22:39:30 node-01-nyc sudo: pam_unix(sudo:auth): authentication failure; logname=bob uid=1003 euid=0 tty=/dev/pts/4 ruser=bob rhost= user=bobTo extract only the lines related to invalid SSH users or abrupt resets in /var/log/auth.log, use:
grep -E 'Invalid user|Connection reset by invalid user' /var/log/auth.logSample Output
2025-06-24T23:10:45 node-01-nyc sshd[3782288]: Connection reset by invalid user admin 45.135.232.177 port 27918 [preauth]
2025-06-24T23:10:49 node-01-nyc sshd[3782294]: Invalid user admin from 45.135.232.177 port 30478Auditing Fail2ban Logs
fail2ban is a service that monitors logs for suspicious activity (e.g., failed SSH logins) and automatically blocks offending IPs via the firewall. This section covers how to review its actions via /var/log/fail2ban.log
View the most recent activity
Follow in real time
IP Banned Example:
Auditing Logrotate Activity
logrotate is responsible for rotating, compressing, and deleting old log files to prevent disk usage from growing uncontrollably.
If logrotate is managed by systemd, you can view the most recent log entries with:
You can also check its dedicated log file at:
If logrotate encounters an error, you'll see entries like this in the journal:
Solana Audit Logs
Solana validators generate comprehensive audit logs that are crucial for monitoring validator performance, detecting anomalies, and maintaining operational security. These logs provide detailed insights into validator operations, consensus participation, and network interactions.
Initial Startup Monitoring
After starting the validator service, immediately check the logs to verify proper initialization:
If the validator is running correctly, you should immediately see snapshot download progress indicating the validator is synchronizing with the network:
The tail command will continue to display the output of a file as the file changes. You should see a continuous stream of log output as your validator runs. Keep an eye out for any lines that say ERROR.
Jito Relayer Logs
To verify correct operation of the validator when connected to the Jito relayer and block engine, monitor these specific metrics in the validator logfile:
These metrics are emitted once per second when properly connected:
block_engine_stage-stats: Indicates active connection to the block engine
relayer_stage-stats: Indicates active connection to the relayer
Additional Verification of Correct Operation
After monitoring the server to identify that it's functioning correctly, in addition to the Jito documentation that indicates when it's running well in Co-hosted relayer, we also capture several lines in the Solana logs that indicate it's functioning correctly:
Monitoring Slot Leadership
Another log that we should consider to verify that the relayer is functioning correctly is the validator's behavior when it becomes a leader slot .
When your validator becomes the slot leader, you should see a message like this:
Once the slot is produced, you should see a message indicating that your validator is no longer the leader:
Authentication Error Handling
It's also good to check the logs of the jito-relayer service:
If you encounter this error: error authenticating and connecting
You must contact the Jito team and provide them with the pubkey that you will use for the relayer.
Last updated
Was this helpful?