search
HomepageStake NowLog in ->

Inspecting Logs

Log analysis is essential to understanding how a server is performing. In any infrastructure, reviewing logs from both the operating system and services is critical for detecting issues, verifying process health, and ensuring operational stability.

hashtag
Audit Auth Logs

Review authentication logs to detect failed login attempts, sudo misuse, and other security-relevant events. Monitoring /var/log/auth.log helps identify unauthorized access and privilege escalation issues.

Show all failed sudo authentications

grep 'sudo: pam_unix(sudo:auth): authentication failure' /var/log/auth.log

Example output:

2025-06-24T22:39:30 node-01-nyc sudo: pam_unix(sudo:auth): authentication failure; logname=bob uid=1003 euid=0 tty=/dev/pts/4 ruser=bob rhost=  user=bob

To extract only the lines related to invalid SSH users or abrupt resets in /var/log/auth.log, use:

grep -E 'Invalid user|Connection reset by invalid user' /var/log/auth.log

Sample Output

2025-06-24T23:10:45 node-01-nyc sshd[3782288]: Connection reset by invalid user admin 45.135.232.177 port 27918 [preauth]
2025-06-24T23:10:49 node-01-nyc sshd[3782294]: Invalid user admin from 45.135.232.177 port 30478
circle-info

These log entries appear when an SSH connection attempt fails during the authentication phase:

Invalid user indicates that the remote host attempted to log in with a non-existent or unauthorized username.

Connection reset by invalid user means the SSH client disconnected abruptly before completing authentication — typically during a brute-force scan or failed login.

Most of the time, these connection attempts are made by automated bots scanning the internet for exposed SSH ports using default usernames like adminorroot.

hashtag
Auditing Fail2ban Logs

fail2ban is a service that monitors logs for suspicious activity (e.g., failed SSH logins) and automatically blocks offending IPs via the firewall. This section covers how to review its actions via /var/log/fail2ban.log

View the most recent activity

Follow in real time

IP Banned Example:

hashtag
Auditing Logrotate Activity

logrotate is responsible for rotating, compressing, and deleting old log files to prevent disk usage from growing uncontrollably.

If logrotate is managed by systemd, you can view the most recent log entries with:

You can also check its dedicated log file at:

triangle-exclamation

If logrotate encounters an error, you'll see entries like this in the journal:

hashtag
Solana Audit Logs

Solana validators generate comprehensive audit logs that are crucial for monitoring validator performance, detecting anomalies, and maintaining operational security. These logs provide detailed insights into validator operations, consensus participation, and network interactions.

hashtag
Initial Startup Monitoring

After starting the validator service, immediately check the logs to verify proper initialization:

If the validator is running correctly, you should immediately see snapshot download progress indicating the validator is synchronizing with the network:

circle-exclamation

The tail command will continue to display the output of a file as the file changes. You should see a continuous stream of log output as your validator runs. Keep an eye out for any lines that say ERROR.

hashtag
Jito Relayer Logs

To verify correct operation of the validator when connected to the Jito relayer and block engine, monitor these specific metrics in the validator logfile:

These metrics are emitted once per second when properly connected:

block_engine_stage-stats: Indicates active connection to the block engine

relayer_stage-stats: Indicates active connection to the relayer

hashtag
Additional Verification of Correct Operation

After monitoring the server to identify that it's functioning correctly, in addition to the Jito documentationarrow-up-right that indicates when it's running well in Co-hosted relayer, we also capture several lines in the Solana logs that indicate it's functioning correctly:

hashtag
Monitoring Slot Leadership

Another log that we should consider to verify that the relayer is functioning correctly is the validator's behavior when it becomes a leader slot .

When your validator becomes the slot leader, you should see a message like this:

Once the slot is produced, you should see a message indicating that your validator is no longer the leader:

hashtag
Authentication Error Handling

It's also good to check the logs of the jito-relayer service:

If you encounter this error: error authenticating and connecting

You must contact the Jito team and provide them with the pubkey that you will use for the relayer.

PreviousSetup Validator MetricsNextBackup & Recovery

Last updated

Was this helpful?