5.5. Logging

During a user session, suSSHi creates three different kinds of log files.

5.5.1. System Log

The system log is a central log in which all system messages of all suSSHid daemons are recorded. Messages vary from received connections and user login attempts to basic system messages like daemon startup. As specified in the configuration SystemLogFile, system messages are written to the local file system.

Path variables

The path specified under SystemLogFile may contain the following tokens that are expanded at runtime once the user has been authenticated at the gateway:

Token

Description

%d

Day of month (01 … 31

%m

Month (01 … 12)

%y

Year (2019 …)

Note

The default path within the containers is /var/log/susshi/system/%y/%m/%d/system.log.

All system log messages are forwarded to Syslog as well. The Syslog facility used for this messages can be set by the configuration file option SystemLogFacility. The default facility is AUTH.

Please refer to the Partitions section for further details.

5.5.2. Session Logs

The session logs are individual logs per suSSHi user session and contain some details about what SSH protocol features the user is making use of, what channels have been opened and closed and whether the requested features have been either accepted or denied by suSSHi, etc. The system messages are written to the local file system as specified in the configuration SessionLogFile.

Path variables

The path may contain the following tokens that are expanded at runtime once the user has been authenticated at the gateway:

Token

Description

%d

Day of month (01 … 31

%m

Month (01 … 12)

%y

Year (2019 …)

%t

Target ip address

%u

Gateway user ID

%s

suSSHid session identifier

Note

The default path within the containers is /var/log/susshi/sessions/%y/%m/%d/%t/%u-%s.log.

Each entry in the session log is prefixed with the time of the event, the gateway user, client IP address, target user, target hostname and target IP address in the following form:

<time> susshid[<pid>]: <gateway_user>@<client_ip> ==> <target_user>@<target_host> (<target_ip>):

With this prefix it is very easy to grep through a bunch of log files and find the session you are looking for.

All session log messages are forwarded to Syslog as well. The Syslog facility used for this messages can be set by the configuration file option SessionLogFacility. The default facility is AUTH.

Please refer to the Partitions section for further details.

5.5.3. Audit Logs

The audit logs are the most advanced logs in suSSHi. They provide a detailed insight of what a user has done during a session, including full logging of interactive and command exec sessions in both directions:

  • the target server’s output, which contains all the data a server sends to the client, including the characters the user has typed unless echo has not been turned off on server side

  • the client user input, which contains any single keystroke typed by the user.

Furthermore, file transfers using SCP or SFTP can be logged so that it is clear which files have been touched and in which way during the file transfer session. Other logging like libpcap dumps of the forwarded IP traffic over a port forwarding channel or X11 session logs are available as well. Not all audit log files are ASCII readable but need some additional software to be read.

To understand audit logging, it is important to know that in a single SSH session, multiple channels can be requested by either the client or the target, and each channel can be of a different type. For example, the user can start an interactive login to a server and receive a shell I/O channel, but next request X11 or port forwarding or SFTP file transfer within the same SSH session, but on different channels.

To obtain a complete and detailed log of an SSH session, suSSHi examines each of these channels and logs the requests and traffic depending on what function the channel is used for. An interactive session results in some sort of ASCII logging of server output and user input, an SFTP protocol leads to a detailed logging of what the user has done within SFTP, etc.

Path variables

Audit messages are written to the local file system as specified in the configuration AuditLogFile. The path may contain the following tokens that are expanded at runtime once the user has been authenticated at the gateway:

Token

Description

%d

Day of month (01 … 31

%m

Month (01 … 12)

%y

Year (2019 …)

%t

Target ip address

%u

Gateway user ID

%s

suSSHid session identifier

%f

File type

Note

The default path within the containers is /var/log/susshi/audit/%y/%m/%d/%t/%u-%s.%f.

The file type (%f token) is expanded as shown in the following table. The %f token must be specified otherwise reading the configuration file will fail.

Type

Data Source

Format

Description

agent

Client & Target

ASCII, line by line

Auth Agent logging

client

Client

ASCII

Client input logging

log

Any

ASCII, line by line

Standard logfile

pcap

Client & Target

Binary (PCAP)

Packet capture file used by many utilities like tcpdump, wireshark etc.

scp

Client & Target

ASCII, line by line

SCP Logging

session

Target

ASCII

Server output logging

sftp

Client & Target

ASCII, line by line

SFTP Logging

time

Client & Target

ASCII

Timing file for server output and client input logging

x11

Client & Target

Binary (PCAP)

X11 data logging in PCAP format

Please refer to the Partitions section for further details.

5.5.3.1. Audit Logging in Detail

With suSSHi it is possible to log all kinds of different SSH protocol features in the file system according to the protocol. The different protocol characteristics require suitable log file formats to write the output files for later analysis. The audit log files are (usually) not typical log files with line-by-line events, but special files.

5.5.3.1.1. Interactive Session Logging

Server Output

If a user logs on to a target server and no optional command is specified by the user, a default interactive shell runs on the remote server. If the client specified a command but requested a PTY (for example, with the -t option in the OpenSSH ssh client), audit logging also becomes this type of session logging.

The interactive logging of the session server output generates two files:

  • The first file contains all output sent by the server without filtering or losing a single character.

  • The second file contains time information about the server output.

This allows the output of the server session to be reproduced 100% in the correct order and at the correct time since the data originally arrived.

Client Input

In the server output log (see above), only the output generated by the server is logged. So if the terminal is set to “no output” (e.g. when entering a password or forced by the user by sending the corresponding terminal control sequences), no output is included in the server output logs. To obtain all data entered by the user, client input logging can be enabled by setting the appropriate logging mask.

In the interactive session client input log, suSSHi also generates two different files:

  • One with the input entered by the user

  • One with the timing information as described above.

If server output logging is also enabled, the time information together with the server output timing go to the same time file.

5.5.3.1.2. Remote Command Execution

Whenever a user instructs his SSH client to execute a remote command at login (e.g. ssh gwuser@root@server@susshi shutdown -h now), suSSHi inspects the session as an exec session instead of an interactive session. Since execution sessions are very often used for remote file copying, and this could naturally fill a journaling file very quickly (think of an user copying an entire DVD by using a tar or cpio pipe over an SSH session), it is important to have a different journaling strategy for the data transferred over these sessions. Only when browsing the command to be executed on the target, it is simply impossible for suSSHi to guess whether the resulting data should be logged or simply ignored to save disk space.

The remote command logging algorithm looks like this:

  • When suSSHi receives a PTY request, the logging is converted to standard interactive session logging, since we now have some kind of interactive I/O that requires a PTY. The user can also request a PTY when executing remote commands (e.g. with the -t option in OpenSSH’s ssh client). This is sometimes done to run a preferred user shell instead of the default user shell in the user profile (e.g. ssh -t gwuser@root@server@susshi bash).

  • If the remote command is a form of SCP call, the logging is converted to SCP logging (see below).

  • If the remote command is retrieved from the user-definable stop patterns (see the Execution Stop Pattern in Partitions configuration), no audit log of the session is performed. It can be very helpful for the administrator to observe the session logs from time to time to look for patterns that need to be added to the configuration to save disk space by logging large file transfers.

  • In all other cases, suSSHi always logs the number of bytes configured with the exec log max-size (see Partitions) and then stops logging to save disk space. If the value is not set, 200MB is the default.

5.5.3.1.3. SFTP

All SCP traffic is analyzed in its function and logged in a human-readable log file. The SCP protocol is very restrictive in the supported functions: a file can be copied or a directory (which is always a recursive copy). There are no functions like file deletion, etc. that you can find in SFTP. To enable SCP audit logging, the logging mask must be set accordingly.

5.5.3.1.4. Port Forwarding

Any port forwarding, whether local or remote, can only transfer TCP traffic. If the logging mask is set accordingly, all traffic on the forwarded port is logged into a libpcap-compatible packet capture which can be read with any software such as tcpdump or Wireshark that can read libpcap-formatted files.

Since the port forwarding within the SSH protocol is ultimately only a bidirectional data stream (socket IO), the actual IP addresses of the endpoints communicating over these sockets cannot be determined. Therefore, always check the session log and be careful when using IP addresses and ports in forensics without performing a thorough traffic analysis.

For output in PCAP format, the IP addresses are therefore always set to 1.1.1.1 for the client side and 2.2.2.2 for the target side. By the way, this also applies if IPv6 is actually used to access the socket.

5.5.3.1.5. X11

X11 logging is essentially the same as standard port forwarding logging. Traffic is logged in a libpcap-compatible file with the X11 file type set. The same applies for IP addresses as for standard port forwarding. The port is always set to 6000, regardless of which display the X server is running on.

The X11 libpcap files can be read by any software that can read libpcap files for further analysis.