Juju logs

There are various logging resources available to the Juju operator. This page will explain these and show how to use them. It will cover:

Model logs

Model logs can be considered as Juju's "regular logs" and are intended to be inspected with the debug-log command. This method provides logs on a per-model basis and is therefore more convenient than reading individual logs on multiple (Juju) machines directly on the file system. The latter can nonetheless be done in exceptional circumstances and some explanation is provided here.

See Controller HA and logging when viewing logs in an HA context.

Juju agents

One of the roles of the Juju agent, 'jujud', is to perform logging. There is one agent for every Juju machine and unit. For instance, for a machine with an id of '2', we see evidence of such agents:

juju ssh 2 ls -lh /var/lib/juju/agents

This example has the following output:

drwxr-xr-x 2 root root 4.0K Apr 28 00:42 machine-2
drwxr-xr-x 4 root root 4.0K Apr 28 00:42 unit-nfs2-0

So there are two agents running on this machine. One for the machine itself and one for a service unit.

The contents of one of these directories

juju ssh 2 ls -lh /var/lib/juju/agents/machine-2

reveals the agent's configuration file:

-rw------- 1 root root 2.1K Apr 28 00:42 agent.conf

Consider keeping backups of these files, especially prior to upgrading the agents. See Upgrading models.

The debug-log command

The debug-log command shows the consolidated logs of all Juju agents (machines and units) running in a model. The switch command is used to change models. Alternatively, a model can be chosen with the '-m' option. The default model is the current model.

The 'controller' model captures logs related to internal management (controller activity) such as adding machines and services to the database. Whereas a hosted model will provide logs concerning activities that take place post- provisioning.

Due to the above, when deploying a service, it is normal for there to be an absence of logging in the workload model since logging first takes place in the 'controller' model.

The output is a fixed number of existing log lines (number specified by possible options; the default is 10) and a stream of newly appended messages. Both existing lines and appended lines can be filtered by specifying options.

The exception to the streaming is when limiting the output (option '--limit'; see below) and that limit is attained. In all other cases the command will need to be interrupted with 'Ctrl-C' in order to regain the shell prompt.

For complete syntax, see the Command reference page.

Examples:

To begin with the last ten log messages:

juju debug-log

To begin with the last thirty log messages:

juju debug-log -n 30

To begin with all the log messages:

juju debug-log --replay

To begin with the last twenty log messages for the 'lxd-pilot' model:

juju debug-log -m lxd-pilot -n 20

To begin with the last 500 lines. The grep utility is used as a text filter:

juju debug-log -n 500 | grep amd64

Logging levels

You can verify the current logging level like this:

juju model-config logging-config

Output will be similar to the following:

<root>=WARNING;unit=DEBUG

The above is the default configuration. It sets the machine agent log level at 'WARNING' and the unit agent log level at 'DEBUG'.

The logging levels, from most verbose to least verbose, are as follows:

  • TRACE
  • DEBUG
  • INFO
  • WARNING
  • ERROR

Change the logging level

When diagnosing an issue (and possibly filing a bug), the first step is to make logging more verbose. For instance, to increase the logging level of the unit agents to 'TRACE' you would enter the following command:

juju model-config logging-config="<root>=WARNING;unit=TRACE"

To avoid filling up the database unnecessarily, when verbose logging is no longer needed, do not forget to return logging to normal levels.

The logging level can also be changed on a per-unit basis. See section Agent logging override below.

Advanced filtering

A Juju log line is written in this format:

<entity> <timestamp> <log-level> <module>:<line-no> <message>

The --include and --exclude options select and deselect, respectively, the entity that logged the message. An entity is a Juju machine or unit agent. The entity names are similar to the names shown by juju status.

Similarly, the --include-module and --exclude-module options can be used to influence the type of message displayed based on a (dotted) module name. The module name can be truncated.

A combination of machine and unit filtering uses a logical OR whereas a combination of module and machine/unit filtering uses a logical AND.

The --level option places a limit on logging verbosity (e.g. --level INFO will allow messages of levels 'INFO', 'WARNING', and 'ERROR' to be shown).

Examples:

To begin with the last 1000 lines and exclude messages from machine 3:

juju debug-log -n 1000 --exclude machine-3

To select all the messages emitted from a particular unit and a particular machine in the entire log:

juju debug-log --replay --include unit-mysql-0 --include machine-1

Note: The unit can also be written 'mysql/0' (as shown by juju status).

To see all WARNING and ERROR messages in the entire log:

juju debug-log --replay --level WARNING

To progressively exclude more content from the entire log:

juju debug-log --replay --exclude-module juju.state.apiserver
juju debug-log --replay --exclude-module juju.state
juju debug-log --replay --exclude-module juju

To begin with the last 2000 lines and include messages pertaining to both the juju.cmd and juju.worker modules:

juju debug-log --lines 2000 \
    --include-module juju.cmd \
    --include-module juju.worker

Agent logging override

As we've seen, the logging level for machine agents and unit agents are specified as a single model configuration setting. However, in some situations (e.g. targeted verbose debugging) it may be desirable to increase the logging level on a per-agent basis. This can also be done after having reduced the model-wide log level (as explained in section Change the logging level above).

For example, let's enable 'TRACE' logging level to a unit of MySQL. We begin by logging in to the unit's machine ('0' in this example):

juju ssh mysql/0

File /var/lib/juju/agents/unit-mysql-0/agent.conf is then edited by adding the line LOGGING_OVERRIDE: juju=trace to the 'values' section.

To be clear, the bottom of the file now looks like:

loggingconfig: <root>=WARNING;unit=DEBUG
values:
  CONTAINER_TYPE: ""
  NAMESPACE: ""
  LOGGING_OVERRIDE: juju=trace
mongoversion: "0.0"

The affected agent will need to be restarted to have this change take effect:

sudo systemctl restart jujud-unit-mysql-0.service

Log files

Log files are located on every machine Juju creates, including the controller. They reside under /var/log/juju and correspond to the machine and any units.

Using the example from a previous section:

juju ssh 2 ls -lh /var/log/juju

Output:

-rw------- 1 syslog syslog  22K Apr 28 00:42 machine-2.log
-rw------- 1 syslog syslog 345K Apr 28 16:58 unit-nfs2-0.log

There is one extra log file on each controller: logsink.log:

juju ssh -m controller 0 ls -lh /var/log/juju

Output:

-rw------- 1 syslog syslog 2.3M Apr 28 17:05 logsink.log
-rw------- 1 syslog syslog  85K Apr 28 17:03 machine-0.log

File logsink.log contains logs for all models managed by the controller. Its contents get sent to the database where it is consumed by the debug-log command.

Note: In a High availability scenario, logsink.log is not guaranteed to contain all messages since agents have a choice of several controllers to send their logs to. The debug-log command should be used for accessing consolidated data across all controllers.

Remote logging

On a per-model basis log messages can optionally be forwarded to a remote syslog server over a secure TLS connection.

See Rsyslog documentation for help with security-related files (certificates, keys) and the configuration of the remote syslog server.

Configuring

Remote logging is configured during the controller-creation step by supplying a YAML format configuration file:

juju bootstrap <cloud> --config logconfig.yaml

The contents of the YAML file is of the form:

syslog-host: <host>:<port>
syslog-ca-cert: |
-----BEGIN CERTIFICATE-----
 <cert-contents>
-----END CERTIFICATE-----
syslog-client-cert: |
-----BEGIN CERTIFICATE-----
 <cert-contents>
-----END CERTIFICATE-----
syslog-client-key: |
-----BEGIN PRIVATE KEY-----
 <cert-contents>
-----END PRIVATE KEY-----

Enabling

To actually enable remote logging for a model a configuration key needs to be set for that model:

juju model-config -m <model> logforward-enabled=True

An initial 100 (maximum) existing log lines will be forwarded.

See Configuring models for extra help on configuring a model.

Note that it is possible to configure and enable forwarding on all the controller's models in one step:

juju bootstrap <cloud> --config logforward-enabled=True --config logconfig.yaml

Audit logging

Juju audit logging provides a chronological account of all events by capturing invoked user commands. These logs reside on the controller involved in the transmission of commands affecting the Juju client, Juju machines, and the controller itself.

The audit log filename is /var/log/juju/audit.log and contains records which are either:

  • a Conversation, a collection of API methods associated with a single top-level CLI command
  • a Request , a single API method
  • a ResponseErrors, errors resulting from an API method

Information can be filtered out of the audit log to prevent its file(s) from growing without bounds and making it difficult to read. See Excluding information from the audit log.

The log is typically viewed by connecting to the controller over SSH and looking at the file:

juju ssh -m controller 0
more /var/log/juju/audit.log