How to Start Logging in Python

Python offers built-in support for logging, providing programmers with critical visibility into their apps without much hassle.

To understand the Python logging module, you need to learn about the corresponding logging APIs in the language and the steps to use them. It is just as vital to learn the best practices when using these APIs for hitch-free logging.

This guide introduces you to the many concepts relevant to logging in Python. While Python has offered logging support built-in since version 2.3, this article is based on Python 3.8. It assumes the reader knows about the concepts and constructs pertaining to general and object-oriented programming.

What is Python Logging?

The standard Python library has a built-in logging module, allowing users to log events from libraries and applications easily. After the logger is configured, it runs as a part of the Python interpreter whenever code runs. Put simply; it is global.

Python also allows users to configure the logging subsystem with an external config file. You can find the specifications of the logging config format in the Python library.

The built-in logging library follows a modular approach and has the following components:

Loggers: These expose the interface that the application uses
Handlers: These send the log records generated by the loggers to the right destination
Filters: They filter the records and decide which to output
Formatters: These define the layout of the final log entry.

The several logger objects are organized into a tree representing the system’s many parts and the various third-party libraries installed on it. When the user sends a message to one of these loggers, the message is output to all the handlers by their formatters.

The message moves up the logger tree until it reaches a logger configured with propagate=False or the root logger.

How Does Python Logging Work?

Here is a breakdown of all the tasks that trigger when a Python user uses a logging library:

A client runs a logging statement and issues a log request. These statements typically invoke a method in the logging library’s API, providing the logging level and log data as the arguments.

The logging level parameter defines the importance of the request. The log data is generally a log message string, but some other data is also logged. The logger objects typically expose the logging API.

The logging library then creates a log record representing the log request and captures the relevant data to enable the processing of the request as it threads through the library.

The log requests/records are filtered as the library indicates. The logging level request is compared to the threshold logging level during filtering. The records are then passed through the user-provided filters.

Finally, the handlers go through the filtered records, typically storing the data by writing it into a file. However, handlers can be set to email the log data to a specific address.

The handlers in some logging libraries may filter the log records a second time based on the logging level and the handler-specific filters by the user. When needed, a handler also relies on the formatters provided by the user to format the log data into log entries.

The Logging Modules in Python

The Python standard library has the following modules to offer support for logging:

The logging module: Has the primary client-facing API.
The logging.config module: It has the API that allows client configuration.
The logging.handlers module: It offers the various handlers that process and store log records in different ways.

These modules collectively are called the Python logging library, and they materialize the concepts discussed in the earlier sections as classes, module-level methods, or constants.

Logging Levels in Python

The logging library in Python supports five logging levels: critical, error, warning, info, and debug.

They are represented by constants having the same name, like so: logging.CRITICAL, logging.ERROR, logging.WARNING, logging.INFO, and logging.DEBUG. The constants have the values 50, 40, 30, 20, and 10, respectively.

During runtime, the value of the logging level indicates the meaning of the level. Clients can also use other logging levels by using values greater than zero and not equal to the pre-defined logging level values.

When logging levels’ names are available, they appear by their names in the log entries. Every pre-existing logging level has the same name as the corresponding constant. For instance, logging.WARNING and 30 levels appear as “WARNING” in the log entries.

On the other hand, the custom logging levels don't have any name by default. Therefore, Python prints a custom level with value n (as "Level n") in the entries.

However, when several custom logging levels are involved, the log entries can become confusing since the same default name is used across the entries.

To prevent this problem, clients can name their custom logging levels using the logging.addLevelName function by passing the level and levelName parameters.

In other words, running logging.addLevelName(43, ‘CUSTOM1’) will record level 43 as ‘CUSTOM1’ in the log entries.

The official documentation outlines community-wide applicability rules for the five logging levels built into the Python logging library. These guidelines indicate:

Use the debug level to log detailed information for diagnosing problems.
Use the info level to confirm that the app is working as expected.
Use the warning level to report unexpected behaviors that may indicate future problems but do not currently affect the proper functioning of the app.
Use the error level to report serious problems in the app’s functioning.
Use the critical level to report errors that indicate that the program may not continue running.

Functioning of Python Loggers

The logging.Logger objects (simply called "loggers") act as the interface between the user and the library. The objects facilitate issuing log requests and also offer methods to query and modify the state of the methods.

Typically, programmers use logging.getLogger(name) factory function to create loggers. Using the function, clients can access and manage loggers by name. No need to store and pass references to the loggers!

The argument passed in the function is generally a hierarchical name separated by dots (such as a.b.c). Passing the values this way allows the library to maintain the logger hierarchy.

When the logger is made, the library ensures that there is a logger at every hierarchy level. It also ensures that all loggers are linked to their parent and child loggers.

This is also where the concept of threshold logging level comes into the picture. Every logger has a threshold level to help determine whether the request needs to be processed.

If the value of a requested logging level is equal to or higher than the threshold level, the logger processes the log request.

Clients can use the Logger.getEffectiveLevel(), and the Logger.setLevel(level) functions to retrieve and change the threshold logging level of a logger.

The factory function sets a logger's threshold level at the same value as

the parent. Python Logging Methods

Every logger has the following methods to facilitate issuing log requests:

Logger.critical(msg, *args, **kwargs)
Logger.error(msg, *args, **kwargs)
Logger.debug(msg, *args, **kwargs)
Logger.info(msg, *args, **kwargs)
Logger.warn(msg, *args, **kwargs)

Users can use these methods to issue requests with pre-defined logging levels. Besides these methods, loggers offer two other methods:

Logger.log(level, msg, *args, **kwargs): It issues log requests with logging levels specified explicitly. It is helpful when using custom logging levels.
Logger.exception(msg, *args, **kwargs): It issues requests with the ERROR level and captures the current exception in the log entries. For this reason, clients should only use this method from an exception handler.

The “msg” and “args" parameters in these methods together create the log messages in the log entries.

Further, all the above methods support the “exc_info” argument, allowing clients to add exception information to the log entries. The “stack_info” and “stacklevel” arguments are also supported, allowing users to add call stack details to the log entries.

The keyword argument “extra” enables clients to pass the relevant filter, handler, and formatter values.

When these methods run, they perform all the tasks mentioned in the previous section and then perform the following tasks:

After the threshold and logging levels are compared, and the logger decides to process the log request, it creates a LogRecord object. The object represents the request in the downstream processing of the request. LogRecord objects are set up to capture the methods’ parameters, call stack details, and exceptions. It can also capture the values and keys in the extra argument as fields.
Every handler processes a log request, and the handlers of its antecedent loggers process the request as appropriate in the logger hierarchy. The Logger controls this aspect of the handlers.propagate field, which is True by default.

Filters are not simply a means to interact with logging levels. They allow fine-grained filtering of the requests and ignore the requests in specific classes. Users can use the Logger.addFilter(filter) and the Logger.removeFilter(filter) methods to add and remove filters from loggers, respectively.

Logging Filters in Python

Any callable that accepts a log as an argument and either returns a zero (rejecting) or a non-zero value (accepting) to admit the record is a filter. Objects having methods with the signature filter(record: LogRecord) -> int can also be a filter.

The subclass of the logging.Filter(name) method that can override the logging.Filter.filter(record) method can also become a filter.

However, this type of filter will set records generated by loggers with the same name as the child filters without overriding the filter method.

If the filter name is empty, the filter admits all records. On the other hand, when the method is overridden, it returns zero to reject the record or non-zero to admit the record.

Logging Handler in Python

The logging.Handler objects are responsible for processing the log records in Python. These components are responsible for logging the log requests. Final processing by the handler typically involves storing the record by writing it to the system logs or other files.

However, final processing can also involve sending it to other entities or emailing the record data to specific email addresses.

Handlers, like loggers, also have a threshold logging level. Clients can set it using the Handler.setLevel(level) method, support the filters using the Handler.addFilter(filter) and Hander.removeFilter(filter) methods.

Handlers use filters and threshold logging levels to filter records for processing. The additional filtering enables context control over the records, meaning the notifying handler will only process requests from a broken or critical module.

When a handler processes log records, the records are formatted into log entries by the formatters. Clients can use the Handler.setFormatter(formatter) method to set the formatter. Handlers without formatters can be used with the default formatter in the Python standard library.

The logging.handler module has over a dozen different handlers that can be used in several cases. Therefore, clients can quickly configure these handlers in most cases.

However, some situations warrant the use of custom handlers. In these situations, developers can extend the Handler class or their pick of the pre-defined Handler classes using the Handler.emit(record) method.

Logging Formatters in Python

The logging.Formatter objects are invaluable to the handlers since they format the log record into a string-based log entry. However, it’s important to remember that formatters have no control over the creation of log messages.

Formatters combine the data or fields in a log record with the format string specified by the user. But unlike handlers, the Python logging library only has one basic formatter to log the level, logger’s name, and the message.

Therefore, if clients need a formatter for applications beyond the simple use cases, they will need to build new logging.Formatter objects with the required format strings.

Formatters support the printf, str.format(), and str.template styles of format strings. Further, the format string can be any field of the LogRecord objects, including the fields based on the keys of the extra argument.

Before the log record is formatted, the formatter uses LogRecord.getMessage() to combine the msg and args arguments of the logging method using the % operator to construct the log message.

The formatter finally combines the log messages with the log record data using the specified format string to create the entry.

Logging Modules in Python

When a client sues the logging library, the library sets up a root logger to maintain the hierarchy of loggers. The default threshold logging level of the root logger of all the loggers is logging.WARNING.

The module supplies all the logging methods of the Logger class as module-level methods having identical names and signatures. Clients can use the methods to issue log requests without creating a logger, with the root logger ultimately servicing all the requests.

That said, if the root logger does not have any handlers when servicing the log requests in the methods, then the logging library adds a logging.StreamHandler instance using the sys.stderr stream as the root logger’s handler. This instance is called the last resort handler.

The log requests given to loggers without handlers are directed to the last resort handler by the logging library. Clients can access the handler with the logging.lastResort attribute.

Good Practices in Python Logging

We’ve highlighted the best practices for logging in Python for you below. However, it’s important to note that there are no silver bullets. Before using any of these practices, it is best to consider their applicability in your program and consider their appropriateness.

#1 Use the getlogger Function

The logging.getLogger() function enables the library to manage mapping logger names to logger instances and maintains the hierarchy of loggers. Mapping and hierarchy thereby offer the following benefits:

Clients can use the function to access the loggers from different parts of the program by retrieving the logger by name.
Finite loggers are created at runtime.
Log requests can propagate high in the logger hierarchy.
The threshold logging level can be inferred from previous loggers when unspecified.
The library’s configuration can be updated at runtime using the logger names.

#2 Use Pre-Defined Logging Levels

The pre-defined logging levels in the library capture nearly all logging scenarios. Furthermore, most developers are familiar with the pre-defined logging levels since logging libraries across programming languages have similar levels.

Using the pre-defined levels makes configuring, deployment, and maintenance easy. Unless necessary, it is best to use the pre-defined levels.

#3 Create Module-Level Loggers

Clients can create loggers for every class and every module. Creating loggers for every class allows for detailed configuration; however, it loads up the program with several loggers.

On the other hand, creating loggers for every module can keep the total number of loggers small. Therefore, using class-level loggers is ideal only when the fine-grained configuration is a must.

#4 Name Module-Level Loggers Same as the Modules

The names of loggers are string values and are not included in the Python namespace. Therefore, they do not clash with the module names.

Using the module’s name for the corresponding module-level loggers makes identifying them in the code a breeze. Naming the loggers this way uses the dot notation and makes referring to the loggers easy.

#5 Inject Local Context Information using the logging.LoggerAdapter method

Inserting contextual information into log records is as simple as using the logging.LoggerAdapter method. Clients can also use it to modify the message and the data in the log request.

The logging library does not manage these adapters, meaning they cannot be accessed using common names. Therefore, developers use them to insert contextual information into a class or module as required.

#6 Avoid Using Filters to Insert Global Contextual Information

Inserting global contextual information into log records is as simple as using filters to modify the record’s arguments provided to the filters.

You could write a filter like this to insert details into incoming log records:

def version_injecting_filter(logRecord):

    logRecord.version = '3'

    return True

However, there are two disadvantages to this approach. Firstly, if the filters take their data from the log records, the filters that inject the data into the records must run before the filters that use the injected data. The order in which the filters are added to the loggers becomes vital.

Secondly, the filters extend the log records, abusing the support to filter them.

There is another approach to injecting contextual information into a logger – you can initialize the logging library using the logging.setLogRecordFactory() function. Given that the injected information is global, clients can inject it into the log records when the factory function creates them.

Using the logging.setLogRecordFactory() function is an excellent way to ensure the data is available in every logging component in the program. However, this approach also comes with a downside:

Clients must ensure that the factory functions from the different parts of the program work well with each other. While clients can chain the log record factory functions, making the program needlessly complex.

#7 Put the Common Handlers to the Loggers High in the Hierarchy

If two loggers have the same handler, and one of the loggers is a descendant of the other, then it’s best to attach the handler to the ascendant logger. Clients can rely on the logging library to propagate the requests from the lower logger to the handlers of the upper logger.

Further, if the propagate attribute is not modified, putting the handlers in the upper loggers prevents duplicate messages from appearing.

#8 Use the logging.disable() Function to Throttle the Logging Output

If the logging level of a request is equal to or higher than the logger’s effective logging level, it will process the request. But the effective logging level is determined by whichever is higher between the threshold logging level and the library-wide logging level.

Clients can use the logging.disable(level) method to set the library-wide logging level. This level is 0 by default; therefore, it processes the requests of every logging level.

Using the function is an excellent way to increase the logging level across the program and throttle the logging output of the program.

#9 Cache References to Loggers Considerately

Caching references to loggers and accessing them using the cached references helps prevent calling the logging.getLogger() function repeatedly to get the same logger, increasing the program’s efficiency.

That said, if the retrievals are not redundant, these eliminations of calling the function can lead to lost log requests. Updating the references to the loggers can help the situation, but using the logging.getLogger() function is the best way to avoid this problem in the first place.