scripts.utils.logging module

Centralized Logging Module

Comprehensive logging system with custom SUCCESS level, dual output (file + console), and intelligent filtering. Features timestamped files and automatic log directory creation.

New in v0.0.12: - Enhanced verbose logging with detailed formatter - VerboseLogger class for tree-view, step-by-step debugging - Better error context and stack traces - Timing information for all operations

Key Features

Custom SUCCESS Level: Between INFO and WARNING for successful operations
Dual Output: Console (clean) and file (detailed) with independent filtering
Verbose Mode: Tree-view logging with context managers (DEBUG level)
Timestamped Logs: Automatic log file creation in .logs/ directory
UTF-8 Support: International character encoding
Progress Bar Integration: Works seamlessly with tqdm

Log Levels

DEBUG (10): Verbose mode only - detailed file processing, timing, metrics
INFO (20): Default - major steps and summaries
SUCCESS (25): Custom level - successful completions (console + file)
WARNING (30): Potential issues
ERROR (40): Failures
CRITICAL (50): Fatal errors

Console vs. File Output

Console: Only SUCCESS, ERROR, and CRITICAL (keeps terminal clean)
File: INFO or DEBUG (depending on –verbose flag) and above

Verbose Logging (VerboseLogger)

The VerboseLogger class provides tree-view formatted output for detailed debugging:

Usage Example:

>>> from scripts.utils import logging as log
>>> vlog = log.get_verbose_logger()
>>>
>>> with vlog.file_processing("data.xlsx", total_records=100):
...     vlog.metric("Total rows", 100)
...     with vlog.step("Loading data"):
...         vlog.detail("Reading sheet 1...")
...         vlog.timing("Load time", 0.45)

Output Format:

├─ Processing: data.xlsx (100 records) │ ├─ Total rows: 100 │ ├─ Loading data │ │ │ Reading sheet 1… │ │ │ ⏱ Load time: 0.45s │ └─ ✓ Complete

VerboseLogger Methods:

file_processing(filename, total_records): Context manager for file-level operations
step(step_name): Context manager for processing steps
detail(message): Log detailed information
metric(label, value): Log metrics/statistics
timing(operation, seconds): Log operation timing
items_list(label, items, max_show): Log lists with truncation

Integration

Used by all pipeline modules:

scripts/load_dictionary.py: Sheet and table processing
scripts/extract_data.py: File extraction and duplicate column removal
scripts/deidentify.py: De-identification and validation

See also

-, -

class scripts.utils.logging.CustomFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)[source]

Bases: Formatter

Custom log formatter that properly handles the SUCCESS log level.

format(record)[source]

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class scripts.utils.logging.VerboseLogger(logger_module)[source]

Bases: object

Centralized verbose logging for detailed output in DEBUG mode.

Provides formatted tree-view output for file processing, step execution, and operation timing. Only logs when logger is in DEBUG mode.

Usage:

vlog = VerboseLogger(log) with vlog.file_processing(“file.xlsx”, total_records=412):

with vlog.step(“Processing step”):
vlog.detail(“Details here”)

__init__(logger_module)[source]: Initialize with logger module.

detail(message)[source]

Log a detail message within a step.

Return type:: None

file_processing(filename, total_records=None)[source]: Context manager for processing a file.

items_list(label, items, max_show=5)[source]

Log a list of items with truncation if too long.

Return type:: None

metric(label, value)[source]

Log a metric/statistic.

Return type:: None

step(step_name)[source]: Context manager for a processing step.

timing(operation, seconds)[source]

Log operation timing.

Return type:: None

scripts.utils.logging.critical(msg, *args, include_log_path=True, **kwargs)[source]

Log a CRITICAL level message with optional log file path.

Return type:: None

scripts.utils.logging.debug(msg, *args, **kwargs)[source]

Log a DEBUG level message.

Return type:: None

scripts.utils.logging.error(msg, *args, include_log_path=True, **kwargs)[source]

Log an ERROR level message with optional log file path.

Return type:: None

scripts.utils.logging.get_log_file_path()[source]

Get the path to the current log file.

Return type:: Optional[str]

scripts.utils.logging.get_logger(name=None)[source]

Get the configured logger instance or set it up if not already done.

Parameters:: name (Optional[str]) – Logger name (optional). Accepted for API compatibility with standard Python logging patterns (e.g., logging.getLogger(__name__)), but currently ignored as this returns a singleton logger instance shared by all modules.
Return type:: Logger
Returns:: The global ‘reportalin’ logger instance

Note

This function implements a singleton pattern - all modules share the same logger instance. The name parameter is accepted for compatibility with standard Python logging but is currently ignored. A debug message is logged if a name is provided and the logger is already initialized.

Examples

Standard usage (singleton pattern):

from scripts.utils import get_logger
logger = get_logger()
logger.info("Processing data...")

Compatible with Python logging pattern:

logger = get_logger(__name__)  # Works but returns singleton logger
logger.info("Message logged")

Overview

The scripts.utils.logging module provides centralized logging infrastructure for RePORTaLiN with custom SUCCESS level, dual output (file + console), and intelligent filtering.

Key Features:

Custom SUCCESS log level (25) between INFO and WARNING
Dual output: console (filtered) + file (complete)
Thread-safe and optimized (no record mutation)
Explicit public API via __all__

Public API

The module exports the following public interface via __all__ (12 exports total):

Setup Functions (3):

setup_logger(log_name, log_level)  # Initialize logging system
get_logger()                        # Get the logger instance
get_log_file_path()                # Get path to current log file

Logging Functions (6):

debug(message, *args, **kwargs)    # Debug-level messages
info(message, *args, **kwargs)     # Informational messages
warning(message, *args, **kwargs)  # Warning messages
error(message, *args, **kwargs)    # Error messages
critical(message, *args, **kwargs) # Critical error messages
success(message, *args, **kwargs)  # Success messages (custom level)

Constants (1):

SUCCESS  # Custom log level constant (25)

Usage Example:

from scripts.utils.logging import setup_logger, info, success, error

# Setup (usually done in main.py)
setup_logger('myapp', logging.INFO)

# Use logging functions
info("Processing started")
success("Operation completed")
error("An error occurred")

Use these functions and constants when importing from the module.

Setup Functions

setup_logger

setup_logger(log_name: str, log_level: int) -> logging.Logger

Initialize the logging system with custom configuration.

Parameters:

log_name (str): Name for the logger (e.g., ‘reportalin’)
log_level (int): Minimum logging level (e.g., logging.INFO, logging.DEBUG)

Returns:

logging.Logger: Configured logger instance

Example:

import logging
from scripts.utils.logging import setup_logger

# Setup with INFO level
logger = setup_logger('myapp', logging.INFO)

get_logger

get_logger() -> Optional[logging.Logger]

Get the current logger instance.

Returns:

logging.Logger: The logger instance, or None if not initialized

Example:

from scripts.utils.logging import get_logger

logger = get_logger()
if logger:
    logger.info("Direct logger access")

get_log_file_path

get_log_file_path() -> Optional[str]

Get the path to the current log file.

Returns:

str: Path to log file, or None if logging not initialized

Example:

from scripts.utils.logging import get_log_file_path

log_path = get_log_file_path()
if log_path:
    print(f"Logs written to: {log_path}")

Custom Log Levels

SUCCESS Level

The logging system adds a custom SUCCESS level (value: 25) between INFO and WARNING:

# Standard levels
logging.INFO = 20
logging.WARNING = 30

# Custom level added by scripts.utils.logging
scripts.utils.logging.SUCCESS = 25

This level is used to highlight successful completion of major operations.

Example:

from scripts.utils import logging as log

log.info("Starting processing...")
log.success("Processing completed successfully!")
log.warning("Warning: some records skipped")
log.error("Error occurred")

Logging Functions

The module provides standard logging functions:

info

log.info(message, *args, **kwargs)

Log an informational message.

Example:

log.info("Processing 43 files...")
log.info(f"Found {count} records in {file}")

success

log.success(message, *args, **kwargs)

Log a success message (custom level).

Example:

log.success("Step 1: Data extraction completed successfully")
log.success(f"Processed {total} records from {file_count} files")

warning

log.warning(message, *args, **kwargs)

Log a warning message.

Example:

log.warning("Empty dataframe detected, skipping file")
log.warning(f"Duplicate column name: {col_name}")

error

log.error(message, *args, **kwargs)

Log an error message.

Example:

log.error("Failed to read Excel file", exc_info=True)
log.error(f"File not found: {file_path}")

debug

log.debug(message, *args, **kwargs)

Log a debug message (only when LOG_LEVEL = DEBUG).

Example:

log.debug(f"Processing row {row_num}")
log.debug(f"Column types: {df.dtypes}")

Logging Configuration

Setup

The logging system is configured in config.py:

# config.py
import logging

LOG_LEVEL = logging.INFO
LOG_NAME = "reportalin"

Log File Format

Log files are created in .logs/ with timestamps:

.logs/
└── reportalin_YYYYMMDD_HHMMSS.log

Example log file: reportalin_20251002_143052.log

Log Message Format

Messages are formatted as:

YYYY-MM-DD HH:MM:SS - LEVEL - Message

Example:

2025-10-02 14:30:52 - INFO - Starting data extraction
2025-10-02 14:30:52 - SUCCESS - Step 0 completed successfully
2025-10-02 14:31:10 - WARNING - Empty dataframe in file 99B_FSB.xlsx
2025-10-02 14:31:15 - ERROR - Failed to process file: permission denied

Output Handlers

The logging system uses two handlers:

Console Handler

Output: stdout
Format: Simple format with level and message (LEVEL: message)
Filter: Shows only SUCCESS, ERROR, and CRITICAL messages (suppresses DEBUG, INFO, WARNING)
Performance: Optimized - no record mutation, direct formatting

Technical Details:

CustomFormatter does not mutate log records (thread-safe)
No unnecessary copying of records (performance optimized)

File Handler

Output: Timestamped file in .logs/
Format: Full format with timestamp, name, level, and message
Encoding: UTF-8 for international characters
Level: All messages (DEBUG and above)

Usage Examples

Basic Logging

from scripts.utils import logging as log

def process_data():
    log.info("Starting data processing")

    try:
        # Process data
        result = do_processing()
        log.success("Processing completed successfully")
        return result

    except Exception as e:
        log.error(f"Processing failed: {e}", exc_info=True)
        raise

Structured Logging

from scripts.utils import logging as log

log.info(f"Processing {file_count} files")

for file in files:
    log.debug(f"Processing file: {file}")

    try:
        result = process_file(file)
        log.info(f"Processed {result['records']} records from {file}")

    except Exception as e:
        log.error(f"Failed to process {file}: {e}")

Conditional Logging

from scripts.utils import logging as log
import logging

# Only log if DEBUG level
if log.getEffectiveLevel() <= logging.DEBUG:
    log.debug(f"Detailed state: {complex_object}")

Context Logging

from scripts.utils import logging as log

def process_with_context(file):
    log.info(f"--- Processing {file} ---")

    try:
        result = do_processing(file)
        log.success(f"Completed {file}: {result['records']} records")

    except Exception as e:
        log.error(f"Error in {file}: {e}", exc_info=True)

    finally:
        log.info(f"--- Finished {file} ---")

Logging Best Practices

Use Appropriate Levels:

log.debug("Detailed diagnostic info")      # Development
log.info("General progress updates")       # Normal operation
log.success("Major milestones")            # Success confirmation
log.warning("Potential issues")            # Non-critical problems
log.error("Errors requiring attention")    # Critical problems

Include Context:

# Good: Includes context
log.error(f"Failed to read {file}: {error}")

# Bad: Missing context
log.error("Failed to read file")

Use Exceptions:

# Include exception details
try:
    process_file(file)
except Exception as e:
    log.error(f"Error processing {file}", exc_info=True)

Don’t Over-Log:

# Good: Log summary
log.info(f"Processed {count} records")

# Bad: Log every record
for record in records:
    log.info(f"Processing record {record['id']}")  # Too verbose

scripts.utils.logging module

Centralized Logging Module

Key Features

Log Levels

Console vs. File Output

Verbose Logging (VerboseLogger)

Integration

Overview

Public API

Setup Functions

setup_logger

get_logger

get_log_file_path

Custom Log Levels

SUCCESS Level

Logging Functions

info

success

warning

error

debug

Logging Configuration

Setup

Log File Format

Log Message Format

Output Handlers

Console Handler

File Handler

Usage Examples

Basic Logging

Structured Logging

Conditional Logging

Context Logging

Logging Best Practices

See Also