Stdout logging

[OliverCole]

Right now the logs specific to my backup get compressed and stored alongside it. Is there any way I can get those logs output to stdout also (or instead)?

I really want to be able to ingest them into other logging tools like Cloudwatch/ELK/Grafana etc.

[kaovilai]

Good news! Velero already implements multi-output logging for backup operations. After investigating the codebase, I found that backup logs are written to both stdout and a compressed file simultaneously.

Current Implementation

Velero uses a DualModeLogger that writes logs to multiple destinations:

1. DualModeLogger Creation

The logger is created with io.MultiWriter to write to both stdout and a gzip file:

velero/pkg/util/logging/dual_mode_logger.go

Lines 47 to 68 in bd3aa00

	// NewTempFileLogger creates a DualModeLogger instance that writes logs to both Stdout and a file in the temp folder.
	func NewTempFileLogger(logLevel logrus.Level, logFormat Format, hook *LogHook, fields logrus.Fields) (DualModeLogger, error) {
	file, err := os.CreateTemp("", "")
	if err != nil {
	return nil, errors.Wrap(err, "error creating temp file")
	}
	w := gzip.NewWriter(file)
	logger := DefaultLogger(logLevel, logFormat)
	logger.Out = io.MultiWriter(os.Stdout, w)
	if hook != nil {
	logger.Hooks.Add(hook)
	}
	return &tempFileLogger{
	FieldLogger: logger.WithFields(fields),
	logger: logger,
	file: file,
	w: w,
	}, nil

// NewTempFileLogger creates a DualModeLogger instance that writes logs to both Stdout and a file
func NewTempFileLogger(logLevel logrus.Level, logFormat Format, hook *LogHook, fields logrus.Fields) (DualModeLogger, error) {
    // ...
    logger := DefaultLogger(logLevel, logFormat)
    logger.Out = io.MultiWriter(os.Stdout, w)  // w is a gzip writer
    // ...
}

2. Backup Controller Usage

The backup controller creates this dual-mode logger for each backup operation:

velero/pkg/controller/backup_controller.go

Lines 676 to 684 in bd3aa00

	b.logger.WithField(constant.ControllerBackup, kubeutil.NamespaceAndName(backup)).Info("Setting up backup log")
	// Log the backup to both a backup log file and to stdout. This will help see what happened if the upload of the
	// backup log failed for whatever reason.
	logCounter := logging.NewLogHook()
	backupLog, err := logging.NewTempFileLogger(b.backupLogLevel, b.formatFlag, logCounter, logrus.Fields{constant.ControllerBackup: kubeutil.NamespaceAndName(backup)})
	if err != nil {
	return errors.Wrap(err, "error creating dual mode logger for backup")
	}

// Log the backup to both a backup log file and to stdout
backupLog, err := logging.NewTempFileLogger(b.backupLogLevel, b.formatFlag, logCounter, logrus.Fields{constant.ControllerBackup: kubeutil.NamespaceAndName(backup)})

3. Log Persistence

After the backup completes, the compressed log file is uploaded to object storage:

velero/pkg/persistence/object_store.go

Lines 249 to 254 in bd3aa00

	func (s *objectBackupStore) PutBackup(info BackupInfo) error {
	if err := seekAndPutObject(s.objectStore, s.bucket, s.layout.getBackupLogKey(info.Name), info.Log); err != nil {
	// Uploading the log file is best-effort; if it fails, we log the error but it doesn't impact the
	// backup's status.
	s.logger.WithError(err).WithField("backup", info.Name).Error("Error uploading log file")
	}

Accessing Backup Logs in Stdout

The backup logs are already being sent to stdout during backup execution. You can access them through:

# View Velero pod logs
kubectl logs -n velero deployment/velero

# Follow logs in real-time during a backup
kubectl logs -n velero deployment/velero -f

These logs can be ingested by your logging infrastructure (CloudWatch, ELK, Grafana) using standard Kubernetes log collection methods.

Configuration Options

You can configure logging behavior using Velero server flags:

• --log-level: Set verbosity (debug, info, warning, error)
• --log-format: Choose format (text or json) - JSON is recommended for log aggregation tools

Example:

velero server --log-level=debug --log-format=json

What You'll See in Stdout

During a backup, you'll see entries like:

• "Setting up backup log"
• Resource discovery and backup progress
• Any warnings or errors
• "Initial backup processing complete"

If you're not seeing backup logs in stdout, please check:

1. Your Velero server log level configuration
2. Whether your log collection is properly configured for the velero namespace
3. The timing - backup logs only appear during active backup operations

The multi-output logging is built into Velero's core backup process, so no additional configuration is needed to get logs to stdout. Let me know if you need help troubleshooting why the logs might not be appearing in your logging system!

[kaovilai]

ie. helm chart, some other operators, argocd, etc.

[pryantcc]

Thanks @kaovilai , I figured it out!
In case anyone else finds this, I found out that I had to scale the deployment to 0 in order for the changes to remain in place
oc scale deployment --replicas=0 openshift-adp-controller-manager
It's important to scale it back to 1 after debugging is finished.

[kaovilai]

OADP does have flag in DPA for log level so you didn't have to go through this trouble. But what you did will also work at least temporarily. Editing replica in OADP ClusterServiceVersion will be more permanent for more extended debugging.

[pryantcc]

@kaovilai Can you provide details please on how to configure it in the DPA? I don't see any logging section in mine.
Thanks.

[kaovilai]

https://docs.okd.io/latest/backup_and_restore/application_backup_and_restore/oadp-api.html#:~:text=the%20Velero%20pod.-,logLevel,-string