Some Background

I’m currently building a small IT infra at home. For logging and monitoring, I decided to use grafana.

It was quite easy to setup. I’ll add another post for that later.

The Problem

This morning, when I checked the log, I got a lot of failing log from grafana-agent.service that could not open two files for permission denied. They were pihole.log and boot.log

Analyze the Problem

Check the Permissions of the files

ls -l shows the overview while stat does the detail.

$ ls -l /var/log/boot.log
-rw------- 1 root root 0 Nov 12 00:00 /var/log/boot.log
$ stat $ stat /var/log/boot.log
  File: /var/log/boot.log
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 179,2   Inode: 1861        Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-10-10 06:06:14.107999998 +0200
Modify: 2023-11-12 00:00:07.077836918 +0100
Change: 2023-11-12 00:00:07.077836918 +0100
 Birth: 2023-10-10 06:06:14.107999998 +0200

boot.log is very stricted. The only the owner user can read and write. Also, it’s only root.

$ ls -al /var/log/pihole/pihole.log
-rw-r----- 1 pihole pihole 4808152 Dec 13 06:30 /var/log/pihole/pihole.log
$ stat /var/log/pihole/pihole.log
  File: /var/log/pihole/pihole.log
  Size: 4812236         Blocks: 9408       IO Block: 4096   regular file
Device: 179,2   Inode: 2101935     Links: 1
Access: (0640/-rw-r-----)  Uid: (  999/  pihole)   Gid: ( 1001/  pihole)
Access: 2023-12-04 11:39:58.836577774 +0100
Modify: 2023-12-13 06:31:03.854792117 +0100
Change: 2023-12-13 06:31:03.854792117 +0100
 Birth: 2023-12-04 11:39:58.836577774 +0100

pihole.log is less strict. It allows the owner group to read the file. But the group is pihole, the newly created group when I installed pihole.

Simple Workaround (which I don’t prefer)

If I check the grafana-agent.service which is located somewhere in /etc/systemd/, it runs grafana-agent application with grafana-agent user. The simplest way to solve the problem might be running the application with root user by chaning User= value under the [Service] section.

9c9
< User=grafana-agent
---
> User=root

But it has a little problem. grafana-agent will get the full access for all my system. I don’t think it is a good practice.

A Mutual Owner Group: the Proper Solution

When I check which groups that the user grafana-agent joined, I found an interesting one.

$ groups grafana-agent
grafana-agent : grafana-agent adm systemd-journal

adm. What is it? According to the Debian Wiki article, it is a group for system monitoring tasks. It sounds appropriate for me. Let’s use this group as a mutual group for the unreadable files!

$ groups root
root : root
$ groups pihole
pihole : pihole

Unfortunately, root and pihole do not join adm just yet. Let them do it.

$ sudo usermod -G adm root
$ sudo usermod -G adm pihole

Please do name mess it with usermod -g which changes the user group instead of adding an additional one.

Also, note that I’m using sudo here. usermod changes /etc/passwd file which needs the root previledge to write.

Now, if you check their groups, you can see the right one.

$ groups root
root : root adm
$ groups pihole
pihole : pihole adm

Finally, it’s time for let the adm group to read the files.

$ sudo chgrp -v adm /var/log/boot.log
changed group of '/var/log/boot.log' from root to adm
$ sudo chgrp -v adm /var/log/pihole/pihole.log
changed group of '/var/log/pihole/pihole.log' from pihole to adm

You may ignore -v because it is for the verbosity of the chgrp command.

We have only one step forward the full solution: let the group to read the file boot.log

$ sudo chmod g+r /var/log/boot.log

Tada! 🎉

Additional Notes

Usually, Linux system keeps separate log files in time. I mean there are some more /var/log/boot?.log files. I think it’d better to change them, too although they will be removed gradually. It’s the same to pihole related files, too.

$ sudo chgrp adm /var/log/boot?.log
$ sudo chmod g+r /var/log/boot?.log
$ sudo chgrp adm /var/log/pihole*
$ sudo chgrp adm /var/log/pihole/*