GlusterFS Troubleshooting

From GlusterDocumentation

1 Introduction
2 Basic Troubleshooting
3 Advanced Troubleshooting
- 3.1 The debug/trace Translator
- 3.2 GNU Debugger

Introduction

GlusterFS provides error information to administrators through its log file. As GlusterFS is a file system, you need to run the command as root user. If the log information is not sufficient, you can debug GlusterFS through the debugging tools on your system.

[edit]

Basic Troubleshooting

[edit]

First Steps

Check the system process table if the file system is still running. You can check this with the ps command or the top command.

Note the following (which provide useful information for reporting):

CPU utilization (if it is using any CPU or 100% CPU)
Memory utilization
Process state (for example, if it has entered D state)

[edit]

Log File Inspection

The next place to look for troubleshooting information is the log file. Both the server and client logs can provide useful information about what has gone wrong or unexpectedly. The log files are located in /usr/local/var/log/glusterfs/ (or possibly other paths depending on the distro packaging). Search for [W] (Warning) and [E] (Error) tags for indications about what might have gone wrong.

[edit]

Debug Mode

If the initial look at the log files does not provide sufficient details, then restart the file system in debugging mode (passing --debug in the command line arguments). This makes the file system log in a more verbose manner and some messages might make sense only to the developers.

[edit]

Advanced Troubleshooting

[edit]

The debug/trace Translator

The 'debug/trace' translator can be loaded in the vol file to get a trace of the system calls and their parameters passing through the translator graph. The information is written to the GlusterFS log file. For example:

   volume brick
     type storage/posix
     option directory /home/export
   end-volume

   ### Trace APIs on storage/posix translator.
   volume trace
     type debug/trace
     subvolumes brick
   end-volume

You can add options to the trace translator to log only specific system calls or exclude a few.

[edit]

GNU Debugger

GNU Debugger (GDB) helps you pin point the problem at the source level. GDB is the last resort in the troubleshooting process. To let GDB properly debug your binary, compile with -g (debug info) and -O0 (disable optimization).

bash# CFLAGS="-O0 -g" ./configure --sysconfdir=/etc 
bash# make install CFLAGS="-g -O0"

If there is a crash, GlusterFS produces a core file under the top level root directory (/core). This file stores the state of the server at the time of failure. Using GDB, you can analyze what went wrong.