|
|
|
|
|
|
2. QUESTIONS RELATING TO SAR/SADC/SADF
2.1. The sar command complains with the following message: "Invalid system activity file: ...".3. QUESTIONS RELATING TO IOSTAT
3.1. I can't see all my disks when I use the iostat command...4. QUESTIONS RELATING TO PIDSTAT
4.1. pidstat -d doesn't report task I/O statistics...make: msgfmt: Command not found make: ***[locales] Error 127The
msgfmt command belongs to the GNU gettext package.
If you don't have it on your system, just configure
sysstat with NLS disabled like this:$ ./configure --disable-nlsor answer 'y' (for "yes") to the question
Disable National Language Support (NLS)? (y/n) [--disable-nls]if you use the Interactive Configuration script (
iconfig), then compile sysstat as usual (make ; make install).
README-nls file included in sysstat
source package to learn some more about National Language Support.
Enable National Language Support (NLS)? [y]during config stage (
make config).
In file included from /usr/include/bits/errno.h:25,
from /usr/include/errno.h:36,
from common.c:26:
/usr/include/linux/errno.h:4: asm/errno.h: No such file or directory
common.c: In function `get_kb_shift':
common.c:180: `PAGE_SIZE' undeclared (first use in this function)
common.c:178: warning: `size' might be used uninitialized in this function
make: *** [common.o] Error 1
Make sure that you have the Linux kernel sources
installed in /usr/src/linux. Also make sure that the
symbolic link exists in the /usr/src/linux/include
directory and points to the right architecture, e.g.:
# ll /usr/src/linux/include/asm lrwxrwxrwx 1 root root 8 May 5 18:31 /usr/src/linux/include/asm -> asm-i386In fact, only the Linux kernel headers should be necessary to be able to compile sysstat.
LANG variable is
perhaps the most widely used. See the following example:
$ export LANG=en_US $ sar Linux 2.4.9 (brooks.seringas.fr) 07/20/04 04:32:11 PM LINUX RESTART 05:00:00 PM CPU %user %nice %system %iowait %idle 05:10:00 PM all 0.24 0.00 89.64 0.00 10.12 Average: all 0.24 0.00 89.64 0.00 10.12 $ export LANG=fr_FR $ sar Linux 2.4.9 (brooks.seringas.fr) 20.07.2004 16:32:11 LINUX RESTART 17:00:00 CPU %user %nice %system %iowait %idle 17:10:00 all 0,24 0,00 89,64 0,00 10,12 Moyenne: all 0,24 0,00 89,64 0,00 10,12As you can notice, the time format but also the date, the decimal point, and even some words (like "Average") have changed according to the specified locale.
kB_read/s, wkB/s, or even sometimes with a lower
'b': kbmemfree, kbavail...), and the manual pages speak
of kilobytes, megabytes, etc., we always actually refer
to kibibytes (kiB), mebibytes (MiB),...2. QUESTIONS RELATING TO SAR/SADC/SADF
2.1. The sar command complains with the following message:Invalid system activity file: ...You are trying to use a file which is not a system activity file, or whose format is no longer compatible with that of files created by current version of
sar./var/log/sa directory then the
solution is easy: Just log in as root and remove by hand
all the files located in the /var/log/sa directory:# rm /var/log/sa/sa??If you are using sysstat 11.1.1 and later, you can also use the
sadf command to convert an old system activity
binary datafile (version 9.1.6 and later) to current
up-to-date format. Use the following syntax:
$ sadf -c old_datafile > new_datafileNote: Starting with sysstat version 8.1.1 and later, it is possible to know which version of
sar or sadc has
been used to create a data file. Just enter the
following command:
$ sadf -H /your/datafile | grep sysstat File created using sar/sadc from sysstat version 8.1.7
Cannot append data to that file [...]The internal structure of the data file does not allow
sar to append data to it. The data file may come from
another machine, or the components of the current box,
such as the number of processors, may have changed. Use
another data file, or delete the current daily data file, and try again.Invalid data formator:
Inconsistent input dataThis error message means that
sadc (the system activity
data collector that sar is using) is not consistent with
the sar command. In most cases this is because the sar
and sadc commands do not belong to the same release of
the sysstat package. Remember that sar may search for
sadc in predefined directories (/usr/local/lib/sa,
/usr/lib/sa, ...) before looking in the current
directory! With sysstat version 11.1.5 and later, enter
sar --sadc to determine which data collector is
called by sar.
Cannot open /var/log/sa/sa30: No such file or directoryPlease read the
sar(1) manual page! Daily data files are
created by default in the /var/log/sa directory using
the data collector (sadc) or using option -o
with sar.
Once they are created, sar can display statistics saved in those files.sar can also display statistics collected "on the
fly": Just enter the proper options on the command line
to indicate which statistics are to be displayed, and
also specify interval and count numbers.sar command is not compatible with formats
from other Unixes, because it contains data which are
closely linked to Linux. For the same reason, sysstat
cannot work on platforms other than Linux...
11:00:00 AM CPU %user %nice %system %idle 11:10:00 AM all 0.54 0.00 0.89 98.57 11:20:01 AM all 3.02 8.05 22.85 66.08 11:30:01 AM all 8.15 0.00 2.31 89.54 11:40:01 AM all 8.03 0.00 2.42 89.55 11:50:01 AM all 16.04 0.00 2.81 81.16 12:00:00 PM all 21.11 0.00 3.23 75.66 03:40:01 PM all 100.01 100.01 100.01 0.00 04:40:00 PM all 100.00 0.00 100.00 0.00 04:50:00 PM all 5.87 0.00 1.26 92.87 05:00:00 PM all 4.70 0.00 1.48 93.82 05:10:00 PM all 4.93 0.00 1.29 93.78 Average: all 100.22 100.20 100.13 0.00Your
sar command was not installed properly. Whenever
your computer is restarted (as it is the case here
between 12:00:00 PM and 03:40:01 PM), the 'sysstat'
shell script must be called by the system, so that the
LINUX RESTART message can be inserted into the daily
data file, indicating that the relevant kernel counters
have been reinitialized...$ ./configure --enable-install-cronOr you can answer 'y' to the question
Set crontab to start sar automatically? (y/n) [--enable-install-cron]if you use the Interactive Configuration script (
iconfig). Then compile sysstat as usual and run
make install as the last stage.Set crontab to start sar automaticallyduring config stage (
make config).
-s and -e are interpreted has
changed with sysstat's versions.sadf, this
is because no data belong to the specified time
interval! Up to sysstat version 12.1.4, the time
specified with options -s and -e was always considered
as being given in local time to be consistent with sar's
default output. Yet sadf displays its timestamps in UTC
(Coordinated Universal Time) by default (and in local
time with option -T). This could lead to some
misunderstandings, as if sadf's options -s
and -e didn't
work properly. So with sysstat version 12.1.5, the time
specified with options -s and -e is now consistent with
the timestamps displayed by sadf (either in UTC by
default or in local time with option -T), even if the
output doesn't match that of sar.
sadf in sysstat version 11.3.1.
Read sadf(1) manual page to learn some more about this new format.isag (a Perl script) or sargraph (a shell script).kSar (a Java application capable of
visualizing a sar file with static graphs), sarjitsu (a
more sophisticated application producing dynamic
visualizations based on Grafana), sarvant, sar2gp,
loadgraph, SysStat Charts, sarplot...PerfMan comes to mind, among others.
flock: Resource temporarily unavailableYou are launching
sadc using -L option. With this
option, sadc tries to get an exclusive lock on the
output file. The above error message indicates that
another sadc process was running and had already locked
the same output file. Stop all sadc instances and try again.
0 * * * * /usr/local/lib/sa/sa1 600 6so that I get an activity report every 10 minutes. When I use sar to get my output, there is no reading for 00:00:00. This means that at midnight everynight there is a spike, or dip, in the graphs. How should I run sysstat / sar so that I get a reading for 00:00:00?
sadc writes its data at the end of the previous
daily data file (/var/log/sa/sa<DD>) and at the
beginning of the new one (/var/log/sa/sa<DD+1>).
Please note that '-' must be used to specify the output
file for sadc to be able to detect such a file rotation.
So a crontab like the following one should enable you to
get the data for midnight at the end of each daily data file:# Activity reports every 10 minutes from 01:00:00 to 22:50:00 0 1-22 * * * /usr/local/lib/sa/sa1 600 6 # Activity reports every 10 minutes from 23:00:00 to 00:00:00 # Reporting until 00:00:00 ensures that a file rotation will be detected # by sadc 0 23 * * * /usr/local/lib/sa/sa1 600 7 # Activity reports every 10 minutes from 00:10:00 to 00:50:00 10 0 * * * /usr/local/lib/sa/sa1 600 5Another possible crontab would be:
*/10 1-22 * * * /usr/lib/sa/sa1 1 1 0,10,20,30,40 23 * * * /usr/lib/sa/sa1 1 1 50 23 * * * /usr/lib/sa/sa1 600 2 10,20,30,40,50 0 * * * /usr/lib/sa/sa1 1 1Things are much easier with recent sysstat versions (12.5.1 and later): You simply have to run
sa1 with its
option --rotate shortly after midnight to add a
statistics record to the system activity daily data file
of the previous day. So your full crontab could be:# Rotate file at midnight 0 0 * * * /usr/lib/sa/sa1 --rotate # Run system activity accounting tool every 10 minutes 0,10,20,30,40,50 * * * * /usr/lib/sa/sa1 1 1 # Generate a text summary of previous day process accounting at 00:07 7 0 * * * /usr/lib/sa/sa2 -A
Requested activities not available in file [...]This error message means that you are trying to extract non-existent activities from the data file. Usually
sadc
reads all the available activities from the system and
stores them in the data file. However, to prevent data
files from taking too much space on disk, some
activities must be explicitly set on the command line to
be read by sadc. To tell sadc that an optional activity
should be collected, use switch -S followed by the
keyword corresponding to that activity (see sadc(8)
manual page). As of this writing, optional activities
are: interrupts, disks, SNMP, IPv6 and power management statistics.sa1(8) or even the script used to insert a RESTART
message when the machine is rebooted).-I and -d) had to be
used with sadc to
indicate which activities had to be collected. Other
activities could not be saved into a file at all even if
you specified the relevant options on the command line.
Process statistics (-x and -X options in old
sysstat versions) were such activities.sar command complains with the error message:
Requested activities not available(without mentioning "in file"), it means that you are trying to display activities that the kernel itself is unable to provide (please see question 2.17 below).
sar doesn't need a lot of CPU to run, nor does it
make your system slow, contrary to what some people
think. In the first place, it only runs every ten
minutes by default. Secondly, when it does run, it is
over and done very quickly. Try:$ time /usr/lib/sa/sa1to verify that for yourself. Nor do you have to be concerned about using up all your disk space.
sar will
use a few hundred kilobytes for a whole day's worth of
data, and it normally only stores one week worth (this
can be configured via the HISTORY variable in the
/etc/sysconfig/sysstat file). It is entirely self
limiting. Moreover, you can ask sar to compress its
datafiles older than a certain number of days: see the
COMPRESSAFTER parameter in the /etc/sysconfig/sysstat
configuration file.
sadc are cumulative values. Moreover all
per-second statistics displayed by sar are average
values on the given time interval. So the value for
counter foo at time T is calculated as:Requested activities not availableThis error message means that you are trying to display activities that the kernel itself is unable to provide.
sar -o datafile
...), this may also be because the target file cannot
accept the requested activities. In this case, just try
to use another file or create a new one. See also question 2.13 above.
sar saves its data in the standard system
activity data file, the /var/log/sa/saDD file, where DD
is the current day in the month. To prevent sar from
overwriting any existing files, just set the variable
HISTORY in /etc/sysconfig/sysstat to the number of days
during which data must be kept. When this variable has a
value greater than 28, sa1 script uses a month-by-month
directory structure; datafiles are named YYYYMM/saDD and
the script maintains links to these datafiles to mimic
the standard sar datafile structure.sa1 now calls
sadc with option -D
set, telling it to use saYYYYMMDD instead of saDD as the
name for the standard system activity daily data files,
where YYYY stands for the current year, MM for the
current month and DD for the current day. All these
files are saved in the same directory (/var/log/sa by default).
sadf (a command
included in sysstat package) with its option -d. It
displays sar data in a format that can easily be
ingested by a relational database system (fields are
separated by a semicolon). It should then be easy for a
tool like SQL*Loader to load the data into the Oracle database.
10:50:01 AM CPU %user %nice %system %iowait %idle 11:00:01 AM all 90.90 0.00 5.17 3.93 0.00 11:00:01 AM 0 39.40 0.00 2.37 2.07 56.17 11:00:01 AM 1 29.71 0.00 1.73 1.17 67.39 11:00:01 AM 2 42.69 0.00 2.34 1.11 53.85 11:00:01 AM 3 26.24 0.00 1.41 1.61 70.74 ...Sysstat may have met an overflow condition while reading CPU usage from your
/proc/stat file. This condition is
all the more likely to happen as your machine uptime is
high and/or there are many processors. Sysstat up to
version 5.0.6 uses 32-bit integer variables to store CPU
usage. Then, beginning with version 5.1.1, sysstat has
shifted to 64-bit variables, which has fixed the
problem. So try to upgrade your version of sysstat to
the latest stable release and check that the problem has gone.
sar because new
commands have been made available. You should now use
the sadf command instead of sar -h or sar -H, and the
pidstat command instead of sar -x or sar -X. Please read
their manual page to learn some more about their respective options.
sadc, count is the number of data samples collected.sar, count is the number of records to
display (a record contains the average values for
counters over the given time interval - See 2.15).sadc the
"valid" count values for sar are 1 through 5.
Any value greater than 5 for sar will give the same
output as 5 in this example. So entering sar -f datafile1 2000
for a file populated with the output of sadc 1 6 datafile
will give the same output as sar -f datafile 1 5.
Note that it all depends on the number of data
samples pre-existing in the data file. If the file is
empty when first running sadc then the above is true.
sar to not handle sub-second intervals:sar's purpose. sar has been created to
give the sys admin a global overview of its machine
daily utilization so that when a problem happens, he has
a benchmark and can compare the statistics gathered by
sar with those saved before. For that reason an interval
of 10 minutes (which is the default for sar) is quite appropriate.sar really collects a lot of data). This can be
resource-consuming and you are all the more prone to
have an influence on the data you are retrieving as the
interval of time is small.
sadc's option -S followed by one or more keywords
(DISK, SNMP...) can already be used to specify which
optional activities are to be collected. Without this
option, sadc collects a default set of activities (CPU
activity, memory activity, network activity, etc.) Yet
it is actually possible to specify explicitly which
activities should be collected by sadc! You have to use
sadc's option -S followed by the report name
corresponding to the activity you want to collect (enter
sar --help to know the formal report names used by
sadc).sadc to collect only temperature sensors activity
in addition to the default set of activities, enter:$ sadc -S A_PWR_TEMP (...)followed by the other classic options (interval, count, filename...) Now assume you want to collect temperature sensors activity without the other activities collected by default, add the special report name
A_NULL to the list passed to sadc, e.g.:$ sadc -S A_NULL,A_PWR_TEMP (...)Of course you can enter as many report names as you want to collect different activities.
$ sadc -S XALL,-A_IRQ (...)This way you can tell
sadc to collect only the desired activities.
End of system activity file unexpected
sadc, the data collector, was unable to write all its
data to the system activity data file (/var/log/sa/saDD
by default). This is probably because there was no space
left on the device where the data file is located.
Make sure there is enough free space on the device.
Sometimes it seems there is enough free space but there
may be some jobs run by crontab (particularly during the
night) that can temporarily consume all of your free
space, making sadc fail.sadc are trying to
update the same data file, especially around midnight
when making a file rotation. See question 2.12 to know
how to make such a file rotation properly.-f with sadc to make sure data
are written to disk immediately. Of course sync'ing each
data sample to disk implies a (probably slight)
performance penalty.
3. QUESTIONS RELATING TO IOSTAT
3.1. I can't see all my disks when I use the iostat command...linux/include/linux/kernel_stat.h and
recompile your kernel. You should no longer have any
problem with post 2.5 kernels, since statistics are
maintained by the kernel for all the devices. In the
particular case of iostat, also be sure to use the ALL
keyword on the command line to display statistical
information relating to every device, including those
that are defined but have never been used by the system.
iostat -x to be able to report extended disk I/O
statistics, it is better to use a recent version of the
Linux kernel (2.6.x). Indeed, iostat tries to read data
from the /proc/diskstats file or from the sysfs
filesystem for that. But iostat may also be able to
display extended statistics with older kernels (e.g.
2.4.x) providing that all the necessary statistical
information is available in the /proc/partitions file,
which requires that a patch be applied to the kernel
(this is often done on kernels included in various
distros). In recent 2.4.x kernels, the /proc/partitions
file has all the necessary data providing that the
kernel has been compiled with CONFIG_BLK_STATS=y.
iostat displays I/O activity in blocks per
second. With old kernels (i.e. older than 2.4.x) a block
is of indeterminate size and therefore the displayed
values are not useful.iostat is
now able to get disk activities from the kernel
expressed in a number of sectors. If you take a look at
the kernel code, the sector size is actually allowed to
vary although I have never seen anything other than 512 bytes.
iostat concerns the time since
system startup, whereas subsequent ones cover only the
time since the previous report (that is to say, the
interval of time entered on the command line).
iostat -x may display
huge I/O response times (svctm) and a bandwidth
utilization (%util) of 100% for some devices. Indeed
these devices have a value for the field #9 (beginning
after the device name) in /proc/{partitions,diskstats}
which is always different from 0, and even negative
sometimes. Yet this field should go to zero, since it
gives the number of I/Os currently in progress (it is
incremented as requests are submitted, and decremented
as they finish). To (temporarily) solve the problem, you
should reboot your system to reset the counters in
/proc/{partitions,diskstats}.
4. QUESTIONS RELATING TO PIDSTAT
4.1. pidstat -d doesn't report task I/O statistics...pidstat -d to be able to report I/O statistics for
tasks, you need a recent Linux kernel (2.6.20 or later)
with the option CONFIG_TASK_IO_ACCOUNTING compiled in.
Requested activities not availableThis message is displayed when the
pidstat command is
unable to display the statistics you have requested.
This may happen when you try to display statistics for
child processes (option -T CHILD) because all options of
pidstat don't necessarily work for child processes. Read
the manual page to know which statistics are available
for child processes.
pidstat only displays statistics
for active tasks by default. If you don't use option -p
on the command line, then pidstat will display only
tasks with non-zero statistics. For example, pidstat -u
will display only tasks that have actually used some
CPU resources since system startup. You should enter
pidstat -u -p ALL to make sure that all the processes
are listed in the report.
pidstat is the CPU to which
the task is attached when the statistics are actually
displayed. This doesn't mean that the task has spent its
whole interval of time attached to it. Hence the CPU
ressource used by a thread on an interval of time as
displayed by pidstat may have concerned several processors.