Linux.pl
Opcje wyszukiwania podręcznika man:
Lista stron man zaczynających się od znaku:
A   B   C   D   E   F   G   H   I   J   K   L   M   N   O   P   Q   R   S   T   U   V   W   X   Y   Z   ALPHA   NUM   OTHER   ALL
INNOTOP(1p)           User Contributed Perl Documentation          INNOTOP(1p)

NAME
       innotop - MySQL and InnoDB transaction/status monitor.

SYNOPSIS
       To monitor servers normally:

        innotop

       To monitor InnoDB status information from a file:

        innotop /var/log/mysql/mysqld.err

       To run innotop non-interactively in a pipe-and-filter configuration:

        innotop --count 5 -d 1 -n

DESCRIPTION
       innotop monitors MySQL servers.  Each of its modes shows you a differ-
       ent aspect of what's happening in the server.  For example, there's a
       mode for monitoring replication, one for queries, and one for transac-
       tions.  innotop refreshes its data periodically, so you see an updating
       view.

       innotop has lots of features for power users, but you can start and run
       it with virtually no configuration.  If you're just getting started,
       see "QUICK-START".  Press '?' at any time while running innotop for
       context-sensitive help.

QUICK-START
       To start innotop, open a terminal or command prompt.  If you have
       installed innotop on your system, you should be able to just type
       "innotop" and press Enter; otherwise, you will need to change to inno-
       top's directory and type "perl innotop".

       The first thing innotop needs to know is how to connect to a MySQL
       server.  You can just enter the hostname of the server, for example
       "localhost" or "127.0.0.1" if the server is on the same machine as
       innotop.  After this innotop will prompt you for a DSN (data source
       name).  You should be able to just accept the defaults by pressing
       Enter.

       When innotop asks you about a table to use when resetting InnoDB dead-
       lock information, just accept the default for now.  This is an advanced
       feature you can configure later (see "D: InnoDB Deadlocks" for more).

       If you have a .my.cnf file with your MySQL connection defaults, innotop
       can read it, and you won't need to specify a username and password if
       it's in that file.  Otherwise, you should answer 'y' to the next couple
       of prompts.

       After this, you should be connected, and innotop should show you some-
       thing like the following:

        InnoDB Txns (? for help) localhost, 01:11:19, InnoDB 10s :-), 50 QPS,

        CXN        History  Versions  Undo  Dirty Buf  Used Bufs  Txns  MaxTxn
        localhost        7      2035  0 0       0.00%     92.19%     1   07:34

        CXN        ID     User   Host       Txn Status  Time   Undo  Query Tex
        localhost  98379  user1  webserver  ACTIVE      07:34     0  SELECT `c
        localhost  98450  user1  webserver  ACTIVE      01:06     0  INSERT IN
        localhost  97750  user1  webserver  not starte  00:00     0
        localhost  98375  user1  appserver  not starte  00:00     0

       (This sample is truncated at the right so it will fit on a terminal
       when running 'man innotop')

       This sample comes from a quiet server with few transactions active.  If
       your server is busy, you'll see more output.  Notice the first line on
       the screen, which tells you what mode you're in and what server you're
       connected to.  You can change to other modes with keystrokes; press 'Q'
       to switch to a list of currently running queries.

       Press the '?' key to see what keys are active in the current mode.  You
       can press any of these keys and innotop will either take the requested
       action or prompt you for more input.  If your system has Term::ReadLine
       support, you can use TAB and other keys to auto-complete and edit
       input.

       To quit innotop, press the 'q' key.

OPTIONS
       innotop is mostly configured via its configuration file, but some of
       the configuration options can come from the command line.  You can also
       specify a file to monitor for InnoDB status output; see "MONITORING A
       FILE" for more details.

       You can negate some options by prefixing the option name with --no.
       For example, --noinc (or --no-inc) negates "--inc".

       --help
           Print a summary of command-line usage and exit.

       --color
           Enable or disable terminal coloring.  Corresponds to the "color"
           config file setting.

       --config
           Specifies a configuration file to read.  This option is non-sticky,
           that is to say it does not persist to the configuration file
           itself.

       --nonint
           Enable non-interactive operation.  See "NON-INTERACTIVE OPERATION"
           for more.

       --count
           Refresh only the specified number of times (ticks) before exiting.
           Each refresh is a pause for "interval" seconds, followed by
           requesting data from MySQL connections and printing it to the ter-
           minal.

       --delay
           Specifies the amount of time to pause between ticks (refreshes).
           Corresponds to the configuration option "interval".

       --mode
           Specifies the mode in which innotop should start.  Corresponds to
           the configuration option "mode".

       --inc
           Specifies whether innotop should display absolute numbers or rela-
           tive numbers (offsets from their previous values).  Corresponds to
           the configuration option "status_inc".

       --version
           Output version information and exit.

HOTKEYS
       innotop is interactive, and you control it with key-presses.

       o   Uppercase keys switch between modes.

       o   Lowercase keys initiate some action within the current mode.

       o   Other keys do something special like change configuration or show
           the innotop license.

       Press '?' at any time to see the currently active keys and what they
       do.

MODES
       Each of innotop's modes retrieves and displays a particular type of
       data from the servers you're monitoring.  You switch between modes with
       uppercase keys.  The following is a brief description of each mode, in
       alphabetical order.  To switch to the mode, press the key listed in
       front of its heading in the following list:

       B: InnoDB Buffers
           This mode displays information about the InnoDB buffer pool, page
           statistics, insert buffer, and adaptive hash index.  The data comes
           from SHOW INNODB STATUS.

           This mode contains the "buffer_pool", "page_statistics",
           "insert_buffers", and "adaptive_hash_index" tables by default.

       C: Command Summary
           This mode is similar to mytop's Command Summary mode.  It shows the
           "cmd_summary" table, which looks something like the following:

            Command Summary (? for help) localhost, 25+07:16:43, 2.45 QPS, 3 thd, 5.0.40
            _____________________ Command Summary _____________________
            Name                    Value    Pct     Last Incr  Pct
            Select_scan             3244858  69.89%          2  100.00%
            Select_range            1354177  29.17%          0    0.00%
            Select_full_join          39479   0.85%          0    0.00%
            Select_full_range_join     4097   0.09%          0    0.00%
            Select_range_check            0   0.00%          0    0.00%

           The command summary table is built by extracting variables from
           "STATUS_VARIABLES".  The variables must be numeric and must match
           the prefix given by the "cmd_filter" configuration variable.  The
           variables are then sorted by value descending and compared to the
           last variable, as shown above.  The percentage columns are percent-
           age of the total of all variables in the table, so you can see the
           relative weight of the variables.

           The example shows what you see if the prefix is "Select_".  The
           default prefix is "Com_".  You can choose a prefix with the 's'
           key.

           It's rather like running SHOW VARIABLES LIKE "prefix%" with memory
           and nice formatting.

           Values are aggregated across all servers.  The Pct columns are not
           correctly aggregated across multiple servers.  This is a known lim-
           itation of the grouping algorithm that may be fixed in the future.

       D: InnoDB Deadlocks
           This mode shows the transactions involved in the last InnoDB dead-
           lock.  A second table shows the locks each transaction held and
           waited for.  A deadlock is caused by a cycle in the waits-for
           graph, so there should be two locks held and one waited for unless
           the deadlock information is truncated.

           InnoDB puts deadlock information before some other information in
           the SHOW INNODB STATUS output.  If there are a lot of locks, the
           deadlock information can grow very large, and there is a limit on
           the size of the SHOW INNODB STATUS output.  A large deadlock can
           fill the entire output, or even be truncated, and prevent you from
           seeing other information at all.  If you are running innotop in
           another mode, for example T mode, and suddenly you don't see any-
           thing, you might want to check and see if a deadlock has wiped out
           the data you need.

           If it has, you can create a small deadlock to replace the large
           one.  Use the 'w' key to 'wipe' the large deadlock with a small
           one.  This will not work unless you have defined a deadlock table
           for the connection (see "SERVER CONNECTIONS").

           You can also configure innotop to automatically detect when a large
           deadlock needs to be replaced with a small one (see
           "auto_wipe_dl").

           This mode displays the "deadlock_transactions" and "deadlock_locks"
           tables by default.

       F: InnoDB Foreign Key Errors
           This mode shows the last InnoDB foreign key error information, such
           as the table where it happened, when and who and what query caused
           it, and so on.

           InnoDB has a huge variety of foreign key error messages, and many
           of them are just hard to parse.  innotop doesn't always do the best
           job here, but there's so much code devoted to parsing this messy,
           unparseable output that innotop is likely never to be perfect in
           this regard.  If innotop doesn't show you what you need to see,
           just look at the status text directly.

           This mode displays the "fk_error" table by default.

       I: InnoDB I/O Info
           This mode shows InnoDB's I/O statistics, including the I/O threads,
           pending I/O, file I/O miscellaneous, and log statistics.  It dis-
           plays the "io_threads", "pending_io", "file_io_misc", and
           "log_statistics" tables by default.

       L: Locks
           This mode shows information about current locks.  At the moment
           only InnoDB locks are supported, and by default you'll only see
           locks for which transactions are waiting.  This information comes
           from the TRANSACTIONS section of the InnoDB status text.  If you
           have a very busy server, you may have frequent lock waits; it helps
           to be able to see which tables and indexes are the "hot spot" for
           locks.  If your server is running pretty well, this mode should
           show nothing.

           You can configure MySQL and innotop to monitor not only locks for
           which a transaction is waiting, but those currently held, too.  You
           can do this with the InnoDB Lock Monitor
           (<http://dev.mysql.com/doc/en/innodb-monitor.html>).  It's not doc-
           umented in the MySQL manual, but creating the lock monitor with the
           following statement also affects the output of SHOW INNODB STATUS,
           which innotop uses:

             CREATE TABLE innodb_lock_monitor(a int) ENGINE=INNODB;

           This causes InnoDB to print its output to the MySQL file every 16
           seconds or so, as stated in the manual, but it also makes the nor-
           mal SHOW INNODB STATUS output include lock information, which
           innotop can parse and display (that's the undocumented feature).

           This means you can do what may have seemed impossible: to a limited
           extent (InnoDB truncates some information in the output), you can
           see which transaction holds the locks something else is waiting
           for.  You can also enable and disable the InnoDB Lock Monitor with
           the key mappings in this mode.

           This mode displays the "innodb_locks" table by default.  Here's a
           sample of the screen when one connection is waiting for locks
           another connection holds:

            _________________________________ InnoDB Locks __________________________
            CXN        ID  Type    Waiting  Wait   Active  Mode  DB    Table  Index
            localhost  12  RECORD        1  00:10   00:10  X     test  t1     PRIMARY
            localhost  12  TABLE         0  00:10   00:10  IX    test  t1
            localhost  12  RECORD        1  00:10   00:10  X     test  t1     PRIMARY
            localhost  11  TABLE         0  00:00   00:25  IX    test  t1
            localhost  11  RECORD        0  00:00   00:25  X     test  t1     PRIMARY

           You can see the first connection, ID 12, is waiting for a lock on
           the PRIMARY key on test.t1, and has been waiting for 10 seconds.
           The second connection isn't waiting, because the Waiting column is
           0, but it holds locks on the same index.  That tells you connection
           11 is blocking connection 12.

       M: Master/Slave Replication Status
           This mode shows the output of SHOW SLAVE STATUS and SHOW MASTER
           STATUS in three tables.  The first two divide the slave's status
           into SQL and I/O thread status, and the last shows master status.
           Filters are applied to eliminate non-slave servers from the slave
           tables, and non-master servers from the master table.

           This mode displays the "slave_sql_status", "slave_io_status", and
           "master_status" tables by default.

       O: Open Tables
           This section comes from MySQL's SHOW OPEN TABLES command.  By
           default it is filtered to show tables which are in use by one or
           more queries, so you can get a quick look at which tables are
           'hot'.  You can use this to guess which tables might be locked
           implicitly.

           This mode displays the "open_tables" mode by default.

       Q: Query List
           This mode displays the output from SHOW FULL PROCESSLIST, much like
           mytop's query list mode.  This mode does not show InnoDB-related
           information.  This is probably one of the most useful modes for
           general usage.

           There is an informative header that shows general status informa-
           tion about your server.  You can toggle it on and off with the 'h'
           key.  By default, innotop hides inactive processes and its own pro-
           cess.  You can toggle these on and off with the 'i' and 'a' keys.

           You can EXPLAIN a query from this mode with the 'e' key.  This dis-
           plays the query's full text, the results of EXPLAIN, and in newer
           MySQL versions, even the optimized query resulting from EXPLAIN
           EXTENDED.  innotop also tries to rewrite certain queries to make
           them EXPLAIN-able.  For example, INSERT/SELECT statements are
           rewritable.

           This mode displays the "q_header" and "processlist" tables by
           default.

       R: InnoDB Row Operations and Semaphores
           This mode shows InnoDB row operations, row operation miscellaneous,
           semaphores, and information from the wait array.  It displays the
           "row_operations", "row_operation_misc", "semaphores", and
           "wait_array" tables by default.

       S: Variables & Status
           This mode calculates statistics, such as queries per second, and
           prints them out in several different styles.  You can show absolute
           values, or incremental values between ticks.

           You can switch between the views by pressing a key.  The 's' key
           prints a single line each time the screen updates, in the style of
           vmstat.  The 'g' key changes the view to a graph of the same num-
           bers, sort of like tload.  The 'v' key changes the view to a piv-
           oted table of variable names on the left, with successive updates
           scrolling across the screen from left to right.  You can choose how
           many updates to put on the screen with the "num_status_sets" con-
           figuration variable.

           Headers may be abbreviated to fit on the screen in interactive
           operation.  You choose which variables to display with the 'c' key,
           which selects from predefined sets, or lets you create your own
           sets.  You can edit the current set with the 'e' key.

           This mode doesn't really display any tables like other modes.
           Instead, it uses a table definition to extract and format the data,
           but it then transforms the result in special ways before outputting
           it.  It uses the "var_status" table definition for this.

       T: InnoDB Transactions
           This mode shows transactions from the InnoDB monitor's output, in
           top-like format.  This mode is the reason I wrote innotop.

           You can kill queries or processes with the 'k' and 'x' keys, and
           EXPLAIN a query with the 'e' or 'f' keys.  InnoDB doesn't print the
           full query in transactions, so explaining may not work right if the
           query is truncated.

           The informational header can be toggled on and off with the 'h'
           key.  By default, innotop hides inactive transactions and its own
           transaction.  You can toggle this on and off with the 'i' and 'a'
           keys.

           This mode displays the "t_header" and "innodb_transactions" tables
           by default.

INNOTOP STATUS
       The first line innotop displays is a "status bar" of sorts.  What it
       contains depends on the mode you're in, and what servers you're moni-
       toring.  The first few words are always the innotop mode, such as "Inn-
       oDB Txns" for T mode, followed by a reminder to press '?' for help at
       any time.

       ONE SERVER

       The simplest case is when you're monitoring a single server.  In this
       case, the name of the connection is next on the status line.  This is
       the name you gave when you created the connection -- most likely the
       MySQL server's hostname.  This is followed by the server's uptime.

       If you're in an InnoDB mode, such as T or B, the next word is "InnoDB"
       followed by some information about the SHOW INNODB STATUS output used
       to render the screen.  The first word is the number of seconds since
       the last SHOW INNODB STATUS, which InnoDB uses to calculate some per-
       second statistics.  The next is a smiley face indicating whether the
       InnoDB output is truncated.  If the smiley face is a :-), all is well;
       there is no truncation.  A :^| means the transaction list is so long,
       InnoDB has only printed out some of the transactions.  Finally, a frown
       :-( means the output is incomplete, which is probably due to a deadlock
       printing too much lock information (see "D: InnoDB Deadlocks").

       The next two words indicate the server's queries per second (QPS) and
       how many threads (connections) exist.  Finally, the server's version
       number is the last thing on the line.

       MULTIPLE SERVERS

       If you are monitoring multiple servers (see "SERVER CONNECTIONS"), the
       status line does not show any details about individual servers.
       Instead, it shows the names of the connections that are active.  Again,
       these are connection names you specified, which are likely to be the
       server's hostname.  A connection that has an error is prefixed with an
       exclamation point.

       If you are monitoring a group of servers (see "SERVER GROUPS"), the
       status line shows the name of the group.  If any connection in the
       group has an error, the group's name is followed by the fraction of the
       connections that don't have errors.

       See "ERROR HANDLING" for more details about innotop's error handling.

       MONITORING A FILE

       If you give a filename on the command line, innotop will not connect to
       ANY servers at all.  It will watch the specified file for InnoDB status
       output and use that as its data source.  It will always show a single
       connection called 'file'.  And since it can't connect to a server, it
       can't determine how long the server it's monitoring has been up; so it
       calculates the server's uptime as time since innotop started running.

SERVER ADMINISTRATION
       While innotop is primarily a monitor that lets you watch and analyze
       your servers, it can also send commands to servers.  The most fre-
       quently useful commands are killing queries and stopping or starting
       slaves.

       You can kill a connection, or in newer versions of MySQL kill a query
       but not a connection, from "Q: Query List" and "T: InnoDB Transactions"
       modes.  Press 'k' to issue a KILL command, or 'x' to issue a KILL QUERY
       command.  innotop will prompt you for the server and/or connection ID
       to kill (innotop does not prompt you if there is only one possible
       choice for any input).  innotop pre-selects the longest-running query,
       or the oldest connection.  Confirm the command with 'y'.

       In "M: Master/Slave Replication Status" mode, you can start and stop
       slaves with the 'a' and 'o' keys, respectively.  You can send these
       commands to many slaves at once.  innotop fills in a default command of
       START SLAVE or STOP SLAVE for you, but you can actually edit the com-
       mand and send anything you wish, such as SET GLOBAL
       SQL_SLAVE_SKIP_COUNTER=1 to make the slave skip one binlog event when
       it starts.

       You can also ask innotop to calculate the earliest binlog in use by any
       slave and issue a PURGE MASTER LOGS on the master.  Use the 'b' key for
       this.  innotop will prompt you for a master to run the command on, then
       prompt you for the connection names of that master's slaves (there is
       no way for innotop to determine this reliably itself).  innotop will
       find the minimum binlog in use by these slave connections and suggest
       it as the argument to PURGE MASTER LOGS.

SERVER CONNECTIONS
       When you create a server connection, innotop asks you for a series of
       inputs, as follows:

       DSN A DSN is a Data Source Name, which is the initial argument passed
           to the DBI module for connecting to a server.  It is usually of the
           form

            DBI:mysql:;mysql_read_default_group=mysql;host=HOSTNAME

           Since this DSN is passed to the DBD::mysql driver, you should read
           the driver's documentation at
           "http://search.cpan.org/dist/DBD-mysql/lib/DBD/mysql.pm" for the
           exact details on all the options you can pass the driver in the
           DSN.  You can read more about DBI at <http://dbi.perl.org/docs/>,
           and especially at <http://search.cpan.org/~timb/DBI/DBI.pm>.

           The mysql_read_default_group=mysql option lets the DBD driver read
           your MySQL options files, such as ~/.my.cnf on UNIX-ish systems.
           You can use this to avoid specifying a username or password for the
           connection.

       InnoDB Deadlock Table
           This optional item tells innotop a table name it can use to delib-
           erately create a small deadlock (see "D: InnoDB Deadlocks").  If
           you specify this option, you just need to be sure the table doesn't
           exist, and that innotop can create and drop the table with the Inn-
           oDB storage engine.  You can safely omit or just accept the default
           if you don't intend to use this.

       Username
           innotop will ask you if you want to specify a username.  If you say
           'y', it will then prompt you for a user name.  If you have a MySQL
           option file that specifies your username, you don't have to specify
           a username.

           The username defaults to your login name on the system you're run-
           ning innotop on.

       Password
           innotop will ask you if you want to specify a password.  Like the
           username, the password is optional, but there's an additional
           prompt that asks if you want to save the password in the innotop
           configuration file.  If you don't save it in the configuration
           file, innotop will prompt you for a password each time it starts.
           Passwords in the innotop configuration file are saved in plain
           text, not encrypted in any way.

       Once you finish answering these questions, you should be connected to a
       server.  But innotop isn't limited to monitoring a single server; you
       can define many server connections and switch between them by pressing
       the '@' key.  See "SWITCHING BETWEEN CONNECTIONS".

       To create a new connection, press the '@' key and type the name of the
       new connection, then follow the steps given above.

SERVER GROUPS
       If you have multiple MySQL instances, you can put them into named
       groups, such as 'all', 'masters', and 'slaves', which innotop can moni-
       tor all together.

       You can choose which group to monitor with the '#' key, and you can
       press the TAB key to switch to the next group.  If you're not currently
       monitoring a group, pressing TAB selects the first group.

       To create a group, press the '#' key and type the name of your new
       group, then type the names of the connections you want the group to
       contain.

SWITCHING BETWEEN CONNECTIONS
       innotop lets you quickly switch which servers you're monitoring.  The
       most basic way is by pressing the '@' key and typing the name(s) of the
       connection(s) you want to use.  This setting is per-mode, so you can
       monitor different connections in each mode, and innotop remembers which
       connections you choose.

       You can quickly switch to the 'next' connection in alphabetical order
       with the 'n' key.  If you're monitoring a server group (see "SERVER
       GROUPS") this will switch to the first connection.

       You can also type many connection names, and innotop will fetch and
       display data from them all.  Just separate the connection names with
       spaces, for example "server1 server2."  Again, if you type the name of
       a connection that doesn't exist, innotop will prompt you for connection
       information and create the connection.

       Another way to monitor multiple connections at once is with server
       groups.  You can use the TAB key to switch to the 'next' group in
       alphabetical order, or if you're not monitoring any groups, TAB will
       switch to the first group.

       innotop does not fetch data in parallel from connections, so if you are
       monitoring a large group or many connections, you may notice increased
       delay between ticks.

       When you monitor more than one connection, innotop's status bar
       changes.  See "INNOTOP STATUS".

ERROR HANDLING
       Error handling is not that important when monitoring a single connec-
       tion, but is crucial when you have many active connections.  A crashed
       server or lost connection should not crash innotop.  As a result, inno-
       top will continue to run even when there is an error; it just won't
       display any information from the connection that had an error.  Because
       of this, innotop's behavior might confuse you.  It's a feature, not a
       bug!

       innotop does not continue to query connections that have errors,
       because they may slow innotop and make it hard to use, especially if
       the error is a problem connecting and causes a long time-out.  Instead,
       innotop retries the connection occasionally to see if the error still
       exists.  If so, it will wait until some point in the future.  The wait
       time increases in ticks as the Fibonacci series, so it tries less fre-
       quently as time passes.

       Since errors might only happen in certain modes because of the SQL com-
       mands issued in those modes, innotop keeps track of which mode caused
       the error.  If you switch to a different mode, innotop will retry the
       connection instead of waiting.

       By default innotop will display the problem in red text at the bottom
       of the first table on the screen.  You can disable this behavior with
       the "show_cxn_errors_in_tbl" configuration option, which is enabled by
       default.  If the "debug" option is enabled, innotop will display the
       error at the bottom of every table, not just the first.  And if
       "show_cxn_errors" is enabled, innotop will print the error text to STD-
       OUT as well.  Error messages might only display in the mode that caused
       the error, depending on the mode and whether innotop is avoiding query-
       ing that connection.

NON-INTERACTIVE OPERATION
       You can run innotop in non-interactive mode, in which case it is
       entirely controlled from the configuration file and command-line
       options.  To start innotop in non-interactive mode, give the L"<--non-
       int"> command-line option.  This changes innotop's behavior in the fol-
       lowing ways:

       o   Certain Perl modules are not loaded.  Term::Readline is not loaded,
           since innotop doesn't prompt interactively.  Term::ANSIColor and
           Win32::Console::ANSI modules are not loaded.  Term::ReadKey is
           still used, since innotop may have to prompt for connection
           passwords when starting up.

       o   innotop does not clear the screen after each tick.

       o   innotop does not persist any changes to the configuration file.

       o   If "--count" is given and innotop is in incremental mode (see "sta-
           tus_inc" and "--inc"), innotop actually refreshes one more time
           than specified so it can print incremental statistics.  This sup-
           presses output during the first tick, so innotop may appear to
           hang.

       o   innotop only displays the first table in each mode.  This is so the
           output can be easily processed with other command-line utilities
           such as awk and sed.  To change which tables display in each mode,
           see "TABLES".  Since "Q: Query List" mode is so important, innotop
           automatically disables the "q_header" table.  This ensures you'll
           see the "processlist" table, even if you have innotop configured to
           show the q_header table during interactive operation.  Similarly,
           in "T: InnoDB Transactions" mode, the "t_header" table is sup-
           pressed so you see only the "innodb_transactions" table.

       o   All output is tab-separated instead of being column-aligned with
           whitespace, and innotop prints the full contents of each table
           instead of only printing one screenful at a time.

       o   innotop only prints column headers once instead of every tick (see
           "hide_hdr").  innotop does not print table captions (see "dis-
           play_table_captions").  innotop ensures there are no empty lines in
           the output.

       o   innotop does not honor the "shorten" transformation, which normally
           shortens some numbers to human-readable formats.

       o   innotop does not print a status line (see "INNOTOP STATUS").

CONFIGURING
       Nearly everything about innotop is configurable.  Most things are pos-
       sible to change with built-in commands, but you can also edit the con-
       figuration file.

       While running innotop, press the '$' key to bring up the configuration
       editing dialog.  Press another key to select the type of data you want
       to edit:

       S: Statement Sleep Times
           Edits SQL statement sleep delays, which make innotop pause for the
           specified amount of time after executing a statement.  See "SQL
           STATEMENTS" for a definition of each statement and what it does.
           By default innotop does not delay after any statements.

           This feature is included so you can customize the side-effects
           caused by monitoring your server.  You may not see any effects, but
           some innotop users have noticed that certain MySQL versions under
           very high load with InnoDB enabled take longer than usual to exe-
           cute SHOW GLOBAL STATUS.  If innotop calls SHOW FULL PROCESSLIST
           immediately afterward, the processlist contains more queries than
           the machine actually averages at any given moment.  Configuring
           innotop to pause briefly after calling SHOW GLOBAL STATUS allevi-
           ates this effect.

           Sleep times are stored in the "stmt_sleep_times" section of the
           configuration file.  Fractional-second sleeps are supported, sub-
           ject to your hardware's limitations.

       c: Edit Columns
           Starts the table editor on one of the displayed tables.  See "TABLE
           EDITOR".  An alternative way to start the table editor without
           entering the configuration dialog is with the '^' key.

       g: General Configuration
           Starts the configuration editor to edit global and mode-specific
           configuration variables (see "MODES").  innotop prompts you to
           choose a variable from among the global and mode-specific ones
           depending on the current mode.

       k: Row-Coloring Rules
           Starts the row-coloring rules editor on one of the displayed ta-
           ble(s).  See "COLORS" for details.

       p: Manage Plugins
           Starts the plugin configuration editor.  See "PLUGINS" for details.

       s: Server Groups
           Lets you create and edit server groups.  See "SERVER GROUPS".

       t: Choose Displayed Tables
           Lets you choose which tables to display in this mode.  See "MODES"
           and "TABLES".

CONFIGURATION FILE
       innotop's default configuration file location is in $HOME/.innotop, but
       can be overridden with the "--config" command-line option.  You can
       edit it by hand safely.  innotop reads the configuration file when it
       starts, and writes it out again when it exits, so any changes you make
       while innotop is running will be lost.

       innotop doesn't store its entire configuration in the configuration
       file.  It has a huge set of default configuration that it holds only in
       memory, and the configuration file only overrides these defaults.  When
       you customize a default setting, innotop notices, and then stores the
       customizations into the file.  This keeps the file size down, makes it
       easier to edit, and makes upgrades easier.

       A configuration file can be made read-only.  See "readonly".

       The configuration file is arranged into sections like an INI file.
       Each section begins with [section-name] and ends with [/section-name].
       Each section's entries have a different syntax depending on the data
       they need to store.  You can put comments in the file; any line that
       begins with a # character is a comment.  innotop will not read the com-
       ments, so it won't write them back out to the file when it exits.  Com-
       ments in read-only configuration files are still useful, though.

       The first line in the file is innotop's version number.  This lets
       innotop notice when the file format is not backwards-compatible, and
       upgrade smoothly without destroying your customized configuration.

       The following list describes each section of the configuration file and
       the data it contains:

       general
           The 'general' section contains global configuration variables and
           variables that may be mode-specific, but don't belong in any other
           section.  The syntax is a simple key=value list.  innotop writes a
           comment above each value to help you edit the file by hand.

           S_func
               Controls S mode presentation (see "S: Variables & Status").  If
               g, values are graphed; if s, values are like vmstat; if p, val-
               ues are in a pivoted table.

           S_set
               Specifies which set of variables to display in "S: Variables &
               Status" mode.  See "VARIABLE SETS".

           auto_wipe_dl
               Instructs innotop to automatically wipe large deadlocks when it
               notices them.  When this happens you may notice a slight delay.
               At the next tick, you will usually see the information that was
               being truncated by the large deadlock.

           charset
               Specifies what kind of characters to allow through the
               "no_ctrl_char" transformation.  This keeps non-printable char-
               acters from confusing a terminal when you monitor queries that
               contain binary data, such as images.

               The default is 'ascii', which considers anything outside normal
               ASCII to be a control character.  The other allowable values
               are 'unicode' and 'none'.  'none' considers every character a
               control character, which can be useful for collapsing ALL text
               fields in queries.

           cmd_filter
               This is the prefix that filters variables in "C: Command Sum-
               mary" mode.

           color
               Whether terminal coloring is permitted.

           cxn_timeout
               On MySQL versions 4.0.3 and newer, this variable is used to set
               the connection's timeout, so MySQL doesn't close the connection
               if it is not used for a while.  This might happen because a
               connection isn't monitored in a particular mode, for example.

           debug
               This option enables more verbose errors and makes innotop more
               strict in some places.  It can help in debugging filters and
               other user-defined code.  It also makes innotop write a lot of
               information to "debugfile" when there is a crash.

           debugfile
               A file to which innotop will write information when there is a
               crash.  See "FILES".

           display_table_captions
               innotop displays a table caption above most tables.  This vari-
               able suppresses or shows captions on all tables globally.  Some
               tables are configured with the hide_caption property, which
               overrides this.

           global
               Whether to show GLOBAL variables and status.  innotop only
               tries to do this on servers which support the GLOBAL option to
               SHOW VARIABLES and SHOW STATUS.  In some MySQL versions, you
               need certain privileges to do this; if you don't have them,
               innotop will not be able to fetch any variable and status data.
               This configuration variable lets you run innotop and fetch what
               data you can even without the elevated privileges.

               I can no longer find or reproduce the situation where GLOBAL
               wasn't allowed, but I know there was one.

           graph_char
               Defines the character to use when drawing graphs in "S: Vari-
               ables & Status" mode.

           header_highlight
               Defines how to highlight column headers.  This only works if
               Term::ANSIColor is available.  Valid values are 'bold' and
               'underline'.

           hide_hdr
               Hides column headers globally.

           interval
               The interval at which innotop will refresh its data (ticks).
               The interval is implemented as a sleep time between ticks, so
               the true interval will vary depending on how long it takes
               innotop to fetch and render data.

               This variable accepts fractions of a second.

           mode
               The mode in which innotop should start.  Allowable arguments
               are the same as the key presses that select a mode interac-
               tively.  See "MODES".

           num_digits
               How many digits to show in fractional numbers and percents.
               This variable's range is between 0 and 9 and can be set
               directly from "S: Variables & Status" mode with the '+' and '-'
               keys.  It is used in the "set_precision", "shorten", and "per-
               cent" transformations.

           num_status_sets
               Controls how many sets of status variables to display in piv-
               oted "S: Variables & Status" mode.  It also controls the number
               of old sets of variables innotop keeps in its memory, so the
               larger this variable is, the more memory innotop uses.

           plugin_dir
               Specifies where plugins can be found.  By default, innotop
               stores plugins in the 'plugins' subdirectory of your innotop
               configuration directory.

           readonly
               Whether the configuration file is readonly.  This cannot be set
               interactively, because it would prevent itself from being writ-
               ten to the configuration file.

           show_cxn_errors
               Makes innotop print connection errors to STDOUT.  See "ERROR
               HANDLING".

           show_cxn_errors_in_tbl
               Makes innotop display connection errors as rows in the first
               table on screen.  See "ERROR HANDLING".

           show_percent
               Adds a '%' character after the value returned by the "percent"
               transformation.

           show_statusbar
               Controls whether to show the status bar in the display.  See
               "INNOTOP STATUS".

           skip_innodb
               Disables fetching SHOW INNODB STATUS, in case your server(s) do
               not have InnoDB enabled and you don't want innotop to try to
               fetch it.  This can also be useful when you don't have the
               SUPER privilege, required to run SHOW INNODB STATUS.

           status_inc
               Whether to show absolute or incremental values for status vari-
               ables.  Incremental values are calculated as an offset from the
               last value innotop saw for that variable.  This is a global
               setting, but will probably become mode-specific at some point.
               Right now it is honored a bit inconsistently; some modes don't
               pay attention to it.

       plugins
           This section holds a list of package names of active plugins.  If
           the plugin exists, innotop will activate it.  See "PLUGINS" for
           more information.

       filters
           This section holds user-defined filters (see "FILTERS").  Each line
           is in the format filter_name=text='filter text' tbls='table list'.

           The filter text is the text of the subroutine's code.  The table
           list is a list of tables to which the filter can apply.  By
           default, user-defined filters apply to the table for which they
           were created, but you can manually override that by editing the
           definition in the configuration file.

       active_filters
           This section stores which filters are active on each table.  Each
           line is in the format table_name=filter_list.

       tbl_meta
           This section stores user-defined or user-customized columns (see
           "COLUMNS").  Each line is in the format col_name=properties, where
           the properties are a name=quoted-value list.

       connections
           This section holds the server connections you have defined.  Each
           line is in the format name=properties, where the properties are a
           name=value list.  The properties are self-explanatory, and the only
           one that is treated specially is 'pass' which is only present if
           'savepass' is set.  See "SERVER CONNECTIONS".

       active_connections
           This section holds a list of which connections are active in each
           mode.  Each line is in the format mode_name=connection_list.

       server_groups
           This section holds server groups.  Each line is in the format
           name=connection_list.  See "SERVER GROUPS".

       active_server_groups
           This section holds a list of which server group is active in each
           mode.  Each line is in the format mode_name=server_group.

       max_values_seen
           This section holds the maximum values seen for variables.  This is
           used to scale the graphs in "S: Variables & Status" mode.  Each
           line is in the format name=value.

       active_columns
           This section holds table column lists.  Each line is in the format
           tbl_name=column_list.  See "COLUMNS".

       sort_cols
           This section holds the sort definition.  Each line is in the format
           tbl_name=column_list.  If a column is prefixed with '-', that col-
           umn sorts descending.  See "SORTING".

       visible_tables
           This section defines which tables are visible in each mode.  Each
           line is in the format mode_name=table_list.  See "TABLES".

       varsets
           This section defines variable sets for use in "S: Status & Vari-
           ables" mode.  Each line is in the format name=variable_list.  See
           "VARIABLE SETS".

       colors
           This section defines colorization rules.  Each line is in the for-
           mat tbl_name=property_list.  See "COLORS".

       stmt_sleep_times
           This section contains statement sleep times.  Each line is in the
           format statement_name=sleep_time.  See "S: Statement Sleep Times".

       group_by
           This section contains column lists for table group_by expressions.
           Each line is in the format tbl_name=column_list.  See "GROUPING".

CUSTOMIZING
       You can customize innotop a great deal.  For example, you can:

       o   Choose which tables to display, and in what order.

       o   Choose which columns are in those tables, and create new columns.

       o   Filter which rows display with built-in filters, user-defined fil-
           ters, and quick-filters.

       o   Sort the rows to put important data first or group together related
           rows.

       o   Highlight rows with color.

       o   Customize the alignment, width, and formatting of columns, and
           apply transformations to columns to extract parts of their values
           or format the values as you wish (for example, shortening large
           numbers to familiar units).

       o   Design your own expressions to extract and combine data as you
           need.  This gives you unlimited flexibility.

       All these and more are explained in the following sections.

       TABLES

       A table is what you'd expect: a collection of columns.  It also has
       some other properties, such as a caption.  Filters, sorting rules, and
       colorization rules belong to tables and are covered in later sections.

       Internally, table meta-data is defined in a data structure called
       %tbl_meta.  This hash holds all built-in table definitions, which con-
       tain a lot of default instructions to innotop.  The meta-data includes
       the caption, a list of columns the user has customized, a list of
       columns, a list of visible columns, a list of filters, color rules, a
       sort-column list, sort direction, and some information about the ta-
       ble's data sources.  Most of this is customizable via the table editor
       (see "TABLE EDITOR").

       You can choose which tables to show by pressing the '$' key.  See
       "MODES" and "TABLES".

       The table life-cycle is as follows:

       o   Each table begins with a data source, which is an array of hashes.
           See below for details on data sources.

       o   Each element of the data source becomes a row in the final table.

       o   For each element in the data source, innotop extracts values from
           the source and creates a row.  This row is another hash, which
           later steps will refer to as $set.  The values innotop extracts are
           determined by the table's columns.  Each column has an extraction
           subroutine, compiled from an expression (see "EXPRESSIONS").  The
           resulting row is a hash whose keys are named the same as the column
           name.

       o   innotop filters the rows, removing those that don't need to be dis-
           played.  See "FILTERS".

       o   innotop sorts the rows.  See "SORTING".

       o   innotop groups the rows together, if specified.  See "GROUPING".

       o   innotop colorizes the rows.  See "COLORS".

       o   innotop transforms the column values in each row.  See "TRANSFORMA-
           TIONS".

       o   innotop optionally pivots the rows (see "PIVOTING"), then filters
           and sorts them.

       o   innotop formats and justifies the rows as a table.  During this
           step, innotop applies further formatting to the column values,
           including alignment, maximum and minimum widths.  innotop also does
           final error checking to ensure there are no crashes due to unde-
           fined values.  innotop then adds a caption if specified, and the
           table is ready to print.

       The lifecycle is slightly different if the table is pivoted, as noted
       above.  To clarify, if the table is pivoted, the process is extract,
       group, transform, pivot, filter, sort, create.  If it's not pivoted,
       the process is extract, filter, sort, group, color, transform, create.
       This slightly convoluted process doesn't map all that well to SQL, but
       pivoting complicates things pretty thoroughly.  Roughly speaking, fil-
       tering and sorting happen as late as needed to effect the final result
       as you might expect, but as early as possible for efficiency.

       Each built-in table is described below:

       adaptive_hash_index
           Displays data about InnoDB's adaptive hash index.  Data source:
           "STATUS_VARIABLES".

       buffer_pool
           Displays data about InnoDB's buffer pool.  Data source: "STA-
           TUS_VARIABLES".

       cmd_summary
           Displays weighted status variables.  Data source: "STATUS_VARI-
           ABLES".

       deadlock_locks
           Shows which locks were held and waited for by the last detected
           deadlock.  Data source: "DEADLOCK_LOCKS".

       deadlock_transactions
           Shows transactions involved in the last detected deadlock.  Data
           source: "DEADLOCK_TRANSACTIONS".

       explain
           Shows the output of EXPLAIN.  Data source: "EXPLAIN".

       file_io_misc
           Displays data about InnoDB's file and I/O operations.  Data source:
           "STATUS_VARIABLES".

       fk_error
           Displays various data about InnoDB's last foreign key error.  Data
           source: "STATUS_VARIABLES".

       innodb_locks
           Displays InnoDB locks.  Data source: "INNODB_LOCKS".

       innodb_transactions
           Displays data about InnoDB's current transactions.  Data source:
           "INNODB_TRANSACTIONS".

       insert_buffers
           Displays data about InnoDB's insert buffer.  Data source: "STA-
           TUS_VARIABLES".

       io_threads
           Displays data about InnoDB's I/O threads.  Data source:
           "IO_THREADS".

       log_statistics
           Displays data about InnoDB's logging system.  Data source: "STA-
           TUS_VARIABLES".

       master_status
           Displays replication master status.  Data source: "STATUS_VARI-
           ABLES".

       open_tables
           Displays open tables.  Data source: "OPEN_TABLES".

       page_statistics
           Displays InnoDB page statistics.  Data source: "STATUS_VARIABLES".

       pending_io
           Displays InnoDB pending I/O operations.  Data source: "STATUS_VARI-
           ABLES".

       processlist
           Displays current MySQL processes (threads/connections).  Data
           source: "PROCESSLIST".

       q_header
           Displays various status values.  Data source: "STATUS_VARIABLES".

       row_operation_misc
           Displays data about InnoDB's row operations.  Data source: "STA-
           TUS_VARIABLES".

       row_operations
           Displays data about InnoDB's row operations.  Data source: "STA-
           TUS_VARIABLES".

       semaphores
           Displays data about InnoDB's semaphores and mutexes.  Data source:
           "STATUS_VARIABLES".

       slave_io_status
           Displays data about the slave I/O thread.  Data source: "STA-
           TUS_VARIABLES".

       slave_sql_status
           Displays data about the slave SQL thread.  Data source: "STA-
           TUS_VARIABLES".

       t_header
           Displays various InnoDB status values.  Data source: "STATUS_VARI-
           ABLES".

       var_status
           Displays user-configurable data.  Data source: "STATUS_VARIABLES".

       wait_array
           Displays data about InnoDB's OS wait array.  Data source:
           "OS_WAIT_ARRAY".

       COLUMNS

       Columns belong to tables.  You can choose a table's columns by pressing
       the '^' key, which starts the "TABLE EDITOR" and lets you choose and
       edit columns.  Pressing 'e' from within the table editor lets you edit
       the column's properties:

       o   hdr: a column header.  This appears in the first row of the table.

       o   just: justification.  '-' means left-justified and '' means
           right-justified, just as with printf formatting codes (not a coin-
           cidence).

       o   dec: whether to further align the column on the decimal point.

       o   num: whether the column is numeric.  This affects how values are
           sorted (lexically or numerically).

       o   label: a small note about the column, which appears in dialogs that
           help the user choose columns.

       o   src: an expression that innotop uses to extract the column's data
           from its source (see "DATA SOURCES").  See "EXPRESSIONS" for more
           on expressions.

       o   minw: specifies a minimum display width.  This helps stabilize the
           display, which makes it easier to read if the data is changing fre-
           quently.

       o   maxw: similar to minw.

       o   trans: a list of column transformations.  See "TRANSFORMATIONS".

       o   agg: an aggregate function.  See "GROUPING".  The default is
           "first".

       o   aggonly: controls whether the column only shows when grouping is
           enabled on the table (see "GROUPING").  By default, this is dis-
           abled.  This means columns will always be shown by default, whether
           grouping is enabled or not.  If a column's aggonly is set true, the
           column will appear when you toggle grouping on the table.  Several
           columns are set this way, such as the count column on "processlist"
           and "innodb_transactions", so you don't see a count when the group-
           ing isn't enabled, but you do when it is.

       FILTERS

       Filters remove rows from the display.  They behave much like a WHERE
       clause in SQL.  innotop has several built-in filters, which remove
       irrelevant information like inactive queries, but you can define your
       own as well.  innotop also lets you create quick-filters, which do not
       get saved to the configuration file, and are just an easy way to
       quickly view only some rows.

       You can enable or disable a filter on any table.  Press the '%' key
       (mnemonic: % looks kind of like a line being filtered between two cir-
       cles) and choose which table you want to filter, if asked.  You'll then
       see a list of possible filters and a list of filters currently enabled
       for that table.  Type the names of filters you want to apply and press
       Enter.

       USER-DEFINED FILTERS

       If you type a name that doesn't exist, innotop will prompt you to cre-
       ate the filter.  Filters are easy to create if you know Perl, and not
       hard if you don't.  What you're doing is creating a subroutine that
       returns true if the row should be displayed.  The row is a hash refer-
       ence passed to your subroutine as $set.

       For example, imagine you want to filter the processlist table so you
       only see queries that have been running more than five minutes.  Type a
       new name for your filter, and when prompted for the subroutine body,
       press TAB to initiate your terminal's auto-completion.  You'll see the
       names of the columns in the "processlist" table (innotop generally
       tries to help you with auto-completion lists).  You want to filter on
       the 'time' column.  Type the text "$set->{time} > 300" to return true
       when the query is more than five minutes old.  That's all you need to
       do.

       In other words, the code you're typing is surrounded by an implicit
       context, which looks like this:

        sub filter {
           my ( $set ) = @_;
           # YOUR CODE HERE
        }

       If your filter doesn't work, or if something else suddenly behaves dif-
       ferently, you might have made an error in your filter, and innotop is
       silently catching the error.  Try enabling "debug" to make innotop
       throw an error instead.

       QUICK-FILTERS

       innotop's quick-filters are a shortcut to create a temporary filter
       that doesn't persist when you restart innotop.  To create a quick-fil-
       ter, press the '/' key.  innotop will prompt you for the column name
       and filter text.  Again, you can use auto-completion on column names.
       The filter text can be just the text you want to "search for."  For
       example, to filter the "processlist" table on queries that refer to the
       products table, type '/' and then 'info product'.

       The filter text can actually be any Perl regular expression, but of
       course a literal string like 'product' works fine as a regular expres-
       sion.

       Behind the scenes innotop compiles the quick-filter into a specially
       tagged filter that is otherwise like any other filter.  It just isn't
       saved to the configuration file.

       To clear quick-filters, press the '\' key and innotop will clear them
       all at once.

       SORTING

       innotop has sensible built-in defaults to sort the most important rows
       to the top of the table.  Like anything else in innotop, you can cus-
       tomize how any table is sorted.

       To start the sort dialog, start the "TABLE EDITOR" with the '^' key,
       choose a table if necessary, and press the 's' key.  You'll see a list
       of columns you can use in the sort expression and the current sort
       expression, if any.  Enter a list of columns by which you want to sort
       and press Enter.  If you want to reverse sort, prefix the column name
       with a minus sign.  For example, if you want to sort by column a
       ascending, then column b descending, type 'a -b'.  You can also explic-
       itly add a + in front of columns you want to sort ascending, but it's
       not required.

       Some modes have keys mapped to open this dialog directly, and to
       quickly reverse sort direction.  Press '?' as usual to see which keys
       are mapped in any mode.

       GROUPING

       innotop can group, or aggregate, rows together (I use the terms inter-
       changeably).  This is quite similar to an SQL GROUP BY clause.  You can
       specify to group on certain columns, or if you don't specify any, the
       entire set of rows is treated as one group.  This is quite like SQL so
       far, but unlike SQL, you can also select un-grouped columns.  innotop
       actually aggregates every column.  If you don't explicitly specify a
       grouping function, the default is 'first'.  This is basically a conve-
       nience so you don't have to specify an aggregate function for every
       column you want in the result.

       You can quickly toggle grouping on a table with the '=' key, which tog-
       gles its aggregate property.  This property doesn't persist to the con-
       fig file.

       The columns by which the table is grouped are specified in its group_by
       property.  When you turn grouping on, innotop places the group_by
       columns at the far left of the table, even if they're not supposed to
       be visible.  The rest of the visible columns appear in order after
       them.

       Two tables have default group_by lists and a count column built in:
       "processlist" and "innodb_transactions".  The grouping is by connection
       and status, so you can quickly see how many queries or transactions are
       in a given status on each server you're monitoring.  The time columns
       are aggregated as a sum; other columns are left at the default 'first'
       aggregation.

       By default, the table shown in "S: Variables & Status" mode also uses
       grouping so you can monitor variables and status across many servers.
       The default aggregation function in this mode is 'avg'.

       Valid grouping functions are defined in the %agg_funcs hash.  They
       include

       first
           Returns the first element in the group.

       count
           Returns the number of elements in the group, including undefined
           elements, much like SQL's COUNT(*).

       avg Returns the average of defined elements in the group.

       sum Returns the sum of elements in the group.

       Here's an example of grouping at work.  Suppose you have a very busy
       server with hundreds of open connections, and you want to see how many
       connections are in what status.  Using the built-in grouping rules, you
       can press 'Q' to enter "Q: Query List" mode.  Press '=' to toggle
       grouping (if necessary, select the "processlist" table when prompted).

       Your display might now look like the following:

        Query List (? for help) localhost, 32:33, 0.11 QPS, 1 thd, 5.0.38-log

        CXN        Cmd        Cnt  ID      User   Host           Time   Query
        localhost  Query      49    12933  webusr localhost      19:38  SELECT * FROM
        localhost  Sending Da 23     2383  webusr localhost      12:43  SELECT col1,
        localhost  Sleep      120     140  webusr localhost    5:18:12
        localhost  Statistics 12    19213  webusr localhost      01:19  SELECT * FROM

       That's actually quite a worrisome picture.  You've got a lot of idle
       connections (Sleep), and some connections executing queries (Query and
       Sending Data).  That's okay, but you also have a lot in Statistics sta-
       tus, collectively spending over a minute.  That means the query opti-
       mizer is having a really hard time optimizing your statements.  Some-
       thing is wrong; it should normally take milliseconds to optimize
       queries.  You might not have seen this pattern if you didn't look at
       your connections in aggregate.  (This is a made-up example, but it can
       happen in real life).

       PIVOTING

       innotop can pivot a table for more compact display, similar to a Pivot
       Table in a spreadsheet (also known as a crosstab).  Pivoting a table
       makes columns into rows.  Assume you start with this table:

        foo bar
        === ===
        1   3
        2   4

       After pivoting, the table will look like this:

        name set0 set1
        ==== ==== ====
        foo  1    2
        bar  3    4

       To get reasonable results, you might need to group as well as pivoting.
       innotop currently does this for "S: Variables & Status" mode.

       COLORS

       By default, innotop highlights rows with color so you can see at a
       glance which rows are more important.  You can customize the coloriza-
       tion rules and add your own to any table.  Open the table editor with
       the '^' key, choose a table if needed, and press 'o' to open the color
       editor dialog.

       The color editor dialog displays the rules applied to the table, in the
       order they are evaluated.  Each row is evaluated against each rule to
       see if the rule matches the row; if it does, the row gets the specified
       color, and no further rules are evaluated.  The rules look like the
       following:

        state  eq  Locked       black on_red
        cmd    eq  Sleep        white
        user   eq  system user  white
        cmd    eq  Connect      white
        cmd    eq  Binlog Dump  white
        time   >   600          red
        time   >   120          yellow
        time   >   60           green
        time   >   30           cyan

       This is the default rule set for the "processlist" table.  In order of
       priority, these rules make locked queries black on a red background,
       "gray out" connections from replication and sleeping queries, and make
       queries turn from cyan to red as they run longer.

       (For some reason, the ANSI color code "white" is actually a light gray.
       Your terminal's display may vary; experiment to find colors you like).

       You can use keystrokes to move the rules up and down, which re-orders
       their priority.  You can also delete rules and add new ones.  If you
       add a new rule, innotop prompts you for the column, an operator for the
       comparison, a value against which to compare the column, and a color to
       assign if the rule matches.  There is auto-completion and prompting at
       each step.

       The value in the third step needs to be correctly quoted.  innotop does
       not try to quote the value because it doesn't know whether it should
       treat the value as a string or a number.  If you want to compare the
       column against a string, as for example in the first rule above, you
       should enter 'Locked' surrounded by quotes.  If you get an error mes-
       sage about a bareword, you probably should have quoted something.

       EXPRESSIONS

       Expressions are at the core of how innotop works, and are what enables
       you to extend innotop as you wish.  Recall the table lifecycle
       explained in "TABLES".  Expressions are used in the earliest step,
       where it extracts values from a data source to form rows.

       It does this by calling a subroutine for each column, passing it the
       source data set, a set of current values, and a set of previous values.
       These are all needed so the subroutine can calculate things like the
       difference between this tick and the previous tick.

       The subroutines that extract the data from the set are compiled from
       expressions.  This gives significantly more power than just naming the
       values to fill the columns, because it allows the column's value to be
       calculated from whatever data is necessary, but avoids the need to
       write complicated and lengthy Perl code.

       innotop begins with a string of text that can look as simple as a
       value's name or as complicated as a full-fledged Perl expression.  It
       looks at each 'bareword' token in the string and decides whether it's
       supposed to be a key into the $set hash.  A bareword is an unquoted
       value that isn't already surrounded by code-ish things like dollar
       signs or curly brackets.  If innotop decides that the bareword isn't a
       function or other valid Perl code, it converts it into a hash access.
       After the whole string is processed, innotop compiles a subroutine,
       like this:

        sub compute_column_value {
           my ( $set, $cur, $pre ) = @_;
           my $val = # EXPANDED STRING GOES HERE
           return $val;
        }

       Here's a concrete example, taken from the header table "q_header" in
       "Q: Query List" mode.  This expression calculates the qps, or Queries
       Per Second, column's values, from the values returned by SHOW STATUS:

        Questions/Uptime_hires

       innotop decides both words are barewords, and transforms this expres-
       sion into the following Perl code:

        $set->{Questions}/$set->{Uptime_hires}

       When surrounded by the rest of the subroutine's code, this is exe-
       cutable Perl that calculates a high-resolution queries-per-second
       value.

       The arguments to the subroutine are named $set, $cur, and $pre.  In
       most cases, $set and $cur will be the same values.  However, if "sta-
       tus_inc" is set, $cur will not be the same as $set, because $set will
       already contain values that are the incremental difference between $cur
       and $pre.

       Every column in innotop is computed by subroutines compiled in the same
       fashion.  There is no difference between innotop's built-in columns and
       user-defined columns.  This keeps things consistent and predictable.

       TRANSFORMATIONS

       Transformations change how a value is rendered.  For example, they can
       take a number of seconds and display it in H:M:S format.  The following
       transformations are defined:

       commify
           Adds commas to large numbers every three decimal places.

       dulint_to_int
           Accepts two unsigned integers and converts them into a single long-
           long.  This is useful for certain operations with InnoDB, which
           uses two integers as transaction identifiers, for example.

       no_ctrl_char
           Removes quoted control characters from the value.  This is affected
           by the "charset" configuration variable.

           This transformation only operates within quoted strings, for exam-
           ple, values to a SET clause in an UPDATE statement.  It will not
           alter the UPDATE statement, but will collapse the quoted string to
           [BINARY] or [TEXT], depending on the charset.

       percent
           Converts a number to a percentage by multiplying it by two, format-
           ting it with "num_digits" digits after the decimal point, and
           optionally adding a percent sign (see "show_percent").

       secs_to_time
           Formats a number of seconds as time in days+hours:minutes:seconds
           format.

       set_precision
           Formats numbers with "num_digits" number of digits after the deci-
           mal point.

       shorten
           Formats a number as a unit of 1024 (k/M/G/T) and with "num_digits"
           number of digits after the decimal point.

       TABLE EDITOR

       The innotop table editor lets you customize tables with keystrokes.
       You start the table editor with the '^' key.  If there's more than one
       table on the screen, it will prompt you to choose one of them.  Once
       you do, innotop will show you something like this:

        Editing table definition for Buffer Pool.  Press ? for help, q to quit.

        name               hdr          label                  src
        cxn                CXN          Connection from which  cxn
        buf_pool_size      Size         Buffer pool size       IB_bp_buf_poo
        buf_free           Free Bufs    Buffers free in the b  IB_bp_buf_fre
        pages_total        Pages        Pages total            IB_bp_pages_t
        pages_modified     Dirty Pages  Pages modified (dirty  IB_bp_pages_m
        buf_pool_hit_rate  Hit Rate     Buffer pool hit rate   IB_bp_buf_poo
        total_mem_alloc    Memory       Total memory allocate  IB_bp_total_m
        add_pool_alloc     Add'l Pool   Additonal pool alloca  IB_bp_add_poo

       The first line shows which table you're editing, and reminds you again
       to press '?' for a list of key mappings.  The rest is a tabular repre-
       sentation of the table's columns, because that's likely what you're
       trying to edit.  However, you can edit more than just the table's
       columns; this screen can start the filter editor, color rule editor,
       and more.

       Each row in the display shows a single column in the table you're edit-
       ing, along with a couple of its properties such as its header and
       source expression (see "EXPRESSIONS").

       The key mappings are Vim-style, as in many other places.  Pressing 'j'
       and 'k' moves the highlight up or down.  You can then (d)elete or
       (e)dit the highlighted column.  You can also (a)dd a column to the ta-
       ble.  This actually just activates one of the columns already defined
       for the table; it prompts you to choose from among the columns avail-
       able but not currently displayed.  Finally, you can re-order the
       columns with the '+' and '-' keys.

       You can do more than just edit the columns with the table editor, you
       can also edit other properties, such as the table's sort expression and
       group-by expression.  Press '?' to see the full list, of course.

       If you want to really customize and create your own column, as opposed
       to just activating a built-in one that's not currently displayed, press
       the (n)ew key, and innotop will prompt you for the information it
       needs:

       o   The column name: this needs to be a word without any funny charac-
           ters, e.g. just letters, numbers and underscores.

       o   The column header: this is the label that appears at the top of the
           column, in the table header.  This can have spaces and funny char-
           acters, but be careful not to make it too wide and waste space
           on-screen.

       o   The column's data source: this is an expression that determines
           what data from the source (see "TABLES") innotop will put into the
           column.  This can just be the name of an item in the source, or it
           can be a more complex expression, as described in "EXPRESSIONS".

       Once you've entered the required data, your table has a new column.
       There is no difference between this column and the built-in ones; it
       can have all the same properties and behaviors.  innotop will write the
       column's definition to the configuration file, so it will persist
       across sessions.

       Here's an example: suppose you want to track how many times your slaves
       have retried transactions.  According to the MySQL manual, the
       Slave_retried_transactions status variable gives you that data: "The
       total number of times since startup that the replication slave SQL
       thread has retried transactions. This variable was added in version
       5.0.4."  This is appropriate to add to the "slave_sql_status" table.

       To add the column, switch to the replication-monitoring mode with the
       'M' key, and press the '^' key to start the table editor.  When
       prompted, choose slave_sql_status as the table, then press 'n' to cre-
       ate the column.  Type 'retries' as the column name, 'Retries' as the
       column header, and 'Slave_retried_transactions' as the source.  Now the
       column is created, and you see the table editor screen again.  Press
       'q' to exit the table editor, and you'll see your column at the end of
       the table.

VARIABLE SETS
       Variable sets are used in "S: Variables & Status" mode to define more
       easily what variables you want to monitor.  Behind the scenes they are
       compiled to a list of expressions, and then into a column list so they
       can be treated just like columns in any other table, in terms of data
       extraction and transformations.  However, you're protected from the
       tedious details by a syntax that ought to feel very natural to you: a
       SQL SELECT list.

       The data source for variable sets, and indeed the entire S mode, is the
       combination of SHOW STATUS, SHOW VARIABLES, and SHOW INNODB STATUS.
       Imagine that you had a huge table with one column per variable returned
       from those statements.  That's the data source for variable sets.  You
       can now query this data source just like you'd expect.  For example:

        Questions, Uptime, Questions/Uptime as QPS

       Behind the scenes innotop will split that variable set into three
       expressions, compile them and turn them into a table definition, then
       extract as usual.  This becomes a "variable set," or a "list of vari-
       ables you want to monitor."

       innotop lets you name and save your variable sets, and writes them to
       the configuration file.  You can choose which variable set you want to
       see with the 'c' key, or activate the next and previous sets with the
       '>' and '<' keys.  There are many built-in variable sets as well, which
       should give you a good start for creating your own.  Press 'e' to edit
       the current variable set, or just to see how it's defined.  To create a
       new one, just press 'c' and type its name.

       You may want to use some of the functions listed in "TRANSFORMATIONS"
       to help format the results.  In particular, "set_precision" is often
       useful to limit the number of digits you see.  Extending the above
       example, here's how:

        Questions, Uptime, set_precision(Questions/Uptime) as QPS

       Actually, this still needs a little more work.  If your "interval" is
       less than one second, you might be dividing by zero because Uptime is
       incremental in this mode by default.  Instead, use Uptime_hires:

        Questions, Uptime, set_precision(Questions/Uptime_hires) as QPS

       This example is simple, but it shows how easy it is to choose which
       variables you want to monitor.

PLUGINS
       innotop has a simple but powerful plugin mechanism by which you can
       extend or modify its existing functionality, and add new functionality.
       innotop's plugin functionality is event-based: plugins register them-
       selves to be called when events happen.  They then have a chance to
       influence the event.

       An innotop plugin is a Perl module placed in innotop's "plugin_dir"
       directory.  On UNIX systems, you can place a symbolic link to the mod-
       ule instead of putting the actual file there.  innotop automatically
       discovers the file.  If there is a corresponding entry in the "plugins"
       configuration file section, innotop loads and activates the plugin.

       The module must conform to innotop's plugin interface.  Additionally,
       the source code of the module must be written in such a way that inno-
       top can inspect the file and determine the package name and descrip-
       tion.

       Package Source Convention

       innotop inspects the plugin module's source to determine the Perl pack-
       age name.  It looks for a line of the form "package Foo;" and if found,
       considers the plugin's package name to be Foo.  Of course the package
       name can be a valid Perl package name, with double semicolons and so
       on.

       It also looks for a description in the source code, to make the plugin
       editor more human-friendly.  The description is a comment line of the
       form "# description: Foo", where "Foo" is the text innotop will con-
       sider to be the plugin's description.

       Plugin Interface

       The innotop plugin interface is quite simple: innotop expects the plu-
       gin to be an object-oriented module it can call certain methods on.
       The methods are

       new(%variables)
           This is the plugin's constructor.  It is passed a hash of innotop's
           variables, which it can manipulate (see "Plugin Variables").  It
           must return a reference to the newly created plugin object.

           At construction time, innotop has only loaded the general configu-
           ration and created the default built-in variables with their
           default contents (which is quite a lot).  Therefore, the state of
           the program is exactly as in the innotop source code, plus the con-
           figuration variables from the "general" section in the config file.

           If your plugin manipulates the variables, it is changing global
           data, which is shared by innotop and all plugins.  Plugins are
           loaded in the order they're listed in the config file.  Your plugin
           may load before or after another plugin, so there is a potential
           for conflict or interaction between plugins if they modify data
           other plugins use or modify.

       register_for_events()
           This method must return a list of events in which the plugin is
           interested, if any.  See "Plugin Events" for the defined events.
           If the plugin returns an event that's not defined, the event is
           ignored.

       event handlers
           The plugin must implement a method named the same as each event for
           which it has registered.  In other words, if the plugin returns
           qw(foo bar) from register_for_events(), it must have foo() and
           bar() methods.  These methods are callbacks for the events.  See
           "Plugin Events" for more details about each event.

       Plugin Variables

       The plugin's constructor is passed a hash of innotop's variables, which
       it can manipulate.  It is probably a good idea if the plugin object
       saves a copy of it for later use.  The variables are defined in the
       innotop variable %pluggable_vars, and are as follows:

       action_for
           A hashref of key mappings.  These are innotop's global hot-keys.

       agg_funcs
           A hashref of functions that can be used for grouping.  See "GROUP-
           ING".

       config
           The global configuration hash.

       connections
           A hashref of connection specifications.  These are just specifica-
           tions of how to connect to a server.

       dbhs
           A hashref of innotop's database connections.  These are actual DBI
           connection objects.

       filters
           A hashref of filters applied to table rows.  See "FILTERS" for
           more.

       modes
           A hashref of modes.  See "MODES" for more.

       server_groups
           A hashref of server groups.  See "SERVER GROUPS".

       tbl_meta
           A hashref of innotop's table meta-data, with one entry per table
           (see "TABLES" for more information).

       trans_funcs
           A hashref of transformation functions.  See "TRANSFORMATIONS".

       var_sets
           A hashref of variable sets.  See "VARIABLE SETS".

       Plugin Events

       Each event is defined somewhere in the innotop source code.  When inno-
       top runs that code, it executes the callback function for each plugin
       that expressed its interest in the event.  innotop passes some data for
       each event.  The events are defined in the %event_listener_for vari-
       able, and are as follows:

       extract_values($set, $cur, $pre, $tbl)
           This event occurs inside the function that extracts values from a
           data source.  The arguments are the set of values, the current val-
           ues, the previous values, and the table name.

       set_to_tbl
           Events are defined at many places in this subroutine, which is
           responsible for turning an arrayref of hashrefs into an arrayref of
           lines that can be printed to the screen.  The events all pass the
           same data: an arrayref of rows and the name of the table being cre-
           ated.  The events are set_to_tbl_pre_filter,
           set_to_tbl_pre_sort,set_to_tbl_pre_group, set_to_tbl_pre_colorize,
           set_to_tbl_pre_transform, set_to_tbl_pre_pivot, set_to_tbl_pre_cre-
           ate, set_to_tbl_post_create.

       draw_screen($lines)
           This event occurs inside the subroutine that prints the lines to
           the screen.  $lines is an arrayref of strings.

       Simple Plugin Example

       The easiest way to explain the plugin functionality is probably with a
       simple example.  The following module adds a column to the beginning of
       every table and sets its value to 1.

        use strict;
        use warnings FATAL => 'all';

        package Innotop::Plugin::Example;
        # description: Adds an 'example' column to every table

        sub new {
           my ( $class, %vars ) = @_;
           # Store reference to innotop's variables in $self
           my $self = bless { %vars }, $class;

           # Design the example column
           my $col = {
              hdr   => 'Example',
              just  => '',
              dec   => 0,
              num   => 1,
              label => 'Example',
              src   => 'example', # Get data from this column in the data source
              tbl   => '',
              trans => [],
           };

           # Add the column to every table.
           my $tbl_meta = $vars{tbl_meta};
           foreach my $tbl ( values %$tbl_meta ) {
              # Add the column to the list of defined columns
              $tbl->{cols}->{example} = $col;
              # Add the column to the list of visible columns
              unshift @{$tbl->{visible}}, 'example';
           }

           # Be sure to return a reference to the object.
           return $self;
        }

        # I'd like to be called when a data set is being rendered into a table, please.
        sub register_for_events {
           my ( $self ) = @_;
           return qw(set_to_tbl_pre_filter);
        }

        # This method will be called when the event fires.
        sub set_to_tbl_pre_filter {
           my ( $self, $rows, $tbl ) = @_;
           # Set the example column's data source to the value 1.
           foreach my $row ( @$rows ) {
              $row->{example} = 1;
           }
        }

        1;

       Plugin Editor

       The plugin editor lets you view the plugins innotop discovered and
       activate or deactivate them.  Start the editor by pressing $ to start
       the configuration editor from any mode.  Press the 'p' key to start the
       plugin editor.  You'll see a list of plugins innotop discovered.  You
       can use the 'j' and 'k' keys to move the highlight to the desired one,
       then press the * key to toggle it active or inactive.  Exit the editor
       and restart innotop for the changes to take effect.

SQL STATEMENTS
       innotop uses a limited set of SQL statements to retrieve data from
       MySQL for display.  The statements are customized depending on the
       server version against which they are executed; for example, on MySQL 5
       and newer, INNODB_STATUS executes "SHOW ENGINE INNODB STATUS", while on
       earlier versions it executes "SHOW INNODB STATUS".  The statements are
       as follows:

        Statement           SQL executed
        =================== ===============================
        INNODB_STATUS       SHOW [ENGINE] INNODB STATUS
        KILL_CONNECTION     KILL
        KILL_QUERY          KILL QUERY
        OPEN_TABLES         SHOW OPEN TABLES
        PROCESSLIST         SHOW FULL PROCESSLIST
        SHOW_MASTER_LOGS    SHOW MASTER LOGS
        SHOW_MASTER_STATUS  SHOW MASTER STATUS
        SHOW_SLAVE_STATUS   SHOW SLAVE STATUS
        SHOW_STATUS         SHOW [GLOBAL] STATUS
        SHOW_VARIABLES      SHOW [GLOBAL] VARIABLES

DATA SOURCES
       Each time innotop extracts values to create a table (see "EXPRESSIONS"
       and "TABLES"), it does so from a particular data source.  Largely
       because of the complex data extracted from SHOW INNODB STATUS, this is
       slightly messy.  SHOW INNODB STATUS contains a mixture of single values
       and repeated values that form nested data sets.

       Whenever innotop fetches data from MySQL, it adds two extra bits to
       each set: cxn and Uptime_hires.  cxn is the name of the connection from
       which the data came.  Uptime_hires is a high-resolution version of the
       server's Uptime status variable, which is important if your "interval"
       setting is sub-second.

       Here are the kinds of data sources from which data is extracted:

       STATUS_VARIABLES
           This is the broadest category, into which the most kinds of data
           fall.  It begins with the combination of SHOW STATUS and SHOW VARI-
           ABLES, but other sources may be included as needed, for example,
           SHOW MASTER STATUS and SHOW SLAVE STATUS, as well as many of the
           non-repeated values from SHOW INNODB STATUS.

       DEADLOCK_LOCKS
           This data is extracted from the transaction list in the LATEST
           DETECTED DEADLOCK section of SHOW INNODB STATUS.  It is nested two
           levels deep: transactions, then locks.

       DEADLOCK_TRANSACTIONS
           This data is from the transaction list in the LATEST DETECTED DEAD-
           LOCK section of SHOW INNODB STATUS.  It is nested one level deep.

       EXPLAIN
           This data is from the result set returned by EXPLAIN.

       INNODB_TRANSACTIONS
           This data is from the TRANSACTIONS section of SHOW INNODB STATUS.

       IO_THREADS
           This data is from the list of threads in the the FILE I/O section
           of SHOW INNODB STATUS.

       INNODB_LOCKS
           This data is from the TRANSACTIONS section of SHOW INNODB STATUS
           and is nested two levels deep.

       OPEN_TABLES
           This data is from SHOW OPEN TABLES.

       PROCESSLIST
           This data is from SHOW FULL PROCESSLIST.

       OS_WAIT_ARRAY
           This data is from the SEMAPHORES section of SHOW INNODB STATUS and
           is nested one level deep.  It comes from the lines that look like
           this:

            --Thread 1568861104 has waited at btr0cur.c line 424 ....

MYSQL PRIVILEGES
       o   You must connect to MySQL as a user who has the SUPER privilege for
           many of the functions.

       o   If you don't have the SUPER privilege, you can still run some func-
           tions, but you won't necessarily see all the same data.

       o   You need the PROCESS privilege to see the list of currently running
           queries in Q mode.

       o   You need special privileges to start and stop slave servers.

       o   You need appropriate privileges to create and drop the deadlock
           tables if needed (see "SERVER CONNECTIONS").

SYSTEM REQUIREMENTS
       You need Perl to run innotop, of course.  You also need a few Perl mod-
       ules: DBI, DBD::mysql,  Term::ReadKey, and Time::HiRes.  These should
       be included with most Perl distributions, but in case they are not, I
       recommend using versions distributed with your operating system or Perl
       distribution, not from CPAN.  Term::ReadKey in particular has been
       known to cause problems if installed from CPAN.

       If you have Term::ANSIColor, innotop will use it to format headers more
       readably and compactly.  (Under Microsoft Windows, you also need
       Win32::Console::ANSI for terminal formatting codes to be honored).  If
       you install Term::ReadLine, preferably Term::ReadLine::Gnu, you'll get
       nice auto-completion support.

       I run innotop on Gentoo GNU/Linux, Debian and Ubuntu, and I've had
       feedback from people successfully running it on Red Hat, CentOS,
       Solaris, and Mac OSX.  I don't see any reason why it won't work on
       other UNIX-ish operating systems, but I don't know for sure.  It also
       runs on Windows under ActivePerl without problem.

       I use innotop on MySQL versions 3.23.58, 4.0.27, 4.1.0, 4.1.22, 5.0.26,
       5.1.15, and 5.2.3.  If it doesn't run correctly for you, that is a bug
       and I hope you report it.

FILES
       $HOMEDIR/.innotop is used to store configuration information.  Files
       include the configuration file innotop.ini, the core_dump file which
       contains verbose error messages if "debug" is enabled, and the plugins/
       subdirectory.

GLOSSARY OF TERMS
       tick
           A tick is a refresh event, when innotop re-fetches data from con-
           nections and displays it.

ACKNOWLEDGEMENTS
       I'm grateful to the following people for various reasons, and hope I
       haven't forgotten to include anyone:

       Allen K. Smith, Aurimas Mikalauskas, Bartosz Fenski, Brian Miezejewski,
       Christian Hammers, Cyril Scetbon, Dane Miller, David Multer, Dr. Frank
       Ullrich, Giuseppe Maxia, Google.com Site Reliability Engineers, Jan
       Pieter Kunst, Jari Aalto, Jay Pipes, Jeremy Zawodny, Johan Idren, Kris-
       tian Kohntopp, Lenz Grimmer, Maciej Dobrzanski, Michiel Betel, MySQL
       AB, Paul McCullagh, Sebastien Estienne, Sourceforge.net, Steven
       Kreuzer, The Gentoo MySQL Team, Trevor Price, Yaar Schnitman, and prob-
       ably more people I've neglected to include.

       (If I misspelled your name, it's probably because I'm afraid of putting
       international characters into this documentation; earlier versions of
       Perl might not be able to compile it then).

COPYRIGHT, LICENSE AND WARRANTY
       This program is copyright (c) 2006 Baron Schwartz.  Feedback and
       improvements are welcome.

       THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
       WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
       MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE.

       This program is free software; you can redistribute it and/or modify it
       under the terms of the GNU General Public License as published by the
       Free Software Foundation, version 2; OR the Perl Artistic License.  On
       UNIX and similar systems, you can issue `man perlgpl' or `man per-
       lartistic' to read these licenses.

       You should have received a copy of the GNU General Public License along
       with this program; if not, write to the Free Software Foundation, Inc.,
       59 Temple Place, Suite 330, Boston, MA  02111-1307  USA.

       Execute innotop and press '!' to see this information at any time.

AUTHOR
       Baron Schwartz.

BUGS
       You can report bugs, ask for improvements, and get other help and sup-
       port at <http://sourceforge.net/projects/innotop>.  There are mailing
       lists, forums, a bug tracker, etc.  Please use these instead of con-
       tacting me directly, as it makes my job easier and benefits others if
       the discussions are permanent and public.  Of course, if you need to
       contact me in private, please do.

perl v5.8.8                       2007-11-09                       INNOTOP(1p)

Time taken: 0.00057 seconds


Created with the man page lookup class by Andrew Collington, php@amnuts.com