Tech/HowTo/Monitoring and Munin
Don’t just page, track those events.
Quis custodiet ipsos custodes? -Juvenal
or translated
Who will watch the watchmen
Monitoring systems and devices is a large idea from physical, environmental and status reporting. Some tools like Nagios are really good at alerting/paging on issue but lack at showing trends. For something as important as UPTIME you need to be using many tools.
Trending VS Alerting
Data, Data, Data How was this last week? Did this ever work?
Nagios has been alarming for 473 days, I wonder if this system is still working?
Munin
Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.
The Name
In Norse mythology Hugin and Munin are the ravens of the god king Odin. They flew all over Midgard for him, seeing and remembering, and later telling him. "Munin" means "memory".
http://hugin.sourceforge.net/ <<< cool stuff
Munin Features
- Fast web based access to graphed data
- Zoomable, scalable graphs
- Data
- Custom Graphs on the fly
- Data
- Historical reports
RRDtool
What RRDtool does: RRDtool is the OpenSource industry standard, high performance data logging and graphing system for time series data. RRDtool can be easily integrated in shell scripts, perl, python, ruby, lua or tcl applications.
Munin Plugins
amavis freeradius_proxy_acct munin_stats pm3users_ sendmail_mailqueue snmp__sensors_mbm_fan apache freeradius_proxy_auth munin_update pop_stats sendmail_mailstats snmp__sensors_mbm_temp apc_envunit_ haproxy_ mysql_ postfix_mailqueue sendmail_mailtraffic snmp__sensors_mbm_volt apc_nis haproxy_ng mysql_bytes postfix_mailstats slapd_ snmp__swap asterisk hddtemp mysql_innodb postfix_mailvolume slapd_bdb_cache_ snmp__uptime bind9 hddtemp2 mysql_isam_space_ postgres_autovacuum slony_lag_ snmp__users bind9_rndc hddtempd mysql_queries postgres_bgwriter smart_ snmp__winload cmc_tc_sensor_ hddtemp_smartctl mysql_slowqueries postgres_cache_ snmp__apc_ups snmp__winmem courier_ hp2000_ mysql_threads postgres_checkpoints snmp__cpuload snort_alerts courier_mta_mailqueue http_loadtime named postgres_connections_ snmp__df snort_bytes_pkt courier_mta_mailstats ifx_concurrent_sessions_ netopia postgres_connections_db snmp__df_ram snort_drop_rate courier_mta_mailvolume ipac-ng netstat postgres_locks_ snmp__eltek_rectifier snort_pattern_match cupsys_pages ipmi_ nginx_request postgres_oldest_prepared_xact_ snmp__fc_if_ snort_pkts debug_sincos ipmi_sensor_ nginx_status postgres_prepared_xacts_ snmp__fc_if_err_ snort_traffic df ircu nomadix_users_ postgres_querylength_ snmp__if_ spamstats df_inode jenkins_builds ntp_ postgres_scans_ snmp__if_err_ squeezebox_ dhcpd3 jenkins_builds_results ntp_kernel_err postgres_size_ snmp__if_multi squid digitemp_ jenkins_builds_results_summary ntp_kernel_pll_freq postgres_streaming_ snmp__load surfboard ejabberd_ jenkins_builds_running ntp_kernel_pll_off postgres_transactions_ snmp__memory tomcat_ example-graphs jenkins_jobs ntp_offset postgres_tuples_ snmp__netapp_diskusage_ tomcat_access exim_mailqueue loggrep ntp_states postgres_users snmp__netapp_inodeusage_ tomcat_jvm exim_mailqueue_alt lpstat nut_misc postgres_xlog snmp__netstat tomcat_threads exim_mailstats mailman nutups_ processes snmp__print_pages tomcat_volume external_ mailscanner nut_volts proxy_plugin snmp__print_supplies users fail2ban mbmon_ nvidia_ ps_ snmp__processes varnish_ foldingathome memcached_ openvpn psu_ snmp__rdp_users vmstat foldingathome_rank mhttping perdition qmailqstat snmp__sensors_fsc_bx_fan zimbra_ foldingathome_wu multiping pgbouncer_connections qmailscan snmp__sensors_fsc_bx_temp freeradius_acct multips pgbouncer_requests qmailscan-simple snmp__sensors_fsc_fan freeradius_auth multips_memory ping_ samba snmp__sensors_fsc_temp
I could keep listing things but it would take forever.
Installation
Installation can be as simple as:
apt install munin-node munin
List of packages on Debian at time of writing
munin - network-wide graphing framework (grapher/gatherer) munin-async - network-wide graphing framework (async master/client) munin-common - network-wide graphing framework (common) munin-doc - network-wide graphing framework (documentation) munin-libvirt-plugins - Munin plugins using libvirt munin-node - network-wide graphing framework (node) munin-node-c - C rewrite of Munin node munin-plugins - munin-plugins-btrfs - munin plugin for monitoring btrfs filesystems munin-plugins-c - C rewrite of some Munin plugins munin-plugins-core - network-wide graphing framework (plugins for node) munin-plugins-extra - network-wide graphing framework (user contributed plugins for node) munin-plugins-java - network-wide graphing framework (java plugins for node)
Installation doc is very good and I would give it a read.
Remember folks, PHBs love data
There are many tools to export the Munin/RRD data if you want the raw data. Use the custom graph generator to make cool graphs.
Scale
Highly scalable note from dev
Beware of shared hardware Munin loves to annihilate any hardware It is designed to be highly scalable... ... but not in a very efficient manner