Memory RAS Plugin Executed Tests

1. Enabled RAS memory plugin by uncommenting 'mcelog' related lines in collectd.conf.
2. Start collectd: "./collectd -C collectd.conf -f" or start as a service "service collectd start".
3. Open collectd csv path, like: "collectd/csv/<DUT>/…".
4. Stop collectd: "pkill collectd" or "service collectd stop". Comment out RAS memory collectd plugin in "collectd.conf" file (mcelog). Delete existing collectd csv files under "collectd/csv" path. Start collectd.
5. Stop collectd: "pkill collectd" or "service collectd stop". Uncomment RAS memory collectd plugin in "collectd.conf" file (mcelog). Start collectd.

1. File is changed.
2. Verify collectd is running: "pidof collectd" returns process ID or "service collectd status" service is running.
3. RAS memory collectd files exists: total and 24 hour for corrected and uncorrected errors. Collectd RAS related files are updated with interval set in "collectd.conf".
4. After collectd start collectd RAS related files are not created/updated.
5. RAS memory collectd files exists: total and 24 hour for corrected and uncorrected errors. Collectd RAS related files are updated with interval set in "collectd.conf".

1. File is changed.
2. collectd is running.
3. mcelog-SOCKET_0_CHANNEL_0_DIMM_0_DIMM_A1
   mcelog-SOCKET_0_CHANNEL_2_DIMM_0_DIMM_C1
   mcelog-SOCKET_1_CHANNEL_0_DIMM_0_DIMM_E1
   mcelog-SOCKET_1_CHANNEL_2_DIMM_0_DIMM_G1
   mcelog-SOCKET_0_CHANNEL_0_DIMM_any
   mcelog-SOCKET_0_CHANNEL_3_DIMM_0_DIMM_D1
   mcelog-SOCKET_1_CHANNEL_0_DIMM_any
   mcelog-SOCKET_1_CHANNEL_3_DIMM_0_DIMM_H1
   mcelog-SOCKET_0_CHANNEL_1_DIMM_0_DIMM_B1
   mcelog-SOCKET_0_CHANNEL_any_DIMM_any
   mcelog-SOCKET_1_CHANNEL_1_DIMM_0_DIMM_F1 mcelog-SOCKET_1_CHANNEL_any_DIMM_any
4. Files are not updated.
5. Files are updated with new values (timestamp and errors).

1. Open "collectd.conf" file to check the collectd update interval.
2. Open collectd csv path, like: "collectd/csv/<DUT>/mce_log…".
3. Change interval in "collectd.conf" to 60 (seconds). Inject few memory errors.
4. Change interval in range 1-300 seconds.  Inject few memory errors.

1. Find line "Interval     <number>".
2. RAS memory collectd files exists: total and 24 hour for corrected and uncorrected errors. RAS memory collectd files are updated with interval set in "collectd.conf".
3. RAS memory collectd files are updated every 60 seconds.
4. RAS memory collectd files are updated every set interval.

1. 10 seconds is by default.
2. Timestamps are updated every 10 second.
3. RAS memory collectd files are updated every 60 seconds.
4. Works correct for 30, 60, 300 seconds.

1. Verify collectd, mcelog are running.
2. Stop mcelog service.
3. Start mcelog service. Restart collectd if needed.
4. Terminate mcelog (pkill mcelog).
5. Restart mcelog service.
6. Repeat test three times. 

1. Collectd, mcelog are running.
2. Service mcelog is stopped. Appropriate messages are printed to syslog with correct severity by collectd RAS memory plugin.
3. Collectd and mcelog are running. RAS memory collectd files are updated with interval set in "collectd.conf".
4. Service mcelog is exited. Appropriate messages are printed to syslog (TBD) with correct severity by RAS memory collectd plugin.
5. Collectd and mcelog are running. RAS memory collectd related files are updated with interval set in "collectd.conf".
6. RAS memory collectd plugin is stopped/started, messages about this are printed. 

1. pidof mcelog, collectd: 207803, 207791
2. syslog messages:
   collectd[207791]: mcelog: Connection to socket is broken
   collectd[207791]: plugin_dispatch_notification: severity = 1; message = Connection to mcelog socket is broken.; time = 1477301194.912; host = silpixa00378251;
   collectd[207791]: plugin_read_thread: Handling `mcelog'. mcelog: mcelog_read
   collectd[207791]: mcelog: MACHINE CHECK INFO NOT AVAILABLE
   collectd[207791]: plugin_read_thread: read-function of the `mcelog' plugin took 0.000027 seconds.
   collectd[207791]: plugin_read_thread: Effective interval of the `mcelog' plugin is 30.000 seconds.
   collectd[207791]: plugin_read_thread: Next read of the `mcelog' plugin at 1477301754.617.

3. RSA memory collectd files are updated with new timestamps. After error injected to DIMM any new values are recorded.

4. systemd[1]: Stopped LSB: Machine Check Exceptions (MCE) collector & decoder.

5. pidof collectd, mcelog: 209386, 209318

PASS

(HAA-1195, Fixed)

1. Enabled RAS memory plugin by uncommenting 'mcelog' related lines in collectd.conf.
2. Start collectd: "./collectd -C collectd.conf -f" or start as a service "service collectd start".
3. Open collectd csv path, like: "collectd/csv/<DUT>/…".
4. Stop collectd: "pkill collectd" or "service collectd stop". 
5. Repeat test three times.

1. File is changed.
2. Verify collectd is running: "pidof collectd" returns process ID or "service collectd status" service is running.
3. Collectd RAS memory files exists: total and 24 hour for corrected and uncorrected errors. Collectd RAS related files are updated with interval set in "collectd.conf".
4. Verify collectd is not running: "pidof collectd" returns nothing or "service collectd status" service is stopped. Collectd RAS related files are not updated.
5. Collectd is functioning correctly. Collectd RAS memory related data is updated in time.

1. Success
2. Collectd service started, mcelog plugin init and read callback calls present in syslog.
3. Mcelog appends data to log files with defined interval.
4. Collectd service is stopped. Logs of mcelog are not updated anymore.
5. Repeating previous steps reproduces same behavior.
   
   

1. Inject a correctable memory errors.

   $ cat mytest/corrected
   CPU 0 BANK 0
   STATUS 0xcc00008000010090
   ADDR 0x0010FFFFFFF

   $ ./mce-inject mytest/corrected

2. Make sure no errors are injected. Wait for while.

3. Repeat test for other correctable memory errors.

1. Memory error recorded to total and 24 hour files for corrected errors with correct timestamp and location (node#/channel#/DIMM#) by collectd RAS memory plugin. Same number of memory errors are recoreded to mcelog and to collectd logfile.
2. No new memory errors are recorded neither to mcelog nor to csv collectd plugin files related to RAS memory.
3. Same as in step#1.

1. Error is logged to /var/log/mcelog and to collectd and values are the same. Same number of memory errors are recoreded to mcelog and to collectd logfile. No server reboot observed.
2. During idle time there's no new records in mcelog plugin log files, neither logged by mcelog itself. Unchanged value is updated to collectd with correct time interval.
3. Once new errors are injected, counting and logging occurs as expected.

1. Inject an uncorrectable non-fatalmemory error.
   $ cat mytest/uncorrected_nonfatal
   CPU 0 BANK 2
   STATUS UNCORRECTED SRAO 0xc0
   MCGSTATUS RIPV MCIP
   ADDR 0x1234
   MISC 0x8c
   RIP 0x73:0x1eadbabe
   $ ./mce-inject mytest/uncorrected_nonfatal
2. Make sure no errors are injected. Wait for while.
3. Repeat error injection three times.

1. Memory error recorded to total and 24 hour files for uncorrected errors with correct timestamp and location (node#/channel#/DIMM#) by collectd RAS memory plugin. Same number of memory errors are recoreded to mcelog and to collectd logfile.
   Note: error injection may cause a system reboot.
2. No new memory errors are recorded neither to mcelog nor to csv collectd plugin files related to RAS memory.
3. Same as in step#1.

1. Error is logged to /var/log/mcelog and to collectd and values are the same. Same number of memory errors are recoreded to mcelog and to collectd logfile. No server reboot observed.
2. During idle time there's no new records in mcelog plugin log files, neither logged by mcelog itself. Unchanged value is updated to collectd with correct time interval.
3. Once new errors are injected, counting and logging occurs as expected.

1. Inject an uncorrectable fatalmemory error.
   $ cat mytest/uncorrected_fatal
   CPU 0 BANK 2
   STATUS UNCORRECTED SRAO 0xc0
   MCGSTATUS MCIP
   ADDR 0x1234
   MISC 0x8c
   $ ./mce-inject mytest/uncorrected_fatal
2. Check server behavior.
3. Repeat step#1 again (same or different memory error).

1. Memory error recorded to total and 24 hour files for uncorrected errors with correct timestamp and location (node#/channel#/DIMM#) by collectd RAS memory plugin. Same number of memory errors are recoreded to mcelog and to collectd logfile.
   Note: error injection may cause a system reboot.
2. No new memory errors are recorded neither to mcelog nor to csv collectd plugin files related to RAS memory.
3. Same as in step#1.

1. Server is rebooted. Uncorrected error is detected by mcelog, logged by collectd after server is up against correct DIMM location. Collectd files don't preserve statistic after error injected and reboot!
   Collectd files don't preserve statistic once mcelog is restarted!
2. During idle time there's no new records in mcelog plugin log files, neither logged by mcelog itself. Unchanged value is updated to collectd with correct time interval.
3. Once new errors are injected, counting and logging occurs as expected.

1. Get prepared a server with faulty DIMM installed to specific slot.
2. Wait for expected memory errors. Check for RAS memory errors in mcelog and in collectd csv files.
3. Repeat observation for a while, overnight.

1. Start the server.
2. Errors are registered in mcelog log file and in "collectd/csv/" files with correct address: node#/channel#/DIMM#.
3. Errors are detected and MCE statistic is updated.

1. Change socket location in mcelog.conf (socket-path = /var/run/mcelog-client) and collectd.conf for mcelog plugin to other location (default: McelogClientSocket "/var/run/mcelog-client"). Restart mcelog/collectd.
2. Inject an error and check the statistic.

1. Configuration changed. Socket is created, mcelog/collectd are running.
2. Memory error recorded to total and 24 hour files for errors with correct timestamp and location (node#/channel#/DIMM#) by collectd RAS memory plugin. Same number of memory errors are recoreded to mcelog and to collectd logfile.

1. Configuration changed. Socket is created, mcelog/collectd are running.
2. Memory error recorded to total and 24 hour files for errors with correct timestamp and location (node#/channel#/DIMM#) by collectd RAS memory plugin. Same number of memory errors are recoreded to mcelog and to collectd logfile.

1. Change log file location in mcelog.conf (logfile = /var/log/newmcelog). Make sure data not updated though socket-path defined mcelog.conf.
2. Change log file location in collectd.conf for mcelog plugin to other location (McelogLogfile "/var/log/newmcelog"). Restart collectd.
3. Inject an error and check the statistic.

1. Configuration changed. Log file is created, mcelog/collectd are running.
2. Memory error notifications dispatched by exec or python plugin. Stats not updated by CSV plugin!!!

1. Log file is not created under new location.
2. Memory error notifications dispatched by exec or python plugin. Stats not updated by CSV plugin!!!

TBD

(PR's awaiting)

1. Comment out "<Plugin mcelog>" section in collectd.conf.
2. Start collectd.
3. Inject a memory error.

2. Collectd started.

Default path for socket, "McelogClientSocket" - "/var/run/mcelog-client".

Default path for log file, "McelogLogfile" - "/var/log/mcelog".

3. Mcelog reports memory error to the "/var/log/mcelog" log file correctly.

2. Collectd started. Socket is created under "/var/run/mcelog-client".

3. Mcelog reports memory error to the "/var/log/mcelog" log file, values are same as reported by collectd plugin.

1. Comment out "McelogClientSocket" field in collectd.conf.
2. Start collectd. Inject a memory error.
3. Comment out  "McelogLogfile" field in collectd.conf.
4. Restart collectd. Inject a memory error.

2.  Collectd started. Socket file is created under "/var/run/mcelog-client" location. Mcelog reports memory error to the "/var/log/mcelog" log file correctly.

4. Collectd started. Mcelog reports memory error to the "/var/log/mcelog" log file correctly.

4. Collectd started. Mcelog reports memory error to the "/var/log/mcelog" log file correctly.

1. Start mcelog, collectd.
2. Inject corrected, uncorrected non fatal and fatal errors.
3. Wait for new day started.
4. Inject corrected, uncorrected non fatal and fatal errors.

2. Memory error recorded to total and 24 hour files for corrected errors by collectd RAS memory plugin. Same number of memory errors are recoreded to mcelog and to collectd log file.

3. All memory corrected/uncorrected errors are copied from previous day to the new day (YYYY-MM-DD) for total timespan.

All memory corrected/uncorrected errors for 24h timespanpreserved values for previous day, but set to zero for a new day.

4. Memory error recorded for new day only to total and 24 hour files for corrected/uncorrected errors by collectd RAS memory plugin. Same number of memory errors are recoreded to mcelog and to collectd log file.

2. Memory error recorded to total and 24 hour files for corrected errors by collectd RAS memory plugin. Same number of memory errors are recoreded to mcelog and to collectd log file.

3. All memory corrected/uncorrected errors are copied from previous day to the new day (YYYY-MM-DD) for total timespan.

All memory corrected/uncorrected errors for 24h timespan preserved values for previous day, but set to zero for a new day.

4. Memory error recorded for new day only to total and 24 hour files for corrected/uncorrected errors by collectd RAS memory plugin. Same number of memory errors are recoreded to mcelog and to collectd log file.

1. Configure mcelog plugin to retrieve data from other socket (collectd.conf). 
2. Open a socket (using mcelog emulator).
3. Start collectd (mcelog service must be stopped).
4. Generate corrected/uncorrected errors through created socket (using mcelog emulator).

3. Collectd started.

4. Generated memory corrected/uncorrected errors are recorded correctly to specified DIMM's location. Number of corrected/uncorrected errors is same retrieved from collectd and generated.

3. Collectd started.

4. Generated memory corrected/uncorrected errors are recorded correctly against DIMM's location. Number of corrected/uncorrected errors is same retrieved from collectd and generated.

1. Enable and configure exec plugin:
   LoadPlugin exec
   <Plugin exec>
            NotificationExec "test" "/home/test/notify.sh" </Plugin>
2. Type in script as below (cat "/home/test/notify.sh"):
   #!/bin/bash
   while read x y
   do
       echo $x$y >> "/home/test/notifications"
   done
3. Start collectd. Wait for few intervals.
4. Inject memory corrected/uncorrected errors.
5. Repeat test with time interval in range of 1 to 60 seconds.

3. Mcelog running. Collectd started without errors in syslog.

Notification(s) recorded every time interval for corrected/uncorrected memory errors.

4. Number of notification(s) increased and is the same as retrieved by mcelog, and updated every time interval for corrected/uncorrected memory errors.

5. Number of notification(s) increased and is the same as retrieved by mcelog, and updated every time interval for corrected/uncorrected memory errors.

3. Mcelog running. Collectd started without errors in syslog.

Notification(s) recorded every time interval for corrected/uncorrected memory errors.

4. Number of notification(s) increased and is the same as retrieved by mcelog, and updated every time interval for corrected/uncorrected memory errors.

5. Number of notification(s) increased and is the same as retrieved by mcelog, and updated every time interval for corrected/uncorrected memory errors.

1. Enable and configure exec plugin to update data every 5 ms.
2. Start collectd. Wait for few intervals.
3. Inject memory corrected/uncorrected errors.

Notification(s) recorded every time interval for corrected/uncorrected memory errors.

3. Number of notification(s) increased and is the same as retrieved by mcelog, and updated every time interval for corrected/uncorrected memory errors.

Notification(s) recorded every time interval for corrected/uncorrected memory errors.

3. Number of notification(s) increased and is the same as retrieved by mcelog, and updated every time interval for corrected/uncorrected memory errors.

Time needed for notification been operated:

corrected - 79-116=37ms / 108-148=40ms

uncorrected - 134-160=26ms / 162-188=26ms

1. Set PersistentNotification to True (collectd.conf)
2. Enable and configure exec plugin:
   LoadPlugin exec
   <Plugin exec>
            NotificationExec "test" "/home/test/notify.sh" </Plugin>
3. Type in script as below (cat "/home/test/notify.sh"):
   #!/bin/bash
   while read x y
   do
       echo $x$y >> "/home/test/notifications"
   done
4. Start collectd. Wait for few intervals.
5. Inject memory corrected and uncorrected errors.

4. Mcelog running. Collectd started without errors in syslog.

5. Notifications recorded every time interval for corrected and uncorrected memory errors.

5. Notifications recorded every time interval for corrected and uncorrected memory errors.

Pass

(dev testing on branch feat_mcelog_mem_notification_level)

1. Set PersistentNotification to False (collectd.conf)
2. Enable and configure exec plugin:
   LoadPlugin exec
   <Plugin exec>
            NotificationExec "test" "/home/test/notify.sh" </Plugin>
3. Type in script as below (cat "/home/test/notify.sh"):
   #!/bin/bash
   while read x y
   do
       echo $x$y >> "/home/test/notifications"
   done
4. Start collectd. Wait for few intervals.
5. Inject memory corrected and uncorrected errors.

4. Mcelog running. Collectd started without errors in syslog.

5. Notifications recorded only once per each error injection for corrected and uncorrected memory errors.

5. Notifications recorded only once per each error injection for corrected and uncorrected memory errors.

Pass

(dev testing on branch feat_mcelog_mem_notification_level)

1. Enable McelogClientSocket and McelogLogfile options.
   <Plugin mcelog>
      <Memory>
        McelogClientSocket "/var/run/mcelog-client"
      </Memory>
      McelogLogfile “/var/log/mcelog"
    </Plugin>
2. Start collectd.

2. Enabling memory socket and log file is prohibited. Error should be received and plugin exited.

2. Enabling memory socket and log file is prohibited. Error should be received and plugin exited. Collectd failed on config stage, no other plugins loaded. 
ERROR: mcelog: Invalid configuration option: "McelogLogfile", Memory option is already configured.

Pass

(dev testing on branch feat_mcelog_mem_notification_level)

1. Define mcelog as:
   <Plugin mcelog>
      McelogLogfile "/var/log/mcelog" </Plugin>

2. Start collectd.

3. Inject memory corrected and uncorrected errors.

2. Mcelog running. Collectd started without errors in syslog.

1. Define mcelog as:
   <Plugin mcelog>
      <Memory>
        PersistentNotification false
      </Memory>
      McelogLogfile "/var/log/mcelog"
   </Plugin>
2. Start collectd.
3. Send to log file corrected/uncorrected memory errors (need to define error format).
4. Set PersistentNotification to True in collectd.conf. Restart collectd.

3. Notification about errors are sent once per injection of corrected/uncorrected error.

1. Define collectd mcelog plugin:
   <Plugin mcelog>
      <Memory>
        McelogClientSocket "/var/run/mcelog-client"
        PersistentNotification false
      </Memory>
   </Plugin>
2. Start collectd.
3. Inject corrected memory errors.
4. Inject uncorrected non-fatal memory errors.

2. Mcelog running. Collectd started without errors in syslog.

3. Notification about corrected error is sent with Warning severity.

4. Notification about uncorrected error is sent with Failure severity.

1. Define collectd mcelog plugin.
   <Plugin mcelog>
      McelogLogfile "/var/log/mcelog"
   </Plugin>
2. Start collectd.
3. Inject corrected memory errors.
4. Inject uncorrected non-fatal memory errors.

2. Mcelog running. Collectd started without errors in syslog.

3. Notification about corrected error is sent with Warning severity.

4. Notification about uncorrected error is sent with Failure severity.

1. Run collectd with enabled mcelog and snmp-agent plugins.

2. Stop snmpd: service snmpd stop

User cant sent snmpwalk

Message appears in log/syslog :"Warning: Failed to connect to the agentx master agent ([NIL])"

1. Stop mcelog: service mcelog stop

2. Run collectd with enabled snmp-agent and mcelog plugin.

Error raises: "Failed to connect to mcelog server. Connection refused"

1. Run collectd with enabled snmp-agent and mcelog plugin.

2. Restart mcelog service.

3. Trigger another count of errors.

4. Verify that mcelog snmp values are equal to triggered errors count.

1. Run collectd with enabled snmp-agent and mcelog plugin.
2. Trigger another errors using mce-inject tool.
3. Restart snmpd service
4. Verify that snmp_agent is resumed.