Projects

marathonmq

Lightweight Messaging

dhruva

AboutDhruva is a web performance testing tool, completely developed in Python. with an easy-to-use GUI interface and a powerful python script recorder, this tool will be useful to those new to the performance testing area while not compromising on the customizability of the tests. Home ... [More] Pagehttp://www.testingperspective.com/tptools FeaturesMulti-Threaded Load Generator IE Session Recorder wxPython based GUI Python Script Recorder In-Built Script Editor Run-time Performance Monitoring HTML Report Generator [Less]

pyrfcon

AboutPyrfcon provides the ability to monitor performance counters data for process and its threads. You can use this module on Windows and Linux. In addition to standard set of counters for process and thread, you can observe two extra counters: cpuUsage (for process and thread) and memUsage (only ... [More] for process). Supported countersStandard countersFor Windows: Process: time, % Processor Time, % User Time, % Privileged Time, Virtual Bytes Peak, Virtual Bytes, Page Faults/sec, Working Set Peak, Working Set, Page File Bytes Peak, Page File Bytes, Private Bytes, Thread Count, Priority Base, Elapsed Time, ID Process, Creating Process ID, Pool Paged Bytes, Pool Nonpaged Bytes, Handle Count, IO Read Operations/sec, IO Write Operations/sec, IO Data Operations/sec, IO Other Operations/sec, IO Read Bytes/sec, IO Write Bytes/sec, IO Data Bytes/sec, IO Other Bytes/sec Thread: time, % Processor Time, % User Time, % Privileged Time, Context Switches/sec, Elapsed Time, Priority Current, Priority Base, Start Address, Thread State, Thread Wait Reason, ID Process, ID Thread For Linux: Process: time, pid,comm, state, ppid, pgrp, session, tty_nr, tpgid, flags, minflt, cminflt, majflt, cmajflt, utime, stime, cutime, cstime, priority, nice, num_threads, itrealvalue, starttime, vsize, rss, rsslim, startcode, endcode, startstack, kstkesp, kstkeip, signal, blocked, sigignore, sigcatch, wchan, nswap, cnswap, exit_signal, processor, rt_priority, policy, delayacct_blkio_ticks, guest_time, cguest_time, size, resident, share, trs, lrs, drs, dt Thread: time, pid,comm, state, ppid, pgrp, session, tty_nr, tpgid, flags, minflt, cminflt, majflt, cmajflt, utime, stime, cutime, cstime, priority, nice, num_threads, itrealvalue, starttime, vsize, rss, rsslim, startcode, endcode, startstack, kstkesp, kstkeip, signal, blocked, sigignore, sigcatch, wchan, nswap, cnswap, exit_signal, processor, rt_priority, policy, delayacct_blkio_ticks, guest_time, cguest_time Extra countersFor Windows: cpuUsage: % Processor Time is the percentage of elapsed time that all of process' threads used the processor to execution instructions. An instruction is the basic unit of execution in a computer, a thread is the object that executes instructions, and a process is the object created when a program is run. Code executed to handle some hardware interrupts and trap conditions are included in this count. memUsage: Private Bytes is the current size, in bytes, of memory that this process has allocated that cannot be shared with other processes. For Linux: cpuUsage: 100 * (jiffies_delta / HZ) / (time_delta / 1e6) time_delta = time(t) - time(t0) jiffies_delta = (utime(t) + stime(t)) - (utime(t0) + stime(t0)) utime: the number of jiffies that this process has been scheduled in user mode stime: the number of jiffies that this process has been scheduled in kernel mode HZ: the number of ticks per second t0,t: two serial time measurements memUsage: vsize/fullmemsize vsize: virtual memory size in bytes fullmemsize: full memory size Module usageModule usage is pretty simple. You can find a working example in example.py. Here you'll find detailed step-by-step analysis of this example with comments on usage. You can start new process for monitoring or attach to already running one. To start new process: import subprocess process = subprocess.Popen(sys.argv[1:4]) pid = process.pid # Save process PID for later usageTo attach to already running process: pid = 228 # PID of existing processTo start monitoring of a process and its threads, you must first create MonitorThread object. MonitorThread is inherited from the standard Thread class and is used to actually collect performance data in background. By default Monitor Thread saves the process info every 0.5 seconds, but you can set necessary update interval (in seconds): import pyrfcon updateInterval = 1 # 1 second update period thread = pyrfcon.MonitorThread(pid, updateInterval)To start the thread’s activity, you must call start() function. This function arranges for the object’s run() method to be invoked in a separate thread of control: thread.start()You can use getData() function to poll for performance counters updates or callbacks to get counters data as they change. startCollectData() starts collecting data for getData() function: thread.startCollectData()getData() returns collected data since startCollectData() call or last getData() call: thread.getData()stopCollectData() stops collect data for getData() function: thread.stopCollectData()setCallback(function) sets callback function: thread.setCallback(function)clearCallback() clears callback: thread.clearCallback()By default getData() returns standard set of counters as a tuple (processData, threadsData). processData: process data numpy array (dtype = typeProc) threadsData: threads data dictionary consists of thread data numpy arrays (dtype = typeThread). Threads IDs are used as a keys. To limit number of monitored counters you can construct different queries from standard and extra (cpuUsage, memUsage) counters. procQuery = ['time', 'Working Set' , 'cpuUsage', 'memUsage'] threadsQuery= ['time', 'cpuUsage', 'memUsage']You can use this queries for getData() to return only necessary set of counters. Popen.wait() waits for child process to terminate. process.wait()setProcess(process) starts or stops collect data. To start collecting data pass process = True, and to stop collecting data pass process = False: thread.setProcess(False) # Stop collecting datajoin(timeout) waits until the thread terminates. This blocks the calling thread until the thread whose join() method is called terminates – either normally or through an unhandled exception – or until the optional timeout occurs. thread.join() [Less]

mysql-search

Purpose The script filters the MySQL Slow Query Log to show queries which impacted performance most and is intended to be used by DB admins and application developers. The log file is analyzed and processed as a stream, line after line, so there is no need to load the whole log file into memory. ... [More] --no-duplicates is a very useful option to see only necessary statistics. --incremental remembers last input file positions and statistics in a SQLite 3 database, so periodical executions on the same files run much faster. The Python version is usually 3-5 times faster than the PHP5 version. You are welcome to contribute a version translated into your favorite scripting language, just open a feature request under Issues i.e. to add a Perl version. Usage Examples # Filter slow queries executed for at least 3 seconds not from root, remove duplicates, # apply execution count as first sorting value and save first 10 unique queries to file. # In addition, remember last input file position and statistics. php mysql_filter_slow_log.php -T=3 -eu=root --no-duplicates --sort-execution-count --top=10 --incremental linux-slow.log > mysql-slow-queries.log # Start permanent filtering of all slow queries from now on: at least 3 seconds or examining 10000 rows, exclude users root and test tail -f -n 0 linux-slow.log | python mysql_filter_slow_log.py -T=3 -R=10000 -eu=root -eu=test & # (-n 0 outputs only lines generated after start of tail) # Stop permanent filtering kill ps auxww | grep 'tail -f -n 0 linux-slow.log' | egrep -v grep | awk '{print $2}' Filter Options -T=min_query_time -R=min_rows_examined -ih, --include-host -eh, --exclude-host -iu, --include-user -eu, --exclude-user -iq, --include-query --date=date_first-date_last Include only queries between date_first (and date_last). Input: Date Range: .11.2006 -> 13.11.2006 - 14.11.2006 (exclusive) .11.2006-15.11.2006 -> 13.11.2006 - 16.11.2006 (exclusive) -11-2006-11/13/2006 -> 13.11.2006 - 16.11.2006 (exclusive) >13.11.2006 -> 14.11.2006 - later .11.2006- -> 13.11.2006 - later earlier - 13.11.2006 (exclusive) -13.11.2006 -> earlier - 14.11.2006 (exclusive) Please do not forget to escape the greater or lesser than symbols (>= 5.1.21 (or patched): 3 seconds = 3000000 microseconds # long_query_time=3.000000 # minimum: 0.000001 (1 microsecond) # Activate the Slow Query Log slow_query_log # >= 5.1.29 # log-slow-queries # deprecated since 5.1.29 # Write to a custom file name (>= 5.1.29) # slow_query_log_file=file_name # default: /data_dir/host_name-slow.log # Log all queries without indexes # log-queries-not-using-indexes # Log only queries which examine at least N rows (>= 5.1.21) # min_examined_row_limit=1000 # default: 0 # Log slow OPTIMIZE TABLE, ANALYZE TABLE, and ALTER TABLE statements # log-slow-admin-statements # Log slow queries executed by replication slaves (>= 5.1.21) # log-slow-slave-statements # MySQL 5.1.6 through 5.1.20 had a default value of log-output=TABLE, so you should force # Attention: logging to TABLE only includes whole seconds information log-output=FILE ## Admin query for online activation is possible since MySQL 5.1 (without server restart) ## SET @@global.slow_query_log=1 ## SET @@global.long_query_time=1 ## Show current variables related to the Slow Query Log ## SHOW GLOBAL VARIABLES WHERE Variable_name REGEXP 'admin|min_examined|log_output|log_queries|log_slave|long|slow_quer' [Less]

PerfMetrics

PerfMetrics is a free/open source performance monitoring system for Windows. It collects, stores, and graphs performance counters from remote Windows machines. It uses WMI to collect server stats and RRDTool for data storage and graphing.

red-stripe

Web Site Profiling

gst-viewperf

AboutThis tool aims at grabbing and displaying performance indicators of a gstreamer pipeline to ease bottleneck debugging. How it worksIt displays performance indicators such as: queue filling states videorate statistics plugin-relative time reference (using progressreport) How to use itJust ... [More] place queues after elements you wish to track the status of, and the colors and statistics will help you understand what is happening. Example reasons of bad performance are: stuck pipeline: an element is blocking the chain (which is often the case with display sinks without queues) cpu is saturated (often encoding elements) In any case, it is advised to use queue and videorate right after your source. Screenshot How to installDependancies: gstreamer python bindings gstmanager TouchWizard [Less]

selenium-profiler

Automated Web/HTTP Profiler with Selenium-RC and Pythonby Corey Goldberg (c) 2009 What is it?Selenium-profiler is a web/http profiler built with Selenium-RC and Python. It profiles page load time and network traffic for a web page. The profiler uses Selenium-RC to automate site navigation (via ... [More] browser), proxy traffic, and sniff the proxy for network traffic stats as requests pass through during a page load. It is useful to answer questions like: how many http requests does that page make? how fast are the http responses coming back? which http status codes are returned? how many of each object type are requested? what is the total page load time? License:GNU GPL v3 This program is Free Open Source software Contents:web_profiler.py : main profiler program selenium.py : Selenium-RC Python client driver selenium-server.jar : Selenium server (a java based server) How do you use it?Install Python (and add it to your path) Install Java (and add it to your path) Get selenium-profiler code from Subversion Unzip Selenium RC and search for 'selenium-server.jar' and 'selenium.py' Copy them to a directory and run 'java -jar selenium-server.jar' to start the server Open web_profiler.py and set your parameters Run web_profiler.py *notes: you may need to adjust browser security settings to get it to work properly to run against an HTTPS/SSL enabled site, you need to install a fake certificate. look inside selenium-server.jar for the cert. Command Line Parameters:web_profiler.py takes 2 command line arguments: web_profiler.py [browser_launcher] Sample usage: > python web_profiler.py www.google.com *firefox (use *firefox to launch Mozilla Firefox) (use *iexplore or *iexploreproxy to launch Internet Explorer) (use *googlechrome to launch Google Chrome) Browsers/Platforms:This profiler seems to run best under MS Windows with either Firefox or IE. I have tested under various combinations with mixed success: Working: Firefox 3.5 / Windows XP Firefox 3.5 / Windows Server 2008 Internet Explorer 8 / Windows XP Internet Explorer 8 / Windows Server 2008 Google Chrome 4 / Windows XP Partially Working: Firefox 3.0 / Ubuntu Linux 9.04 Sample Output:-------------------------------- results for http://www.google.com/ content size: 31.096 kb http requests: 7 status 200: 6 status 204: 1 profiler timing: 0.344 secs (page load) 0.328 secs (network: end last request) 0.110 secs (network: end first request) file extensions: (count, size) gif: 1, 3.011 kb ico: 1, 1.150 kb js: 2, 18.083 kb png: 1, 5.401 kb unknown: 2, 3.451 kb http timing detail: (status, method, doc, size, time) 204, GET, /generate_204, 0, 62 ms 200, GET, /favicon.ico, 1150, 31 ms 200, GET, /barcode09.gif, 3011, 31 ms 200, GET, /, 3451, 110 ms 200, GET, /2cca7b2e99206b9c.js, 3451, 78 ms 200, GET, /nav_logo7.png, 5401, 16 ms 200, GET, /f_wkquEsVv8.js, 14632, 47 ms -------------------------------- [Less]

ioapps

IOapps consists of two applications: ioreplay and ioprofiler, both of them sharing common code base. The tools are intended to run under Linux. * ioreplay is mainly intended for replaying of recorded (using strace) IO traces, which is useful for standalone benchmarking. It provides many ... [More] features to ensure validity of such measurements. * ioprofiler is a GUI application for profiling application's IO access pattern. It shows all the read/written files by the traced application, how many reads were performed, how much data were read and how much time was spent reading for each file. Most importantly, it produces nice diagrams showing read/write access pattern (i.e. whether the application was reading sequentially or randomly) in how big chunks etc [Less]

PerfLog

Performance Analysis Tool for Web Server Logs. Produces metrics and graphs from web logs (W3C Extended Log File Format). This is useful during performance testing and analysis. Output is created in XHTML/CSS with embedded PNG images. PerfLog is written in Python and uses Matplotlib for graphs and plotting