Projects tagged ‘performance’ and ‘python’

KCachegrind is a KDE viewer of profiling data generated by Callgrind.

Pocoo is an open-source bulletin board software (aka message board) written in Python. It provides an advanced plugin system with a component architecture which allows other developers to modify Pocoo ... [More] to their liking without the need to touch existing source code. Because it uses SQLAlchemy, it is possible to use either MySQL, SQLite, Oracle or Postgres as the storage backend. [Less]

Psyco is a Python extension module which can massively speed up the execution of any Python code.

Pylot is a free open source tool for testing performance and scalability of web services. It runs HTTP load tests, which are useful for capacity planning, benchmarking, analysis, and system tuning. ... [More] Pylot generates concurrent load (HTTP Requests), verifies server responses, and produces reports with metrics. Tests suites are executed and monitored from a GUI. [Less]

Shed Skin is an experimental compiler, that can translate pure, but implicitly statically typed Python programs into optimized C++. It can generate stand-alone programs or extension modules that can ... [More] be imported and used from larger Python programs. Besides the typing restriction, programs cannot freely use the Python standard library (although about 17 common modules, such as random and re, are currently supported). Also, not all Python features, such as nested functions and variable numbers of arguments, are supported. For a set of 40 non-trivial test programs (at over 12,000 lines in total; see the shedskin-examples-0.2.tgz download), measurements show a typical speedup of 2-40 times over Psyco, and 2-220 times over CPython. Because Shed Skin is still in an early stage of development, however, many other programs will not compile out-of-the-box. Shed Skin consists of only 7,000 lines of Python code, and about 20,000 lines of C++ code (implementation of builtins and library modules). Unfortunately, there is still only one active developer, so please consider joining the project! [Less]

Performance Analysis Tool for Web Server Logs. Produces metrics and graphs from web logs (W3C Extended Log File Format). This is useful during performance testing and analysis. Output is created ... [More] in XHTML/CSS with embedded PNG images. PerfLog is written in Python and uses Matplotlib for graphs and plotting [Less]

DBUtils is a suite of tools providing solid, persistent and pooled connections to a database that can be used in all kinds of multi-threaded environments like Webware for Python or other web ... [More] application servers. The suite supports DB-API 2 compliant database interfaces and the classic PyGreSQL interface. [Less]

PerfMetrics is a free/open source performance monitoring system for Windows. It collects, stores, and graphs performance counters from remote Windows machines. It uses WMI to collect server stats and RRDTool for data storage and graphing.

Lets you see your Python program's execution as a tree of function invocations, each tree node exposing the real time, and CPU time (user/sys) of that call. This project consists of two main ... [More] components: A Python tracer that can run your Python programs (much like "cProfile" and friends). A Gtk+ based GUI that can show the trace results. It uses a tiny auxiliary library written for it "graphfile" to allow append-only writing and reading static DAG's directly from file without reading it whole into memory at any stage. [Less]

AboutPyrfcon provides the ability to monitor performance counters data for process and its threads. You can use this module on Windows and Linux. In addition to standard set of counters for process ... [More] and thread, you can observe two extra counters: cpuUsage (for process and thread) and memUsage (only for process). Supported countersStandard countersFor Windows: Process: time, % Processor Time, % User Time, % Privileged Time, Virtual Bytes Peak, Virtual Bytes, Page Faults/sec, Working Set Peak, Working Set, Page File Bytes Peak, Page File Bytes, Private Bytes, Thread Count, Priority Base, Elapsed Time, ID Process, Creating Process ID, Pool Paged Bytes, Pool Nonpaged Bytes, Handle Count, IO Read Operations/sec, IO Write Operations/sec, IO Data Operations/sec, IO Other Operations/sec, IO Read Bytes/sec, IO Write Bytes/sec, IO Data Bytes/sec, IO Other Bytes/sec Thread: time, % Processor Time, % User Time, % Privileged Time, Context Switches/sec, Elapsed Time, Priority Current, Priority Base, Start Address, Thread State, Thread Wait Reason, ID Process, ID Thread For Linux: Process: time, pid,comm, state, ppid, pgrp, session, tty_nr, tpgid, flags, minflt, cminflt, majflt, cmajflt, utime, stime, cutime, cstime, priority, nice, num_threads, itrealvalue, starttime, vsize, rss, rsslim, startcode, endcode, startstack, kstkesp, kstkeip, signal, blocked, sigignore, sigcatch, wchan, nswap, cnswap, exit_signal, processor, rt_priority, policy, delayacct_blkio_ticks, guest_time, cguest_time, size, resident, share, trs, lrs, drs, dt Thread: time, pid,comm, state, ppid, pgrp, session, tty_nr, tpgid, flags, minflt, cminflt, majflt, cmajflt, utime, stime, cutime, cstime, priority, nice, num_threads, itrealvalue, starttime, vsize, rss, rsslim, startcode, endcode, startstack, kstkesp, kstkeip, signal, blocked, sigignore, sigcatch, wchan, nswap, cnswap, exit_signal, processor, rt_priority, policy, delayacct_blkio_ticks, guest_time, cguest_time Extra countersFor Windows: cpuUsage: % Processor Time is the percentage of elapsed time that all of process' threads used the processor to execution instructions. An instruction is the basic unit of execution in a computer, a thread is the object that executes instructions, and a process is the object created when a program is run. Code executed to handle some hardware interrupts and trap conditions are included in this count. memUsage: Private Bytes is the current size, in bytes, of memory that this process has allocated that cannot be shared with other processes. For Linux: cpuUsage: 100 * (jiffies_delta / HZ) / (time_delta / 1e6) time_delta = time(t) - time(t0) jiffies_delta = (utime(t) + stime(t)) - (utime(t0) + stime(t0)) utime: the number of jiffies that this process has been scheduled in user mode stime: the number of jiffies that this process has been scheduled in kernel mode HZ: the number of ticks per second t0,t: two serial time measurements memUsage: vsize/fullmemsize vsize: virtual memory size in bytes fullmemsize: full memory size Module usageModule usage is pretty simple. You can find a working example in example.py. Here you'll find detailed step-by-step analysis of this example with comments on usage. You can start new process for monitoring or attach to already running one. To start new process: import subprocess process = subprocess.Popen(sys.argv[1:4]) pid = process.pid # Save process PID for later usageTo attach to already running process: pid = 228 # PID of existing processTo start monitoring of a process and its threads, you must first create MonitorThread object. MonitorThread is inherited from the standard Thread class and is used to actually collect performance data in background. By default Monitor Thread saves the process info every 0.5 seconds, but you can set necessary update interval (in seconds): import pyrfcon updateInterval = 1 # 1 second update period thread = pyrfcon.MonitorThread(pid, updateInterval)To start the thread’s activity, you must call start() function. This function arranges for the object’s run() method to be invoked in a separate thread of control: thread.start()You can use getData() function to poll for performance counters updates or callbacks to get counters data as they change. startCollectData() starts collecting data for getData() function: thread.startCollectData()getData() returns collected data since startCollectData() call or last getData() call: thread.getData()stopCollectData() stops collect data for getData() function: thread.stopCollectData()setCallback(function) sets callback function: thread.setCallback(function)clearCallback() clears callback: thread.clearCallback()By default getData() returns standard set of counters as a tuple (processData, threadsData). processData: process data numpy array (dtype = typeProc) threadsData: threads data dictionary consists of thread data numpy arrays (dtype = typeThread). Threads IDs are used as a keys. To limit number of monitored counters you can construct different queries from standard and extra (cpuUsage, memUsage) counters. procQuery = ['time', 'Working Set' , 'cpuUsage', 'memUsage'] threadsQuery= ['time', 'cpuUsage', 'memUsage']You can use this queries for getData() to return only necessary set of counters. Popen.wait() waits for child process to terminate. process.wait()setProcess(process) starts or stops collect data. To start collecting data pass process = True, and to stop collecting data pass process = False: thread.setProcess(False) # Stop collecting datajoin(timeout) waits until the thread terminates. This blocks the calling thread until the thread whose join() method is called terminates – either normally or through an unhandled exception – or until the optional timeout occurs. thread.join() [Less]