Projects tagged ‘benchmarking’, ‘performance’, and ‘testing’
Projects tagged ‘benchmarking’, ‘performance’, and ‘testing’
Filtered by
benchmarking
performance
testing
Refine results
database (3)
continuous-performance-testing (3)
performance-testing (3)
automation (3)
data_generation (2)
oracle (2)
data (2)
framework (2)
regression-testing (2)
test (2)
development (2)
tool (2)
[6 total ]
1 Users
- Project
- View
- |
- Download
- |
- Code Metrics
benerator is a framework for creating realistic and valid high-volume test data, used for testing (unit/integration/load) and showcase setup.
Metadata constraints are imported from systems and/or
... [More]
configuration files. Data can be imported from and exported to files and systems, anonymized or generated from scratch. Domain packages provide reusable generators for creating domain-specific data as names and addresses internationalizable in language and region. It is strongly customizable with plugins and configuration options. [Less]
Created
about 1 year
ago.
0 Users
- Project
- View
- |
- Download
- |
- Code Metrics
With DBFeeder you can generate testdata for Oracle Databases which fits primary and foreign keys of tables. A file-based configuration systems allows in-depth customization of the type of data which is generated.
Created
about 1 year
ago.
0 Users
- Project
- View
- |
- Code Metrics
A set of scripts for easy benchmarking of VIFF protocols.
ViffBench is a set of scripts that makes it easy to write, execute, and visulize benchmarks of VIFF protocols. Features include
*
... [More]
Benchmarking using arbitrary VIFF revisions.
* Based on ssh and thus benefits from key agents like ssh-agent.
* Benchmarking of distributed VIFF protocols as well as local VIFF code.
* Automatically stores benchmark results in central database.
* Stores benchmark results for every single run, e.g. no info is lost. [Less]
Created
10 months
ago.
0 Users
- Project
- View
Tryouts - v0.8 BETATryouts is a high-level testing library (DSL) for your Ruby codes and command-line applications.
A tryout is made up of one of more drills. The return value of the drill block is
... [More]
compared to the expectations defined by one or more dreams. The goal is to achieve your dreams.
TerminologyTryout: a set of drills (think: basketball tryouts) Drill: a single test. Drill Sergeant: The class responsible for executing a drill. Dream: the expected outcome of a drill. There's always one or more dream per drill. Installation $ gem install tryoutsWriting TestsThe examples below are a complete overview of Tryouts syntax.
Testing Ruby Codes (:api) library :gibbler, "../path/to/gibbler/lib"
tryouts "Common Usage", :api do
# Define variables available only within
# in the current tryout definition.
set :local_variable, 3
setup do
# do stuff before all drills
end
clean do
# do stuff after all drills
end
# This drill block should return 3.
drill "Maths R Us", local_variable do
12 / 4
end
# You can specify a method to execute
# on the return value of the drill block.
drill "We want a symbol", :class, Symbol do
Class.methods.first
end
# Dreams can also be specified explicitly which is
# important b/c it's possible to specify multiple.
dream :class, Array
dream ['b', 'c', 'd']
drill "Should return a list of 3" do
letters = 'a'..'z'
letters.to_a[1..local_variable]
end
# Drills can pass based on an exception too.
dream :exception, NameError
drill "Something failed" do
raise NameError
end
# We can even put simple drills on a single line.
drill 'Small, fast, and furious', 'Muggsy Bogues', :match, /Mug+sy Bogu?es/
endTesting Command-Line Applications (:cli)You can execute drills on command-line applications.
command :rudy, "path/2/rudy"
tryouts "Rudy CLI", :cli do
# We have specified the name of the command above so
# we only need to specify the arguments in the drill.
# $ rudy myaddress -e
drill "can display my IP addresses", :myaddress, :e
# The drill returns an Array of lines from STDOUT so
# we can use Array#grep to check the command output.
# If any lines match, this test will pass.
dream :grep, /No machines running/
drill "can call with no arguments"
# Drills can also be defined with a block which gets
# executed via Rye::Box#batch. Methods in the block
# are executed as commands. The return value of the
# block is considered the test output.
dream :match, /\d+\.\d+\.\d+\.\d+/
drill "can execute a block of commands" do
ret = rudy :q, :myaddress, :e
ret.first
end
endRunning Performance Benchmarks (:benchmark)You can also use Tryouts to run benchmarks. The tryouts method takes a second parameter which specifies which drill sergeant to use. Below we specify :benchmark so each drill is timed and executed multiple times.
tryouts "Benchmark examples", :benchmark do
# We create an Array and store it in a class
# variable so it's available to other drills.
drill "Create test array" do
@@array = (1..100000).map { rand }
end
dream :mean, 3.0 # The mean should be <= 3.0 seconds
dream :sdev, 0.1 # and the standard deviation <= 0.1
drill("Array sort!") { @@array.dup.sort! }
# You can also include a dream inline
drill("Array sort", :mean, 3.0) { @@array.dup.sort }
# The 3rd argument is the number of times to
# execute the drill block. The mean and sdev
# are calculate based on all iterations. The
# default is 5 and the maximum is 30.
dream :sdev, 0.1, 15
drill("Array sort") { @@array.dup.sort }
endThe drill blocks in a benchmark Tryout return Tryouts::Stats objects. See: Tryouts::Drill::Sergeant::Benchmark
Running TestsTryouts comes with an executable called sergeant. This is the drill sergeant that tells you what percentage of your dreams come true.
$ sergeant
Gibbler Gazette
Basic syntax with SHA1
"Object" PASS
"Class" PASS
...
"Knows when an Hash has changed" PASS
All 9 dreams came true The sergeant looks in the current working directory for a tryouts directory and will automatically load all files ending in tryouts.rb.
$ ls -l tryouts/
01_mixins_tryouts.rb
10_syntax_tryouts.rb
20_cli_tryouts.rb
30_benchmark_tryouts.rb
50_class_context_tryouts.rbYou can also specify specific files to load:
$ sergeant any/file.rbVerbose mode gives you some extra information, including the return values from the drills. The more v's, the more information:
$ sergeant -vIn quiet mode, the sergeant returns only a PASS or FAIL message (in the case of a FAIL, it will also return a non-zero exit code):
$ sergeant -q
PASSScreenshots:api Tryouts
from gibbler
:cli Tryouts (verbose)
from tryouts
:benchmark Tryouts (verbose)
from gibbler
News2009-07-20: New DSL keyword "set"The "set" keyword has been introduced in 0.8.4 for defining variables available only to the Tryout definition they were declared in. See Example 1.
2009-07-01: Drills for command-line applicationsSupport for testing command-line applications has been re-introduced in 0.8. See example code further down in this README.
BETA NoticeTryouts is very new (est. 2009-05-19) and has not been vetted by the scrutiny of time. In particular you can expect:
The test definition syntax may change in future releases. Unexpected errors. Bugs! I love fixing bugs so if you find one let me know. On ThreadsTryouts does some funky stuff to make it simple to write tests. This "funky stuff" means that this library is not thread-safe at definition-time. However, once all tryouts files are parsed (or in OO-syntax, once all objects are created), this class should be thread-safe at drill-time.
More InfoCheck out the sourcecodes Read the rdocs Inspiration ThanksEveryone at Utrecht.rb and Amsterdam.rb for putting up with my Ruby questions :] Sam Aaron (http://sam.aaron.name) for the early feedback. Kalin Harvey for the encouragement. Credits Delano (@solutious.com)
Related ProjectsContext Testy Expectations Zebra LicenseSee: LICENSE.txt [Less]
Created
4 months
ago.
0 Users
- Project
- View
Why would I use this rather than tools like jMeter or jUnitPerf?
There are a variety of different performance testing tools available in the Java world, some are more useful than others for
... [More]
different tasks, but we have run into a number of limitations. We have found that both load testing and micro-benchmarking tools are not designed with generalized performance benchmarking in mind. To be fair, they also don't intend to be generalized benchmarking tools, which is good. A benchmark is not necessarily a load test, a micro-benchmark test, or any kind of test. Load testing tools like jMeter take the client-server black-box approach; driven by the model of load testing the server side of web apps. Micro-benchmarking tools like jUnitPerf are test method approach; driven by the model of the annotation based mechanism that the unit test framework uses. This forces the performance test to include setup and tear-down times since it does not allow special setup hooks for the benchmark that are not included in the timings of the benchmark.
From: http://clarkware.com/software/JUnitPerf.html#limitations
JUnitPerf has the following known limitations:
The elapsed time measured by a TimedTest decorating a single testXXX() method of a TestCase includes the total time of the setUp(), testXXX(), and tearDown() methods, as this is the granularity offered by decorating any Test instance. The expected elapsed time measurements should be adjusted accordingly to account for the set-up and tear-down costs of the decorated test. You can check out some examples of jUnitPerf style tests here: http://clarkware.com/software/JUnitPerf.html#howtouse
What is cool about speedy mcbenchmark?
run a benchmark and report its performance declare a constraint that an operation must complete within a certain amount of time run a benchmark a given number of times in order to smooth out noise in individual runs compare the performance of n different concrete implementations to see which is fastest declare a constraint that a particular implementation is the fastest among a group of n concrete implementations observe how an algorithm scales relative to its theoretical "Big O" complexity and see where the scaling function breaks down more cool stuff in the backlog at the end of this doc - go check it out! How do I impress my friends with the sweet nectar of benchmarking goodness?
Let's look at different benchmarking scenarios from the code base that this benchmarking project was extracted out of. These examples come from our development of a very fast K-Way Merge algorithm to serve at the core of our discrete event simulator for trading systems. The main problem in the simulator portion of a discrete event simulation environment is to take K collections of elements that are arranged temporally (in our case, this is historical time series data from stock, commodity, and currency markets) and perform a merge so that the simulator can emit the events from all the time series in the correct order. http://www.site.uottawa.ca/~nat/Courses/DFS-Course/DFS-Lecture-9/tsld013.htm
You want to run a benchmark and report the timings.
[TestFixture, Category("Performance")]
public class PriorityQueueImplementationComparison
{
private PriorityQueue queue;
private const int YEARS = 30;
private const int TRADING_DAYS_PER_YEAR = 265;
private const int INSTRUMENTS = 10000;
private const int TIMES = 1;
public void EnumerateDataStructure()
{
var events = new Collection();
var count = 0;
while (queue.HasNext)
{
events.Add(queue.Dequeue().Object);
count++;
}
Assert.That(events.Count, Is.EqualTo(count));
}
[Test]
public void KWayMerge()
{
new Benchmark(EnumerateDataStructure)
.beforeEachRun(()=>
{
queue = new KWayMerge(FakeInstruments.produceNSeries(
INSTRUMENTS, YEARS * TRADING_DAYS_PER_YEAR));
})
.whenRun(TIMES.times()).ReportIndividualTimings();
}Notes:
you can call ReportAggreatedTimings or ReportIndividualTimings - aggregated timings aggregates timings for all runs of a benchamrk, whereas individual timings reports the timings for each individual run. Notice that EnumerateDataStructure is the name of the method that I have created a benchmark scenario in. Reports and test failures include the name of the benchmark, which, if you do not specify it in the constructor of benchmark that takes a string name, will default to the last method up the call stack - the name of your test. Create a well named test to compose your benchmarks in. If you prefer, you can just construct the benchmark with lambda and avoid creating a named method for your benchmark. the beforeEachRun() method is a hook that allows you to run a code block before each run of the benchmark. This is where you should prepare data sets and do any other work that is required to for each benchmark run, but that you do not want to include in the timings for each benchmark run. The times() extention method on the int type is a higher order loop generator function. If you are curious about it, you can find more in the section on looping constructs at the end of this doc. You have a benchmark that must complete within a specified time limit.
new Benchmark(MethodToBenchmark).Should(complete.within(2.seconds())).whenRun(10.times());Notes:
Here we are introducing the Should method, which takes a constrain, and applies the constraint to the Benchmark results after the run finishes. Note the argument to the complete.within() constraint. We have added extension methods on the int type for milliseconds, seconds, minutes, etc. You have several implementations and you want to benchmark to see which is fastest:
You can assert that one benchmark is faster than another like this:
new Benchmark(FasterBenchmark).whenRun(17.times()).isFasterThan(new Benchmark(SlowerBenchmark).whenRun(17.times()));You can assert that a benchmark is the fastest among a list of many benchmarks like this:
var slowerBenchmark = new Benchmark(SlowerBenchmark);
var slowestBenchmark = new Benchmark(SlowestBenchmark);
var fasterBenchmark = new Benchmark(FasterBenchmark);
fasterBenchmark.isFastestAmong(new Benchmarks{
slowerBenchmark,
slowestBenchmark,
fasterBenchmark
}
.whenRun(20.times()));Notes:
note that isFasterThan (which takes another benchamrk as an argumkent) and isFastestAmong (which takes a list of benchmarks as an argument) are methods called directly on the Benchmark itself, which is different from the way that we pass a constraint into the Should method. This is a bit inconsistent, and we are not sure yet whether this is the best way to do things, but it leads to a level of readability that we are happy with for now. You want to see how the performance of an implementation scales with increasing n.
You might know the theoretical Big O complexity of your implementation and you want to see how different it is from the real life scaling properties of the implementation. You might also want to discover where performance starts to break down as n gets very large and issues such as context switching, page swapping, and being pushed up the memory hierarchy start to invalidate the performance scaling function. Ideally, you would fit a function to the sample of benchmark runs at intervals of increasing n, but for now we only provide a mechanism to run a benchmark a number of times with increasing n and report results.
[Test]
public void FindOofNforJustSort()
{
new from(1000)
.to(10000)
.by(i=>i+1000)
(numberOfInstruments =>
new Benchmark(JustSort)
.beforeEachRun(() =>
{
all_the_series = FakeInstruments.produceNSeries(numberOfInstruments, YEARS * TRADING_DAYS_PER_YEAR);
})
.whenRun(1.times())
.ReportIndividualTimings());
}Notes:
If you are curious about the from.to.by construct see the section on loop generator constructs at the end of the doc. [Test]
public void FindOofNforIteratorHeap()
{
new from(2)
.to(2^15)
.by(i=>i*2)
(numberOfInstruments =>
new Benchmark(JustSort)
.beforeEachRun(() =>
{
all_the_series = FakeInstruments.produceNSeries(numberOfInstruments, YEARS * TRADING_DAYS_PER_YEAR);
})
.whenRun(1.times())
.ReportIndividualTimings());
}How do I avoid making some of the same mistakes that you gentlemen have made?
use the beforeEachRun() hook to reset data between each run and make sure that you are not including time to set data up for your benchmark in your benchmark timings. put asserts in the benchmark scenario itself to make sure that the benchmark runs through all the data you expect it to. Especially when you specify multiple runs of a benchmark, you need to ensure that your benchmark runs as you expect. In one mistake, we noticed that a benchmark was getting faster and faster at an incredible rate and we thought that it must be an issue where the JIT was distorting my benchmark timings. Closer inspection revealed that we had completely forgotten to create fresh data for each run of the benchmark and some benchmark runs had been timing runs for empty data sets. As a side effect of putting asserts in your benchmarks, it might also help to decrease the likelihood of invalid benchmarks by stoping the JIT from optimizing out code because it is forced to execute operations in order to produce the result for the assert at the end of the benchmark. make sure that what you are benchmarking fully encompasses all the work that you want to benchmark. In one mistake, we had one implementation among our competing implementations of k-way merge algorithms that was doing a lot of work in its constructor, which was called in the beforeEachRun() hook and not included in the timing. It is bad style to be doing lots of work in constructors anyway so we changed things to be a more accurate benchmark. monitor any running processes on the machine you are using to run your benchmarks in order to ensure you are not experiencing interference. if you look in the core whenRun method on Benchmark, you can see that there is a call to GC.Collect before each run of the benchmark - this makes sure that inconsistent GC's don't add noise to your benchmark numbers. This can be a confusing source of inconsistent timings when benchmarking on garbage collected runtimes. What sort of benchmarking process do you follow when working in a code base with very high performance requirements?
Use the profiler to isolate hot spots. Write benchmarks around the hot spot and use the benchmarks as your feedback generator while you measure, profile, and tune. Once you get to the point of acceptable performance, specify constraints on the benchmark and turn it into a regression test. This approach maximizes our development speed while tuning and stops us from experiencing performance regressions or "creep." As a side effect of working in this style, we ended up writing a lot of benchmarking code, so we just factored it out into infrastructure as we went rather than trying to copy some other performance testing framework over to .Net. The result is a flexible, lightweight declarative benchmark specification language. It leverages existing NUnit infrastructure for the test runner and asserts (and could easily do the same for xUnit.net).
What is up with these gnarly loop generator constructs?
int.times()
The times method is a higher order extension method on the integer type that, when called on an integer n, returns a function encapsulating an imperative for loop from 0 to n-1 that takes another function as an argument, and executes the function n times. This allows the higher order function that times() returns to be passed around and applied at a later time - in our case, it is applied within the whenRun method in order to run the benchmark n times.
from().to().by()
The from.to.by construct is interesting because the by() method is a higher order function (it both takes a function as input and returns a function as output) that takes a scalar function for the iterator i and generates a looping function that takes a function and runs the function inside the loop - passing the function the value of the current iterator created by the scalar function. There is an interesting reason that we made the design choice to create the from.to.by looping structure rather than providing this functionality on the benchmark API directly. The whole point of testing the scaling properties of your benchmark is that you feed in increasing amounts of data, so you probably want to generate that data using the beforeEachRun() hook. As a result of this, you need the number, n, to be in scope for your data generator to generate increasingly large data sets in the before hook. This is also a strange brew of OO and functional style. We tried to implement the from.to.by construct in a purely functional curried style, but it does not read as well as from(10)(20)(5). We also though about using a style using extension methods; for example, 10.to(20).by(i=>i+5). For now we like the "from" at the front of the construct. So we ended up with this hybrid approach using an OO style builder syntax together with higher order functions.
What is in the backlog for this project?
historical benchmarks with regression constraints. o requires persisting benchmark data after each run
o requires the ability to make asserts based on historical data with the intent to fail if performance decreases outside acceptable bads from the last run or from the historical best runs.
constraint composition o right now you can only add one constraint to a Benchmark, it would be good to allow composition of arbitrary constraints.
function fitting and constraints for scaling order o right now we only print out the list of results for increasing n for scaling benchmarks.
o It would be nice to be able to fit an actual function to the output data and compare its order against the theoretical order.
load testing for web apps and such? Should I create something like this for
? where foo : programming_language
yes.
What about all those crazy issues with optimizing compilers, JIT, garbage collected runtimes, and whatnot?
We are pooling together all our links and working on munging together a killer blogpost.
What about and environment-dependent timings? Don't benchmark timings depend both on the machines and what is happening on the machines at the time of the benchmarking?
See above. [Less]
Created
11 months
ago.
Recent Posts
-
MonoDevelop project not updating
by
Lluis Sanchez
It has been stopped for several months. https://www.ohloh.net/p/monodevelop -
candycane not updated
by
yandod
I push some commits recentyly.But not appear in ohloh. Is there any reason? http://thechaw.com/c... -
3 failing projects
by
Robin Luckey
Hi chorny, I've rescheduled these updates -- they all appear to have become blocked by temporary...
Recent Reviews
-
BioPerl
by
Mark A. Jensen
Like any good organism, BioPerl appears to be evolving in response to selection. I love BioPerl, ... -
Joomla!
by
Michael Shigorin
Short version: among php/mysql ones, look closer at Drupal for something like Joomla but created ... -
ply
by
eliben
For me, PLY is the best Python parser/lexer library out there. I'm using it for pycparser and oth...
Copyright © 2009 Geeknet, Inc., All Rights Reserved.