GBench = @Benchmark Annotation + BenchmarkBuilder

I released GBench, a benchmarking framework for Groovy. Now GBench has two features, @Benchmark Annotation and BenchmarkBuilder. @Benchmark Annotation is an annotation that allows to measure execution time of methods without modifying production code and is already published. Please read the previous post for more detail information. BenchmarkBuilder is a convenient builder to write benchmark code easily and now appears for the first time.


The following code is an example to get benchmark string concatenation by repeating them 1000 times:
----
def strings = ['GBench', ' = ', '@Benchmark Annotation', ' + ', 'BenchmarkBuilder']
def benchmark = new BenchmarkBuilder()
benchmark.run times: 1000, {
    with '+=', { 
        def s = ''
        for (string in strings) {
            s += string    
        }
    }
    with 'concat', { 
        def s = ''
        for (string in strings) {
            s.concat string    
        }
    }
    with 'string builder', {
        def sb = new StringBuilder()    
        for (string in strings) {
            sb << string    
        }
        sb.toString() 
    }
    with 'join', {
        strings.join()
    }
}
println benchmark
----


The output will be like this:
----
                 time
+=             18197705
concat         7669621
string builder 9420539
join           5248348
----

Of course you can sort the results:
----
println benchmark.sort({ lhs, rhs -> lhs.time <=> rhs.time })
----


----
                 time
join            5248348
concat          7669621
string builder  9420539
+=             18197705
----

and also you can handle the results as you want!:
----
new File('benchmark.csv').withWriter { file ->
    file.writeLine 'label,time(ms)'
    benchmark.sort().each { bm ->
        file.writeLine "${bm.label},${bm.time / 1000000}"
    }
}
----


----
> cat benchmark.csv
label,time(ms)
join,5.248348
concat,7.669621
string builder,9.420539
+=,18.197705
----


For now, GBench can measure only wall-clock time but I'm planning to also support CPU time and user time in a future release.


You can download GBench from here. Please try and let me know your feedback!

@Benchmark annotation for Groovy gets a big update!

I released @Benchmark annotation for Groovy v11.06.11. This release includes some big changes and a big bug fix:


- Changes
  - Added classes / methods information to benchmark results
  - New option to customize handling of benchmark results
  - Supported class annotation


- Bug fixes
  - Fixed a problem that not working with methods of classes


Added classes / methods information to benchmark results


You can get benchmarked methods and their classes information as benchmark results. For example with the following code,
----
package foo


class Foo {
    @Benchmark
    def foo() {
    }
}
----


the output will be:
----
foo.Foo java.lang.Object foo(): xxx ns
----


New option to customize handling of benchmark results


Added "value" option to customize handling of benchmark results instead of poor options, "prefix" and "suffix". There are three ways to assign value to it:


- Use handler classes
- Use closures
- Use a system property


In the case of using handler classes, create classes that implement Benchmark.BenchmarkHandler interface and add two methods, handle() and getInstance() to them:
----
class MyHandler implements Benchmark.BenchmarkHandler {
    static def instance = new MyHandler()
    static MyHandler getInstance() {
        instance    
    }
    void handle(klass, method, time) {
        println("${method} of ${klass}: ${(time/1000000) as long} ms")
    }    
}
----


Yes, singleton classes as the above example can be writtern shorter by using @Singleton annotation in Groovy :-)
----
@Singleton
class MyHandler implements Benchmark.BenchmarkHandler {
    void handle(klass, method, time) {
        println("${method} of ${klass}: ${(time/1000000) as long} ms")
    }
}
----


In the end, assign handler classes to @Benchmark:
----
@Benchmark(MyHandler.class)
def foo() {
}
----


Since Groovy 1.8, you can also use closures instead of handler classes. With closures, you just need to assign closures that handle benchmark results:
----
@Benchmark({println("${method} of ${klass}: ${(time/1000000) as long} ms")})
def foo() {
}
----


And you can replace the default handling operation with a system property, "groovybenchmark.sf.net.defaulthandle":
----
> groovy -cp groovybenchmark-11.06.11.jar 
-Dgroovybenchmark.sf.net.defaulthandle="println(method + ' of ' + klass + ': ' + ((time/1000000) as long) + ' ms')" foo\Foo.groovy
----


In the cases of the examples, the outputs will be:
----
java.lang.Object foo() of foo.Foo: xxx ms
----
  
Supported class annotation


By annotating classes, you can get benchmark results of all the methods of the classes:
----
package foo


@Benchmark
class Foo {
    def foo() {
    }
    def bar() {
    }
}
----


In the case of the example, you will get the benchmark results of foo() and bar() methods:
----
foo.Foo java.lang.Object foo(): xxxx ns
foo.Foo java.lang.Object bar(): xxxx ns
----


And this means, now you've got a power to benchmark all the methods of your program without modifying code and profiling tools! Because Groovy 1.8 provides a compilation customizer to apply annotations to all the classes:
----
// BenchmarkGroovyc.groovy
import groovybenchmark.sf.net.Benchmark


import org.codehaus.groovy.control.CompilerConfiguration
import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer
import org.codehaus.groovy.tools.FileSystemCompiler


def cc = new CompilerConfiguration()
cc.addCompilationCustomizers(new ASTTransformationCustomizer(Benchmark))
new FileSystemCompiler(cc).commandLineCompile(args)
----


----
> groovy -cp groovybenchmark-11.06.11.jar BenchmarkGroovyc.groovy MyApp.groovy
> java -cp .;%GROOVY_HOME%\embeddable\groovy-all-1.8.0.jar;groovybenchmark-11.06.11.jar MyApp 
----


----
Xxx xxxx foo(): xxx ns
Xxx xxxx bar(): xxx ns
Xxx xxxx baz(): xxx ns
MyApp java.lang.Object main(java.lang.Object): xxx ns
----


Please try and let me know your feedback!