I think most people would agree that having some unit tests, which ensures that non-functional requirements are met, is a good idea. With processing time people usually use time in order to measure whether a particular function runs within the requirements. However, everybody knows that measuring time is inaccurate, since it varies depending on the machine the tests are running and whether there are other process consuming resources in parallel.
Wouldn't it be cool to just count the atomic operations per thread executed by e.g. the Java VM?
We could than have a reference system, which is used to give each atomic operation a value (e.g. nanoseconds).
Then I would like to write tests like this:
@ExecutesIn(23000)
public void testFoo() {
doStuff("Foo "+" bar");
}
Where the number 23000 is the time it would take to run the code on the reference system. The test would always take the same time, no matter it was run on an x386 machine or a modern multi GHz box.
Questions:
Does anyone know whether such a tool/library/framework already exists?
And if not, do you have any idea of how this could be implemented?
8 Kommentare:
I am not sure, whether such an approach can satisfy your needs. In native environments one usually counts cpu cycles of the measured thread. It seems that JVMPI offers something similar with JVMPI_EVENT_INSTRUCTION_START. Nevertheless, accurate measurement gets harder when adding more variables to your runtime system. Transparent parallel execution, JITting of hotspots or different underlying physical machines and version of your framework (read:JVM) greatly affect the runtime behavior of the given byte code.
Since these characteristics do not scale proportional a metric on this level of detail (instructions of byte code) may not offer a reliable answer to "is this fast enough in production". Counting them seems worthless to me.
To get an impression of how modifications influence the actual execution speed you can use a stable reference implementation instead. In phonet4n http://www.1160pm.net/?p=34 I am testing the execution time of the .net library against a given reference implementation in C running on the same machine. Surely, this is approximately as precise as an order of magnitude ;)
Yes, JITing needs to be turned off, of course.
Different hardware shouldn't affect the results and getting different result, because we changed a library would be an interesting outcome, wouldn't it?
JVM PI really seems to be a good starting point.
To me, counting byte code instruction without JITting would be like counting lines of code of a C++ program. Depending on the included macros, optimization level and chosen target architecture the actual output and its speed differs significantly in an unpredictable way.
What do you think about a calibration of the execution speed of your testing framework on startup using a stable implementation? You can than express execution times in multiples of this?
I don't want to check how good the VM's optimizations are or how fast my hardware is.
Therefore I want to mock the underlying platform, by just counting bytecode instructions multiplying each with a reasonable value (the time it takes on a reference system).
Such an approach is just useful for automated tests, ensuring that you don't have complicated computation accidentally. You'll of course still have to test and profile things on the real machines.
Interesting idea. If JVMPI can give you more than just CPU cycle counts, I would suggest also counting memory allocations and generated garbage.
I've spent quite a bit of time performance tuning various aspects of EMF. Getting accurate and useful measurements to help guide the search for bottlenecks is a huge challenge. Turning off the JIT will give you a grossly distorted view of actual performance to such an extent that you'd tend to focus in issues that simply won't arise in an actual deployed application. So if the actual aim is to produce code that performs well, you simply can't get away from measuring it under real live conditions. Even things like the time spent on garbage collection has to be counted somewhere. I just wish we had really good tools at Eclipse for doing this type of work...
Hi Sven,
I am often using the YourKit Profiler to profile my apps and I read that they also have an API for programming against the profiling agent. Maybe this is simpler that using JVMPI directly.
Just an idea...
Cheers,
-Martin
Did you try using PerformanceTestCase from org.eclipse.test.performance? They have a way of hooking into OS performance counters. Also, the framework allows you to track your performance results over time, crank out nice graphs, etc.
Post a Comment