FHOJ: A New Java Benchmark Framework

Authors: Vinh Quang La, Dirk Jansen

Abstract:

There are some existing Java benchmarks, application benchmarks as well as micro benchmarks or mixture both of them,such as: Java Grande, Spec98, CaffeMark, HBech, etc. But none of them deal with behaviors of multi tasks operating systems. As a result, the achieved outputs are not satisfied for performance evaluation engineers. Behaviors of multi tasks operating systems are based on a schedule management which is employed in these systems. Different processes can have different priority to share the same resources. The time is measured by estimating from applications started to it is finished does not reflect the real time value which the system need for running those programs. New approach to this problem should be done. Having said that, in this paper we present a new Java benchmark, named FHOJ benchmark, which directly deals with multi tasks behaviors of a system. Our study shows that in some cases, results from FHOJ benchmark are far more reliable in comparison with some existing Java benchmarks.

Keywords: Java Virtual Machine, Java benchmark, FHOJ framework.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1061404

References:

[1] Q. Li and C. Yao, Real-Time Concepts for Embedded Systems. CMP Books, 2003.
[2] H. resolution timers, "http://high-res-dimers.sourceforge.net/."
[Online]. Available: http://high-res-timers.sourceforge.net/
[3] J. P. Charles Daly, Jane Horgan and J. Waldron, "Platform Independent Dynamic Java Virtual Machine Analysis: the Java Grande Forum Benchmark Suite," In Proceedings of the 2001 joint ACM-ISCOPE conference on Java Grande, pp. 106-115, 2001.
[4] M. D. W. D. S. H. J. M. Bull, L. A. Smith and R. A. Davey, "A Methodology for Benchmarking Java Grande Applications," In Proceedings of the ACM 1999 conference on Java Grande, pp. 81-88, 1999.
[5] P. S. Corporation, "CaffeineMark," http://www.benchmarkhq.ru/cm30/,2002.
[6] S. Org, "Spec JVM 98," http://www.spec.org/osg/jvm98, 1998.
[7] X. Zhang and M. Seltzer, "HBench: Java: An Application-Specific Benchmarking Framework for Java Virtual Machine," In Proceedings of the ACM 2000 conference on Java Grande, 2000.
[8] R. W. Jack Dongarra and P. McMahan, "LINPACK Benchmarks," http://www.netlib.org/benchmark/linpackjava/.
[9] N. V. G. Chen, M. Kandemir and M. J. Irwin, "PennBench: A Benchmark Suite for Embedded Java," IEEE International Workshop on Workload characterization, pp. 71-80, 2002.
[10] S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khang, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss,B. Moss, A. Phansalkar, D. Stefanovi'c, T. VanDrunen, D. von Dincklage, and B. Wiedermann, "The DaCapo benchmarks: Java benchmarking development and analysis," in OOPSLA -06: Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications. New York, NY, USA: ACM, 2006, pp. 169-190.
[11] S. M. Blackburn, K. S. McKinley, R. Garner, C. Hoffmann, A. M. Khan, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanovik, T. VanDrunen, D. von Dincklage, and B. Wiedermann, "Wake up and smell the coffee: evaluation methodology for the 21st century," Commun. ACM, vol. 51, no. 8, pp. 83-89, 2008.
[12] M. S. Shuichi Oikawa and T. Nakajima, "Accounting system: a finegrained CPU resource protection mechanism for embedded system," In Proceedings of the Ninth IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing, 2006.
[13] G. M. I. Bate, G.Bernat and P. Puschner, "Low-Level Analysis of a Portable Java Byte Code WCET Analysis Framework," In Proceedings of the Seventh International Conference on Real-Time Systems and Applications, 2000.
[14] Jack Shirazi and Kirk Pepperdine, "Eye on performance: Micro performance benchmarking," IBM journal, 2003.
[15] Jack Shizazi and Kirk Pepperdine, "Eye on performance: When good benchmark go bad," IBM journal, 2005.
[16] A. G. Lieve Eeckhout and K. D. Bosschere, "How Java Programs Interact with Virtual Machines at the Microarchitectural Level," In Proceedings of the OOPSLA 03 conference, 2003.
[17] R. Pozo and B. Miller, "SciMark 2.0 Benchmark," http://math.nist.gov/scimark2, 2004.
[18] M. G. Yefim Shuf, Maurico J. Serrano and J. P. Singh, "Characterizing the Memory Behavior of Java Workloads: A Structured View and Opportunities for Optimizations," ACM Sigmetrics, 2001.
[19] S. M. Y. T. T. Y. Morgan Hirosuke Miki, Mamoru Sakamoto and I. Shirakawa, "Evaluation of Processor Code Efficiency for Embedded Systems," In Proceedings of the 15th international conference on Supercomputing, 2001.
[20] V. Q. La, "Design Virtual Machine for Java Processing for a Small Embedded Microprocessor Core," Master-s thesis, University Of Applied Sciences Offenburg, 2007.
[21] V.-Q. La, "A study on Java Virtual Machine for Real-time embedded systems," IEEE International Conference on Computer Science and Software Engineering (CSSE 2008), Accepted 2008.