Updated: AUGUST 7th, 1997
Features JON BODNER
([email protected])
JVM Showdown
Now that Java has been officially proclaimed the Next Big Thing, every company is making sure that Java applets run best on their system. While the Mac world got a late start on the Java revolution (and is finally catching up with the Windows and Unix worlds' support for the latest Java standard, version 1.1.3), there are now many different Java environments available for Macintosh browsers.
JVM? JIT?
You'll notice a huge difference in performance between a standard Java Virtual Machine (JVM) and a Just In Time compiler (JIT). When a programmer writes Java applications and applets, they are not compiled to native machine code. Instead, the instructions are converted to a special Java instruction set. An emulator is then written to allow a computer to run the Java programs.
In a standard JVM, each Java instruction is converted into a native instruction and then forgotten. If this instruction is in a loop (and in a typical program, there are lots of loops), it could be converted millions of times. A JIT converts the Java instruction into a native instruction and then remembers it. This saves the overhead of each conversion after the first time the instruction is seen, and as the chart shows, it certainly makes a difference.
So, if JITs are so much better, why did anyone release a standard JVM? Writing a good JIT takes a lot of time, and in today's computer industry, you need to get a product out quickly if you want it to make a splash. Sun needed to get Java released. Although Java was designed with the JIT concept in mind, they released the much simpler standard JVM so people could start experimenting with Java sooner. It is also easier to make a debugger work with a standard JVM, which makes programmers happy. Now that Java has past its second birthday, virtually everyone has converted to JITs for end users, while developers will use a standard JVM for debugging and a JIT for testing code performance.
Since writing a JIT can be complicated, companies have been licensing JITs from Java compiler writers instead of creating their own. While Microsoft wrote its own JIT for WinTel boxes, they partnered with Mac compiler giant Metrowerks for their Mac Java support. Netscape used their own port of Sun's JVM in the Mac and Unix release of Navigator 3.0.1, but switched over to Symantec's JIT with the release of version 3.0 for Windows and version 4.0 for Macintosh. Apple originally planned to follow a similar path with Macintosh Runtime for Java (MRJ). Apple licensed Roaster's Java technologies, but then decided to develop a Java environment in-house and keep the Roaster code for an emergency backup. (MRJ is the JVM for Cyberdog, and can be used by any Macintosh application which supports Apple's JManager programming interface. Microsoft supports MRJ in Internet Explorer as an alternative to its own JVMs.)
Java Versions
Java is a rapidly maturing language and the MacOS JVMs have done a poor job at playing catch-up. The latest version, 1.1.3, adds many features to the original 1.0.2 system which make it possible for developers to create more powerful Java applications. While there are several Java 1.1.3 environments available for Windows, the only JVM which supports Java 1.1.3 is the one included with MSIE 4.0 Preview Release 1.
Over the last several months, the state of Just-In-Time compilers on the Mac has improved markedly. There are now release-quality JITs available in Java environments from Apple, Metrowerks, Symantec, Netscape, Microsoft, and Roaster. While these JITs vary in terms of stability, most of them are capable performers for everyday use.
Apple's own Java strategy is progressing nicely. There are three different JVMs available from Apple. MRJ 1.0.2, which ships with Mac OS 8, implements a standard Java 1.0.2 JVM. MRJ 1.5 was released in late July. Like MRJ 1.0.2, it also supports Java 1.0.2. However it includes numerous bug fixes and performance improvements, including an optional JIT for PowerPC Macs.
Finally, Apple has released MRJ 2.0d2 to developers. This release finally implements (incompletely) Java 1.1.1, but its JIT has been delayed for a future release. MRJ 2.0d2 does include Sun's javac Java compiler. Later releases promise to include other standard Java development tools. If Apple does in fact release the final version of MRJ 2.0 with all of these tools, it will mark Apple's first giveaway of programming tools since the early versions of HyperCard. The public alpha of MRJ 2.0 is due in August and the final MRJ 2.0 should be released sometime following.
While it's nice that Apple is catching up, the fact remains that the MacOS is seriously behind Windows in the realm of Java development. Apple has taken steps to make sure that this does not happen again. For the development of Java 1.2 (due out in December), Sun and Apple have agreed to intersperse their Java releases. As Sun finishes a release (development, alpha, or beta), Apple's Java team will get the source code and release the Mac version. Furthermore, Sun has lent Javasoft programmers to Apple to work on MRJ with Apple's own engineers. This plan should ensure that the Mac release of Java 1.2 will coincide with the release for Windows and Solaris.
Third-party Mac Java developers have announced their own plans for Java 1.1.3 support. Metrowerks CodeWarrior Pro 1 ships with beta-quality support for Java 1.1.1. An online update to a more-stable Java 1.1.3 release is expected very shortly. Roaster Release 4 will have Java 1.1.1 support; Roaster's full plans for future Java support will be revealed at MacWorld Boston. Symantec has decided to cast its lot with Apple; future versions of Cafe and Visual Cafe will rely on MRJ for a 1.1.1-compliant JVM.
CaffeineMarks
Now that many different companies have released JVMs, it was inevitable that a method to benchmark them would be created. Enter the CaffeineMark, developed by Pendragon Software. The latest release, version 3.0, is designed to exercise the most commonly used functions of a Java applet or application, and determine what JVMs perform the best. There are several changes from the earlier CaffeineMark 2.5 which previous editions of the JVM Shootout used. CaffeineMark 3.0 is designed to give an accurate measure of JIT performance. The final CaffeineMark score is now generated by computing the geometric mean of the individual scores, i.e., it is the 9th root of the product of all the scores. The base system, which gets a 100 on all tests, has also been changed. It is now a Pentium 133 running Windows 95 using Sun's JDK 1.1 interpreter. Because of these changes, it is not possible to compare scores from CaffeineMark 2.5 test runs to CaffeineMark 3.0 test runs.
The Tests
Sieve
The classic sieve of eratosthenes finds prime numbers.
Loop
The loop test uses sorting and sequence generation as to measure compiler optimization of loops.
Logic
Tests the speed with which the virtual machine executes decision-making instructions.
Method
The Method test executes recursive function calls to see how well the VM handles method calls. |
Float
Simulates a 3D rotation of objects around a point.
Graphics
Draws random rectangles and lines.
Image
Draws a sequence of three graphics repeatedly.
Dialog
Writes a set of values into labels and editboxes on a form. |
How did they do?
Here are the results for each of the available Mac JVMs:
|
CaffeineMark 3.0 Scores
|
|
Sieve |
Loop |
Logic |
String |
Floating
Point |
Method |
Graphics |
Image |
Dialog |
CaffeineMark |
| MRJ 1.5 JIT |
601 |
1476 |
2001 |
266 |
908 |
1055 |
45 |
60 |
13 |
292 |
| Roaster R3 JIT |
795 |
3615 |
2731 |
93 |
2164 |
120 |
32 |
60 |
16 |
262 |
| MRJ 1.5 JIT (MSIE 3.0.1) |
564 |
1409 |
1863 |
264 |
821 |
972 |
27 |
49 |
11 |
254 |
| MSIE 4.0p1 JIT |
886 |
1031 |
3158 |
76 |
576 |
423 |
46 |
25 |
14 |
211 |
| MW CW Pro 1 JIT |
430 |
611 |
2971 |
207 |
443 |
468 |
43 |
34 |
17 |
211 |
| Cafe 1.0.1 JIT |
437 |
639 |
875 |
281 |
451 |
228 |
36 |
52 |
11 |
173 |
| Netscape 4.0.1 |
446 |
701 |
721 |
240 |
512 |
410 |
10 |
21 |
9 |
140 |
| MW CW Pro 1 |
193 |
204 |
200 |
73 |
200 |
212 |
45 |
32 |
17 |
94 |
| Roaster R3 |
106 |
95 |
102 |
113 |
93 |
90 |
44 |
53 |
53 |
72 |
| Netscape 3.0.1 |
108 |
110 |
113 |
89 |
107 |
98 |
37 |
24 |
20 |
65 |
| MRJ 1.5 |
77 |
77 |
82 |
79 |
80 |
80 |
42 |
67 |
14 |
59 |
| MRJ 1.0.2 |
81 |
77 |
64 |
84 |
79 |
79 |
17 |
38 |
11 |
48 |
| Cafe 1.0.1 |
65 |
31 |
63 |
64 |
61 |
45 |
33 |
54 |
15 |
43 |
| MSIE 4.0p1 |
45 |
35 |
138 |
37 |
32 |
77 |
40 |
27 |
13 |
40 |
| MSIE 3.0.1 JIT |
|
|
|
|
|
|
|
|
|
* |
| MSIE 3.0.1 |
|
|
|
|
|
|
|
|
|
* |
* - Did not successfully compete test
These tests were run on a PowerBook 1400c/133, with 16M of RAM (RAM Doubler 2 set to triple RAM), 128K L2 cache, 1G hard drive, and Mac OS 8.
While the changes in CaffeineMark have significantly reduced the gap between the performance of the JITs and the standard virtual machines, it is clear that a JIT provides significant improvement. Apple's MRJ 1.5 JIT, running within the Apple-supplied Apple Applet Runner is far and away the best performer. Bug fixes in MRJ 1.5 now allow it to be run correctly within Internet Explorer, and as can be expected, performance is significantly slower inside a browser rather than standalone. Even with the slowdown, MRJ 1.5 is faster than any other JVM embedded in a WWW browser.
However, the biggest surprise is the complete failure of the Microsoft/Metrowerks VMs (JIT and standard) in MSIE 3.0.1 when run under Mac OS 8. Despite repeated attempts and installations, we were unable to get the CaffeineMark 3.0 tests to complete. During either the Graphics or Image test, the browser would always quit with an error of either type 2 or type 3. Since these virtual machines are nearly identical to the ones included with CodeWarrior Pro 1--and CodeWarrior Pro 1 had no problem completing these tests--the bug must be linked to something in the MSIE 3.0.1 browser itself. Luckily for Microsoft, MRJ 1.5 worked fine, and for the adventurous, the preview release of MSIE 4.0 was able to complete the tests with its new 1.1.3 JVMs.
Microsoft, which had been championing its JIT from Metrowerks as the performance leader, finds itself in an interesting position. Their current release (3.0.1) is severely broken when used with Mac OS 8, but the preview release (4.0) has very good performance and the only Java 1.1.3 support on the Mac. Their JIT bests the MRJ JIT (when used with MSIE 3.0.1) in 4 out of the 9 categories, including a trouncing in the logic category, but it is still slower overall. Since MSIE 4.0 is still in early release, it's likely that Java performance will continue to improve in later versions.
Netscape, which has aligned itself very closely with the Java revolution, is in serious trouble. Its JIT delivers only 55% of the performance of MRJ 1.5 when MRJ is run inside of MSIE 3.0.1. Since, Navigator does not support JManager, it cannot take advantage of the speedy MRJ 1.5 JIT. If Netscape intends to provide a viable Java option on the Mac, it will need to either support MRJ or improve its own JIT in a hurry.
Roaster's performance is very exciting. While there's nothing spectacular about the performance of Roaster's standard JVM, the JIT is easily the fastest available for the Mac. However, there have been problems with the Roaster JIT's stability (see below). Furthermore, both the JIT and the standard JVM can only be used in conjunction with the stand-alone Roaster Runner application. Metrowerks' JIT and JVM are in the same boat, as the only way to invoke either one is by using the less-than-spectacular Metrowerks Java application. Symantec's Visual Cafe 1.0.1 is a so-so performer and its Java environment is equally unimpressive. It comes with an unremarkable applet/application runner which is nearly identical to the Sun AppletViewer.
The graphic performance of all of the tested VMs is poor in general, due to the inferior video controller in the PowerBook 1400. Desktop Macs with their much faster video boards will have much better video scores. However, the video benchmarks do not benefit as much as CPU-bound tasks from a JIT, so they will continue to remain a limiting factor in Java performance as JVMs continue to mature.
What about stability?
It's nice to be fast, but without stability, speedy performance is useless. As previously mentioned, the VMs included with MSIE 3.0.1 were unable to complete the CaffeineMark tests, which indicates that they will probably fail on many other applets. The Roaster JIT, which is one of the fastest performers, proved shaky when reviewed (see the Roaster Release 3 review) and needs significant improvement before it's usable.
During testing, each VM would display some fonts and windows slightly differently. Some VMs draw the menus inside the CaffeineMark window while others would append the menus onto whatever menus were currently on the menubar. It's hard to judge which VM is acting correctly when each one acts differently. This is the biggest problem facing Java today. If an applet doesn't seem to play correctly with one virtual machine, try using a different one to see if the display problems go away. Another part of the problem is that it's hard to evaluate stability with the current crop of publicly available applets, since many are too simple to stress a JVM. These problems should disappear with time, as Java continues to mature as a language and environment and the various VM makers agree on conventions.
A few months ago, there was only JIT in final release for the Mac: the one included with Internet Explorer 3.0.1. Now, there are four. The best choice for Mac users is clearly MRJ 1.5, but it will be interesting to see what Microsoft and Metrowerks ship in the upcoming months.
In conclusion
Apple has come a long way in a short time; MRJ 1.5 moves Apple from one of the slowest JVMs to one of the fastest, without sacrificing stability. Release-quality Java 1.1.3 is the next item on the agenda, and if Apple's Java team continues to perform this well, Mac users have nothing to worry about. The Microsoft/Metrowerks combo is also doing well; it already has Java 1.1.3 support and is still one of the top performers. The failure of the MSIE 3.0.1 VMs is disheartening, but now that Internet Explorer and JManager are working well together, this issue becomes less important. With the upcoming releases of Roaster Release 4 and continued improvements to Metrowerks CodeWarrior Pro 1 and MSIE 4.0, the Java Wars on the Mac are just starting to heat up. It's going to be an interesting summer.
All information on the base system and the test details are from Pendragon Software WWW site and are Copyright 1996-1997, Pendragon Software. For more information on the CaffeineMark(tm) benchmark, take a look at Pendragon's WWW site. |