tag:blogger.com,1999:blog-68173312024-02-09T02:59:03.598-08:00Java Performance blogJava and android performance analysis, Eclipse Memory AnalyzerAnonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.comBlogger41125tag:blogger.com,1999:blog-6817331.post-91575331170644226102011-06-22T07:38:00.000-07:002011-06-22T07:42:07.619-07:00Memory Analyzer 1.1Memory Analyzer 1.1. was just released!<br />
Check the <a class="vt-p" href="http://www.eclipse.org/mat/1.1/noteworthy.html">New and noteworthy</a> page and download it from <a class="vt-p" href="http://www.eclipse.org/mat/downloads.php">here</a>.<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com3tag:blogger.com,1999:blog-6817331.post-57005288340991224132010-11-11T01:56:00.000-08:002010-11-11T01:57:03.165-08:00Extending the Eclipse Memory AnalyzerI usally don't post links here but I always wanted to describe here how to extend the Eclipse Memory Analyzer. Since there's now a Slide Deck available that describes briefly how it can be done at<br /><a href="http://www.slideshare.net/AJohnson1/extending-eclipse-memory-analyzer">http://www.slideshare.net/AJohnson1/extending-eclipse-memory-analyzer</a> it's not that urgent anymore to post a description here ;-)<br /><br /><br />I believe there's a huge potential in extending MAT for specific use cases, such as Android, Jruby, Scala etc.Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com14tag:blogger.com,1999:blog-6817331.post-74835212216735525042010-06-09T23:53:00.000-07:002010-06-09T23:53:36.768-07:00Talk at Eclipse demo camp in MannheimI will give a short talk/demo of how to use the Eclipse Memory Analyzer for analyzing android applications today, June 10 (19.00 - 21.30) in Mannheim, University A5, C013 at the demo camp event:<br />
<br />
http://wiki.eclipse.org/Eclipse_DemoCamps_Helios_2010/Mannheim<br />
<br />
Hope to see some of you there.<br />
MarkusAnonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com1tag:blogger.com,1999:blog-6817331.post-16045852978095360882010-02-24T11:09:00.000-08:002010-02-25T07:50:37.993-08:00Android memory usage analysis slides availableYesterday, I gave a talk at the mobile monday android developer day in Düsseldorf.The slides about <a href="http://docs.google.com/fileview?id=0B6tt5piUV3RWZGYxZTg1ZjQtY2E0ZS00NzlkLWJjMzUtZTYxYWE4YTVjZmQ2&hl=en_GB">analysing the memory usage of android</a> applications with the <a href="http://www.eclipse.org/mat/">Eclipse Memory Analyzer</a> are available now. Note it might be worth to download the pdf because it contains some notes.<br />The slides should be interesting for general java developers, not only for android developers. <br />Also, I want to thank everyone who showed up. It was great talking to all of you!<br /><br /><script type="text/javascript"> var dzone_url = 'http://kohlerm.blogspot.com/2010/02/android-memory-usage-analysis-slides.html';</script><br /> <br /><script type="text/javascript"> var dzone_style = '1';</script><br /> <br /><script language="javascript" src="http://widgets.dzone.com/widgets/zoneit.js"> </script>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com11tag:blogger.com,1999:blog-6817331.post-81087580813066895592010-01-25T17:50:00.000-08:002010-01-25T17:50:00.291-08:00New Memory Analyzer releaseA new version of the Eclipse Memory analyzer is available for <a href="http://www.eclipse.org/mat/downloads.php">download</a> and from the <a href="http://download.eclipse.org/technology/mat/latest/update-site/%20%20">update site</a>.<br />This is the last 0.8 release and from now on working on the <a href="http://www.eclipse.org/projects/project-plan.php?projectid=technology.mat">1.0 release</a> will start.<br />This is a bug fix release and here's the list of the most important bugs fixed:<br /><div class="MsoNormal"><br /></div><div class="MsoNormal"><a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=294350" target="_blank"><span lang="EN-US">294350</span></a> <span lang="EN-US">StackOverflowError in ArrayUtils.hybridsort</span><br /></div><div class="MsoNormal"><a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=8108758081306689559" name="126558df2abc6336_b270777"></a><a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=270777" target="_blank"><span lang="EN-US">270777</span></a> <span lang="EN-US">The dialog font not apply to the ArgumentWizard <a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=8108758081306689559" name="126558df2abc6336_b291481"></a></span><br /></div><div class="MsoNormal"><a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=8108758081306689559" name="126558df2abc6336_b281311"></a><a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=281311" target="_blank"><span lang="EN-US">281311</span></a> <span lang="EN-US">Merged Paths from GC roots calculation may slow down the Leak Suspects report <a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=8108758081306689559" name="126558df2abc6336_b273915"></a></span><br /></div><div class="MsoNormal"><a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=273915" target="_blank"><span lang="EN-US">273915</span></a> <span lang="EN-US">New resolvers URL, Point and Rectangle <a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=8108758081306689559" name="126558df2abc6336_b299283"></a></span><br /></div><div class="MsoNormal"><a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=299283" target="_blank"><span lang="EN-US">299283</span></a> <span lang="EN-US">Allow dynamic specification of file extensions for parsers <a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=8108758081306689559" name="126558df2abc6336_b299497"></a></span><br /></div><div class="MsoNormal"><a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=299497" target="_blank"><span lang="EN-US">299497</span></a> <span lang="EN-US">DTFJ Adapter retains memory <a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=8108758081306689559" name="126558df2abc6336_b272887"></a></span><br /></div><div class="MsoNormal"><a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=272887" target="_blank"><span lang="EN-US">272887</span></a> <span lang="EN-US">No JvmInfo in Heap Dump details pane <a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=8108758081306689559" name="126558df2abc6336_b274539"></a></span><br /></div><div class="MsoNormal"><a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=274539" target="_blank"><span lang="EN-US">274539</span></a> <span lang="EN-US">Overview pane is missing scroll bars <a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=8108758081306689559" name="126558df2abc6336_b297458"></a></span><br /></div><div class="MsoNormal"><a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=297458" target="_blank"><span lang="EN-US">297458</span></a> <span lang="EN-US">Single object queries have a confusing dialog box <a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=8108758081306689559" name="126558df2abc6336_b254864"></a></span><br /></div><div class="MsoNormal"><a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=254864" target="_blank"><span lang="EN-US">254864</span></a> <span lang="EN-US">Leaking Memory after closing the Heap Dump Editor</span><br /></div><div class="MsoNormal"><br /></div><div class="MsoNormal"><br /></div><div class="MsoNormal"><br /></div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com13tag:blogger.com,1999:blog-6817331.post-91952588340839000402010-01-21T01:02:00.000-08:002010-01-22T00:04:27.722-08:00First talk about analysing Memory usage on android ever?I am pleased to announce that I will be speaking at the <a href="http://www.m2-d2.de/"><b>MobileMonday Developer Day</b></a><br />
in Düsseldorf<b> on 23.2.2010</b><b>. </b><br />
The talk (<s>in German </s>in English!) will be about analyzing the memory usage analysis of android applications using the <a href="http://www.eclipse.org/mat/">Eclipse Memory Analyzer</a>.<br />
<br />
<div style="font-family: Arial,Helvetica,sans-serif;">Titel:<br />
Optimierung des Speicherverbrauchs von Android Anwendungen mit Hilfe des Eclipse Memory Analyzers<br />
<br />
Abstract:<br />
Die Minimierung des Speicherverbrauchs ist ein sehr wichtiger Aspekt bei der Entwicklung performanter Android Anwendungen.<br />
Im Vortrag werden die wichtigsten Konzepte zur Analyse des Speicherverbrauchs mittels Heap Dumps vorgestellt.<br />
Anhand von realen Android Anwendungen wird live demonstriert wie der Eclipse Memory Analyzer die Identifikation von "Hot Spots" unterstützt und damit eine wichtige Grundlage zur Optimierung liefert.<b> </b><br />
</div><div style="font-family: Arial,Helvetica,sans-serif;"><br />
</div><div style="font-family: Arial,Helvetica,sans-serif;"><b><br />
</b><br />
</div><div style="font-family: Arial,Helvetica,sans-serif;"><br />
</div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com6tag:blogger.com,1999:blog-6817331.post-45296145179131636542009-07-09T13:35:00.000-07:002009-07-10T07:47:58.584-07:00Eclipse Memory Analyzer, 10 useful tips/articles<div xmlns="http://www.w3.org/1999/xhtml">The <a href="http://eclipse.org/mat/" target="_blank">Eclipse Memory Analyzer</a> has been shipped with Eclipse 3.5 Galileo and I planned to start a new series here about memory usage antipatterns and how to analyze them with MAT. <br />
Unfortunately I'm pretty busy these days and I will need more time for those posts. <br />
But, due to <a href="http://twitter.com/dzonejames" target="_blank">popular demand</a> I decided it would make sense to post some good links (ok, I admit some of them are from myself) about how to use the Eclipse Memory Analyzer. <br />
Please forgive me if I missed some important ones, but I'm really hungry now and therefore just stopped after 10 tips ;)<br />
<br />
<ol><li><a href="http://kohlerm.blogspot.com/2009/02/how-to-really-measure-memory-usage-of.html" target="_blank">How to really measure memory usage</a>: The absolute fundamentals</li>
<li>Check the online help. It has a pretty good introduction chapter. </li>
<li><a href="http://kohlerm.blogspot.com/2009/02/memory-leaks-are-easy-to-find.html" target="_blank">Memory leaks are easy to find</a>: Learn how to analyze memory leaks</li>
<li>Check also <a href="http://dev.eclipse.org/blogs/memoryanalyzer/2008/05/27/automated-heap-dump-analysis-finding-memory-leaks-with-one-click/" target="_blank">one click leak analysis</a> if this is your problem :]</li>
<li><a href="http://kohlerm.blogspot.com/2009/03/leaks-are-easy-to-find-but-memory-usage.html" target="_blank">Memory usage analysis</a>: Learn the fundamental approach for finding the owner of a set of objects</li>
<li><a href="http://kohlerm.blogspot.com/2009/04/analyzing-memory-usage-off-your-android.html">A typical issue in Goggles Android UI framework</a>: Learn how to analyze memory usage on Googles android platform. Shows a typical issue that can be solved by using lazy initialization. </li>
<li><a href="http://kohlerm.blogspot.com/2009/01/is-javalangstringintern-really-evil.html" target="_blank">Never forget to take a look at your Strings</a>: Learn why String.intern() can be useful</li>
<li><a href="http://dev.eclipse.org/blogs/memoryanalyzer/2008/11/28/analyzing-the-equinox-bundle-dependencies-with-memory-analyzer/" target="_blank">A tip for analysis Equinox based applications</a>: Learn about a special feature for Eclipse based applications. </li>
<li><a href="http://dev.eclipse.org/blogs/memoryanalyzer/2008/05/17/the-unknown-generation-perm/" target="_blank">Analysing perm size/classloader problems</a>: Perm size problems can be nasty. Check why a Jruby core developer likes MAT ;)</li>
<li>If you like webinar's <a href="http://live.eclipse.org/node/520" target="_blank">here</a>'s an introduction presented by the developers. </li>
</ol><br />
Just read my <a href="http://kohlerm.blogspot.com/search/label/memory" target="_blank">memory related blogs</a>: Oh that was number 11, but who cares? ;)<br />
<br />
<br />
<br />
<script type="text/javascript">
var dzone_url = 'http://kohlerm.blogspot.com/2009/07/eclipse-memory-analyzer-10-useful.html';
</script><br />
<script type="text/javascript">
var dzone_style = '1';
</script><br />
<script language="javascript" src="http://widgets.dzone.com/widgets/zoneit.js">
</script></div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com36tag:blogger.com,1999:blog-6817331.post-13577277492285314802009-04-27T08:20:00.000-07:002011-01-19T00:44:29.337-08:00Analyzing the memory usage of your Android application<div class="separator" style="clear: both; text-align: center;"></div><br />The new <a href="http://developer.android.com/sdk/preview/" target="_blank">Android 1.5 Early Look SDK </a> is out since a few weeks. The <a href="http://developer.android.com/sdk/preview/features.html" target="_blank">"Android 1.5 highlights" page</a> does not mention one highlight, which IMHO will become very important for all developers on the Android platform because it allows you to find memory leaks easily and analyze the memory usage of your Android applications.<br /><br />I'm talking about the <b>hprof-conv tool </b>that allows you to convert an Android/Dalvik heap dump into an .hprof heap dump. The tool does not preserve all the information, because Dalvik has some unique features such as cross-process data sharing, that are not available in a "standard" JVM. Still the .hprof file is a good starting point and can be read with the <a href="http://www.eclipse.org/mat/">Eclipse Memory Analyzer</a>, which will interpret it like a normal file from a Sun 32 bit JVM. In the future it should also be not that difficult to read the Dalvik heap dump format directly and provide more information for memory analysis, such as which objects are shared and also maybe how much memory is used by native Objects (haven't checked that).<br /><br /><b><span style="font-size: x-large;">How to get a heap dump</span></b><br /><br />First we must of course get a heap dump. I do not yet own an Android phone, also I plan to get on this year. If you want to donate one, please let me know. Donating lot's of money would also help ;)<br />I therefore created the heap dump using the emulator.<br />I created a new avd for the 1.5 preview:<br /><br />android create avd -n test -t 2<br />emulator -avd test<br /><br />Then you can logon to the emulated phone and make sure that the /data/misc directory is writable :<br /><br />adb shell<br />chmod 777 /data/misc<br />exit<br /><br />Heap dumps are actually created by sending the process a SIGUSR1 (-10) signal.<br /><br />adb shell ps<br /><br />check for the pid of your application<br /> <br />adb shell kill -10 pid<br /><br /><br />You can check with<br /><br />adb shell logcat<br /><br />whether it works. You should see something like:<br /><br /><span style="font-family: -webkit-monospace; font-size: 12px; line-height: 15px; white-space: pre-wrap;">I/dalvikvm( <pid>): threadid=7: reacting to signal 10 I/dalvikvm( <pid>): SIGUSR1 forcing GC and HPROF dump I/dalvikvm( <pid>): hprof: dumping VM heap to "/data/misc/heap-dump-tm<nr>- pid<pid>.hprof-hptemp". I/dalvikvm( <pid>): hprof: dumping heap strings to "/data/misc/heap-dump-tm124026 3144-pid<pid>.hprof". I/dalvikvm( <pid>): hprof: heap dump completed, temp file removed D/dalvikvm( <pid>): GC freed 1292 objects / 92440 bytes in 11 sec D/dalvikvm( <pid>): GC freed 215 objects / 9400 bytes in 963ms</pid></pid></pid></pid></pid></pid></nr></pid></pid></pid></span><br /><br />now you can copy the heap dump from the (emulated) phone to the Desktop machine:<br /><br /><span style="font-family: -webkit-monospace; font-size: 12px; line-height: 15px; white-space: pre-wrap;">adb pull /data/misc/heap-dump-tm<nr>-pid<pid>.hprof address.hprof</pid></nr></span></pid></pid></pid></pid></pid><br /><pid><pid><pid><pid><span style="font-family: -webkit-monospace; font-size: 12px; line-height: 15px; white-space: pre-wrap;"><nr><pid> </pid></nr></span> <br />Now the file you will get does not conform to the "standard" Sun .hprof format but is written in Dalvik's own format and you need to convert it:</pid></pid></pid></pid><br /><pid><pid><pid><pid><br />hprof-conv heap-dump-tm<nr>-pid<pid>.hprof 4mat.hprof<br /><br />Now you can use the Eclipse Memory analyzer to analyze the memory usage of your application.<br /><br /><b><span style="font-size: medium;"> Typical optimization opportunities</span></b><br /><br />I described some typical issues already here in this blog for example a problem with <a href="http://kohlerm.blogspot.com/2008/06/classical-finalizer-problem-in-netbeans.html">Finalizers</a> in Netbeans,<br /> or finding <a href="http://kohlerm.blogspot.com/2008/05/analyzing-memory-consumption-of-eclipse.html">"String duplicates"</a> in Eclipse (my favourite memory analysis trick).<br /><br />You might also take the time and check <a href="http://kohlerm.blogspot.com/search/label/memory">all my memory related posts</a>.<br /><b><span style="font-weight: normal;"> </span></b><br />Now I would like to present you a new typical memory usage issue that I just found when playing around with the new Android cupcake 1.5 prerelease.<br /><br /><span style="font-size: x-large;"><b> YouShouldHaveDoneLazyInitialization</b><br /></span></pid></nr></pid></pid></pid></pid><br /></pid></pid><br /><div style="margin: 0px;"><b><span style="font-weight: normal;"><br /></span></b></div>I did a heap dump of the android <b>gmail </b>application and loaded it into the Eclipse Memory Analyzer.<br />In the histogram view I filtered the android classes:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXxNSFASKyDzo8qLZOK9NdgrxytcTP8oeX-bnACfkEJI_t8DpVhlIxDDf1_dLRWUjQA6uhoqCJzIsJ_b7HQMI1ePVO0k2dvwTqUrRjRwjelKFGvhMcnrmzMJPnNpknxk_wQ2U/s1600-h/android.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXxNSFASKyDzo8qLZOK9NdgrxytcTP8oeX-bnACfkEJI_t8DpVhlIxDDf1_dLRWUjQA6uhoqCJzIsJ_b7HQMI1ePVO0k2dvwTqUrRjRwjelKFGvhMcnrmzMJPnNpknxk_wQ2U/s320/android.PNG" /></a></div><br /><br />The high number of Rect (Rectangle) objects looked suspicious to me. Also they retain not that much memory, I thought it would be interesting why such a high number of Rect instances was alive.<br />When checking the Rect objects I immediately saw that most of them seemed to be empty e.g. bottom=0 and left=0 and right=0 and top=0.<br /><br />It seemed to me that this was the time to again use the most advanced feature in MAT. I'm talking about the OQL view, which allows you to execute SQL like queries against the objects in your heap dump.<br />I very rarely used this feature in the past because the standard queries are good enough almost all times and because writing your own command in Java is not that difficult.<br />Still here it would help me to find how many of the Rect instances where empty.<br /><br />I entered<br /><br /><span style="font-family: -webkit-monospace; font-size: 12px; line-height: 15px; white-space: pre-wrap;">select * from android.graphics.Rect where bottom=0 and left=0 and right=0 and top=0 </span><br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcyQHya138GVVviqfdvMdpxcRCf9AXx8XstVfNEBaQ-DTnnEq4GFgFZLCy5VSILvliao-fra91mL_Ria3czP_Fo-_chd6OmatRdmO7Bw9oH8ESO6bBCnvOhgxLYNyKOzbl_Gc/s1600-h/osql.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcyQHya138GVVviqfdvMdpxcRCf9AXx8XstVfNEBaQ-DTnnEq4GFgFZLCy5VSILvliao-fra91mL_Ria3czP_Fo-_chd6OmatRdmO7Bw9oH8ESO6bBCnvOhgxLYNyKOzbl_Gc/s320/osql.png" width="320" /></a></div><br /><br /><br />Which showed me that out of 1320 Rect instances 941 where "empty", which means that over 70% of the instances where "useless".<br />I used "immediate dominators" on the first 20 of those 941 instances, which showed me that those empty Rect instances were retained by several UI related<br />classes which are subclasses of <b>android.graphics.drawable.Drawable</b>.<br /><br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2J7VM3TogzguyzMMb2Jd4ReakAVz4YG42QKwi7yL36D6Aa7AULIDvGUxP1HaATsFeL7vjkoQcCHeEqbCpGo040E2GQXCiMM6PtGdDKMJDVAZbIW-0vpYEgL4FGUrpAZzESW4/s1600-h/immeditateOSQL.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="117" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2J7VM3TogzguyzMMb2Jd4ReakAVz4YG42QKwi7yL36D6Aa7AULIDvGUxP1HaATsFeL7vjkoQcCHeEqbCpGo040E2GQXCiMM6PtGdDKMJDVAZbIW-0vpYEgL4FGUrpAZzESW4/s320/immeditateOSQL.png" width="320" /></a></div><br /><br />I checked the source and quickly found that Drawable <b>always</b> instantiates a Rect using the default constructor:<br /><br /><br /><br /><pre style="white-space: pre-wrap; word-wrap: break-word;">public abstract class Drawable { private int[] mStateSet = StateSet.WILD_CARD; private int mLevel = 0; private int mChangingConfigurations = 0; <span style="background-color: yellow;"> private Rect mBounds = new Rect();</span></pre><br /><br />Also this issue might not increase the memory usage of gmail that much it's still a fundamental/systematic problem, which IMHO should be fixed.<br />Even if we would tolerate the memory overhead, we are still left with the problem that the number of objects the garbage collector has to trace during Full GC is higher than needed, which at least potentially can lead to sluggish UI.<br />It's also a kind of fundamental problem because <b>all subclasses of Drawable inherit the problem.</b><br /><br />So instead of always using the default constructor it would <i>most likely</i> better to use <a href="http://en.wikipedia.org/wiki/Lazy_initialization">lazy initalization</a> and only initialize the field when it's first needed.<br />A <a href="http://en.wikipedia.org/wiki/Copy-on-write">copy on write </a>strategy could also be used by sharing an "empty" Rect instance in the beginning, which would be replaced by a new one as soon as the field is written ( haven't checked the code whether this can still be done).<br /><br /><b>Disclaimer:</b><br />This is not a sign that Android is a poor platform. This is a kind of a problem that I've seen in the past more than once and I'm sure this kind of antipattern can be found in a lot of software out there.<br /><br /><b>UPDATE:</b><br /><b><br /></b><br />The issue has been fixed a few hours (or minutes?) after my post!<br />Check<br />https://review.source.android.com/Gerrit#patch,sidebyside,9684,2,graphics/java/android/graphics/drawable/Drawable.java<br /><br />To see how simple the change was!<br /><br /><br />Just take the Eclipse Memory Analyzer and have a look at your application(s).<br />May the force be with you! ;)<br /><br />Update 2: <br />In Android 2.3 (Gingerbread) <a href="http://android.git.kernel.org/?p=platform/dalvik.git;a=blob_plain;f=docs/heap-profiling.html;hb=HEAD">a new command</a> is available to trigger the heap dump. <br />The kill command is not supported anymore. <br /><br /><br /><script type="text/javascript">var dzone_url = 'http://kohlerm.blogspot.com/2009/04/analyzing-memory-usage-off-your-android.html';</script><br /><script type="text/javascript">var dzone_style = '1';</script><br /><script language="javascript" src="http://widgets.dzone.com/widgets/zoneit.js"></script>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com34tag:blogger.com,1999:blog-6817331.post-71407632927426687422009-04-08T08:16:00.000-07:002009-04-14T23:59:20.531-07:00Some facts about the JVM used by Google's App engine<div xmlns="http://www.w3.org/1999/xhtml">As you probably have noticed Google has released <a href="http://googleappengine.blogspot.com/2009/04/seriously-this-time-new-language-on-app.html" target="_blank">Java support for it's App Engine</a>!<br />
<br />
I wrote a small servlet to find out more about which Java they are running: <br />
<br />
java.specification.version: 1.6<br />
java.vendor: Sun Microsystems Inc.<br />
line.separator: <br />
<br />
java.class.version: 50.0<br />
java.util.logging.config.file: WEB-INF/logging.properties<br />
java.specification.name: Java Platform API Specification<br />
java.vendor.url: http://java.sun.com/<br />
java.vm.version: 1.6.0_13<br />
os.name: Linux<br />
java.version: 1.6.0_13<br />
java.vm.specification.version: 1.0<br />
user.dir: /base/data/home/apps/wanlatency/1.332645732335305520<br />
java.specification.vendor: Sun Microsystems Inc.<br />
java.vm.specification.name: Java Virtual Machine Specification<br />
java.vm.vendor: Sun Microsystems Inc.<br />
file.separator: /<br />
path.separator: :<br />
java.vm.specification.vendor: Sun Microsystems Inc.<br />
java.vm.name: Java HotSpot(TM) Client VM<br />
file.encoding: ANSI_X3.4-1968<br />
Total Memory 104857600<br />
Free Memory 6293011<br />
<br />
Note that the Free Memory does not change if you allocate 10Mbyte (nor the Total memory). It might therefore by faked. <br />
Allocating 1Gbyte results in a very simplistic error.<br />
<br />
They use jetty!<br />
<br />
<span style="border-collapse: separate; color: black; font-family: 'Times New Roman'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="font-family: Arial; font-size: 13px; line-height: 19px; text-align: left;"></span></span><br />
<pre class="ae-logs-expanded ae-pre-wrap" style="font-family: 'Courier New'; margin: 0px; padding: 0.3em 0px 0.5em 1.5em; white-space: pre-wrap;"><span style="border-collapse: separate; color: black; font-family: 'Times New Roman'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="font-family: Arial; font-size: 13px; line-height: 19px; text-align: left;">java.lang.OutOfMemoryError: Java heap space at org.kohlerm.wanTestGoogleServlet.doGet(wanTestGoogleServlet.java:24) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1093) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:237) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:313) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:506) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:830) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:63) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:125) at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:235) at com.google.apphosting.base.RuntimePb$EvaluationRuntime$6.handleBlockingRequest(RuntimePb.java:4547) at com.google.apphosting.base.RuntimePb$EvaluationRuntime$6.handleBlockingRequest(RuntimePb.java:4545) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest(BlockingApplicationHandler.java:24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:359) at com.google.net.rpc.impl.Server$2.run(Server.java:792) at com.google.tracing.LocalTraceSpanRunnable.run(LocalTraceSpanRunnable.java:56) at com.google.tracing.LocalTraceSpanBuilder.internalContinueSpan(LocalTraceSpanBuilder.java:489) at com.google.net.rpc.impl.Server.startRpc(Server.java:748) at com.google.net.rpc.impl.Server.processRequest(Server.java:340)</span></span></pre><span style="border-collapse: separate; color: black; font-family: 'Times New Roman'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="font-family: Arial; font-size: 13px; line-height: 19px; text-align: left;"></span></span></div><br />
<br />
<br />
Because my "hobby" is <a href="http://kohlerm.blogspot.com/search/label/memory">memory usage analysis</a>, I ran another test on the appengine and it seems you can allocate about 110Mbyte. <br />
Allocation seems to be very slow. I first tried to allocate a byte array in 1Mbyte steps, but until it reached 100 Mbyte it would need 30 seconds "CPU time". <br />
Maybe they persist larger blocks to disk???<br />
<br />
<br />
<script language="javascript" src="http://widgets.dzone.com/widgets/zoneit.js">
</script>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com14tag:blogger.com,1999:blog-6817331.post-2671434041349743072009-03-23T16:25:00.000-07:002009-05-05T02:19:24.398-07:00Leaks are easy to find, but memory usage analysis is bit more difficult<big><strong>Leaks again</strong></big><br /><br />Last time I talked about how easy it is to <a href="http://kohlerm.blogspot.com/2009/02/memory-leaks-are-easy-to-find.html" target="_blank">find memory leaks in Java</a> using the <a href="http://en.wikipedia.org/wiki/Dominator_(graph_theory)" target="_blank">dominator</a> tree feature of the <a href="http://www.eclipse.org/mat/" target="_blank">Eclipse Memory Analyzer</a>.<br />If you haven't read this post, I recommend you to do so, because this post will assume that you know the meaning of the terms "retained size" and "dominator tree".<br /><br /><br />Why does this work so well? The reason is that memory leaks in Java are not really "classical" leaks in the strict sense. Let's check what <a href="http://en.wikipedia.org/wiki/Memory_leak" target="_blank">Wikipedia says about memory leaks</a>:<br /><blockquote>"many people refer to <em>any</em> unwanted increase in memory usage as a memory leak, even if this is not strictly accurate"<br /><br />and<br /><br />"Typically, a memory leak occurs because <a href="http://en.wikipedia.org/wiki/Dynamic_memory_allocation" title="Dynamic memory allocation">dynamically allocated</a> memory has become <a href="http://en.wikipedia.org/wiki/Unreachable_memory" title="Unreachable memory">unreachable</a>."</blockquote>The later cannot happen in languages such as Java that have built-in automatic <a href="http://en.wikipedia.org/wiki/Garbage_collection_(computer_science)" title="Garbage collection (computer science)">garbage collection</a>, also <a href="http://www.infoq.com/presentations/preston-werner-conceptual-algorithms" target="_blank">Ruby does not seem to be bug free in this area</a>.<br /><br />So because those leaks in Java are "only" unwanted (unbound) increase of memory usage, the typical reason for them is that people forget to remove an object out of some collection/array or a recursive data structure,such as a tree. This might sound stupid and you (and me) would of course never make such a simple mistake ;)<br /><br />But look at the following example:<br /><pre name="code" class="java"><br />try<br />{<br />doSomething(thing); // does IO<br />collection.remove(thing);<br />}<br />catch (IOException e)<br />{<br />// should not happen<br />}<br /></pre><br /><br />"thing" will not be removed if "doSomething" throws an IOException (or any other exception). OMG <a href="http://www.joelonsoftware.com/items/2003/10/13.html" target="_blank">Joel Spolsky</a> was right when he said:<blockquote><b><span>"I consider exceptions to be no better than "goto's""</span></b></blockquote><br />The correct way would be:<br />try<br />{<br /><em>doSomething</em>(thing); <span style="color:#3f7f5f;">// does IO</span><br />}<br /><strong><span style="color:#7f0055;">catch</span></strong> (IOException e)<br />{<br /><span style="color:#3f7f5f;">// should not happen</span><br />}<br /><span style="color:#7f0055;">finally</span><br />{<br />collection.remove(thing);<br />}<br />So I talked enough of leaks. I promise you if you regulary analyze heap dumps taken at the end of a performance test run, after while of fixing, you will not see a lot of leaks anymore If you still think that you need to know more a about leaks. I recommend you to check this <a href="http://java.dzone.com/news/how-fix-memory-leaks-java">excellent tutorial</a>.<br /><big><strong>High memory usage</strong></big><br /><br />You might still see high memory usage, and your users might hate that as much as leaks, because performance degradation can be similiar.<br /><br />OK, why is high memory more difficult to analyze?<br /><br />You might use the dominator tree to find some big objects and you might also use it to figure out some cause of high memory usage. Because it's a tree it's easy to see where the memory is used:<br /><br /><br /><img alt="domtree.PNG" height="245" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiesF4zT3EUu_yAecJQlhvjBzr9rF2S2HuggBcW45-yPBByi6wApL0LJCP-zKEsWjd0zjTKjZs8O5Ii1rDwHwPRCIaIL9LShPs4NjaMudEuIMh7BG8P8mmx0BHZzxAfTIM4xxY/" width="405" /><br /><br />You just have to look at the pathes down the tree to find out where the most memory is used/<a href="http://kohlerm.blogspot.com/2009/02/how-to-really-measure-memory-usage-of.html">retained</a> by single objects.<br /><br /><strong>But</strong> in general the dominator tree view alone (without using some advanced functions, that I will skip for now) will not help you to find the reason why for example all those Strings are there:<br /><img alt="Strings.PNG" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgY6AmjJe7Dx1LEXcK0JkTrAmQWb9292DrZyIURmx3qZeyjRHOhP9T28OgB_01bA6bmkszVNIW5wgwqeE-nwmoVPO3BWdLYZwMeV7oN3YnaXVWl50vbPszvKA6q_dF2Mufkf08/" /><br /><br />Fortunately there is the <strong>"immediate dominators"</strong> query in the <a href="http://www.eclipse.org/mat/" target="_blank">Eclipse Memory Analyzer</a> that is based on the dominator tree that can help here and that also is the used internally by most of the advanced queries. The <strong>"immediate dominators"</strong> query is one of the key innovations in the the <a href="http://www.eclipse.org/mat/" target="_blank">Eclipse Memory Analyzer</a>. Even the commercial <a href="http://yourkit.com/" target="_blank">Yourkit profiler</a> does not seem to have it yet, also they now also have a dominator tree functionality.<br /><br /><big><strong>Immediate Dominators</strong></big><br /><big><span>So what is a "immediate dominator"? Let's have look at a simple example where the "business object" of class "Order" references a LinkedList of Strings:</span></big><br /><img alt="imm1.png" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMCxQDSfXdey4KtGqqurY8MtuyA03W1EZRnDIhnmA5OnNnMpfGU9G6N76BODJ6lP403-OUjsvfF-p1W2m0XXNZTOq7kAE8o2-Uzk1aIOvQ6YjdvQ5MdsXd3r2ThtsWo4JX6Pk/?imgmax=800" width="400" /><br />If we look at String2 first we can find the LinkedList$Entry 2 is the "closest" object hat dominates it. If we could remove LinkedList$Entry 2, the Object String 2 would also be reclaimed by the garbage collector. We say "LinkedList$Entry 2" is the immediate dominator of "Object String 2". Note that there's always one<br />Let's have a look at the immediate dominators up to the Order object:<br /><img alt="imm2.png" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhG7Mor9j1wgOo6SBzQJ7uzdNOiIl3IcK3bn1v_ANjAIFMDXOmjMtONqzGBHEqCL-8YzQZnYdwOrsWtp97PdUtuyfpeUVjpE_2ziwQ3E5uP9igxN4Z-YhfobiZBZRZlJoFQZ3A/?imgmax=800" width="400" /><br />Note that LinkedList$Entry 1 is not an immediate dominator for LinkedList$Entry 2, because after removing it there would still be links from LinkedList$Entry 0 to LinkedList$Entry 2. We can do the same for the String 0 and String 1 and we will get the dominator tree:<br /><br /><img alt="imm3.png" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyi71p6Azc-D3ieyKScdyijwrpDsTm4NfhIqCzn-Ssm6BjrhPVY0AOIE7m0gZiWVqCK8aMFMDMAu6Oh9al_9GwH1qfbCTDe3BkF0XpT9ZZBebqiRiZvft3u7d0pWdoI7UC3Q4/?imgmax=800" width="400" /><br /><br />No if we ask ourself the question "why are all those Strings still there", we see that if we filter all JDK classes out of the dominator tree it's easy:<br /><img alt="imm4.png" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxTUKAtBi0eeqINQnjDJ1Ab-zfdNgNkb3eIjvBF8EXHp5QSIvtSWjPXSNYLFR6T55IlEy978QOAhnH3lARCMfx2PlyFmhCaDUCeLgxHwSpk0qEG1dsjhhfGmL7CGxAWbHrPhg/?imgmax=800" width="400" /><br /><br /><big><span>The immediate dominators query in MAT basically lets you walk up the dominator tree and shows you the dominators aggregated by the class:</span></big><br /><br /><big><img alt="immMAT.PNG" height="205" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixCENdit4UYnfKf2lTkwjyf27wBRiN9L99ExZGg4hZuNFJ2oD9ZvdpCplbWf5yg7AEBA9a-m_INpyCV3isG5tbAObwQTmIH3koWubGZt4XXTE5-YA2xhuuaMvx8MrkcoZVwJA/" width="400" /></big><br /><br />This is really a screenshot from an existing heap dump that I took some time ago from Eclipse. You can see for example the <a href="http://kohlerm.blogspot.com/2008/05/analyzing-memory-consumption-of-eclipse.html">famous Dictionary</a> of the spell checker plugin retaining 74393 Strings.<br />So now how can I find out where memory usage could be reduced?<br />With Strings it's pretty easy, you use the group_by_value in MAT. For the example above I applied it to the Strings dominated by ResolvedBinaryField in the first line:<br /><img alt="group_by_value.PNG" height="204" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiel8X_-nH4SEWf9qlERw8mYL2KQ-BZ8Vc_GWAZfCn-SrXmnltkelr6MOHjR4Xma9zhjzKR8SPgq1lliZMg01tKj93CeuHhHl6xTP4yInaddzk9KszuKRsB1h0CEg08SAlw3nE/" width="400" /><br /><br />Yes,there are really 6969 duplicates of "Ljava.lang.String;" retained only by instances of this class! Disclaimer: And no dear Netbeans "fanboys", Eclipse is not really worse than your beloved IDE in this area ;)<br />Strings are immutable and I wonder what would happen if people would really use more <a href="http://clojure.org/">immutable data structures</a>.<br /><br />But not only Strings are interesting when you look at minimizing memory usage. Strings are just convenient because they are (usually) human readable. You can still often use Strings to find Objects which are equal but not identical, because if equal but not identical objects are created usually those objects also reference Strings that are equal but not identical.<br />The main question that you always have to ask yourself when trying to minimize memory usage is :<br /><br /><b><span class="Apple-style-span" style="font-size:x-large;">Do I need these equal but not identical objects?</span></b><br /><b><br /></b><br /><b><span class="Apple-style-span" style="font-weight: normal;">In a single threaded environment the answer is usually that you don't those copies of objects. </span></b><br />In a highly concurrent environment, reducing the copies might introduce contention, because you have to share objects and you will need to check whether you already have this object. Strings again are relatively safe to optimize in this regard, because they are immutable, so no synchronization is needed to access them.<br /><b><span class="Apple-style-span" style="font-weight: normal;"><br /></span></b><br />Having a query in MAT for the "algorithm" I described here for finding duplicated Strings, would be very helpful (there is a similiar but simpler "duplicated Strings" query already built in) .<br />I have done exactly that quite some time ago, but the query was not yet "production ready". There's some hope that it will appear in the standard MAT soon, stay tuned!<br /><br /><script type="text/javascript">var dzone_url = 'http://kohlerm.blogspot.com/2009/03/leaks-are-easy-to-find-but-memory-usage.html';</script><br /><script type="text/javascript">var dzone_style = '1';</script><br /><script language="javascript" src="http://widgets.dzone.com/widgets/zoneit.js"></script>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com14tag:blogger.com,1999:blog-6817331.post-67636400686797054502009-03-09T16:56:00.000-07:002009-03-11T05:28:00.717-07:00JavaFX's memory overhead, some (high) numbers<div xmlns="http://www.w3.org/1999/xhtml">Lately I took a look at the <a target="_blank" href="http://tweetbox.org/">TweetBox </a> Twitter client which is build with <a href="http://javafx.com/" target="_blank">JavaFX</a>(<span class="copy">© 2008-2009 Sun Microsystems)</span>. I thought it would be fun to check whether a Java based Twitter client would have less memory usage problems than those popular <a target="_blank" href="http://www.adobe.com/products/air/">Adobe Air</a>(<span class="copy">© Adobe)</span> based clients, such as <a target="_blank" href="http://www.tweetdeck.com/beta/">Tweetdeck</a>. If you search for memory leak in <a target="_blank" href="http://twitter.com/">Twitter</a> you will get a lot of references about <a target="_blank" href="http://www.tweetdeck.com/beta/">Tweetdeck</a>, which might only be caused by it's a popularity. Because <a href="http://javafx.com/" target="_blank">JavaFX</a> is JVM based it actually has an advantage in the area of memory usage analysis over <a target="_blank" href="http://www.adobe.com/products/air/">Adobe Air</a>.<br /><br /><br /><a target="_blank" href="http://tweetbox.org/">TweetBox </a> currently does not perform very well, but the author is working on improving it and I also tried to help him out a little bit.<br /><br />When looking for memory usage using my <a href="http://www.eclipse.org/mat/" target="_blank">favorite tool</a> I was quite surprised to see the high overhead of variables in JavaFX.<br /><br />Let's have a look at some numbers for the shallow size of variables (32 bit CPU):<br /><b><br /><big>Shallow Size</big><br /><br />Boolean:</b> 24 bytes (Yes, this is for one bit of information)<br /><b>Integer: </b>24 bytes<br /><b>Objects:</b> 24 bytes<br /><b>Floats:</b> 24 bytes<br /><b>Sequence: </b>32 bytes(never smaller than 48 bytes)<br /><br /><br />JavaFX objects seems to have a relatively large number of fields, so this overhead might be really become significant, also as always, it depends on your application.<span style="font-weight: bold;"> For comparsion a boolean in Java takes 1 byte and Objects have a general overhead of 8 bytes (SUN 32 bit JVM). [updated] Note that even a Boolean object in Java would in practice only require 4 Bytes for the Reference, because only 2 of them (for true and false) are needed. </span></div><div xmlns="http://www.w3.org/1999/xhtml"><span class="Apple-style-span" style="font-weight: bold;">An idea could be to have a native BitSet in JavaFX, which would only need one bit for each boolean value (plus a fixed overhead). </span></div><div xmlns="http://www.w3.org/1999/xhtml"><br /></div><div xmlns="http://www.w3.org/1999/xhtml">For details about the memory usage of Java objects check my post <a href="http://kohlerm.blogspot.com/2008/12/how-much-memory-is-used-by-my-java.html" target="_blank">here</a>.<br /><br />But when looking at the retained size it even gets worse. Due to JavaFX's fancy binding mechanism those objects often reference other objects which causes the retained size to "explode".<br /><br /><br /><b><big>Average "<a href="http://kohlerm.blogspot.com/2009/02/how-to-really-measure-memory-usage-of.html" target="_blank">Retained size</a>"<br /></big><br />Boolean:</b> <span style="color: rgb(204, 0, 0);">111 bytes </span><br /><br /><b>Integer: </b>53 bytes<br /><br /><b>Objects:</b> 76 bytes<br /><br /><b>Floats:</b> 53 bytes<br /><br /><b>Sequence: </b>84 bytes<br /><br />111 bytes average (!) for a Boolean is absurdly high and I'm not sure yet whether that's because Tweetbox using it in the wrong way or because of a bug. It is caused by long chain's of com.sun.javafx.runtime.location.DynamicDependentLocation.<br />I have no clue what that is and even Google doesn't seem to know.<br /><br />I hope JavaFX's memory usage will improve over time. Right know you have to be very careful to not run into memory usage problems.</div><div xmlns="http://www.w3.org/1999/xhtml"><div xmlns="http://www.w3.org/1999/xhtml">Note that other dynamic languages on the Java stack also currently have a higher memory overhead than Java, because most of them can't directly map to the Java object model.</div><div><br /></div></div><div xmlns="http://www.w3.org/1999/xhtml">I assume that the footprint of JavaFX Mobile is similiar. </div><div xmlns="http://www.w3.org/1999/xhtml">IMHO Googles decision to go for plain Java on it's Android platform was a smart decision</div><br /><br /><br />[update]: Michael Heinrichs just blogged about <a href="http://blogs.sun.com/michaelheinrichs/entry/best_practices_for_javafx_mobile4">the difference between "def" and "var" in JavaFX Mobile</a>. It seems to me that "def" could help to optimize memory usage. <br /><br /><script type="text/javascript" src="http://www.reddit.com/button.js?t=1"></script><br /><script type="text/javascript">var dzone_url = 'http://kohlerm.blogspot.com/2009/03/javafx-memory-overhead-some-high.html';</script><br /><script type="text/javascript">var dzone_style = '1';</script><br /><script language="javascript" src="http://widgets.dzone.com/widgets/zoneit.js"></script>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com11tag:blogger.com,1999:blog-6817331.post-63208680283172093152009-02-16T14:24:00.000-08:002009-03-30T13:49:13.938-07:00Memory leaks are easy to findLast time I talked <a href="http://kohlerm.blogspot.com/2009/02/how-to-really-measure-memory-usage-of.html">about the dominator tree </a> and how it can be used to find the biggest objects in your heap easily.<br />
<br />
So what exactly is the dominator tree?<br />
<br />
<b><span style="font-size:large;">Dominators<br />
</span></b><br />
An object A <b>dominates </b>on an object B if all the paths to object B pass through object A.<br />
<br />
Remember that the Garbage Collector removes all objects that are not referenced anymore. If A dominates B and A could be removed from memory, that means that there's no path anymore that leads to B. B is therefore unreachable and would be reclaimed by the Garbage Collector.<br />
One could also say that A is the single object that is responsible for B still being there!<br />
<br />
<b><span style="font-size:large;">The Dominator Tree</span></b><br />
<br />
Using the the "dominates" relationship we can create a dominator tree out of the the graph of objects in memory. At each node of this tree we store the amount of memory that would be freed (= <a href="http://kohlerm.blogspot.com/2009/02/how-to-really-measure-memory-usage-of.html">retained size</a>).<br />
At the top of the tree we have a "virtual" root object, which we also use to represent objects that don't have "real" single dominator.<br />
Here's an example of an object tree (on the left) and the corresponding dominator tree (on the right) :<br />
<br />
<br />
<span><a href="http://www.blogger.com/post-edit.g?blogID=6817331&postID=6320868028317209315" name="acrocheckReadOnlyRange"></a></span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9Q7vfSOPQCWAITc26Hf0YnYsSVi-wYjImOkwU4a0j2tZL-QS4csgj9HxLRKEVuxJ8KPJV45C9dr4eMOF-yUMdjEiB-y_RQwPqEdsgok0lRIbiLexxAlQ2bxXVQiMmn_nenno/s1600-h/dominator.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9Q7vfSOPQCWAITc26Hf0YnYsSVi-wYjImOkwU4a0j2tZL-QS4csgj9HxLRKEVuxJ8KPJV45C9dr4eMOF-yUMdjEiB-y_RQwPqEdsgok0lRIbiLexxAlQ2bxXVQiMmn_nenno/s320/dominator.gif" border="0" /></a></div><br />
<ol><li>Note that A, B and C are dominated by a "virtual" root object.<br />
</li>
<li>Note that the dominator relationship is <b>transitive;</b>C dominates E which dominates G therefore C also dominates G. </li>
</ol><br />
Because of the transitivity of "dominated", the retained size of a parent object within the dominator tree is always greater than the sum of it's child objects.<br />
<br />
<b>To get the biggest objects we just need to sort the second level of the dominator tree (the first level is the "virtual" root object) by retained size. </b><br />
<br />
Now if you are looking to find a memory leak, and you have no <a href="http://android-developers.blogspot.com/2009/01/avoiding-memory-leaks.html">a priori knowledge</a> that could help you, the typical approach is to run a test that reproduces the leak, and then try to somehow figure out what is leaking.<br />
<br />
<span style="font-size:130%;"><span style="font-weight: bold;">Do we really need Object allocations tracing?</span></span><br />
In my experience people often seem to believe that finding leaks requires recording object creations, also called "object allocations tracing "sometimes, because you want to know where in the code objects are always allocates but never released.<br />
Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft) has <a href="http://blogs.msdn.com/tess/archive/2009/02/03/net-memory-leak-to-dispose-or-not-to-dispose-that-s-the-1-gb-question.aspx">an example of how this method for finding leaks can be applied to .NET</a> applications. <i>Dear Microsoft I encourage you to look at the </i><a href="http://www.eclipse.org/mat/">Eclipse Memory Analyzer</a> ;)<br />
<br />
<br />
<span style="font-size:130%;"><span style="font-weight: bold;">All you need is the dominator tree</span></span><br />
<br />
With the dominator tree finding these kind of leaks is much easier and <b>you don't need the high overhead of allocation tracing</b>, which is typically not acceptable in a production environment. Allocation tracing has it's uses, but in general IMHO it is overrated.<br />
<br />
It's almost guaranteed that a significant memory leak will show up at the very top of the dominator tree!<br />
The <a href="http://www.eclipse.org/mat/">Eclipse Memory Analyzer</a> not only supports the dominator tree but can even <a href="http://dev.eclipse.org/blogs/memoryanalyzer/2008/05/27/automated-heap-dump-analysis-finding-memory-leaks-with-one-click/">find memory leaks automatically</a>, based on some heuristics.<br />
<br />
The dominator tree can also be used to find <b>high memory usage </b>in certain areas, which is a <b>much harder problem</b>. More on this in the next post.<br />
<br />
<br />
<script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script><br />
<script type="text/javascript">var dzone_url = 'http://kohlerm.blogspot.com/2009/02/memory-leaks-are-easy-to-find.html';</script><br />
<script type="text/javascript">var dzone_style = '1';</script><br />
<script language="javascript" src="http://widgets.dzone.com/widgets/zoneit.js"></script>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com22tag:blogger.com,1999:blog-6817331.post-14818679250453296472009-02-03T12:28:00.001-08:002009-03-06T15:24:14.747-08:00How to really measure the memory usage of your objects<div xmlns="http://www.w3.org/1999/xhtml">Measuring and analyzing the Memory consumption in Java and other OOP languages is not an easy task, because all the objects form a big highly connected graph. Only <a href="http://kohlerm.blogspot.com/2008/12/how-much-memory-is-used-by-my-java.html" target="_blank">counting the flat, also called "shallow" size, of the objects</a>, is not always helpful:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRZMqgRe6IXe2ZfBc-RWeq3KMqJIuCpU41fK6qKSd6iBXaHFmZj1QOctkdIj3TPTZknDoS8KVS3BdHFEiMDnax2ohJUc9YO59FLhi0wnqsMeopSGZ-2V8buUw3cPc-A-ox2No/s1600-h/histo.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRZMqgRe6IXe2ZfBc-RWeq3KMqJIuCpU41fK6qKSd6iBXaHFmZj1QOctkdIj3TPTZknDoS8KVS3BdHFEiMDnax2ohJUc9YO59FLhi0wnqsMeopSGZ-2V8buUw3cPc-A-ox2No/s320/histo.png" border="0" /></a></div><br />
This figure shows a typical memory histogram. In general this table does not help very much to figure out which objects really cause the high memory usage. Typically String and char[] will show up in the list of the classes with the highest shallow size, just because they are frequently used.<br />
There is probably a correlation between the high number of char[] instances and the high number of String instances(indicated by the yellow arrows), but we cannot really know from the information in the table. We also don't know whether the Order object could potentially be the root cause for all those Objects there.<br />
<br />
So how can we find out, who is really responsible for the memory usage?<br />
<br />
Let's take a step back and ask ourselves how Garbage Collection works in a language like Java (and also in other "modern" languages ;)).<br />
<big><b><br />
Garbage Collection</b></big><br />
<br />
The general idea is that the Garbage Collector will reclaim all objects that cannot be reached from a so called "GC root object".<br />
<br />
GC root objects are<br />
<ul><li>Local variables on the stack</li>
<li>Parameters of functions on the stack</li>
<li>JNI native references</li>
<li>All classes loaded by the bootstrap Loader </li>
</ul>Now, let's come back to our original problem, we can ask us the interesting question:<br />
<b>"How much memory would be freed, if we would remove all those "Order" objects and afterwards a full GC would run?"</b><br />
This would give us a measure for how much memory would be freed if those objects would not be there.<br />
We call this measure the <b>"retained size"</b>. The set of objects that would be freed is called the <b>"retained set"</b>.<br />
With the "retained set" we can figure out how many Strings would be freed if those "Order" objects would not be there. We can therefore solve the problem to find the objects "responsible" for the memory usage.<br />
<big><b><br />
Definition "Retained size/set"</b></big><br />
<br />
<b>Retained set</b> of X is the set of objects which would be collected by the GC<br />
<b>Retained size</b> of X is the sum of shallow sizes of all objects in the retained set of X, i.e. memory kept alive by X.<br />
Generally speaking, shallow size of an object is its "flat" size in the heap whereas the retained size of the same object is the amount of heap memory that will be freed when the object is garbage collected.<br />
The retained set for a leading set of objects, such as all objects of a particular class or all objects of all classes loaded by a particular class loader or simply a bunch of arbitrary objects, is the set of objects that is released if all objects of that leading set become inaccessible. The retained set includes these objects as well as all other objects only accessible through these objects. The retained size is the total heap size of all objects contained in the retained set.<br />
<br />
<big><b>How to find the object with the biggest retained size?<br />
</b></big><br />
The <a href="http://www.eclipse.org/mat/" target="_blank">Eclipse Memory Analyzer</a> was built from the beginning with support for computing the retained set/size of objects.<br />
The retained size was pioneered, as far as I know, by the commercial<br />
<a href="http://yourkit.com/" target="_blank">Yourkit</a> profiler(great product!). When we started to develop the Eclipse Memory Analyzer (SAP Memory Analyzer at that point in time), Yourkit already had this very nice feature to find the single object with the biggest retained size. We still don't know how Yourkit does it, but for sure at that point in time they used a different approach than we do now. The reason is that it took Yourkit several minutes to get the first 10 (I think) biggest (in terms of retained size) objects in the heap. We knew that this was already still much faster than the naive method,<b> that would have to simulate a Garbage Collector run each of the object in the heap. With millions of objects on the heap this would have taken years!<br />
</b><br />
So I went on to research, what kind of Graph algorithms could help us to speed things up. After a lot of Google searches I finally found the <a href="http://en.wikipedia.org/wiki/Dominator_%28graph_theory%29" target="_blank">dominator tree</a>.<br />
<br />
<big><b>The dominator tree<br />
</b></big><br />
The <a href="http://en.wikipedia.org/wiki/Dominator_%28graph_theory%29" target="_blank">dominator tree</a> is a datastructure that allows you to answer the question about the biggest objects in almost no time. After further research I found that it could be build in (almost) O(n) time, by a complicated algorithm invented by <a href="http://www.cl.cam.ac.uk/~mr10/lengtarj.pdf" target="_blank" title="LengauerTarjan ">Lengauer</a> and Tarjan. Since the plan by the architect of the project, was always to store precomputed information on disk, we could compute the dominator tree once and then store it on disk. Therefore opening a heap dump for the first time is a little bit slow (but still often faster than with any other tool), but opening it again is blazing fast (just a few seconds).<br />
Still implementing the algorithm was a high risk, because at that point in time relatively new paper <a href="http://www.springerlink.com/content/6lywrux2mg92kmfc/" target="_blank">"Finding Dominators in Practice"</a> was talking about graphs with at most a few thousand nodes, whereas we wanted to run it on a few millions of nodes on a 32bit machine.<br />
<br />
Still the team would get it done (I was always only in "consultant" role), and it turned out that the dominator tree would not only allow us to speed up the process of finding the biggest objects, but would also allow us to really significantly simplify the process of finding the root causes for high memory usage.<br />
<br />
More on this in the <a href="http://kohlerm.blogspot.com/2009/02/memory-leaks-are-easy-to-find.html">next post</a>.<br />
Stay tuned!</div><br />
<br />
<script type="text/javascript">var dzone_url = 'http://kohlerm.blogspot.com/2009/02/how-to-really-measure-memory-usage-of.html';</script><br />
<script language="javascript" src="http://widgets.dzone.com/widgets/zoneit.js"></script>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com16tag:blogger.com,1999:blog-6817331.post-34000661936048243122009-01-26T15:07:00.000-08:002009-02-02T14:17:28.303-08:00First Android/Dalvik heap dump loaded into the Memory Analyzer<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCFacJ_DlioHKI5GKylGFHF_RtaQG4Y0yEOhgxNuFCj3Qf8qhRWf3HvG9fmm_3Q-_mIg3r3QkWBmP3EC6QctQJVBSOED77mKqGFjPoc5RqblHB0f8RXva4QSNpWZ5UCYPNKU4/s1600-h/MatFirstAndroidDump.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 307px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCFacJ_DlioHKI5GKylGFHF_RtaQG4Y0yEOhgxNuFCj3Qf8qhRWf3HvG9fmm_3Q-_mIg3r3QkWBmP3EC6QctQJVBSOED77mKqGFjPoc5RqblHB0f8RXva4QSNpWZ5UCYPNKU4/s400/MatFirstAndroidDump.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5295743574030472722" /></a><br />Here you go. <br />I thought that Linux would be a better choice then Windows :)<br /><br />Ok, to be honest the converter tool from the Dalvik format to hprof did not work on Windoof.<br /><br />This is showing an old version of the SAP Memory Analyzer, because I'm lazy and no newer version was available on the Linux machine. <br /><br />Thanks to Romain Guy for his assistance. <br />More on this tomorrow.<br /><br />[update:] Could not resist and looked for duplicated Strings: <br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2pn4LRN2uQvxUh5LVg4ctycc_IlaIAAwF72wFSghTXUOF6JDCyG1XkwJWAsLo3nhEeRWCP2on0UgN8sEoKj_0CLyf3QEDPDZleoSjSfGXTZ8nL5QwcdJ9165NVqWu6K2HD0U/s1600-h/AndroidDuplStrings.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 279px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2pn4LRN2uQvxUh5LVg4ctycc_IlaIAAwF72wFSghTXUOF6JDCyG1XkwJWAsLo3nhEeRWCP2on0UgN8sEoKj_0CLyf3QEDPDZleoSjSfGXTZ8nL5QwcdJ9165NVqWu6K2HD0U/s400/AndroidDuplStrings.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5295748008903943682" /></a><br /><br />There's quite some potential for optimizations ...Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com7tag:blogger.com,1999:blog-6817331.post-51444780586611388712009-01-07T00:40:00.000-08:002009-02-16T00:59:17.026-08:00Is java.lang.String.intern() really evil?<div xmlns='http://www.w3.org/1999/xhtml'>Domingos Neto just posted<br /><a href='http://www.codeinstructions.com/2009/01/busting-javalangstringintern-myths.html'>Busting java.lang.String.intern() Myths</a>. <br />In general I like the post,because I think this is an important topic, because in my experience <span style="font-weight:bold;">Strings typically consume about 20% to 50% of the memory in Java applications</span>. It's therefore important to avoid <b>useless</b> copies of the same String, to reduce memory usage. <br />But first some comments to the post above: <br /><br /><blockquote>Myth 1: Comparing strings with == is much faster than with equals()</blockquote><br /><blockquote>busted! Yes, == is faster than String.equals(), but in general it isn't near a performance improvement as it is cracked up to be.<br /></blockquote><br /><br />I agree, it doesn't make sense to intern Strings to be able to use == instead of equals. But the real reason is that String.equals already does == in the first place. If your Strings are identical you automatically get the speed advantage because usually equals will be inlined!<br /><br /><blockquote>Myth 2: String.intern() saves a lot of memory<br /></blockquote><br /><br />Here I disagree. <span style="font-weight:bold;">String.intern() can help you to save a lot of memory</span> because it can be used to avoid holding duplicates of Strings in memory. <br />Imagine you read a lot of Strings from some File and some (or a lot) of these Strings might actually be identifiers such as the name of a City or type(class). If you don't use String.intern()(or a similiar mechanism using a Set), you will hold copies of those Strings in memory. The number of this unnecessary copies will often increase with the number of Strings you read, and therefore you will really save a significant amount of memory. <br /><br /><br />In my experience duplicated Strings are one of the most common memory usage problems in Java applications. <br />Check for example my blog post about the <a target='_blank' href='http://kohlerm.blogspot.com/2008/05/memory-consumption-of-netbeans-versus.html '>Memory usage of Netbeans versus Eclipse</a>.<br /><br /><br />That those interned Strings end up in Perm space IMHO is not a big issue. You need to setup perm space these days to pretty high values anyway, check for example my blog post about the <a target='_blank' href='http://kohlerm.blogspot.com/2008/11/blog-post.html'>perm space requirements of Eclipse</a>.<br />Still in the SAP JVM we also introduced a new option to store those interned Strings in the old space. <br />Maybe someone wants to implement this option for the OpenJDK as well ;)<br /><br /><br /><b><big>Issues with String.intern()</big></b><br />Now you might think that String.intern() is not problematic at all, but unfortunately there are a few issues. <br /><br /><ul><li>Not all JVM's have fast implementations for String.intern(). For example HP's JVM used to have problems until recently. </li><li/><br /><li>Additional contention is introduced and you have no control over it because String.intern() is native </li><br /><br /><li/></ul>Unfortunately I'm not aware of a good pure Java replacement for String.intern(), because what really would be needed is a memory efficient ConcurrentWeakHashSet. Such a Collection would need to use WeakReferences which have a relatively high memory overhead. <span style="font-weight:bold;">Therefore my advice is still to use String.intern() if you need to avoid duplicated Strings.</span></div><br /><br /><br /><br /><script type="text/javascript">var dzone_url ="http://kohlerm.blogspot.com/2009/01/is-javalangstringintern-really-evil.html";</script><br /><script type="text/javascript">var dzone_style = '1';</script><br /><script language="javascript" src="http://widgets.dzone.com/widgets/zoneit.js"></script>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com9tag:blogger.com,1999:blog-6817331.post-44331366208085464612008-12-15T05:24:00.001-08:002008-12-16T02:22:52.831-08:00How much memory is used by my Java object?<div xmlns='http://www.w3.org/1999/xhtml'>There's currently some <a href='http://www.codeinstructions.com/2008/12/size-of-boolean-object-in-java.html' target='_blank'>buzz </a> around the size of certain Java objects <b>(Update: more on this below)</b>. <a href='http://www.codeinstructions.com/2008/12/java-objects-memory-structure.html' target='_blank'>Here</a> is some very good description of what you have to take into account to compute the (shallow) size of an java object. <br />I repost here the rules from my old blog at SDN, because I think they are a more compact description of the memory usage for java objects. <br /><br />The general rules for computing the size of an object on the SUN/SAP VM are :<br /><br /><big>32 bit</big> <br /><br /><blockquote>Arrays of boolean, byte, char, short, int: 2 * 4 (Object header) + 4 (length-field) + sizeof(primitiveType) * length -> align result up to a multiple of 8<br /><br />Arrays of objects: 2 * 4 (Object header) + 4 (length-field) + 4 * length -> align result up to a multiple of 8<br /><br />Arrays of longs and doubles: 2 * 4 (Object header) + 4 (length-field) + 4 (dead space due to alignment restrictions) + 8 * length<br /><br />java.lang.Object: 2 * 4 (Object header) <br /><br />other objects: sizeofSuperClass + 8 * nrOfLongAndDoubleFields + 4 * nrOfIntFloatAndObjectFields + 2 * nrOfShortAndCharFields + 1 * nrOfByteAndBooleanFields -> align result up to a multiple of 8</blockquote><br /><br /><big>64 bit<br /></big><br /><blockquote>Arrays of boolean, byte, char, short, int: 2 * 8 (Object header) + 4 (length-field) + sizeof(primitiveType) * length -> align result up to a multiple of 8<br /><br />Arrays of objects: 2 * 8 (Object header) + 4 (length-field) + 4 (dead space due to alignment restrictions) + 8 * length<br /><br />Arrays of longs and doubles: 2 * 8 (Object header) + 4 (length-field) + 4 (dead space due to alignment restrictions) + 8 * length<br /><br />java.lang.Object: 2 * 8 (Object header)<br /><br />other objects: sizeofSuperClass + 8 * nrOfLongDoubleAndObjectFields + 4 + nrOfntAndFloatFields + 2 * nrOfShortAndCharFields + 1 * nrOfByteAndBooleanFields -> align result up to a multiple of 8<br /></blockquote><br /><br /><br />Note that an object might have unused space due to alignment at every inheritance level (e.g. imagine a class A with just a byte field and class B has A as it's superclass and declares a byte field itself -> 14 bytes 'wasted on 64 bit system).<br /><br />In practice 64 bit needs 30 to 50% more memory due to references being twice as large. <br /><br /><br /><b>[UPDATE]</b><br /><br /><big><br /><b>How much memory does a boolean consume?</b></big><br /><br />Coming back to the <a href='http://www.codeinstructions.com/2008/12/size-of-boolean-object-in-java.html' target='_blank'>question of how much a boolean consumes</a>, yes it does consume at least one byte, but due to alignment rules it may consume much more. IMHO it is more interesting to know that <span style="font-weight:bold;">a boolean[] will consume one byte per entry and not one bit</span>,plus some overhead due to alignment and for the size field of the array. There are graph algorithms where large fields of bits are useful, and you need to be aware that, if you use a boolean[] you need almost exactly 8 times more memory than really needed (1 byte versus 1 bit). <br /><br /><big><b>Alternatives to boolean[]</b></big><br /><br />To store large sets of booleans more efficiently a standard <a target='_blank' href='http://java.sun.com/j2se/1.4.2/docs/api/java/util/BitSet.html'>BitSet</a> can be used. This will pack the bits into a compact representation that only uses one bit per entry (plus some fixed overhead)<br /><br />For the <a target='_blank' href='http://www.eclipse.org/mat/'>Eclipse Memory Analyzer</a> we had the need to use large fields of booleans (Millions of entries) and we know in advance how many entries we need. <br />We therefore implemented the class <a target='_blank' href='http://dev.eclipse.org/viewcvs/index.cgi/trunk/plugins/org.eclipse.mat.report/src/org/eclipse/mat/collect/BitField.java?root=Technology_MAT&view=markup'>BitField</a> that is faster than <a target='_blank' href='http://java.sun.com/j2se/1.4.2/docs/api/java/util/BitSet.html'>BitSet</a>. <br /><br />Still we only use this class on single core machines, because the disadvantage of it is that it is not thread save. On multicore machines we use boolean[] and I will explain in a later post how that works, because it's pretty tricky. <br /><br /><br />Note that these rules tell you only the flat (=shallow) size of an object, which is in practice pretty useless. <br />I will explain in my next post, how you can better measure how much memory your Java objects consume.</div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com9tag:blogger.com,1999:blog-6817331.post-72217118058681075822008-11-27T12:10:00.001-08:002008-12-01T15:00:08.927-08:00Slides for my MAJUG talk about the Eclipse Memory Analyzer available<div xmlns='http://www.w3.org/1999/xhtml'>I just uploaded the Slides that I used 2 days ago at my talk for the Mannheim Java User Group (In German) <br />
, to <a href='http://www.slideshare.net/' target='_blank'>slideshare</a>. There's a little bit of information lost, because of the conversion to PDF, but overall the content should be there:<br />
<br />
<div style='width: 477px; text-align: left;' id='__ss_793999'><a style='margin: 12px 0pt 3px; font-family: Helvetica,Arial,Sans-serif; font-style: normal; font-variant: normal; font-weight: normal; font-size: 14px; line-height: normal; font-size-adjust: none; font-stretch: normal; display: block; text-decoration: underline;' href='http://www.slideshare.net/kohlerm/majug112008-presentation?type=document' title='Eclipse Memory Analyzer MAJUG November 2008'>Eclipse Memory Analyzer MAJUG November 2008</a>[removed the live view because of performance problems with slideshare]</div><br />
<br />
The Slides are also available from <a href='http://majug.de/files/20081125.pdf'>here</a> in PDF format.</div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com4tag:blogger.com,1999:blog-6817331.post-6990163630137874442008-11-06T07:30:00.000-08:002009-02-02T14:15:20.948-08:00Is Eclipse bloated? Some numbers and a first analysisOn the e4 (Eclipse 4) mailing list there were lately some discussions about whether Eclipse is bloated or not, and what could be done to minimize bloat for e4. Some summary information can be found <a href="http://wiki.eclipse.org/E4/Pervasive_Themes#Reducing_Bloat" target="_blank">here</a>.<br /><br />I promised to get some numbers about how much memory is used by the JIT for the classes.<br /><br />I finally managed it :)<br /><br />So I took Eclipse 3.5M2 (without the web tools) configured it to use the <a href="http://help.sap.com/saphelp_nwce10/helpdata/en/47/dc90b4ef17452289f9128b8c2bbd77/content.htm">SAP JVM</a> (which is based on SUN's Hotspot, therefore the numbers should be comparable) and used the <a href="http://www.eclipse.org/mat" target="_blank">Eclipse Memory Analyzer</a> (plus the non open source, but free SAP extensions) to generate a csv file that I then imported into IBM's <a href="http://services.alphaworks.ibm.com/manyeyes/home" target="_blank">manyeyes</a> (great stuff!) to get the data visualized as a <a href="http://en.wikipedia.org/wiki/Treemap" target="_blank">treemap</a>.<br /><br />The numbers do not include the overhead of the compiled code cache which was around 10 Mbyte.<br /><br />Here comes the memory usage(in bytes) per bundle for Eclipse 3.5M2 after clicking around to activate as many plugins as possible (a more well defined test would be good idea) :<br /><script type="text/javascript" src="http://manyeyes.alphaworks.ibm.com/manyeyes/visualizations/89ade5ae1d71e980011d726605870128/comments/dd44e478ada811ddb553000255111976.js?width=400&height=350"></script><br /><br /><br />A detailed analysis would be very time consuming, because you would need to check for a lot of classes whether there's anything that could be done to make them smaller. So for now here are just some interesting observations that I made, when quickly looking over the results:<br /><ul><li><span style="font-weight: bold;">12649 classes</span> were loaded and consumed around <span style="font-weight: bold;">64 Mbyte </span>in Perm space<br /></li><li>during the test I run out of permspace, which was configured by default to 64Mbyte and the UI would freeze :(<br /></li><li>there does not seem to be any large amount of generated code, and therefore optimizing classes might be difficult<br /></li><li> the help system uses JSP's (generats classes) which are relatively big, also only of few of them where in memory </li><li> <span style="font-weight: bold;">247 relatively big classes</span> in org.eclipse.jdt.internal.compiler were <span style="font-weight: bold;">loaded twice</span>,once by the jasper plugin and once by the jdk core plugin</li></ul>I also made a detailed (per class) visualization of the relatively large jdt bundle :<br /><br /><script type="text/javascript" src="http://manyeyes.alphaworks.ibm.com/manyeyes/visualizations/89ade5ae1d71e980011d7256c27800aa/comments/dce4908cada811ddb553000255111976.js?width=400&height=350"></script><br /><br /><br />So is Eclipse really bloated?<br />To me it seems it's overall not very bloated. Remember that a developer PC these days probably has 2 Gbyte of RAM, so 64Mbyte hardly matter.<br /><br />The help system could be probably optimizied because it uses a complete servlet implementation (Jetty) including JSP's as well as a complete Text search engine (lucene).<br /><br />The heap consumption could be a bigger issue. I've seen things that you should almost always avoid like holding parsed DOM trees in memory. I may explain these issues in a later post, and I will probably show some examples in my next <a href="http://majug.de/">talk</a> .Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com17tag:blogger.com,1999:blog-6817331.post-1846118666016689982008-11-04T07:06:00.000-08:002009-02-02T14:16:46.151-08:00Eclipse Memory Analyzer Talk<div xmlns='http://www.w3.org/1999/xhtml'>On 25 November I will give a talk(in German) about the Eclipse Memory Analyzer at the Mannheim Java User Group.<br />See <a target='_blank' href='http://majug.de/talks'>here</a> for the details. <br /><br />I will explain how MAT works and I also will also show some real world examples.</div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com0tag:blogger.com,1999:blog-6817331.post-12205596430218148342008-10-27T14:01:00.000-07:002008-10-28T00:44:39.004-07:00The knowledge about how much memory things need in Java is surprisingly low<div xmlns='http://www.w3.org/1999/xhtml'>I just came across the question <a target='_blank' href='http://stackoverflow.com/questions/229886/size-of-a-byte-in-memory-java'>"Size of a byte in memory - Java"</a> at stackoverflow. <br />
Ok, stackoverflow might not be the place for high quality answers, but almost nobody getting close to a meaningful answer for this day to day question, is a little bit shocking. <br />
<br />
I documented the <a target='_blank' href='http://www.sdn.sap.com/irj/sdn/weblogs?blog=/pub/wlg/5163'>rules for java memory usage</a> of the SUN JVM at my old blog 2 years ago.<br />
<br />
I don't ask for people knowing the rules exactly (I don't either), but some feeling about where a JVM would align objects, and what kind of overhead an object has is essential Java programmers knowledge. <br />
It's not really necessary that you know all the rules because the <a target='_blank' href='http://www.eclipse.org/mat/'>Eclipse Memory Analyzer</a> knows about them. <br />
<br />
Still, I think I understand that people might not know the details, because they are platform (32 bit versus 64 bit) as well as JVM implementation depended and have not been documented for a long time. <br />
<br />
Not everybody has access to JVM hackers ;)<br />
<br />
<br />
Lets come back to the question of how much a byte costs. A byte costs really 8 bits=1 byte (that is defined by the Java Spec), but the SUN JVM (on other JVM's do that as well) aligns to 8 byte. <br />
So a it all depends on what other fields you have defined in your object. If you have 8 byte fields and nothing else everything is properly aligned and you would not waste memory. <br />
<br />
On the other side you could add a byte field to an object and not consume more memory. You could have for example one field that only consumes 4 bytes and due to alignment you already waste for 4 bytes but adding the byte field actually costs nothing because it fills in the padded bytes. <br />
<br />
The same is true for byte arrays, they are aligned as well and there's additional space needed for the length of the array. But for large byte arrays the average consumption is very close to one byte. <br />
<br />
<br />
<br />
</div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com2tag:blogger.com,1999:blog-6817331.post-66450710143895517082008-10-27T06:23:00.000-07:002008-10-27T06:24:55.145-07:00New language for Google App Engine coming before Q2/2009<div xmlns='http://www.w3.org/1999/xhtml'>There were rumors lately that Google will support Java on it's <a href='http://code.google.com/appengine' target='_blank'>App Engine</a>. Now at least the time frame is clear. Google <a href='http://code.google.com/appengine/docs/roadmap.html' target='_blank'>just announced</a> the roadmap for <a href='http://code.google.com/appengine' target='_blank'>App Engine</a> :<br />
<br />
<blockquote><span style='border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;' class='Apple-style-span'><h3 style='margin: 1.5em 0px 0px; position: relative; font-size: 130%; font-weight: bold; top: 0em;'>10/08 - 3/09</h3><ul style='margin: 0.5em 0px 0px 15px; padding: 0px; line-height: 125%;'><li style='margin: 0.3em 0px 0px 1.5em; padding: 0px;'>Service for storing and serving large files</li>
<li style='margin: 0.3em 0px 0px 1.5em; padding: 0px;'>Datastore import and export utility for large datasets</li>
<li style='margin: 0.3em 0px 0px 1.5em; padding: 0px;'>Billing: developers can pay for more resource usage</li>
<li style='margin: 0.3em 0px 0px 1.5em; padding: 0px;'><b>Support for a new runtime language</b></li>
<li style='margin: 0.3em 0px 0px 1.5em; padding: 0px;'>Uptime monitoring site</li>
</ul><br />
</span><br />
</blockquote><br />
<br />
Given that Google internally only allows to use <a href='http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html' target='_blank'>C++, Python, Java and Javascript</a> I think it's save to bet that either Java and/or Javascript is coming. My guess is that Java will be coming and on top of it they will offer a Rhino based web framework(<a href='http://steve-yegge.blogspot.com/2007/06/rhino-on-rails.html' target='_blank'>Rhino on Rails</a>). <br />
The rational for this is that they already showed that they can make Java work in a virtual environment. <a href='http://code.google.com/android/' target='_blank'>Android</a>'s Dalvik VM supports VM's running in separate processes to share data, which is something that is crucial for reducing the costs of running Java (securely) in a hosted environment. They also run <a href='http://www.google.com/calendar/' target='_blank'>some of their applications</a> already on Java. <br />
It's also very likely that the will support Java (at least a subset of it) because <a href='http://code.google.com/webtoolkit/' target='_blank'>GWT</a>, also it can be used on the server with other languages,is Java based.</div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com0tag:blogger.com,1999:blog-6817331.post-58593234397665718442008-10-21T12:10:00.000-07:002008-10-21T12:18:34.092-07:00Funny comment in the Android sources<a href="http://android.kernel.org">Android is now open source</a>. <br /><br />Just by accident I ran into the following <a href="http://android.kernel.org/?p=platform/dalvik.git;a=blob;f=dexdump/DexDump.c;h=372799307dbc97d9edef92b20199da0b61233c2c;hb=HEAD">funny comment</a>: <br /><br /><div><a id="l98" href="#l98" class="linenr"> 98</a> /*</div><br /><div class="pre"><a id="l99" href="#l99" class="linenr"> 99</a> * Count the number of '1' bits in a word.</div><div class="pre"><a id="l100" href="#l100" class="linenr"> 100</a> *</div><div class="pre"><a id="l101" href="#l101" class="linenr"> 101</a> * Having completed this, I'm ready for an interview at Google.</div><div class="pre"><a id="l102" href="#l102" class="linenr"> 102</a> *</div><br /><div class="pre"><a id="l103" href="#l103" class="linenr"> 103</a> * TODO? there's a parallel version w/o loops. Performance not currently</div><div class="pre"><a id="l104" href="#l104" class="linenr"> 104</a> * important.</div><div class="pre"><a id="l105" href="#l105" class="linenr"> 105</a> */</div><div class="pre"><a id="l106" href="#l106" class="linenr"> 106</a> static int countOnes(u4 val)</div><br /><div class="pre"><a id="l107" href="#l107" class="linenr"> 107</a> {</div><div class="pre"><a id="l108" href="#l108" class="linenr"> 108</a> int count = 0;</div><div class="pre"><a id="l109" href="#l109" class="linenr"> 109</a> </div><div class="pre"><a id="l110" href="#l110" class="linenr"> 110</a> while (val != 0) {</div><br /><div class="pre"><a id="l111" href="#l111" class="linenr"> 111</a> val &= val-1;</div><div class="pre"><a id="l112" href="#l112" class="linenr"> 112</a> count++;</div><div class="pre"><a id="l113" href="#l113" class="linenr"> 113</a> }</div><div class="pre"><a id="l114" href="#l114" class="linenr"> 114</a> </div><br /><div class="pre"><a id="l115" href="#l115" class="linenr"> 115</a> return count;</div><div class="pre"><a id="l116" href="#l116" class="linenr"> 116</a> }</div><br /><br />:)Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com5tag:blogger.com,1999:blog-6817331.post-31471902059645006482008-10-21T06:09:00.000-07:002008-10-21T06:12:43.578-07:00Support for IBM JVM's now available for the Eclipse Memory Analyzer<div xmlns="http://www.w3.org/1999/xhtml">Just a short note.<br />It was just brought to my attention, that the <a href="http://www.eclipse.org/mat" target="_blank">Eclipse Memory Analyzer</a> now also supports system dumps from IBM Virtual Machines for Java version 6, version 5.0 and version 1.4.2.<br /><br />The IBM DTFJ adapter is available as an Eclipse plugin and can be downloaded from <a href="https://www.ibm.com/developerworks/java/jdk/tools/mat.html" target="_blank">developerworks</a>.<br /><br />This is great news, because a lot of people had asked for IBM support, and because it means that MAT is now modular enough to support other Heap dump formats as well.<br /><br />Maybe Google would like to support Dalvik (<a href="http://code.google.com/android/" target="_blank">Android</a>)? ;)</div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com1tag:blogger.com,1999:blog-6817331.post-67969752996364923502008-09-29T12:51:00.000-07:002008-09-29T12:53:00.919-07:00It's official, Neal Gafter works for Microsoft<div xmlns="http://www.w3.org/1999/xhtml">There were already rumors that Neil Gafter would leave Google and join Microsoft.<br /><br />It's now <a target="_blank" href="http://gafter.com/~neal/">official</a><br /><br /><blockquote>"I work for Microsoft on the dotNet platform languages. To balance my life, my hobby is designing and developing the future of the Java programming language."</blockquote><br /><br />and on <a href="http://twitter.com/gafter/statuses/936090878" target="_blank">twitter</a><br /><br /><blockquote>"working on Microsoft Visual Studio Managed Languages with Anders Hejlsberg, on C# and other languages."<br /></blockquote><br /><br /><br /><a target="_blank" href="http://javaposse.com/index.php?post_id=385201">The Java Posse</a> seems to have an interview with Neal ( I haven't checked yet).<br /><br />Neal had a great influence on the Java language and lead one of the <a target="_blank" href="http://gafter.blogspot.com/2008/08/java-closures-prototype-feature.html">Closure proposals</a>.<br />Unfortunately <a target="_blank" href="http://tech.puredanger.com/2008/08/02/java7-prediction-update/">Closures will probably not make it into Java 7</a>.<br /><br />This is really a bad day for Java. Microsoft's .NET is already moving quickly to support <a target="_blank" href="http://research.microsoft.com/fsharp/fsharp.aspx">new interesting languages</a><br />whereas Java is falling behind. For example, a real <a target="_blank" href="http://www.se-radio.net/podcast/2007-10/episode-72-erik-meijer-linq">LINQ</a> for Java will not be possible without closures.<br />With Neal working on .NET/C# Microsoft will probably advance even further.</div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com0tag:blogger.com,1999:blog-6817331.post-25868763842046742962008-09-19T01:08:00.000-07:002008-09-19T01:13:47.968-07:00Re: AMF vs. JSON vs. XML<div xmlns="http://www.w3.org/1999/xhtml"><a href="http://www.oreillynet.com/pub/au/251" style="color: #0077dd; text-decoration: none;">Richard Monson-Haefel</a> blogged about the advantages and disadvantages of AMF versus JSON versus XML on <a href="http://www.insideria.com/2008/09/amf-vs-json-vs-xml.html" target="_blank">InsideRIA</a>.<br />
<br />
Also I agree with some of his points, IHMO he also misses some important ones.<br />
<br />
<blockquote>For example, there is the "batch" pattern which says that it is sometimes cheaper to batch many I/O operation than to do them individually. But again using that pattern requires planning and a specific context to make it effective.</blockquote><br />
Batching is an important pattern, but the point is that the current implementations using AMF(BlazeDS) are simply Remote Procedure calls over HTTP POST. <br />
<br />
As I said before the consequence is that BlazeDS, <a href="http://kohlerm.blogspot.com/2008/08/blazeds-does-not-make-use-of-http.html" target="_blank">does not play very well with the HTTP caching infrastructure</a>. <br />
In short the <b>POST's are never cached. </b><br />
<br />
I just cannot see how <b>automatic</b> batching of RPC calls can be easily implemented. <a href="http://en.wikipedia.org/wiki/Representational_State_Transfer" target="_blank">Restful</a> API's based on JSON or XML can more easily support automatic batching and even could potentially (not sure whether the limitations of the Flex HTTPService would allow this) make use of <a href="http://en.wikipedia.org/wiki/HTTP_pipelining" target="_blank">http pipelining</a>. <a href="http://en.wikipedia.org/wiki/Representational_State_Transfer" target="_blank">Restful</a> API's could also better use HTTP's caching features as well as lead to more scalable implementations on the server side, because to much state would be avoided there. <br />
<br />
<blockquote>The truth is that performance differences between JSON and AMF are not that wide.</blockquote><br />
I agree, and would add "in practice" and "for most use cases". <br />
<br />
I think that <a href="http://www.jamesward.com/census/" target="_blank">some benchmarks</a> are a bit misleading. Who would really want to load that much data in one request to the client, if the client typically cannot show all the data? What you want to do is to use some kind of paging or on demand loading support, because most users in typical applications are not going to look at all the data anyway. <br />
<br />
I bet that then the differences between the formats will become minor and probably negligible as soon as you use paging or a similar mechanism. <br />
The reference to another <a href="http://www.5etdemi.com/blog/archives/2006/12/clearing-the-fud-on-amfphps-speed-versus-json-and-xml/" target="_blank">performance analysis</a> that Richard provides seems to go in this direction as well. <br />
<br />
So at the end I would rather agree with one of the <a href="http://www.insideria.com/2008/09/amf-vs-json-vs-xml.html#comment-2042930" target="_blank">commenters </a>on Richard's blog<br />
<br />
<blockquote>Yeah, having a JSON implementation natively in Flash would be great.</blockquote><br />
There is one other requirement that you need to take into account and that is security.<br />
Using Webservice Remoting with authentication doesn't seem to work very well at the moment with Flex. <br />
Therefore If you are going to use HTTP's authentication mechanism, you currently have to use BlazeDS (or the like) or use HTTPService and parse the XML manually.</div>Anonymoushttp://www.blogger.com/profile/06348916709650836506noreply@blogger.com4