Memory Analyzer 1.1. was just released!
Check the New and noteworthy page and download it from here.
Showing posts with label performance. Show all posts
Showing posts with label performance. Show all posts
Wednesday, June 22, 2011
Thursday, November 11, 2010
Extending the Eclipse Memory Analyzer
I usally don't post links here but I always wanted to describe here how to extend the Eclipse Memory Analyzer. Since there's now a Slide Deck available that describes briefly how it can be done at
http://www.slideshare.net/AJohnson1/extending-eclipse-memory-analyzer it's not that urgent anymore to post a description here ;-)
I believe there's a huge potential in extending MAT for specific use cases, such as Android, Jruby, Scala etc.
http://www.slideshare.net/AJohnson1/extending-eclipse-memory-analyzer it's not that urgent anymore to post a description here ;-)
I believe there's a huge potential in extending MAT for specific use cases, such as Android, Jruby, Scala etc.
Wednesday, February 24, 2010
Android memory usage analysis slides available
Yesterday, I gave a talk at the mobile monday android developer day in Düsseldorf.The slides about analysing the memory usage of android applications with the Eclipse Memory Analyzer are available now. Note it might be worth to download the pdf because it contains some notes.
The slides should be interesting for general java developers, not only for android developers.
Also, I want to thank everyone who showed up. It was great talking to all of you!
The slides should be interesting for general java developers, not only for android developers.
Also, I want to thank everyone who showed up. It was great talking to all of you!
Monday, March 23, 2009
Leaks are easy to find, but memory usage analysis is bit more difficult
Leaks again
Last time I talked about how easy it is to find memory leaks in Java using the dominator tree feature of the Eclipse Memory Analyzer.
If you haven't read this post, I recommend you to do so, because this post will assume that you know the meaning of the terms "retained size" and "dominator tree".
Why does this work so well? The reason is that memory leaks in Java are not really "classical" leaks in the strict sense. Let's check what Wikipedia says about memory leaks:
So because those leaks in Java are "only" unwanted (unbound) increase of memory usage, the typical reason for them is that people forget to remove an object out of some collection/array or a recursive data structure,such as a tree. This might sound stupid and you (and me) would of course never make such a simple mistake ;)
But look at the following example:
"thing" will not be removed if "doSomething" throws an IOException (or any other exception). OMG Joel Spolsky was right when he said:
The correct way would be:
try
{
doSomething(thing); // does IO
}
catch (IOException e)
{
// should not happen
}
finally
{
collection.remove(thing);
}
So I talked enough of leaks. I promise you if you regulary analyze heap dumps taken at the end of a performance test run, after while of fixing, you will not see a lot of leaks anymore If you still think that you need to know more a about leaks. I recommend you to check this excellent tutorial.
High memory usage
You might still see high memory usage, and your users might hate that as much as leaks, because performance degradation can be similiar.
OK, why is high memory more difficult to analyze?
You might use the dominator tree to find some big objects and you might also use it to figure out some cause of high memory usage. Because it's a tree it's easy to see where the memory is used:
You just have to look at the pathes down the tree to find out where the most memory is used/retained by single objects.
But in general the dominator tree view alone (without using some advanced functions, that I will skip for now) will not help you to find the reason why for example all those Strings are there:
Fortunately there is the "immediate dominators" query in the Eclipse Memory Analyzer that is based on the dominator tree that can help here and that also is the used internally by most of the advanced queries. The "immediate dominators" query is one of the key innovations in the the Eclipse Memory Analyzer. Even the commercial Yourkit profiler does not seem to have it yet, also they now also have a dominator tree functionality.
Immediate Dominators
So what is a "immediate dominator"? Let's have look at a simple example where the "business object" of class "Order" references a LinkedList of Strings:
If we look at String2 first we can find the LinkedList$Entry 2 is the "closest" object hat dominates it. If we could remove LinkedList$Entry 2, the Object String 2 would also be reclaimed by the garbage collector. We say "LinkedList$Entry 2" is the immediate dominator of "Object String 2". Note that there's always one
Let's have a look at the immediate dominators up to the Order object:
Note that LinkedList$Entry 1 is not an immediate dominator for LinkedList$Entry 2, because after removing it there would still be links from LinkedList$Entry 0 to LinkedList$Entry 2. We can do the same for the String 0 and String 1 and we will get the dominator tree:
No if we ask ourself the question "why are all those Strings still there", we see that if we filter all JDK classes out of the dominator tree it's easy:
The immediate dominators query in MAT basically lets you walk up the dominator tree and shows you the dominators aggregated by the class:
This is really a screenshot from an existing heap dump that I took some time ago from Eclipse. You can see for example the famous Dictionary of the spell checker plugin retaining 74393 Strings.
So now how can I find out where memory usage could be reduced?
With Strings it's pretty easy, you use the group_by_value in MAT. For the example above I applied it to the Strings dominated by ResolvedBinaryField in the first line:
Yes,there are really 6969 duplicates of "Ljava.lang.String;" retained only by instances of this class! Disclaimer: And no dear Netbeans "fanboys", Eclipse is not really worse than your beloved IDE in this area ;)
Strings are immutable and I wonder what would happen if people would really use more immutable data structures.
But not only Strings are interesting when you look at minimizing memory usage. Strings are just convenient because they are (usually) human readable. You can still often use Strings to find Objects which are equal but not identical, because if equal but not identical objects are created usually those objects also reference Strings that are equal but not identical.
The main question that you always have to ask yourself when trying to minimize memory usage is :
Do I need these equal but not identical objects?
In a single threaded environment the answer is usually that you don't those copies of objects.
In a highly concurrent environment, reducing the copies might introduce contention, because you have to share objects and you will need to check whether you already have this object. Strings again are relatively safe to optimize in this regard, because they are immutable, so no synchronization is needed to access them.
Having a query in MAT for the "algorithm" I described here for finding duplicated Strings, would be very helpful (there is a similiar but simpler "duplicated Strings" query already built in) .
I have done exactly that quite some time ago, but the query was not yet "production ready". There's some hope that it will appear in the standard MAT soon, stay tuned!
Last time I talked about how easy it is to find memory leaks in Java using the dominator tree feature of the Eclipse Memory Analyzer.
If you haven't read this post, I recommend you to do so, because this post will assume that you know the meaning of the terms "retained size" and "dominator tree".
Why does this work so well? The reason is that memory leaks in Java are not really "classical" leaks in the strict sense. Let's check what Wikipedia says about memory leaks:
"many people refer to any unwanted increase in memory usage as a memory leak, even if this is not strictly accurate"The later cannot happen in languages such as Java that have built-in automatic garbage collection, also Ruby does not seem to be bug free in this area.
and
"Typically, a memory leak occurs because dynamically allocated memory has become unreachable."
So because those leaks in Java are "only" unwanted (unbound) increase of memory usage, the typical reason for them is that people forget to remove an object out of some collection/array or a recursive data structure,such as a tree. This might sound stupid and you (and me) would of course never make such a simple mistake ;)
But look at the following example:
try
{
doSomething(thing); // does IO
collection.remove(thing);
}
catch (IOException e)
{
// should not happen
}
"thing" will not be removed if "doSomething" throws an IOException (or any other exception). OMG Joel Spolsky was right when he said:
"I consider exceptions to be no better than "goto's""
The correct way would be:
try
{
doSomething(thing); // does IO
}
catch (IOException e)
{
// should not happen
}
finally
{
collection.remove(thing);
}
So I talked enough of leaks. I promise you if you regulary analyze heap dumps taken at the end of a performance test run, after while of fixing, you will not see a lot of leaks anymore If you still think that you need to know more a about leaks. I recommend you to check this excellent tutorial.
High memory usage
You might still see high memory usage, and your users might hate that as much as leaks, because performance degradation can be similiar.
OK, why is high memory more difficult to analyze?
You might use the dominator tree to find some big objects and you might also use it to figure out some cause of high memory usage. Because it's a tree it's easy to see where the memory is used:
You just have to look at the pathes down the tree to find out where the most memory is used/retained by single objects.
But in general the dominator tree view alone (without using some advanced functions, that I will skip for now) will not help you to find the reason why for example all those Strings are there:
Fortunately there is the "immediate dominators" query in the Eclipse Memory Analyzer that is based on the dominator tree that can help here and that also is the used internally by most of the advanced queries. The "immediate dominators" query is one of the key innovations in the the Eclipse Memory Analyzer. Even the commercial Yourkit profiler does not seem to have it yet, also they now also have a dominator tree functionality.
Immediate Dominators
So what is a "immediate dominator"? Let's have look at a simple example where the "business object" of class "Order" references a LinkedList of Strings:
If we look at String2 first we can find the LinkedList$Entry 2 is the "closest" object hat dominates it. If we could remove LinkedList$Entry 2, the Object String 2 would also be reclaimed by the garbage collector. We say "LinkedList$Entry 2" is the immediate dominator of "Object String 2". Note that there's always one
Let's have a look at the immediate dominators up to the Order object:
Note that LinkedList$Entry 1 is not an immediate dominator for LinkedList$Entry 2, because after removing it there would still be links from LinkedList$Entry 0 to LinkedList$Entry 2. We can do the same for the String 0 and String 1 and we will get the dominator tree:
No if we ask ourself the question "why are all those Strings still there", we see that if we filter all JDK classes out of the dominator tree it's easy:
The immediate dominators query in MAT basically lets you walk up the dominator tree and shows you the dominators aggregated by the class:
This is really a screenshot from an existing heap dump that I took some time ago from Eclipse. You can see for example the famous Dictionary of the spell checker plugin retaining 74393 Strings.
So now how can I find out where memory usage could be reduced?
With Strings it's pretty easy, you use the group_by_value in MAT. For the example above I applied it to the Strings dominated by ResolvedBinaryField in the first line:
Yes,there are really 6969 duplicates of "Ljava.lang.String;" retained only by instances of this class! Disclaimer: And no dear Netbeans "fanboys", Eclipse is not really worse than your beloved IDE in this area ;)
Strings are immutable and I wonder what would happen if people would really use more immutable data structures.
But not only Strings are interesting when you look at minimizing memory usage. Strings are just convenient because they are (usually) human readable. You can still often use Strings to find Objects which are equal but not identical, because if equal but not identical objects are created usually those objects also reference Strings that are equal but not identical.
The main question that you always have to ask yourself when trying to minimize memory usage is :
Do I need these equal but not identical objects?
In a single threaded environment the answer is usually that you don't those copies of objects.
In a highly concurrent environment, reducing the copies might introduce contention, because you have to share objects and you will need to check whether you already have this object. Strings again are relatively safe to optimize in this regard, because they are immutable, so no synchronization is needed to access them.
Having a query in MAT for the "algorithm" I described here for finding duplicated Strings, would be very helpful (there is a similiar but simpler "duplicated Strings" query already built in) .
I have done exactly that quite some time ago, but the query was not yet "production ready". There's some hope that it will appear in the standard MAT soon, stay tuned!
Monday, February 16, 2009
Memory leaks are easy to find
Last time I talked about the dominator tree and how it can be used to find the biggest objects in your heap easily.
So what exactly is the dominator tree?
Dominators
An object A dominates on an object B if all the paths to object B pass through object A.
Remember that the Garbage Collector removes all objects that are not referenced anymore. If A dominates B and A could be removed from memory, that means that there's no path anymore that leads to B. B is therefore unreachable and would be reclaimed by the Garbage Collector.
One could also say that A is the single object that is responsible for B still being there!
The Dominator Tree
Using the the "dominates" relationship we can create a dominator tree out of the the graph of objects in memory. At each node of this tree we store the amount of memory that would be freed (= retained size).
At the top of the tree we have a "virtual" root object, which we also use to represent objects that don't have "real" single dominator.
Here's an example of an object tree (on the left) and the corresponding dominator tree (on the right) :
Because of the transitivity of "dominated", the retained size of a parent object within the dominator tree is always greater than the sum of it's child objects.
To get the biggest objects we just need to sort the second level of the dominator tree (the first level is the "virtual" root object) by retained size.
Now if you are looking to find a memory leak, and you have no a priori knowledge that could help you, the typical approach is to run a test that reproduces the leak, and then try to somehow figure out what is leaking.
Do we really need Object allocations tracing?
In my experience people often seem to believe that finding leaks requires recording object creations, also called "object allocations tracing "sometimes, because you want to know where in the code objects are always allocates but never released.
Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft) has an example of how this method for finding leaks can be applied to .NET applications. Dear Microsoft I encourage you to look at the Eclipse Memory Analyzer ;)
All you need is the dominator tree
With the dominator tree finding these kind of leaks is much easier and you don't need the high overhead of allocation tracing, which is typically not acceptable in a production environment. Allocation tracing has it's uses, but in general IMHO it is overrated.
It's almost guaranteed that a significant memory leak will show up at the very top of the dominator tree!
The Eclipse Memory Analyzer not only supports the dominator tree but can even find memory leaks automatically, based on some heuristics.
The dominator tree can also be used to find high memory usage in certain areas, which is a much harder problem. More on this in the next post.
So what exactly is the dominator tree?
Dominators
An object A dominates on an object B if all the paths to object B pass through object A.
Remember that the Garbage Collector removes all objects that are not referenced anymore. If A dominates B and A could be removed from memory, that means that there's no path anymore that leads to B. B is therefore unreachable and would be reclaimed by the Garbage Collector.
One could also say that A is the single object that is responsible for B still being there!
The Dominator Tree
Using the the "dominates" relationship we can create a dominator tree out of the the graph of objects in memory. At each node of this tree we store the amount of memory that would be freed (= retained size).
At the top of the tree we have a "virtual" root object, which we also use to represent objects that don't have "real" single dominator.
Here's an example of an object tree (on the left) and the corresponding dominator tree (on the right) :
- Note that A, B and C are dominated by a "virtual" root object.
- Note that the dominator relationship is transitive;C dominates E which dominates G therefore C also dominates G.
Because of the transitivity of "dominated", the retained size of a parent object within the dominator tree is always greater than the sum of it's child objects.
To get the biggest objects we just need to sort the second level of the dominator tree (the first level is the "virtual" root object) by retained size.
Now if you are looking to find a memory leak, and you have no a priori knowledge that could help you, the typical approach is to run a test that reproduces the leak, and then try to somehow figure out what is leaking.
Do we really need Object allocations tracing?
In my experience people often seem to believe that finding leaks requires recording object creations, also called "object allocations tracing "sometimes, because you want to know where in the code objects are always allocates but never released.
Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft) has an example of how this method for finding leaks can be applied to .NET applications. Dear Microsoft I encourage you to look at the Eclipse Memory Analyzer ;)
All you need is the dominator tree
With the dominator tree finding these kind of leaks is much easier and you don't need the high overhead of allocation tracing, which is typically not acceptable in a production environment. Allocation tracing has it's uses, but in general IMHO it is overrated.
It's almost guaranteed that a significant memory leak will show up at the very top of the dominator tree!
The Eclipse Memory Analyzer not only supports the dominator tree but can even find memory leaks automatically, based on some heuristics.
The dominator tree can also be used to find high memory usage in certain areas, which is a much harder problem. More on this in the next post.
Thursday, November 06, 2008
Is Eclipse bloated? Some numbers and a first analysis
On the e4 (Eclipse 4) mailing list there were lately some discussions about whether Eclipse is bloated or not, and what could be done to minimize bloat for e4. Some summary information can be found here.
I promised to get some numbers about how much memory is used by the JIT for the classes.
I finally managed it :)
So I took Eclipse 3.5M2 (without the web tools) configured it to use the SAP JVM (which is based on SUN's Hotspot, therefore the numbers should be comparable) and used the Eclipse Memory Analyzer (plus the non open source, but free SAP extensions) to generate a csv file that I then imported into IBM's manyeyes (great stuff!) to get the data visualized as a treemap.
The numbers do not include the overhead of the compiled code cache which was around 10 Mbyte.
Here comes the memory usage(in bytes) per bundle for Eclipse 3.5M2 after clicking around to activate as many plugins as possible (a more well defined test would be good idea) :
A detailed analysis would be very time consuming, because you would need to check for a lot of classes whether there's anything that could be done to make them smaller. So for now here are just some interesting observations that I made, when quickly looking over the results:
So is Eclipse really bloated?
To me it seems it's overall not very bloated. Remember that a developer PC these days probably has 2 Gbyte of RAM, so 64Mbyte hardly matter.
The help system could be probably optimizied because it uses a complete servlet implementation (Jetty) including JSP's as well as a complete Text search engine (lucene).
The heap consumption could be a bigger issue. I've seen things that you should almost always avoid like holding parsed DOM trees in memory. I may explain these issues in a later post, and I will probably show some examples in my next talk .
I promised to get some numbers about how much memory is used by the JIT for the classes.
I finally managed it :)
So I took Eclipse 3.5M2 (without the web tools) configured it to use the SAP JVM (which is based on SUN's Hotspot, therefore the numbers should be comparable) and used the Eclipse Memory Analyzer (plus the non open source, but free SAP extensions) to generate a csv file that I then imported into IBM's manyeyes (great stuff!) to get the data visualized as a treemap.
The numbers do not include the overhead of the compiled code cache which was around 10 Mbyte.
Here comes the memory usage(in bytes) per bundle for Eclipse 3.5M2 after clicking around to activate as many plugins as possible (a more well defined test would be good idea) :
A detailed analysis would be very time consuming, because you would need to check for a lot of classes whether there's anything that could be done to make them smaller. So for now here are just some interesting observations that I made, when quickly looking over the results:
- 12649 classes were loaded and consumed around 64 Mbyte in Perm space
- during the test I run out of permspace, which was configured by default to 64Mbyte and the UI would freeze :(
- there does not seem to be any large amount of generated code, and therefore optimizing classes might be difficult
- the help system uses JSP's (generats classes) which are relatively big, also only of few of them where in memory
- 247 relatively big classes in org.eclipse.jdt.internal.compiler were loaded twice,once by the jasper plugin and once by the jdk core plugin
So is Eclipse really bloated?
To me it seems it's overall not very bloated. Remember that a developer PC these days probably has 2 Gbyte of RAM, so 64Mbyte hardly matter.
The help system could be probably optimizied because it uses a complete servlet implementation (Jetty) including JSP's as well as a complete Text search engine (lucene).
The heap consumption could be a bigger issue. I've seen things that you should almost always avoid like holding parsed DOM trees in memory. I may explain these issues in a later post, and I will probably show some examples in my next talk .
Friday, September 19, 2008
Re: AMF vs. JSON vs. XML
Richard Monson-Haefel blogged about the advantages and disadvantages of AMF versus JSON versus XML on InsideRIA.
Also I agree with some of his points, IHMO he also misses some important ones.
Batching is an important pattern, but the point is that the current implementations using AMF(BlazeDS) are simply Remote Procedure calls over HTTP POST.
As I said before the consequence is that BlazeDS, does not play very well with the HTTP caching infrastructure.
In short the POST's are never cached.
I just cannot see how automatic batching of RPC calls can be easily implemented. Restful API's based on JSON or XML can more easily support automatic batching and even could potentially (not sure whether the limitations of the Flex HTTPService would allow this) make use of http pipelining. Restful API's could also better use HTTP's caching features as well as lead to more scalable implementations on the server side, because to much state would be avoided there.
I agree, and would add "in practice" and "for most use cases".
I think that some benchmarks are a bit misleading. Who would really want to load that much data in one request to the client, if the client typically cannot show all the data? What you want to do is to use some kind of paging or on demand loading support, because most users in typical applications are not going to look at all the data anyway.
I bet that then the differences between the formats will become minor and probably negligible as soon as you use paging or a similar mechanism.
The reference to another performance analysis that Richard provides seems to go in this direction as well.
So at the end I would rather agree with one of the commenters on Richard's blog
There is one other requirement that you need to take into account and that is security.
Using Webservice Remoting with authentication doesn't seem to work very well at the moment with Flex.
Therefore If you are going to use HTTP's authentication mechanism, you currently have to use BlazeDS (or the like) or use HTTPService and parse the XML manually.
Also I agree with some of his points, IHMO he also misses some important ones.
For example, there is the "batch" pattern which says that it is sometimes cheaper to batch many I/O operation than to do them individually. But again using that pattern requires planning and a specific context to make it effective.
Batching is an important pattern, but the point is that the current implementations using AMF(BlazeDS) are simply Remote Procedure calls over HTTP POST.
As I said before the consequence is that BlazeDS, does not play very well with the HTTP caching infrastructure.
In short the POST's are never cached.
I just cannot see how automatic batching of RPC calls can be easily implemented. Restful API's based on JSON or XML can more easily support automatic batching and even could potentially (not sure whether the limitations of the Flex HTTPService would allow this) make use of http pipelining. Restful API's could also better use HTTP's caching features as well as lead to more scalable implementations on the server side, because to much state would be avoided there.
The truth is that performance differences between JSON and AMF are not that wide.
I agree, and would add "in practice" and "for most use cases".
I think that some benchmarks are a bit misleading. Who would really want to load that much data in one request to the client, if the client typically cannot show all the data? What you want to do is to use some kind of paging or on demand loading support, because most users in typical applications are not going to look at all the data anyway.
I bet that then the differences between the formats will become minor and probably negligible as soon as you use paging or a similar mechanism.
The reference to another performance analysis that Richard provides seems to go in this direction as well.
So at the end I would rather agree with one of the commenters on Richard's blog
Yeah, having a JSON implementation natively in Flash would be great.
There is one other requirement that you need to take into account and that is security.
Using Webservice Remoting with authentication doesn't seem to work very well at the moment with Flex.
Therefore If you are going to use HTTP's authentication mechanism, you currently have to use BlazeDS (or the like) or use HTTPService and parse the XML manually.
Wednesday, June 25, 2008
Robust Java benchmarking
Doing Micro benchmarks on the JVM is very hard, due to the Hotspot compiler (assuming you are on a SUN/SAP JVM) doing very advanced optimizations at runtime.
I usually recommend to be very careful about doing any micro benchmarks without having a real JVM hacker nearby ;)
Brent Boyer has written an article series at developerworks that explains how to do it correctly and he also presents a framework to make it easier to write robust microbenchmarks :
Robust Java benchmarking, Part 1
Robust Java benchmarking, Part 2
Also I have not yet tried the framework, it looks promising.
I usually recommend to be very careful about doing any micro benchmarks without having a real JVM hacker nearby ;)
Brent Boyer has written an article series at developerworks that explains how to do it correctly and he also presents a framework to make it easier to write robust microbenchmarks :
Robust Java benchmarking, Part 1
Robust Java benchmarking, Part 2
Also I have not yet tried the framework, it looks promising.
Wednesday, May 28, 2008
Memory consumption of Netbeans versus Eclipse an analysis
I recently analyzed the memory consumption of Eclipse and found that it should be easy to optimize it.
This time I will take a look at Netbeans (6.1, the Java SE pack) using the exact same setup.
First the overall memory consumption of Netbeans is only a little bit higher 24 Mbyte versus 22,7 Mbyte for eclipse :
Keep in mind that the Eclipse memory usage includes the spell checker which needs 5,6Mbyte, which can easily turned off. Without the spell checker Eclipse would need only 17,1Mbyte
The overview of the Memory Analyzer shows that the biggest memory consumer, with around 5.4 Mbyte is sun.awt.image.BufImgSurfaceData:
Swing overhead(?)
This seems to be caused by the fact that Swing uses java2d which does it's own image buffering independent of the OS. I could easily figure this out by using the Memory Analyzers "path to GC roots query":
So maybe we pay here for the platform independence of Swing?
I quickly checked using Google whether there are ways around this Image buffering, but I couldn't find any clear guidance that would avoid this. If there are Swing experts reading this, please let me know your advise.
Duplicated Strings again
Again I check for the duplicated Strings using the "group_by_value" feature of the Memory Analyzer.
Again some Strings are there many times :
This time I selected all Strings, which are there more than once, then called the "immediate dominators" query and afterwards I used the "group by package" feature in the resulting view:
This view shows you the sum of the number of duplicates in each package and therefore gives you an good overview of which packages waste the most memory because of objects keeping duplicates of Strings alive.
You can see for example that
org.netbeans.modules.java.source.parsing.FileObjects$CachedZipFileObject keeps alive a lot of duplicated Strings.
Looking at some of these objects you can for example see that one problem is the instance variable ext, which contains very often duplicates of the String "class".
Summary
So at the end we found for this probably simplistic scenario that both Netbeans and Eclipse don't consume that much memory. 24Mbyte is really not that much these days.
Eclipse seems to have a certain advantage, because turning of the spell checker is easy and then it needs almost 30% less memory than Netbeans.
To be clear, it is still much to early to declare a real winner here. My scenario is just to simple.
The only conclusion that we can draw for sure for now is, that this kind of analysis is pretty easy with the Eclipse Memory Analyzer :)
This time I will take a look at Netbeans (6.1, the Java SE pack) using the exact same setup.
First the overall memory consumption of Netbeans is only a little bit higher 24 Mbyte versus 22,7 Mbyte for eclipse :
Keep in mind that the Eclipse memory usage includes the spell checker which needs 5,6Mbyte, which can easily turned off. Without the spell checker Eclipse would need only 17,1Mbyte
The overview of the Memory Analyzer shows that the biggest memory consumer, with around 5.4 Mbyte is sun.awt.image.BufImgSurfaceData:
Swing overhead(?)
This seems to be caused by the fact that Swing uses java2d which does it's own image buffering independent of the OS. I could easily figure this out by using the Memory Analyzers "path to GC roots query":
So maybe we pay here for the platform independence of Swing?I quickly checked using Google whether there are ways around this Image buffering, but I couldn't find any clear guidance that would avoid this. If there are Swing experts reading this, please let me know your advise.
Duplicated Strings again
Again I check for the duplicated Strings using the "group_by_value" feature of the Memory Analyzer.
Again some Strings are there many times :
This time I selected all Strings, which are there more than once, then called the "immediate dominators" query and afterwards I used the "group by package" feature in the resulting view:
This view shows you the sum of the number of duplicates in each package and therefore gives you an good overview of which packages waste the most memory because of objects keeping duplicates of Strings alive.You can see for example that
org.netbeans.modules.java.source.parsing.FileObjects$CachedZipFileObject keeps alive a lot of duplicated Strings.
Looking at some of these objects you can for example see that one problem is the instance variable ext, which contains very often duplicates of the String "class".
Summary
So at the end we found for this probably simplistic scenario that both Netbeans and Eclipse don't consume that much memory. 24Mbyte is really not that much these days.
Eclipse seems to have a certain advantage, because turning of the spell checker is easy and then it needs almost 30% less memory than Netbeans.
To be clear, it is still much to early to declare a real winner here. My scenario is just to simple.
The only conclusion that we can draw for sure for now is, that this kind of analysis is pretty easy with the Eclipse Memory Analyzer :)
Thursday, May 22, 2008
Eclipse memory leaks?
"ecamacho" picked up my post about the memory consumption of Eclipse on the spanish website http://javahispano.org/
Google translation works pretty well for the post and the comments are quite interesting.
Leaks in Eclipse?
One question was, whether I was talking about leaks in Eclipse.
Actually I myself did not, but my colleague and project lead of the Eclipse Memory Analyzer project, "Andreas, Buchen", blogged about analyzing an actual leak in Eclipse. I can highly recommend the article, because it also shows how powerful the Memory Analyzer is.
Does turning of the Spellchecker help?
From what I have seen, yes it should help. Check http://bugs.eclipse.org/bugs/show_bug.cgi?id=233156 for the progress on the spellchecker issue.
Should we wait until the end to analyze the memory consumption of our applications?
IMHO we should not, because as we all know, fixes at the end of an software project are much more expensive than in the beginning.
I often hear :
"premature optimization is the root of all evil."
This has been misquoted just too often.
Hoare said (according to wikipedia)
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil."
"small efficiencies"
That quote clearly doesn't say, you should not do any investigations on how much resources your applications consumes in the early phase of your project.
Actually we have some experience with the automatic evaluation of heap dumps and we found that it pays off very well. This could be a topic for another blog post.
Google translation works pretty well for the post and the comments are quite interesting.
Leaks in Eclipse?
One question was, whether I was talking about leaks in Eclipse.
Actually I myself did not, but my colleague and project lead of the Eclipse Memory Analyzer project, "Andreas, Buchen", blogged about analyzing an actual leak in Eclipse. I can highly recommend the article, because it also shows how powerful the Memory Analyzer is.
Does turning of the Spellchecker help?
From what I have seen, yes it should help. Check http://bugs.eclipse.org/bugs/show_bug.cgi?id=233156 for the progress on the spellchecker issue.
Should we wait until the end to analyze the memory consumption of our applications?
IMHO we should not, because as we all know, fixes at the end of an software project are much more expensive than in the beginning.
I often hear :
"premature optimization is the root of all evil."
This has been misquoted just too often.
Hoare said (according to wikipedia)
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil."
"small efficiencies"
That quote clearly doesn't say, you should not do any investigations on how much resources your applications consumes in the early phase of your project.
Actually we have some experience with the automatic evaluation of heap dumps and we found that it pays off very well. This could be a topic for another blog post.
Wednesday, May 14, 2008
GMail got faster, one trick that they didn't tell us
GMail got even faster and here are some details about how they did it :
Official Gmail Blog: A need for speed: the path to a faster loading sequence
One trick that the obviosouly did, but didn't tell us, is that they send your user name asynchronously in the background as soon as you enter the password field.
This is a GMail specific optimization. It's not available on the generic Google account page (yet?) .
My guess is that they prefetch some stuff under the assumption that most logon attempts will work. Even if the password is wrong, they minimum work, that has to be done is to check whether the account is there.
The funny thing is, that I've seen this just a few days ago, when I checked how much data Google Mail sends, to get a better feeling what would a good goal for a high performance Web application.
Regards,
Markus
Official Gmail Blog: A need for speed: the path to a faster loading sequence
One trick that the obviosouly did, but didn't tell us, is that they send your user name asynchronously in the background as soon as you enter the password field.
This is a GMail specific optimization. It's not available on the generic Google account page (yet?) .
My guess is that they prefetch some stuff under the assumption that most logon attempts will work. Even if the password is wrong, they minimum work, that has to be done is to check whether the account is there.
The funny thing is, that I've seen this just a few days ago, when I checked how much data Google Mail sends, to get a better feeling what would a good goal for a high performance Web application.
Regards,
Markus
Subscribe to:
Posts (Atom)