synchronize(this);
If you're using Domino today, chances are
you're working extensively with Java code. Oh, you might not be writing
Java code yourself. But you use it when you go to work, when you go to
church, when you pay your taxes.... errr, sorry... when you open your Notes
mail, when you go to Designer, when you access an Xpages app.
Java is all around us. The very fabric of the Eclipse user experience is based on Java code. And whenever Java has to interact with information inside an NSF, there is but one way to do it: notes.jar. The lotus.domino package is the standard, both inside IBM and among the Lotus community, for accessing Notes/Domino data. It is our desert of the real.
But what if I were to tell you that this .jar is the world that has been pulled over your eyes to blind you from the truth?
You'd probably say "Nathan, you're really going overboard with this Matrix crap. Get to the point."
The truth that Notes & Domino has a backdoor -- a SECRET API that is only used by a select few. And that it is this API that makes it possible to scale Xpages and keep Designer from processing at glacial speeds.
Click Permalink to continue...
This secret API is com.ibm.designer.domino.napi,
and it is deployed in a Designer client as "com.ibm.domino.napi"
If you have
Designer installed, you can find
this for yourself.
If you take a closer look at this eclipse/shared/plugin, you'll make a few discoveries. Of course, the most interesting discoveries might be found using the JD-Eclipse plugin, but I would never advise that, as it is most assuredly a violation of both your license agreement, and whatever version of the DMCA might apply to your local jurisdiction. Besides, it's difficult to install it in Designer with the File - Application - Install menu select. So stay far away from those kinds of hacker tools, okay? No good can come of it.
But even simply including the com.ibm.domino.napi plugin as a referenced library in Eclipse can provide some valuable information. It turns out that there are classes for NotesSession, NotesDatabase, NotesCollection, NotesCollectionIterator, NotesCollectionEntry, NotesNote, NotesItem and a few other less-familiar constructs. These classes all use a more direct path to access the underlying NSF structure, as they rely on a Java Native Interface approach to make calls to the C API. This is the same way that the Eclipse RCP itself handles standard UI components -- by providing a common Java abstraction layer to the platform-specific C libraries via SWT and Jface. And just as in Eclipse, it is dramatically faster.
Unfortunately, the NAPI classes are more akin to the C API approach to doing things than they are to the Java approach to doing things, so lots of stuff doesn't work the way we'd really like. Fortunately, the architecture of the Domino Java API leaves the door open for us to fix this.
You see, all the various classes of the lotus.domino package are not truly classes at all. They are Interfaces. In Java terms, that means that they are simply rule sets for how any actual objects should behave. So when you say Session.getDatabase("White", "rabbit.nsf"), you're really just referring to a method which by prior contract is known to exist on whatever actual object you got for a Session. Part of that contract is that the method will return an object that conforms to the rules of a Database.
The real objects are defined by different code. You'll find these packages in the notes.jar. They are lotus.domino.local and lotus.domino.cso. The lotus.domino.local package is an implementation of the lotus.domino ruleset.
So... could we make our OWN implementation of that ruleset? Nothing about lotus.domino prevents us from doing it. Last week, I tried it out. Instead of using the lotus.domino.local objects to access NSF data, I imported the NAPI classes, and put a wrapper around them that implements the lotus.domino classes such as Session, Database, Document, Item, View, ViewEntryCollection and ViewEntry. So now I can take the exact same code that I'd been running with the notes.jar, and run it with my new implementation. As far as my higher level agent or plugin code is concerned, there is no difference.
What was the outcome? See for yourself.
This simple Eclipse component defines 4 Jobs: two ordinary Eclipse Jobs and two NotesSessionJobs. The first two jobs access a copy of the events4.nsf, in which I have copy & pasted every event message, so there's just shy of 15,000 documents in that database. The second two jobs access an exact copy of that first NSF. I point them at a different file simply to avoid any file lock conflicts between the two implementations.
For each job, the task is simple: open up a View, start at the beginning, and read out a piece of information from each row in the view, then proceed to the next row. Each implementation has two versions of the process, one that gets the information via a ViewEntry.getColumnValues and another that gets the Document and does a .getFirstItem().getText(). Here's a snippet directly from the source code...
ViewEntryCollection localColl = localView.getAllEntries();
ViewEntry curEntry = localColl.getFirstEntry();
ViewEntry nextEntry;
String whatever;
while (curEntry != null) {
nextEntry = localColl.getNextEntry(curEntry);
whatever = curEntry.getColumnValues().elementAt(4).toString();
curEntry.recycle();
curEntry = nextEntry;
}
The developers among us have all seen this pattern about a billion times. But even the end-users (oh, and administrators) among us run code that does this, because it's part of how your email and calendar applications display their information. So you can imagine what kind of impact it would have on all aspects of the modern Notes/Domino experience if NSF data could be access 3 to 6 times faster than it is today! Tragically, the NAPI implementation is only about 0.1% of the overall API, so it's only useful for a few specialized tasks. For instance, right now, it can only read information from an NSF, not write it.
When I stumbled upon the NAPI package, I asked IBM about it. "This looks really cool, but it's only a tiny part of the API. When are you going to finish it?" The reply was basically "we'd like to finish it because it's much faster, but there's no customer demand for it."
No customer demand!? To make the API that underpins every aspect of the Notes 8 messaging UI, Designer in Eclipse, and every single aspect of every single Xpages application FOUR TIMES FASTER!??!?!?
Do you, dear reader, demand to make the single most important piece of Notes/Domino 8+ way, way better? I certainly do. And if we want to convince them of it, we will need to file PMRs, call sales reps, email product managers, buy developers a beer, blog, tweet, and stick your head out your window and yell "THIS API IS SLOW AS HELL, AND I'M NOT GONNA TAKE IT ANYMORE!!"
Incidentally, while I was at it, I made my implementation do away with .recycle(). Well, technically it doesn't do away with it, so much as make it totally irrelevant to the average user of the API. How? By making all the underlying objects recycle-proof. Just like the current API, the underlying code still performs a recycle to release the C memory handle. But even when it does, it retains enough information about where it got that handle in the first place to re-establish it as needed. So you never get a "object has been recycled" error. Because of this, we can aggressively recycle automatically behind the scenes. So, for instance, when you are looping through an EntryCollection, a call to .getNextEntry() will automatically recycle the previous entry, since you probably don't need it anymore. But if it turns out that we guessed wrong, and you DO need it, no problem. It's totally invisible to you when we re-establish the binding.
I'm sure someone is wondering when they'll be able to download this from OpenNTF. I'm looking into it. But one thing that would certainly help would be if all of your very loudly and very publically insisted that IBM get to work on making the NAPI package complete and fully supported. If they'd commit to implementing a few key features at the JNI layer, I will try to get the executives at GROUP to allow me to open source our implementation layer, and we can all pile on and do the grunt work of implementing every little nuance convenience method defined by the lotus.domino Interfaces.
Have a great weekend!
If you take a closer look at this eclipse/shared/plugin, you'll make a few discoveries. Of course, the most interesting discoveries might be found using the JD-Eclipse plugin, but I would never advise that, as it is most assuredly a violation of both your license agreement, and whatever version of the DMCA might apply to your local jurisdiction. Besides, it's difficult to install it in Designer with the File - Application - Install menu select. So stay far away from those kinds of hacker tools, okay? No good can come of it.
But even simply including the com.ibm.domino.napi plugin as a referenced library in Eclipse can provide some valuable information. It turns out that there are classes for NotesSession, NotesDatabase, NotesCollection, NotesCollectionIterator, NotesCollectionEntry, NotesNote, NotesItem and a few other less-familiar constructs. These classes all use a more direct path to access the underlying NSF structure, as they rely on a Java Native Interface approach to make calls to the C API. This is the same way that the Eclipse RCP itself handles standard UI components -- by providing a common Java abstraction layer to the platform-specific C libraries via SWT and Jface. And just as in Eclipse, it is dramatically faster.
Unfortunately, the NAPI classes are more akin to the C API approach to doing things than they are to the Java approach to doing things, so lots of stuff doesn't work the way we'd really like. Fortunately, the architecture of the Domino Java API leaves the door open for us to fix this.
You see, all the various classes of the lotus.domino package are not truly classes at all. They are Interfaces. In Java terms, that means that they are simply rule sets for how any actual objects should behave. So when you say Session.getDatabase("White", "rabbit.nsf"), you're really just referring to a method which by prior contract is known to exist on whatever actual object you got for a Session. Part of that contract is that the method will return an object that conforms to the rules of a Database.
The real objects are defined by different code. You'll find these packages in the notes.jar. They are lotus.domino.local and lotus.domino.cso. The lotus.domino.local package is an implementation of the lotus.domino ruleset.
So... could we make our OWN implementation of that ruleset? Nothing about lotus.domino prevents us from doing it. Last week, I tried it out. Instead of using the lotus.domino.local objects to access NSF data, I imported the NAPI classes, and put a wrapper around them that implements the lotus.domino classes such as Session, Database, Document, Item, View, ViewEntryCollection and ViewEntry. So now I can take the exact same code that I'd been running with the notes.jar, and run it with my new implementation. As far as my higher level agent or plugin code is concerned, there is no difference.
What was the outcome? See for yourself.
This simple Eclipse component defines 4 Jobs: two ordinary Eclipse Jobs and two NotesSessionJobs. The first two jobs access a copy of the events4.nsf, in which I have copy & pasted every event message, so there's just shy of 15,000 documents in that database. The second two jobs access an exact copy of that first NSF. I point them at a different file simply to avoid any file lock conflicts between the two implementations.
For each job, the task is simple: open up a View, start at the beginning, and read out a piece of information from each row in the view, then proceed to the next row. Each implementation has two versions of the process, one that gets the information via a ViewEntry.getColumnValues and another that gets the Document and does a .getFirstItem().getText(). Here's a snippet directly from the source code...
ViewEntryCollection localColl = localView.getAllEntries();
ViewEntry curEntry = localColl.getFirstEntry();
ViewEntry nextEntry;
String whatever;
while (curEntry != null) {
nextEntry = localColl.getNextEntry(curEntry);
whatever = curEntry.getColumnValues().elementAt(4).toString();
curEntry.recycle();
curEntry = nextEntry;
}
The developers among us have all seen this pattern about a billion times. But even the end-users (oh, and administrators) among us run code that does this, because it's part of how your email and calendar applications display their information. So you can imagine what kind of impact it would have on all aspects of the modern Notes/Domino experience if NSF data could be access 3 to 6 times faster than it is today! Tragically, the NAPI implementation is only about 0.1% of the overall API, so it's only useful for a few specialized tasks. For instance, right now, it can only read information from an NSF, not write it.
When I stumbled upon the NAPI package, I asked IBM about it. "This looks really cool, but it's only a tiny part of the API. When are you going to finish it?" The reply was basically "we'd like to finish it because it's much faster, but there's no customer demand for it."
No customer demand!? To make the API that underpins every aspect of the Notes 8 messaging UI, Designer in Eclipse, and every single aspect of every single Xpages application FOUR TIMES FASTER!??!?!?
Do you, dear reader, demand to make the single most important piece of Notes/Domino 8+ way, way better? I certainly do. And if we want to convince them of it, we will need to file PMRs, call sales reps, email product managers, buy developers a beer, blog, tweet, and stick your head out your window and yell "THIS API IS SLOW AS HELL, AND I'M NOT GONNA TAKE IT ANYMORE!!"
Incidentally, while I was at it, I made my implementation do away with .recycle(). Well, technically it doesn't do away with it, so much as make it totally irrelevant to the average user of the API. How? By making all the underlying objects recycle-proof. Just like the current API, the underlying code still performs a recycle to release the C memory handle. But even when it does, it retains enough information about where it got that handle in the first place to re-establish it as needed. So you never get a "object has been recycled" error. Because of this, we can aggressively recycle automatically behind the scenes. So, for instance, when you are looping through an EntryCollection, a call to .getNextEntry() will automatically recycle the previous entry, since you probably don't need it anymore. But if it turns out that we guessed wrong, and you DO need it, no problem. It's totally invisible to you when we re-establish the binding.
I'm sure someone is wondering when they'll be able to download this from OpenNTF. I'm looking into it. But one thing that would certainly help would be if all of your very loudly and very publically insisted that IBM get to work on making the NAPI package complete and fully supported. If they'd commit to implementing a few key features at the JNI layer, I will try to get the executives at GROUP to allow me to open source our implementation layer, and we can all pile on and do the grunt work of implementing every little nuance convenience method defined by the lotus.domino Interfaces.
Have a great weekend!
Comments
Posted by Jake At 11:30:41 AM On 03/19/2010 |
As part of a small Notes development team fighting against a new IT Director (a self proclaimed Notes hater), this is the sort of help from IBM we need to stay using Notes/in a job!
Tell me which window to stick my head out of and I'll fill up on throat sweets and scream until my larynx dissolves!
Posted by Mike At 11:34:25 AM On 03/19/2010 |
I guess that com.ibm.designer.domino.napi is unsupported and internal IBM code, but there's nobody stopping you from writing your own Eclipse plugin that bundles native libraries (dll's for Windows, .so for Linux, .jnilib for Mac) that calls C API functions, together with a Java API that calls them using the JNI framework (Java Native Interface).
Just search the web for "Bundle-NativeCode" to see how to define them. And the cool thing is that XPages with support those Eclipse plugins with bundled native libs in 8.5.2 (based on announces at Lotusphere 2010). There will even be support for the Eclipse plugin system (OSGi) on the Domino server.
Posted by Karsten Lehmann At 11:44:50 AM On 03/19/2010 |
@mike - been there several times with the notes haters. I feel for ya.
Posted by Bill Wheaton At 11:55:39 AM On 03/19/2010 |
Posted by Michael Bourak At 12:34:46 PM On 03/19/2010 |
While I appreciate the performance gains, I'm personally more interested in having access to more of the c API via JNI and building APIs on top of that.
Didn't Bill Buchan build something like a c API to LS "translator" to generate LS stubs for c API calls? Maybe something similar could be done for JNI.
Posted by Dan Sickles At 12:36:26 PM On 03/19/2010 |
btw, if you're scared like me of messing with eclipse plugins, JD-GUI may be an alternative to JD-Eclipse plugin
Posted by Giuseppe Grasso At 12:44:32 PM On 03/19/2010 |
Fredrik
Posted by Fredrik Stöckel At 02:28:27 PM On 03/19/2010 |
@2 - Thanks Mike. One part of it that really gets me is that the difference in ViewEntry iteration is key to the myth that Domino doesn't scale. In fact, my understanding is that Xpages tries to use napi for rendering Domino Views, but I could be mistaken about this.
@3 - The performance difference is emphasized by the fact that everything in lotus.domino.local is synchronized to the entire object. But I can't tell you how I know this.
And yes, building your own JNI is definitely possible. But I wouldn't want to have to support it. If I were going to bother with that, why even go to NSF? Pick a storage medium that has a native Java API and be done with it.
@5 - rofl
@6 - JNI code generators abound. Two immediate hits from Google: { Link } { Link }
@8 - The only thing I can figure is that since a better Java API doesn't make Domino a better messaging server or lower TCO, none of IBM's enterprise customers care about it.
Posted by Nathan T. Freeman At 03:11:42 PM On 03/19/2010 |
The reason I ask is that I recall Gary Devendorf once telling me that the back-end classes would be much, much faster if they didn't have lots of ECL checks built into them for when they run in front-end code. A good reason why IBM would have a "secret" set of classes could be that they are doing an end-run around the ECLs.
But then again, you're talking about Java, so you're not talking about code that runs in the front-end...
-rich
Posted by Richard Schwartz At 03:22:17 PM On 03/19/2010 |
If ECLs are causing a runtime of 3 to 6 times greater than without, then dear god we need a way to turn them OFF! 99 out of 100 Notes shops just tell their users to click "trust signer" on the dialog anyway. And they don't even exist on server-based processes!
But honestly, I don't think you can attribute a 6000ms vs 1000ms runtime to just ECL checks.
Posted by Nathan T. Freeman At 03:59:07 PM On 03/19/2010 |
Posted by Richard Schwartz At 05:45:17 PM On 03/19/2010 |
*Sigh.* I don't whether to feel exasperated or resigned.
Oh wait, perhaps I've figured it out... more speed might cut into per-processor server license sales. And that would be bad, right?
You should definitely check this against a server.
Posted by Erik Brooks At 07:56:19 PM On 03/19/2010 |
@13 - Yeah, like I said... the support process is the Achilles' heel for Lotus right now. Basing all your product decisions on PMR demand is great if you're a middleware company, but it fails completely in the realm of business solutions, where end-user experience rules the day. The end-user doesn't open a PMR. He just switches to a different solution and never tells you until it's time to pay maintenance.
And yeah, I'll run it on a server. I just have to set up an 8.5.2 server first, because the security manager prevents the napi package from being called from Xpages. It has to be deployed as an OSGi plugin first, and that means the new server build. (I only know this because a very senior developer told me so.)
That being said, I expect that the issue will be even more emphasized in a server context because of the synchronization issue. You simply can't do real multi-threading on the same NSF in the existing API. Every call pushes to the same thread, which has the same Session object, which serves up the same Database objects, all of which have every method synchronized to the entire object. Hence the title of this post.
It's a Java 1.1 API that's hasn't had a close look in about a decade, and in the meantime, the entire platform has become achingly dependent on it.
By the way, I literally laughed out loud at your PMR proposal.
Posted by Nathan T. Freeman At 09:50:03 PM On 03/19/2010 |
Posted by Keith Smillie At 07:47:58 AM On 03/20/2010 |
Glad to see the 'decompiling a jar file' cat is out of the bag.
If you have lots and lots of jars that need decompiling this will help:
{ Link }
Keith
Posted by Keith Smillie At 08:07:50 AM On 03/20/2010 |
Posted by Richard Schwartz At 05:37:07 PM On 03/20/2010 |
ps: Will this new find allow us to do a JOIN?
Posted by Peter Presnell At 12:42:03 PM On 03/21/2010 |
No one person can really know the power of what we have today. It's far too much. Do you know how many Java packages are available in Xpages on a Domino 8.5.1 server? About 1500. The core Harmony runtime alone has 900 packages! (And yes, dear readers, I actually counted.) It's way, way too big for anyone to digest any more. It will never be entirely documented.
Posted by Nathan T. Freeman At 06:12:40 PM On 03/21/2010 |
Interesting find, Nathan.
I think if I were making a product that does a whole lot of the same thing sequentially -- like searching a bunch of databases or something -- I would find this pretty interesting.
Posted by Andrew Pollack At 09:15:29 PM On 03/23/2010 |
Posted by Devin Olson At 06:02:12 PM On 03/26/2010 |