Mindoo Blog - Cutting edge technologies - About Java, Lotus Notes and iPhone

  • XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 2

    Karsten Lehmann  April 27 2012 07:00:09 PM
    This is part 2 of an article about using MongoDB in Notes/Domino. Click here for part 1.


    Diving into the code: client side

    The UI stuff is stored in an NSF database with basic page layout defined in an XPage and the client-side application logic code stored as Dojo class file in the Java perspective of Domino Designer.



    The Dojo class location is defined and the class is loaded with the following code in the XPage "start.xsp":

    <xp:this.resources>
            <xp:dojoModulePath prefix="mongo">
                    <xp:this.url><![CDATA[#{javascript:'/dojo/mongo';}]]></xp:this.url>
            </xp:dojoModulePath>
                    <xp:dojoModule name="mongo.test.PageHandler">
            </xp:dojoModule>
    </this.resources>

    After that, the class can be instantiated in a script block:

    <xp:scriptBlock type="text/javascript">
            <xp:this.value><![CDATA[
            dojo.addOnLoad(function() {
                    window.pageHandler=new mongo.test.PageHandler();
                    window.pageHandler.init();
            });
            ]]></xp:this.value>
    </xp:scriptBlock>

    In the init() method, the page handler class registers event code for the form fields and buttons in the UI and creates the Dojo grid components (dojox.grid.LazyTreeGrid) to display the database content.

    Now let's take a look at two UI operations in detail, to see how the UI communicates with the server-side code.

    Adding data to the database

    When places get added through the Administration tab, we first use the servlet "/mongogeo" to convert the addresses into coordinates:



    What we get back is the JSON response of  the Google Geocoding API. It does not only contain the coordinates of an address, but other quite useful information like the name of the federal state, the country code and coordinates of a bounding box (for general addresses like city names that don't have exact coordinates).

    Next, a POST operation to "/mongotest/addplaces" creates the actual place documents in the database followed by a reload operation of our Dojo grid to reflect the new data in the UI:



    Query the database

    The REST service "/mongotest/queryplaces" is used both for reading all places (on the Administration tab) and finding the nearest places for a certain position (on the Search tab). For the latter, we specify additional longitude/latitude arguments in the URL:

    http://localhost/dev/ec12/nosql/mongodb/geo.nsf/mongotest/queryplaces?distance=3&longitude=8.3799444&latitude=49.009148&type=Shop&start=0&count=100

    Dynamic sorting can be applied to the result with and without longitude/latitude arguments by specifying a sort parameter.
    If no sort parameter and no position is specified, places are sorted by name as default sorting. For queries with longitude/latitude parameters, the distance between both points is used as default sorting.

    http://localhost/dev/ec12/nosql/mongodb/geo.nsf/mongotest/queryplaces?distance=3&longitude=8.3799444&latitude=49.009148&type=Shop&start=0&count=100&sort=name

    In the UI, data can be sorted by clicking one of the grid column headers.




    Enough technical details, let's come to an end with this long blog article! :-)

    If you would like to try out the sample in your own environment, here are the setup instructions:


    Setup instructions - Prerequisites

    I expect that you have downloaded and installed the latest version of MongoDB for your operating system. The sample was built for version 2.0.3 and I just found out that 2.0.4 is already available.

    The server code is using localhost and port 27017 by default, but this can be changed by setting the environment variable NOSQL_MONGO_SERVER to something like "hostname1,hostname2:27123,hostname3", a comma separated list with hostname and optional port. The Mongo driver will use the list of servers for failover in case a server goes down.

    Plugin installation

    Download and extract the archive file from the specified download link.

    Then follow the instructions in the Domino wiki article XPages Extension Library Deployment in Domino 8.5.3 and IBM XWork Server to create an NSF based update site database on the Domino server and import the update site from the download archive into the update site database, followed by a HTTP task restart (restart task http).

    Database installation

    Copy the sample database from the download archive to your Domino server and sign it with your Notes ID. Now open the database in the browser.

    There is a section "Initialization" on the Administration tab which lets you import a set of default addresses (from Mindoo's home town Karlsruhe, Germany). This step is optional.

    Download link

    Click here to download the sample application



    Phew... That was a long text. Thanks for reading until here!

    And stay tuned for an article about my second demo from Entwicklercamp 2012:

    Using Neo4J to solve the travelling salesman problem in XPages apps


    XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1

    Karsten Lehmann  April 27 2012 06:59:38 PM
    This article presents the first demo of my session about NoSQL databases at the German Entwicklercamp conference in March 2012. It demonstrates how the document-oriented NoSQL database MongoDB can be used in an XPages web application for the IBM Lotus Domino server.

    The rise of location based services

    Location based services have become quite popular in the past years: Most of the smartphones carry a GPS sensor, and there are a lot of popular apps out there (e.g. Foursquare, Yelp) that use cloud services to find people, restaurants and places nearby.

    For a number of reasons (mostly scalability/performance related), the NSF database of Lotus Notes/Domino is not the best choice to implement these kind of applications/services. MongoDB is much more suited for this use case and the article shows how you can integrate MongoDB into Notes/Domino to combine both worlds and get the best from both of them.

    The Dojo based web application that I am going to discuss in detail uses MongoDB's geospatial indexing feature to easily find the nearest points of interest for a mobile web user.

    Here is how our application looks like:



    The "Administration" tab provides a user interface to add/remove places to/from the MongoDB database. A place is stored as a document in a MongoDB document collection and consists of a name, a type (e.g. shop/gas station) and the position information as [longitude, latitude] value pair.

    The database can be queried on the "Search" tab by entering an address, a distance in kilometers and an optional type. The address will automatically get converted to [longitude, latitude] coordinates by using the Google Geocoding API, which is then used to find places within the specified distance.

    Our MongoDB document collection is indexed in a way so that we can quickly query the database for places that surround the current user's position and sort them by ascending distance between both points.



    Since geospatial indexing is a built-in feature of MongoDB, the solution is very easy to implement.

    Limitations of NSF

    While it is not impossible to build this application in pure Lotus Notes/Domino technology with an NSF database, the missing geo index and limited scalability of NSF would make it difficult to keep response times low with large data sets, e.g. millions or even billions of places worldwide. MongoDB instead provides a sharding feature to distribute the data evenly to several servers and reduce server load.

    Additional indexer like Apache Lucene in combination with the OSGi tasklet service plugin's extension hooks might help to improve Domino's limited indexing support (we already used a Lucene index with Domino data in a big customer project) , but using MongoDB for this use case is just so much easier.

    Architecture

    The application consists of client-side and server-side code:
    Client-side code is written in JavaScript language as a Dojo class using the class declaration system of Dojo 1.6. This Dojo class handles all user interface operations.

    Server-side code is written in Java as a servlet and is exposed to the client-side via REST API's (here: GET/POST requests sent via dojo.xhrGet / dojo.xhrPost).

    We developed two servlets for this application. The first servlet handles init/add/remove/query operations in MongoDB. The second servlet simply acts as a proxy to access the Google Geocoding API from client-side Javascript in order to convert between addresses and geo coordinates (longitude/latitude).

    Since the REST protocol is completely stateless, the server-side code is very easy to test. With a bit of refactoring, it's even possible to run the whole application in alternative servlet containers like Jetty, Tomcat or directly within the Eclipse IDE instead of the Domino server.

    Personally, I really like this REST API based approach because it's clean, transparent and keeps technologies separated.
    It enables you to replace the server environment, the database system and the client-side UI toolkit at any time. You can even build multiple UIs (e.g. based on Dojo 1.6, 1.7 and Sencha's ExtJS) or mobile clients that work with the same REST API.

    Diving into the code: server side
    Our sample comes as an Eclipse plugin "com.mindoo.mongo.test" to be run on Domino's OSGi framework (tested with Domino 8.5.3 GA without any fixpacks or XPages extension library):



    As you can see above, I added the Mongo API classes to the plugin's classpath.
    Adding the Mongo API classes to an NSF and using them directly from SSJS code might also be possible, but I prefer the plugin solution, because the MongoDB access is supposed to be a central service on the Domino server. In addition, by using a plugin we don't have any issues with Domino's restricted security manager that prevents operations to run properly in an XPages context (e.g. a lot of drivers are using Log4J for logging, which does not run well within an XPages application).

    Our two servlets are defined in the plugin.xml file of the plugin:

    <?xml version="1.0" encoding="UTF-8"?>
    <?eclipse version="3.4"?>
    <plugin>
            <extension point="org.eclipse.equinox.http.registry.servlets">
                    <servlet alias="/mongotest" class="com.mindoo.mongo.test.servlet.MongoTestServlet"
                            load-on-startup="true">
                    </servlet>
            </extension>
            <extension point="org.eclipse.equinox.http.registry.servlets">
                    <servlet alias="/mongogeo" class="com.mindoo.mongo.test.servlet.GeoQueryServlet"
                            load-on-startup="true">
                    </servlet>
            </extension>
    </plugin>

    That's all you need to declare your own servlet on Domino.
    After that, the servlet can either be accessed on the URL base level (http://server/mongogeo) or in the context of a Domino database (http://server/path/to/db.nsf/mongotest).

    If called in the context of a database, the Domino HTTP server first makes sure that the current user is allowed to access the database and if not, it displays a login prompt. A utility class (com.ibm.domino.osgi.core.context.ContextInfo) can be used to get session/database objects for the current HTTP request.

    In our sample, we use the first format for our Geo API query, because it's an open service of the server without access restriction. Anyone can use it and the URL is simple to remember.

    The second URL format (/path/to/db.nsf/mongotest) is used to access MongoDB data. This enables us to check the user against the Notes database ACL to see if he is allowed to read or write MongoDB data.

    Here is the method of our MongoDB access servlet that handles POST requests:

    protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
            Session session=ContextInfo.getUserSession();
            Database dbContext=ContextInfo.getUserDatabase();
            if (dbContext==null) {
                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                            "Servlet must be run in the context of a Domino database", null);
                    return;
            }

            try {
                    String pathInfo=req.getPathInfo();
                    if ("/deleteplaces".equals(pathInfo)) {
                            if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                            "Current user is not allowed to write data", null);
                                    return;
                            }
                            doDeleteGridData(session, dbContext, req, resp);
                    }
                    else if ("/addplaces".equals(pathInfo)) {
                            if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                            "Current user is not allowed to write data", null);
                                    return;
                            }
                            doAddGridData(session, dbContext, req, resp);
                    }
                    else if ("/queryplaces".equals(pathInfo)) {
                            if (!isUserAuthenticated(MongoAccessMode.Read, session, dbContext)) {
                                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                            "Current user is not allowed to read data", null);
                                    return;
                            }
                            doQueryPlaces(session, dbContext, req, resp);
                    }
                    else if ("/initdb".equals(pathInfo)) {
                            if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                            "Current user is not allowed to write data", null);
                                    return;
                            }
                            doInitDbWithValues(session, dbContext, req, resp);
                    }
                    else if ("/cleardb".equals(pathInfo)) {
                            if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                    ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                            "Current user is not allowed to write data", null);
                                    return;
                            }
                            doClearDb(session, dbContext, req, resp);
                    }
                    else {
                            ServletUtils.sendHttpError(resp, 500, "Unsupported command "+pathInfo, null);
                    }
            }
            catch (Throwable e1) {
                    e1.printStackTrace();
                    //avoid posting the full stacktrace to the user (for security reasons)
                    ServletUtils.sendHttpError(resp, 500, "An error occurred processing the request. The error has been logged. Please refer to the server log files/console for details", null);
                    return;
            }
    }

    The code is pretty simple: We check if the user is allowed to call the specific service (read operations require the role [MongoDBRead], write operations the role [MongoDBWrite] ) and then forward the request to helper methods.

    Note that I am not going into full detail in this blog article how we actually access data in MongoDB. You can find that in the provided download archive and there is a great tutorial at the MongoDB website about using the Java API.

    Believe me, this stuff is really easy to use!


    Click here for part 2 of this article!

    English slides for my Entwicklercamp 2012 session about NoSQL databases

    Karsten Lehmann  April 5 2012 12:07:41 AM
    As promised, here are the translated files for my NoSQL session at this year's Entwicklercamp (developer camp) conference in Gelsenkirchen, Germany.

    Image:English slides for my Entwicklercamp 2012 session about NoSQL databases

    I am still searching for time to polish the two demos: MongoDb integration in XPages apps and NoSQL integration in XPages apps.
    Next week is blocked for German holidays. Need to work on the demos afterwards.

    Download link for my Entwicklercamp 2012 session slides about NoSQL databases (German)

    Karsten Lehmann  March 27 2012 07:00:43 PM
    Here are the slides for my session about NoSQL databases at Entwicklercamp 2012 today. For the non-German audience: sorry for the German content, I hope Google translate will help. :-)
    I plan to translate them to English, but this may take some time.

    Image:Download link for my Entwicklercamp 2012 session slides about NoSQL databases (German)


    I will also write blog entries about my two demos: leveraging the MongoDb geo index feature from an XPages application and embedding the Neo4J graph databases in XPages (travelling salesman algorithm to find the best route in a graph).

    Update:
    The English slides of the session are now online.

    Speaking at Entwicklercamp 2012 about NoSQL databases

    Karsten Lehmann  March 19 2012 09:08:18 AM
    Yesterday I submitted my slides for this years Entwicklercamp in Gelsenkirchen, Germany. My session will provide an overview of the NoSQL market for the database types
    • key value stores
    • column oriented databases
    • document oriented databases
    • and graph database

    It discusses Redis, Apache Cassandra, MongoDB and Neo4j in detail and I will be demo'ing, how to integrate MongoDB's geo-spatial indexing feature and Neo4j's graph search algorithms in XPages applications.

    Finally, the session compares the feature sets of Lotus Notes with other NoSQL to see how compatitive Lotus Notes, one of the oldest NoSQL database systems, is today in this moving market.

    Working on this topic was quite challenging, because there is A LOT of material to check out on the web, but it also was a lot of fun playing with the databases and building the demos.

    Icon resources on the web for commercial and non-commercial use

    Karsten Lehmann  February 8 2012 08:44:18 AM
    As Web and Notes developers, we often need to find icons of various sizes to use them in our applications.
    Since I found a new massive resource of free icons yesterday, I thought it might be a good idea so share some URLs.

    For commercial applications, we bought the Icon Experience library a few years ago, which contains about 2500+ icons in 7 sizes at a very affordable price:
    http://www.iconexperience.com

    Free icons with varying licenses can be found here:

    http://openiconlibrary.sourceforge.net
    http://www.freeiconsweb.com
    http://www.famfamfam.com
    http://p.yusukekamiyamane.com

    For the latter, licenses reach from GPL, LGPL, Creative Commons to Public Domain. I guess for pure inhouse development, this is not that important, but if you plan to sell your application or publish it on OpenNTF, you should doublecheck if the icon license is ok for your use case.

    Feel free to add your own favorites by commenting on this article!

    XPages series #13: XPiNC app development tool

    Karsten Lehmann  February 2 2012 04:24:12 PM
    I was just working on an XPiNC integration of a quite large application and had some trouble getting it to work in the Notes Client (the app was working well on the web already). Finding out why it was not working was even harder in this case than when dealing with "normal" XPages applications, because the application is not based on the Dojo web toolkit, but uses Sencha's Ext JS for the UI and is completely based on our own web app development framework.
    The framework does not use XPages design elements at all, but follows a simple REST API architecture with standard servlets to produce the data and user interface.

    The benefit of this approach is that web applications can be developed, run and debugged from a pure Eclipse environment. They can even run on an different servlet engine than the Domino server's http task and - thanks to our data abstraction layer - they can even store the whole app data in a non-Domino database and mix/merge data between different database types.

    Another benefit is that almost all the code has been written by ourselves. So it's not a kind of blackbox, made by IBM, where it's hard to work around occuring issues, but we are able to track issues down right until the service(HttpServletRequest req, HttpServletResponse response) call coming from the web container.

    The downside is of course, that apps developed with the framework do not make use of IBM's XPages Extension Library, that contains various powerful UI controls. So we needed to create them ourselves.

    And regarding XPiNC development, there is another downside: JavaScript errors do not get logged/displayed in the Notes Client, unless you register your own global error handler (providing a function for window.onerror).
    Normal XPiNC applications do already contain such an error handler (somewhere within the XSP API object), which uses an internal bridge to post the error content to the Notes Client's status bar.

    The development tool

    To make development of XPiNC applications easier, I have created a small Eclipse plugin that displays three icons in the Client's toolbar:
    Image:XPages series #13: XPiNC app development tool


    The third icon lauches a piece of code that injects Firebug lite into the currently visible XPage:

            public void run(IAction action) {
                    IWorkbenchPart part =
                      PlatformUI.getWorkbench().getActiveWorkbenchWindow().getPartService().getActivePart();

                    if (part instanceof XspViewPart) {
                            XspViewPart xPart=(XspViewPart)part;
                            XspXulRunnerBrowser browser=xPart.getWebBrowser();
                           
                            HTMLDocument doc=browser.getDocument();
                            NodeList headNodes=doc.getElementsByTagName("head");
                            if (headNodes!=null && headNodes.getLength()>0
                               && headNodes.item(0) instanceof Element) {

                                    //add this snippet to the HTML DOM tree:
                                    //< script type="text/javascript"
                                    // src="https://getfirebug.com/firebug-lite.js" > < /script >
                                    String firebugUrl="https://getfirebug.com/firebug-lite-debug.js";
                                    Element headNode=(Element) headNodes.item(0);
                                    Element scriptNode=doc.createElement("script");
                                    scriptNode.setAttribute("type", "text/javascript");
                                    scriptNode.setAttribute("src", firebugUrl);
                                    headNode.appendChild(scriptNode);
                            }
                    }
                    else {
                            MessageDialog.openError(Display.getDefault().getActiveShell(), "Error",
                              "This action does only work within an XPiNC application!");
                    }
            }

    The result looks like this:

    Image:XPages series #13: XPiNC app development tool


    Using Firebug Lite, you can easily inspect the HTML DOM tree and CSS attributes of the current page. Another very useful feature is the Console API, which you may know already from classic browser development:
    it lets you write log/debug messages to the browser console (by calling console.log('...')) and has other nice features like stacktrace dumping of JavaScript calls.

    To our surprise we found out that the pure Xulrunner engine, that is used to display XPages in the Notes Client, does not register the global "console" at all. So Firebug lite came to the rescue. You can find all your log messages in the Console tab.

    The other two toolbar actions are even more powerful. One lets you define a custom script library URL, e.g. to a script library design element on a public or intranet web server. The other actions will then create a script tag in the current XPage that points to the library. That way, you can inject any code you like into the XPiNC application.

    Download

    Finally, here is the download link with the Eclipse plugin project, feature project and update site:

    xpinc-firebuglite-helper.zip

    Hope this helps!

    Lotusphere 2012: Download links for 105 additional session slides

    Karsten Lehmann  January 22 2012 10:59:10 PM
    I just compared the current state of the Lotusphere 2012 website with my previously released list of session slide download links.
    I found 105 new slide downloads:

    ls12_20120122.csv
    ls12_20120122.html

    There are still a few missing slide decks, but it's a big step forward.

    Lotusphere 2012: download links for session slides

    Karsten Lehmann  January 17 2012 04:50:09 PM
    I just spent some time to grab the download links of the sessions slides from the Lotusphere 2012 website. Since my Macbook Pro could not get a proper connection to the wireless network in Dolphin (network is quite bad both at Lotusphere and in the Yacht Club we are staying in), I had to do this on the iPhone, because it was the only device that got an IP address. That was fun. ;-)

    So here are the links to the slides that are available so far, sorted by session ID. Unfortunately, many are still missing. From our last years experience, the speakers are not the one to blame here. All had to submit their slides back in December.

    Hopefully, the remaining session slides will follow shortly.

    ls12_20120117.csv
    ls12_20120117.html

    Use your preferred download utility (e.g. DownThemAll) for download. You need to be logged in on the LS12 website before downloading.

    Status report / no session submission for LS12

    Karsten Lehmann  November 7 2011 10:27:35 AM
    It's been some time since the last blog post in July. The last weeks have been incredibly busy, working five days a week on 3-4 projects at customers on-site does not leave much time for blogging and it does not look like this will change very soon.
    To give you an impression, here are a few things that we've been working on:

    Development of an OSGi based web application framework with Ext.js UI
    Abstraction layer for web applications that unifies data access across document-oriented and relational database systems, allows for tracking of data object changes (old/new values, group multi-object changes as transactions), different kinds of data serialization like XML or JSON and visualizing data with an Ext.js based, dynamically created web UI (using Jamon template language) that reads and writes contents through REST APIs. The framework can run from pure Eclipse and also in Domino's OSGi container.

    Development of an OSGi based XPages application framework with Dojo UI
    XPages extensibility API add-on to develop the backend code of XPages directly in Java instead of writing SSJS code or Expression Language to bridge between UI element events/properties and application code.
    The framework uses Java Annotations and Java Reflection APIs for UI/backend code weaving.

    Development of an IBM Websphere portal server applications
    Custom development of market-analysis application for consumer goods industry.

    Development of dynamic web application based on Glassfish and JPA, with jQuery and Highcharts
    Data analysis tool for the buying department of a German automotive company.

    Plugin development for the Lotus Notes Client
    Several productivity enhancement plugins for the Lotus Notes Client

    Plugin deployment support
    Autodetection routine for XPages applications in Client/Server to automatically check for required Eclipse features/plugins when an application is opened (implemented in Lotusscript) so that the XPages runtime does not display those confusing error messages about missing XPages extensions. Instead, errors are properly handled without user interaction and missing features are automatically deployed on the machine.

    XPiNC development framework
    Spent time to work on enhancements for the XPages2Eclipse toolkit that provides Eclipse APIs to XPages applications in the Lotus Notes Client.

    Classic Domino web application development
    Improved round tripping quality of richtext editing between web browser and Notes Client including paste optimization from MS Word. Solution uses a combination of CKeditor add-ons and HTML/DXL conversions developed in Lotussript code for R7.  Customer: print industry

    Classic Notes Client development
    Built a Notes application to automatically deploy and update a list of mail folders to iOS users on Traveler.


    A word about Lotusphere 2012
    Since time is so limited, we decided not to submit a session proposal for Lotusphere 2012.

    We will however participate with full conference pass and I just booked hotel and flight two days ago. In contrast to previous years, we will arrive one day earlier, on Friday, 13th of January 2012 (hopefully not a bad day for travel, at least it's a direct flight :-)).
    Are there any plans yet for a blogger meeting for a beer or two on Saturday 14th like in previous years?

    Another reason not to submit a session is that things get more complicated year after year. To learn about OSGi development or XPages extensibility APIs in a hands-on session for a day or two does make much more sense than watching a one hour presentation.

    We could have submitted our plugin development session from last year another time (overview of plugin development for Notes Client and DDE with 14 demos), because slides and demos are already done. But that session (which got very good feedback at LS11) has now been presented four times at two conferences (Lotusphere and German Entwicklercamp 2010/2011). It's hard to get motivated for the 5th time :-).


    So, to sum up this long blog entry, I am really looking forward to seeing many of you guys at LS12 - and to having more time to work on concepts and product prototypes in the next weeks.

Tags

Blog Navigation