Goal :

  • Persist incoming stream of Objects of into the Database.
  • Parse the incoming XML/Json data to extract “Object of similar type” (i.e. a particular Business Entity) .
  • The Business Entity may have a fixed set of attributes or a dynamically changing set of attributes.
  • We should not create/maintain ORM layer and no need for POJO conversions from/to wsdl/http streams.
  • We should keep the persistence service layer very very light-weight.
Setup :
  • mongodb2.0.1 (A database named “sampledb” needs to be created in MongoDB)
  • spring-data-mongodb 1.0.0.M4, mongo-java-driver 2.5.2 (and all other dependencies specified in pom.xml)
Development Environment : SpringSource Tool Suite – 2.8.1.RELEASE
Implementation Approach :
  • SimpleJdbcInsert Command make things really simple when Objects of same type has a fixed set of attributes.

Create the singleton instance of the SimpleJdbcInsert Command for the corresponding table.Prepare the Map of key-value pairs by parsing the incoming data and simply push the data into the tables against the corresponding columns (keys in the map).

Number num = insertCommand.executeAndReturnKey(parameters);
Or insertCommand.executeBatch(…)

  • It may be tweaked to leverage DatabaseMetadata and perform DDL on RDBMS programmatically.
  • But its very costly to Create/Update/Delete columns in relational data-store on-the-fly.
  • So finally we take resort to MongoDB.
  • If the Business Entity has a pre-defined set of attributes we should leverage the well-known annotated document object.
  • Otherwise if Entity structure is changing constantly to accommodate newer attributes, then just create new DBOnjects and DBRefs.
MongoConfig creates a MongoTemplate against the database “staging” MongoDataAccessService provides all CRUD operations.MongoRestController specifies all the Rest operations.
Sneak Peak into the code .. org.jdom.Element rootElement = jdomDocument.getRootElement(); String rootElemName = rootElement.getName();

if (!mongoDataAccessService.collectionExists(rootElemName)) {
// we assume the element contains all the values for the table-columns as the attributes.
List list = rootElement.getAttributes(); DBObject parentDocument = new BasicDBObject();
// suppress _ClassName in the DbObject
for (Object attr : list) {
org.jdom.Attribute source = (org.jdom.Attribute) attr;
if (source.getName().equals(“Id”)) { parentDocument.put(“_id”, source.getValue());
// use the original incoming Id // candidate for shard key — TODO
} else { parentDocument.put(source.getName(), source.getValue()); }
// Now iterate through the Child Elements and store them as Database
References DBRef childDocumentRef = new DBRef( mongoDataAccessService.getDB(“sampledb”),
elementName, elementId); parentDocument.put(elementName + “_ref”, childDocumentRef);

Runtime Steps :
  • target folder already contains a pre-built war file.
  • just drop the war inside an app server context root.
  • http://localhost:8080/dynamic-data-collection/mongo/entities/add
  • Run the MongoRestClient to dynamically extract an entity from XML fragment and add it to the corresponding Collection.
  • Simple MongoClient just creates instance of a predefined Mongo Document. This comes handy when Entity has a fixed set of attributes.

Code : https://github.com/kaniska/Dynamic-Object-Persistence

Take Away :
  • Analyze the Business requirements and accordingly categorize the types of Entities.
  • If all Entities can have pre-defined schema then use SimpleJdbcInsert Command along with MySQL / Oracle for best performance.
  • But in case the Business Process consists of both fixed and dynamic entities, then consider the following approach.
  • Define annotated POJO (Mongo Document) for the Entities having a fixed structure.
  • If same type of Entity has different set of attributes for different entities then consider creating them dynamically.
  • So the choice of database (MongoDB Vs RDBMS) – depends upon how much dynamism one needs to accommodate in the application.
Future Improvements
  1. Externalize the MongoDB configuration in external properties file.
  2. Use the latest Spring-MongoDB release
  3. Work with XPathOperations instead of parsing xml fragments manually.
  4. Figure out if Spring-Data MongoDB API leverages latest Morphia annotations for journal-sync, suppressing _ClassName in objects etc.