Friday, January 6, 2012

Conveniently Processing Large XML Files with Java

When processing XML data I find it most convenient to load the whole document using a DOM parser and fire some XPath-queries against the result. However, since we're building a multi-tenant eCommerce plattform we regularly have to handle large XML files, with file sizes above 1 GB. You certainly don't want to load such a beast into the heap of a production server, since it easily grows up to 3GB+ as DOM representation.

So what to do? Well, SAX to the rescue! Processing a large XML file using a SAX parser still requires constant (low) memory, since it only invokes callback for detected XML tokens. But, on the other hand, parsing complex XML really becomes a mess.

To resolve this problem we need to have a closer look at our XML input data. Most of the time, at least in our cases, you don't need the whole DOM at once. Say your importing product informations, it sufficient to look at one product at a time. Example:

<nodes>
    <node>
        <name>Node 1</name>
        <price>100</price>
    </node>
    <node>
        <name>Node 2</name>
        <price>23</price>
    </node>
    <node>
        <name>Node 3</name>
        <price>12.4</price>
        <resources>
            <resource type="test">Hello 1</resource>
            <resource type="test1">Hello 2</resource>
        </resources>
    </node>
</nodes>

When processing Node 1, we don't need access to any attribute of Node 2 or three, respectively when processing Node 2, we don't need access to Node 1 or 3, and so on. So what we want is a partial DOM, in our example for every <node>.

What we've therefore built is a SAX parser, for which you can specify in which XML elements you are interested. Once such an element starts, we record the whole sub-tree. When this completes we notify a handler which then can run XPath expressions against this partial DOM. After that, the DOM is released and the SAX parser continues.

Here is a shortened example of how you could parse the XML above - one "<node>" at a time:

   XMLReader r = new XMLReader();

   r.addHandler("node", new NodeHandler() {

     @Override
     public void process(StructuredNode node) {
       System.out.println(node.queryString("name"));
       System.out.println(node.queryValue("price").asDouble(0d));
     }
   });

   r.parse(new FileInputStream("src/examples/test.xml"));

The full example, along with the implementation is open source (MIT-License) and available here:
https://github.com/andyHa/scireumOpen/tree/master/src/com/scireum/open/xml
https://github.com/andyHa/scireumOpen/blob/master/src/examples/ExampleXML.java

We successfully handle up to five parallel imports of 1GB+ XML files in our production system, without measurable heap growth. (Instead of using a FileInputStream, we use JAVAs ZIP capabilities and directly open and process ZIP versions of the XML file. This shrinks those monsters down to 20-50MB and makes uploads etc. much easier.)


This post is the first part of the my series "Enterprisy Java" - We share our hints and tricks how to overcome the obstacles when trying to build several multi tenant web applications out of a set of common modules.

125 comments:

  1. Nice little bit of code, thanks for sharing. Apache Jersey has similar capabilities for stream parsing JSON and building a tree of just the bits you are interested in but I've always been surprised why there isn't this type of support in a core Java XML library.

    ReplyDelete
    Replies
    1. Great Article android based projects

      Java Training in Chennai

      Project Center in Chennai

      Java Training in Chennai

      projects for cse

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. Hi Andy,
    That's a nice approach indeed. I would also recommend you having a look at our approach that was specifically designed to handle very large XML input using SAX. The product allows you to specify (runtime config subscription) which types you want to process and sends these as JavaBeans to the processor component. It also allows you to specify how to handle 1..many relationships. So, for instance, say I'd like to process the element and I'd also want to process each of the elements, I could do that detaching the elements from their parent. Well, have a look hre: http://xml2java.net or here: http://xml2java.net/documents/XMLParserTechnologyForProcessingHugeXMLfiles.pdf

    Kind regards,
    Lolke Dijkstra

    ReplyDelete
  3. Hi how to get the values in one Time loading.the hanlder is not working in the order.

    ReplyDelete
    Replies
    1. Sorry, I don't get the question. The handler behaves pretty much like a sax parser. Therefore you get all values in the order of the document

      regards Andy

      Delete
  4. Not SAX, it is so 1990s, vtd-xml is the answer

    ReplyDelete
    Replies
    1. Nope not to this question...

      The first sentence on the website says:
      The world's most memory-efficient (1.3x~1.5x the size of an XML document) random-access XML parser.

      Having to deal with 1 GB large XML files, that still too much and the SAX/partial DOM approch is better for this purpose.

      Delete
  5. Thanks for the useful information, give more updates like First time I visit your site really nice, here after a daily visit.
    ecommerce website development company in chennai

    ReplyDelete
  6. Great job for publishing such a beneficial web site. Your web log isn’t only useful but it is additionally really creative too.
    Twin Cities Web Design

    ReplyDelete
  7. I feel satisfied to read your blog, you have been delivering a useful & unique information to our vision.keep blogging.
    Digital Marketing Course In Kolkata
    Web Design Course In Kolkata

    ReplyDelete
  8. There is always a smarter way to win the task! And if you are looking for ways to unveil that secret, you have landed on the right page. Best Search Engine Optimization services can make your online website perform better. The Web Mines is the company full of creative minds, we are catering SEO services in different countries of the USA. We assure our clients to give the best SEO services. The services we offer our clients are designed according to your business requirements that can fit best in Google algorithms.
    local seo company in usa, local seo the web mines, local seo services pricing, local seo services toronto, local seo services, local seo services atlanta, local seo services companies, local seo services st louis, local seo services in cape town, local seo services usa, local seo services uk, best local seo services, local seo expert company, local seo services for small business, affordable local seo services, local seo marketing company,industrial battery manufacturers usa

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. They follow the footstep of great people, read all scriptures by heart, believe in the almighty, study and remember the latest scientific studies to find some way which can lead them to success and happiness. share more details.
    Ai & Artificial Intelligence Course in Chennai
    PHP Training in Chennai
    Ethical Hacking Course in Chennai Blue Prism Training in Chennai
    UiPath Training in Chennai

    ReplyDelete
  11. I’m excited to uncover this page. I need to to thank you for ones time for this particularly fantastic read !! I definitely really liked every part of it and i also have you saved to fav to look at new information in your site.

    Data Science Course

    ReplyDelete
  12. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    Data Science Training Institute in Bangalore

    ReplyDelete
  13. It's really nice and meanful. it's really cool blog. Linking is very useful thing.you have really helped lots of people who visit blog and provide them usefull information.

    Data Science Training

    ReplyDelete
  14. Thanks for your post! Through your pen I found the problem up interesting! I believe there are many other people who are interested in them just like me! Thanks your shared!..
    acte velachery reviews

    acte tambaram reviews

    acte anna nagar reviews

    acte porur reviews

    acte omr reviews

    acte chennai reviews

    acte student reviews

    ReplyDelete
  15. You have shared a nice article here. After reading your article I got very much information and It resolved many of my doubts. Thanks for sharing this article here.
    IELTS Coaching in chennai

    German Classes in Chennai

    GRE Coaching Classes in Chennai

    TOEFL Coaching in Chennai

    spoken english classes in chennai | Communication training


    ReplyDelete

  16. Fantastic article I ought to say and thanks to the info. Instruction is absolutely a sticky topic. But remains one of the top issues of the time. I love your article and look forward to more.
    360DigiTMG Data Science Training Institute in Bangalore

    ReplyDelete
  17. Very good web site with the exceptional good quality goods along with I’m confident this is drastically valuable. oracle training in chennai

    ReplyDelete
  18. I was looking at some of your posts on this website and I conceive this web site is really instructive! Keep putting up. oracle training in chennai

    ReplyDelete
  19. Additionally, this is an excellent article which I truly like studying. It's not everyday I have the option to see something similar to this.
    Data Science Course In Bangalore With Placement

    ReplyDelete
  20. Excellent Blog! I would Thanks for sharing this wonderful content.its very useful to us.I gained many unknown information, the way you have clearly explained is really fantastic.keep posting like this information.
    Full Stack Training in Chennai

    Full Stack Course Chennai
    Full Stack Training in Bangalore

    Full Stack Course in Bangalore

    Full Stack Training in Hyderabad

    Full Stack Course in Hyderabad

    Full Stack Training

    Full Stack Course

    Full Stack Online Training

    Full Stack Online Course

    ReplyDelete
  21. Thanks for sharing amazing article it was very useful thank you.

    Data Science Training in Hyderabad

    ReplyDelete
  22. Nice article i was really impressed by seeing this article, it was very interesting and it is very useful for me.This is incredible,I feel really happy to have seen your webpage.I gained many unknown information, the way you have clearly explained is really fantastic.keep posting like this information.
    Full Stack Training in Chennai

    Full Stack Course Chennai
    Full Stack Training in Bangalore

    Full Stack Course in Bangalore

    Full Stack Training in Hyderabad

    Full Stack Course in Hyderabad

    Full Stack Training

    Full Stack Course

    Full Stack Online Training

    Full Stack Online Course



    ReplyDelete
  23. I simply want to mention I am just all new to blogging and site-building and truly loved you’re web page. Almost certainly I’m planning to bookmark your site . You really have outstanding stories. Many thanks for revealing your webpage.…
    Azure Training in Chennai

    Azure Training in Bangalore

    Azure Training in Hyderabad

    Azure Training in Pune

    Azure Training | microsoft azure certification | Azure Online Training Course

    Azure Online Training

    ReplyDelete
  24. I simply want to mention I am just all new to blogging and site-building and truly loved you’re web page. Almost certainly I’m planning to bookmark your site . You really have outstanding stories. Many thanks for revealing your webpage.…
    Azure Training in Chennai

    Azure Training in Bangalore

    Azure Training in Hyderabad

    Azure Training in Pune

    Azure Training | microsoft azure certification | Azure Online Training Course

    Azure Online Training

    ReplyDelete
  25. I'd love to thank you for the efforts you've made in composing this post. I hope the same best work out of you later on too. I wished to thank you with this particular sites! Thank you for sharing. Fantastic sites!
    Data Science Course in Bangalore

    ReplyDelete
  26. This is a great post. This post gives a truly quality information. I am certainly going to look into it. Really very helpful tips are supplied here. Thank you so much. Keep up the great works
    Data Science Training in Bangalore

    ReplyDelete
  27. Thanks for sharing such an great blog enjoined reading it thank you.
    Data Analytics Certification Training 360DigiTMG

    ReplyDelete
  28. I am impressed by the information on the blog keep up the good work.
    Data Analytics Certification Training

    ReplyDelete
  29. I would Thanks for sharing this wonderful content.its very useful to us.This is incredible,I feel really happy to have seen your webpage.I gained many unknown information, the way you have clearly explained is really fantastic.keep posting like this information.
    python training in bangalore

    python training in hyderabad

    python online training

    python training

    python flask training

    python flask online training

    python training in coimbatore
    python training in chennai

    python course in chennai

    python online training in chennai

    ReplyDelete
  30. Wonderful blog found to be very impressive to come across such an awesome blog. I should really appreciate the blogger for the efforts they have put in to develop such an amazing content for all the curious readers who are very keen of being updated across every corner. Ultimately, this is an awesome experience for the readers. Anyways, thanks a lot and keep sharing the content in future too.

    360DigiTMG Artificial Intelligence Course

    ReplyDelete
  31. Thanks for such awesome blog. Your article is very easy to understand, informative and provide complete overview about software testing. Please consider including rss feed in your website, so I get your recent post on my site.
    Salesforce Training in Chennai

    Salesforce Online Training in Chennai

    Salesforce Training in Bangalore

    Salesforce Training in Hyderabad

    Salesforce training in ameerpet

    Salesforce Training in Pune

    Salesforce Online Training

    Salesforce Training

    ReplyDelete
  32. It is really a great and helpful piece of info. I am glad that you shared this helpful information with us. Please keep us informed like this. Thank you for sharing.
    PMP Training

    ReplyDelete
  33. This is a really explainable very well and i got more information from your site.Very much useful for me to understand many concepts and helped me a lot.Best data science courses in hyerabad

    ReplyDelete
  34. Thanks For sharing a nice post about Course.It is very helpful and useful for us.data science courses

    ReplyDelete
  35. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our site please visit to know more information
    data science training in courses

    ReplyDelete
  36. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our site please visit to know more information
    data science training

    ReplyDelete
  37. We are well established IT and outsourcing firm working in the market since 2013. We are providing training to the people ,
    like- Web Design , Graphics Design , SEO, CPA Marketing & YouTube Marketing.Call us Now whatsapp: +(88) 01537587949
    :Freelancing training in Bangladesh
    Free bangla sex video:careful
    good post outsourcing institute in bangladesh

    ReplyDelete
  38. With its huge capabilities, Python is also a favorite among web developers to build various types of web applications. The web application framework, Django has been developed using Python and serves as the foundation for popular websites like 'The Guardian', 'The NY Times', 'Pinterest' and more. data science course in india

    ReplyDelete
  39. digital marketing company in chennai
    mobile app development company in chennai
    ios app development company in chennai
    shakthi tech

    shakthi technologies

    ReplyDelete
  40. android app development company in chennai
    payroll software development companies in chennai
    payroll application development company in chennai
    shakthitech

    website development company in chennai

    ReplyDelete
  41. Useful article which was very helpful. also interesting and contains good information.
    to know about python training course , use the below link.

    Python Training in chennai

    Python Course in chennai

    ReplyDelete
  42. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    tally training in chennai

    hadoop training in chennai

    sap training in chennai

    oracle training in chennai

    angular js training in chennai




    ReplyDelete
  43. Superb article and I would really like to thank for your article it’s really helpful.

    Java Training in Chennai

    Java Course in Chennai

    ReplyDelete
  44. Thank you for excellent article.You made an article that is interesting.
    data science course in noida

    ReplyDelete
  45. Fantastic blog extremely good well enjoyed with the incredible informative content which surely activates the learners to gain the enough knowledge. Which in turn makes the readers to explore themselves and involve deeply in to the subject. Wish you to dispatch the similar content successively in future as well.

    Data Science training

    ReplyDelete
  46. Truly incredible blog found to be very impressive due to which the learners who ever go through it will try to explore themselves with the content to develop the skills to an extreme level. Eventually, thanking the blogger to come up with such an phenomenal content. Hope you arrive with the similar content in future as well.

    Digital Marketing training in Raipur

    ReplyDelete
  47. Terrific post thoroughly enjoyed reading the blog and more over found to be the tremendous one. In fact, educating the participants with it's amazing content. Hope you share the similar content consecutively.

    Data Analytics Course in Raipur

    ReplyDelete
  48. Highly appreciable regarding the uniqueness of the content. This perhaps makes the readers feels excited to get stick to the subject. Certainly, the learners would thank the blogger to come up with the innovative content which keeps the readers to be up to date to stand by the competition. Once again nice blog keep it up and keep sharing the content as always.

    Data Science training in Bhilai

    ReplyDelete
  49. Wonderful blog found to be very impressive to come across such an awesome blog. I should really appreciate the blogger for the efforts they have put in to develop such an amazing content for all the curious readers who are very keen of being updated across every corner. Ultimately, this is an awesome experience for the readers. Anyways, thanks a lot and keep sharing the content in future too.

    Digital Marketing Course in Bhilai

    ReplyDelete
  50. Honestly speaking this blog is absolutely amazing in learning the subject that is building up the knowledge of every individual and enlarging to develop the skills which can be applied in to practical one. Finally, thanking the blogger to launch more further too.

    Data Analytics training in bhilai

    ReplyDelete

  51. This is a really very nice post you shared, i like the post, thanks for sharing..

    Data Science Training

    ReplyDelete
  52. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    best data science courses in bangalore

    ReplyDelete
  53. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    best data science courses in bangalore

    ReplyDelete
  54. I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
    best data science courses in hyderabad

    ReplyDelete
  55. It is amazing and wonderful to visit your site. Thanks for sharing information; this is useful to us....
    graphic design training in Delhi

    FOR MORE INFO:

    ReplyDelete
  56. Wonderful article. Very interesting to read this article. I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries. Keep it up.
    Data Science Training in Hyderabad
    Data Science Course in Hyderabad

    ReplyDelete
  57. I've read this post and if I could I desire to suggest you some interesting things or suggestions. Perhaps you could write next articles referring to this article. I want to read more things about it!
    Artificial Intelligence Training in Hyderabad
    Artificial Intelligence Course in Hyderabad

    ReplyDelete
  58. Very nice article. I enjoyed reading your post. very nice share. I want to twit this to my followers. Thanks
    Data Science Training in Hyderabad
    Data Science Course in Hyderabad

    ReplyDelete
  59. It is better to engaged ourselves in activities we like. I liked the post. Thanks for sharing.
    DevOps Training in Hyderabad
    DevOps Course in Hyderabad

    ReplyDelete
  60. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
    Data Science Training in Bangalore

    ReplyDelete
  61. I am a new user of this site, so here I saw several articles and posts published on this site, I am more interested in some of them, hope you will provide more information on these topics in your next articles.
    data analytics training in bangalore

    ReplyDelete
  62. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
    data analytics course in bangalore

    ReplyDelete
  63. I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
    data analytics courses in bangalore

    ReplyDelete
  64. I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
    data science in bangalore

    ReplyDelete
  65. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    Data Science Course in Bangalore

    ReplyDelete
  66. Highly appreciable regarding the uniqueness of the content. This perhaps makes the readers feels excited to get stick to the subject. Certainly, the learners would thank the blogger to come up with the innovative content which keeps the readers to be up to date to stand by the competition. Once again nice blog keep it up and keep sharing the content as always.

    data analytics courses in bangalore with placement

    ReplyDelete
  67. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    Data Science Course in Bangalore

    ReplyDelete
  68. Amazing! Its really amazing piece of writing, I have got much clear idea concerning from this piece of writing.
    data scientist course in hyderabad

    ReplyDelete
  69. This piece of writing will assist the internet visitors for setting up new webpage or even a weblog from start to end
    data scientist course in hyderabad

    ReplyDelete
  70. Thanks for sharing useful information. I learned something new from your bog. Its very interesting and informative. keep updating.Amazon Web Services Training in Chennai

    ReplyDelete
  71. Thank you for taking the time to publish this information very useful!
    data scientist training and placement in hyderabad

    ReplyDelete
  72. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
    data analytics course in bangalore

    ReplyDelete
  73. I am a new user of this site, so here I saw several articles and posts published on this site, I am more interested in some of them, hope you will provide more information on these topics in your next articles.
    data analytics training in bangalore

    ReplyDelete
  74. I was very pleased to find this site.I wanted to thank you for this great read!! I definitely enjoy every little bit of it and I have you bookmarked to check out new stuff you post.
    business analytics course

    ReplyDelete
  75. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
    Pmp training in hyderabad

    ReplyDelete
  76. I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
    data science training in chennai

    ReplyDelete
  77. Really impressed! Everything is a very open and very clear clarification of the issues. It contains true facts. Your website is very valuable. Thanks for sharing.
    Business Analytics Course

    ReplyDelete
  78. I have voiced some of the posts on your website now, and I really like your blogging style. I added it to my list of favorite blogging sites and will be back soon ...

    Data Analytics Course in Bangalore

    ReplyDelete
  79. It is late to find this act. At least one should be familiar with the fact that such events exist. I agree with your blog and will come back to inspect it further in the future, so keep your performance going.
    https://terrabellatech.blogspot.com/2015/02/mapreduce-for-c-run-native-code-in.html?showComment=1624163721414#c6260691746116257168

    ReplyDelete
  80. You actually make it seem like it's really easy with your acting, but I think it's something I think I would never understand. I find that too complicated and extremely broad. I look forward to your next message. I'll try to figure it out!

    Best Data Science Courses in Bangalore

    ReplyDelete
  81. The writer is enthusiastic about purchasing wooden furniture on the web and his exploration about the best wooden furniture has brought about the arrangement of this article.
    data scientist course in hyderabad

    ReplyDelete
  82. Train yourself in specific software modules to brush up your skills & shine in your career growth with the best software training institute in Chennai, Infycle Technologies. Infycle offers the Best Data Science Training in Chennai to serve the candidate's job profile requirements, including the top job placements in the MNC's. Rather than just teaching the theories, our fundamental aim is to make you a master by giving you live hands-on training. Therefore, individuals will be asked to work on the live tasks & real-time use cases that bring out the definite coder in you! To grab all these, call 7502633633 for a free demo.

    ReplyDelete
  83. Train yourself in specific software modules to brush up your skills & shine in your career growth with the best software training institute in Chennai, Infycle Technologies. Infycle offers the Best Data Science Training in Chennai to serve the candidate's job profile requirements, including the top job placements in the MNC's. Rather than just teaching the theories, our fundamental aim is to make you a master by giving you live hands-on training. Therefore, individuals will be asked to work on the live tasks & real-time use cases that bring out the definite coder in you! To grab all these, call 7502633633 for a free demo.

    ReplyDelete
  84. Thanks for posting the best information and the blog is very important.digital marketing institute in hyderabad

    ReplyDelete
  85. Thank you for sharing such a informative post with us, it will beneficial for everyone, It is one of the best sites that I have visited. I am looking forward to read more blogs post from here
    Python Training in Hyderabad
    Python Course in Hyderabad

    ReplyDelete
  86. Thanks for sharing your wealthy information. This is one of the excellent posts which I have seen. I go through your all of your blog, but this blog is the best one. It is really what I wanted to see hope in future you will continue for sharing such an excellent post
    thời gian bay từ los angeles về việt nam

    đặt vé máy bay từ đức về việt nam

    ve may bay tu Anh ve Viet Nam

    có chuyến bay từ úc về việt nam không

    Giá vé máy bay Vietnam Airline tu Dai Loan ve Viet Nam

    giá vé máy bay từ canada về việt nam

    ReplyDelete
  87. Thanks for posting the best information and the blog is very important.artificial intelligence course in hyderabad

    ReplyDelete
  88. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one.
    data scientist training and placement

    ReplyDelete
  89. Thanks for posting the best information and the blog is very important.data science institutes in hyderabad

    ReplyDelete