Comments on: XML input and Hadoop – custom InputFormat

By: Laure Drenth

Laure Drenth — Tue, 08 Mar 2011 00:07:42 +0000

Hello there, Are you going to be publishing a follow up piece? My husband and me have squandered some time browsing over your web page and surprisingly enough you touched on some thing we were discussing only the other week with our accountant. We often notice ourselves quarrelling over the smallest of issues, isn’t it childish? At any rate we wish you greatest wishes from the Usa.

By: Paul Ingles

Paul Ingles — Wed, 20 Jan 2010 02:36:52 +0000

Hi,Just found your post. I’ve been trying to do the same thing. In the end, I went with the XmlInputFormat from Mahout’s Bayesian Classifier. Seems to do everything I need it to (and works without going screwy like the streaming one).I posted about it here: http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html

By: Roman Kirillov

Roman Kirillov — Fri, 06 Nov 2009 11:58:21 +0000

Actually I fixed it – there’s no newline replacing now (mainly because it serves no purpose and we still can handle it with no problems at all). Also, for TPWMNBN we can instead read content straight into objects (by changing the RecordReader to return some Object rather than Text). Once you figure all this stuff out it’s quite simple!

By: Holger Dürer

Holger Dürer — Fri, 06 Nov 2009 11:53:23 +0000

For TPWMNBN you also need to sort out the newline thing. You cannot just replace newlines with spaces — that can be changing the meaning of the contents.

By: Roman Kirillov

Roman Kirillov — Fri, 06 Nov 2009 11:39:56 +0000

I think I’ll give it a try – for now I’m not that really concerned about it, but in future (i.e. when we’ll deal with you-know-which-product) we’ll probably need to sort it out in a proper way.

By: Holger Dürer

Holger Dürer — Fri, 06 Nov 2009 11:33:29 +0000

I am still convinced you need to change the “while (pos < end)” to keep looking past the split’s end if you are in the middle of parsing a document — otherwise your last document in the split will be truncated. I.e. something like “while (pos < end || xmlRecordStarted)

By: Roman Kirillov

Roman Kirillov — Wed, 04 Nov 2009 21:08:20 +0000

pos and end are members from LineRecordReader. Don’t have the source handy, but I they declare end = Long.MAX_VALUE; (or expect user to specify the end-offset but this isn’t the case).

By: Roman Kirillov

Roman Kirillov — Wed, 04 Nov 2009 21:04:15 +0000

Pretty much so. For most of standard Hadoop jobs you have a flat text file where every record is a line. In this example we deal with an XML file where every record is represented by one element and all it’s content.

By: Holger Dürer

Holger Dürer — Wed, 04 Nov 2009 21:03:15 +0000

Also: what are pos and end? The approximate byte range from which to read the records, right? A record will usually not fall on these boundaries. That's why you skip lines at the beginning, right? But doesn't that mean you should finish reading a record past 'end'? Otherwise your last record might terminate prematurely….

By: Roman Kirillov

Roman Kirillov — Wed, 04 Nov 2009 21:02:42 +0000