HTML-Parser
-----------
This is a collection of modules that parse HTML text documents. These
modules used to be part of the libwww-perl distribution, but are now
unbundled in order to facilitate a separate development track. Bug
reports and discussions about these modules can still be sent to the
mailing list. Remember to also take a look at
the HTML-Tree module collection that create and extract information from
HTML syntax trees.
The modules present in this collection are:
HTML::Parser - The parser base class. It receives arbitrary sized
chunks of the HTML text and will tokenize it by calling
appropriate methods on itself.
HTML::Entities - Provides functions to encode and decode text with
embedded HTML <entities>.
HTML::Filter - An HTML::Parser subclass that filters HTML text. You
will need to make a subclass if you want it to do more than
cat(1).
HTML::HeadParser - A lightweight HTML::Parser subclass that extract
information from the section of an HTML document.
HTML::LinkExtor - An HTML::Parser subclass that extract links from
an HTML document.
PREREQUISITES
In order to install and use this package you will need Perl version
5.004 or better. If you intend to use the HTML::HeadParser you need to
have the libwww-perl distribution installed.
INSTALLATION
Just follow the usual procedure:
perl Makefile.PL
make
make test
make install
COPYRIGHT
© 1995-1998 Gisle Aas. All rights reserved.
This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself.
-------------------------------------------------------------------------
NAME
HTML::Parser - SGML parser class
SYNOPSIS
require HTML::Parser;
$p = HTML::Parser->new; # should really a be subclass
$p->parse($chunk1);
$p->parse($chunk2);
#...
$p->eof; # signal end of document
# Parse directly from file
$p->parse_file("foo.html");
# or
open(F, "foo.html") || die;
$p->parse_file(\*F);
DESCRIPTION
The `HTML::Parser' will tokenize an HTML document when the
parse() method is called by invoking various callback methods.
The document to be parsed can be supplied in arbitrary chunks.
The external interface the an *HTML::Parser* is:
$p = HTML::Parser->new
The object constructor takes no arguments.
$p->parse( $string );
Parse the $string as an HTML document. Can be called
multiple times. The return value is a reference to the
parser object.
$p->eof
Signals end of document. Call eof() to flush any remaining
buffered text. The return value is a reference to the parser
object.
$p->parse_file( $file );
This method can be called to parse text from a file. The
argument can be a filename or an already opened file handle.
The return value from parse_file() is a reference to the
parser object.
$p->strict_comment( [$bool] )
By default we parse comments similar to how the popular
browsers (like Netscape and MSIE) do it. This means that
comments will always be terminated by the first occurence of
"-->". This is not correct according to the "official" HTML
standards. The official behaviour can be enabled by calling
the strict_comment() method with a TRUE argument.
The return value from strict_comment() is the old attribute
value.
In order to make the parser do anything interesting, you must
make a subclass where you override one or more of the following
methods as appropriate:
$self->declaration($decl)
This method is called when a *markup declaration* has been
recognized. For typical HTML documents, the only declaration
you are likely to find is . The initial "" is not part of the string passed as argument.
Comments are removed and entities will not be expanded.
$self->start($tag, $attr, $attrseq, $origtext)
This method is called when a complete start tag has been
recognized. The first argument is the tag name (in lower
case) and the second argument is a reference to a hash that
contain all attributes found within the start tag. The
attribute keys are converted to lower case. Entities found
in the attribute values are already expanded. The third
argument is a reference to an array with the lower case
attribute keys in the original order. The fourth argument is
the original HTML text.
$self->end($tag, $origtext)
This method is called when an end tag has been recognized.
The first argument is the lower case tag name, the second
the original HTML text of the tag.
$self->text($text)
This method is called when plain text in the document is
recognized. The text is passed on unmodified and might
contain multiple lines. Note that for efficiency reasons
entities in the text are not expanded. You should call
HTML::Entities::decode($text) before you process the text
any further.
$self->comment($comment)
This method is called as comments are recognized. The
leading and trailing "--" sequences have been stripped off
the comment text.
The default implementation of these methods do nothing, i.e.,
the tokens are just ignored.
There is really nothing in the basic parser that is HTML
specific, so it is likely that the parser can parse other kinds
of SGML documents. SGML has many obscure features (not
implemented by this module) that prevent us from renaming this
module as `SGML::Parser'.
EFFICIENCY
The parser is fairly inefficient if the chunks passed to $p-
>parse() are too big. The reason is probably that perl ends up
with a lot of character copying when tokens are removed from the
beginning of the strings. A chunck size of about 256-512 bytes
was optimal in a test I made with some real world HTML
documents. (The parser was about 3 times slower with a chunck
size of 20K).
SEE ALSO
the HTML::Entities manpage, the HTML::Filter manpage, the
HTML::HeadParser manpage, the HTML::LinkExtor manpage
the HTML::TreeBuilder manpage (part of the *HTML-Tree*
distribution)