NAME File::BOM - Utilities for reading Byte Order Marks SYNOPSIS use File::BOM qw( :all ) high-level functions # read a file with encoding from the BOM: *FH = open_bom($file) *FH = open_bom($file, ':utf8') # the same but with a default encoding # get encoding too (*FH, $encoding) = open_bom($file, ':utf8'); # open a potentially unseekable file: (*FH, $encoding, $spillage) = open_bom($file, ':utf8', 1); # slurp an encoded file my $text = eval { local $/ = undef; my $whole_file = ; decode_from_bom($whole_file, 'UTF-8', 1); } PerlIO::via interface # Read the Right Thing from a unicode file with BOM: open(HANDLE, '<:via(File::BOM)', $filename) # Writing little-endian UTF-16 file with BOM: open(HANDLE, '>:encoding(UTF-16LE):via(File::BOM)', $filename) lower-level functions # read BOM encoding from filehandle: open FH, '<:bytes', $some_file; $encoding = get_encoding_from_filehandle(*FH) # get encoding and BOM length from BOM at start of string: ($encoding, $offset) = get_encoding_from_bom($string); variables # print a BOM for a known encoding print FH $enc2bom{$encoding}; # get an encoding from a known BOM $enc = $bom2enc{$bom} EXPORTS Nothing by default. * open_bom() * decode_from_bom() * get_encoding_from_filehandle() * get_encoding_from_stream() * get_encoding_from_bom() * %bom2enc * %enc2bom * :all All of the above * :subs subroutines only * :vars just %bom2enc and %enc2bom VARIABLES %bom2enc Maps Byte Order marks to their encodings. See for details The keys of this hash are strings which represent the BOMs, the values are their encodings, in a format which is understood by Encode The encodings represented in this hash are: UTF-8, UTF-16BE, UTF-16LE, UTF-32BE and UTF-32LE %enc2bom A reverse-lookup hash for bom2enc, with a few aliases used in Encode, namely utf8, iso-10646-1 and UCS-2. Note that UTF-16, UTF-32 and UCS-4 are not included in this hash. Mainly because Encode::encode automatically puts BOMs on output. See Encode::Unicode FUNCTIONS open_bom *FH = open_bom($name, $default_mode, $try_unseekable) (*FH, $encoding, $spill) = open_bom($name, $default_mode, $try_unseekable) opens $name for reading, setting the mode to the appropriate encoding for the BOM stored in the file. If the file doesn't contain a BOM, $default_mode is used instead. Hence: open_bom('my_file.txt', ':utf8') Opens my_file.txt for reading in an appropriate encoding found from the BOM in that file, or as a UTF-8 file if none is found. If no default mode is specified and no BOM is found, the filehandle is opened using :bytes croaks on errors, returns the filehandle in scalar context or the filehandle and the encoding in list context. The filehandle will be cued up to read after the BOM. Unseekable files (e.g. sockets) will cause croaking, unless $try_unseekable is set in which case any spillage is returned after the encoding (in scalar context the spillage is lost!) e.g. # croak if my_socket is unseekable *FH = open_bom('my_socket') # keep spillage if my_socket is unseekable (*FH, $encoding, $spillage) = open_bom('my_socket', undef, 1); # discard spillage is my_socket is unseekable - not recommended *FH = open_bom('my_socket', undef, 1); decode_from_bom() $unicode_string = decode_from_bom($string, $default, $check) ($unicode_string, $encoding) = decode_from_bom($string, $default, $check) Reads a BOM from the beginning of $string, decodes $string (minus the BOM) and returns it to you as a perl unicode string. if $string doesn't have a BOM, $default is used instead. $check, if supplied, is passed to Encode::decode If there's no BOM and no default, the original string is returned and encoding is ''. See Encode get_encoding_from_filehandle $encoding = get_encoding_from_filehandle(HANDLE) ($encoding, $spillage) = get_encosing_from_filehandle(HANDLE) Returns the encoding found in the given filehandle. The handle should be opened in a non-unicode way (e.g. mode '<:bytes') so that the BOM can be read in its natural state. After calling, the handle will be set to read at a point after the BOM (or at the beginning of the file if no BOM was found) If called in scalar context, unseekable handles cause a croak(). If called in list context, unseekable handles will be read byte-by-byte and any spillage will be returned. See get_encoding_from_stream get_encoding_from_stream ($encoding, $spillage) = get_encoding_from_stream(*FH); Read a BOM from an unrewindable source. This means reading the stream one byte at a time until either a BOM is found or every possible BOM is ruled out. Any non-BOM characters read from the handle will be returned in $spillage. This function is less efficient than get_encoding_from_filehandle, but should work just as well on a seekable handle as on an unseekable one. get_encoding_from_bom ($encoding, $offset) = get_encoding_from_bom($string) Returns the encoding and length in bytes of the BOM in $string. If there is no BOM, an empty string is returned and $offset is zero. To get the data from the string, the following should work: use Encode; my($encoding, $offset) = get_encoding_from_bom($string); if ($encoding) { $string = decode($encoding, substr($string, $offset)) } PerlIO::via interface File::BOM can be used as a PerlIO::via interface. # Read from a handle in a open(HANDLE, '<:via(File::BOM)', 'my_file.txt'); open(HANDLE, '>:encoding(UTF-16LE):via(File::BOM)', 'out_file.txt) print "foo\n"; # BOM is written to file This method is less prone to errors on non-seekable files, but doesn't give you any information about the encoding being used, or indeed whether or not a BOM was present. Reading The via(File::BOM) layer must be added before the handle is read from, otherwise any BOM will be missed. If there is no BOM, no decoding will be done. Writing Add the via(File::BOM) layer on top of a unicode encoding layer to print a BOM at the start of the output file. This needs to be done before any data is written. The BOM is written as part of the first print command on the handle, so if you don't print anything to the handle, you won't get a BOM. SEE ALSO * Encode * Encode::Unicode * ERROR HANDLING The default behaviour on encountering an IO error of any sort is to croak $! but this is subject to change in future versions. BUGS None known. AUTHOR Matt Lawrence