@(#) $Id: CHANGES,v 1.68 2005/05/20 06:39:04 vern Exp $

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

0.9a9 Thu May 19 23:31:33 PDT 2005

- First cut at analyzer for NFS (Vern Paxson).  It generates the following
  events:
	event nfs_request_null(n: connection)
	event nfs_request_getattr(n: connection, fh: string, attrs: nfs3_attrs)
	event nfs_request_lookup(n: connection, req: nfs3_lookup_args,
				rep: nfs3_lookup_reply)
	event nfs_request_fsstat(n: connection, root_fh: string,
				stat: nfs3_fsstat)
	event nfs_attempt_null(n: connection, status: count)
	event nfs_attempt_getattr(n: connection, status: count, fh: string)
	event nfs_attempt_lookup(n: connection, status: count,
				req: nfs3_lookup_args,
				dir_attrs: nfs3_opt_attrs)
	event nfs_attempt_fsstat(n: connection, status: count,
				root_fh: string, obj_attrs: nfs3_opt_attrs)

- The new script OS-fingerprint.bro integrates Bro's new passive OS
  fingerprinting mechanism with the software.bro framework (Vern Paxson).

- You can now operate on patterns using && and || (Vern Paxson).
  If p1 and p2 are patterns, then p1 && p2 yields a pattern that matches
  their concatenation and p1 || p2 yields a pattern that matches either.
  Note that the syntax for this may change in the future to a single '&'
  or '|', which would be more consistent with the use of '|' in
  constructing pattern constants.

- An experimental "connection compressor" tracks not-yet-established
  connections using much less memory than Bro normally does (Robin Sommer).
  This is potentially a major win during flooding attacks and high-speed
  scans.  You activate it by setting use_connection_compressor to T.  You
  can then control the granularity of its processing using the variables
  cc_handle_resets, cc_handle_only_syns, and cc_instantiate_on_data.  See
  bro.init for brief discussion of these.

- The experimental new script firewall.bro supports firewall-rule-like
  processing of connections in terms of allow/deny (Robin Sommer).  It is
  not particularly efficient.

- sensor-sshd.bro provides an experimental interface for receiving
  events from instrumented SSH servers that communicate with Bro via
  the Broccoli client library (Christian Kreibich and Robin Sommer).
  Supporting this also entailed extensions to login.bro so it can
  process the events even though they don't correspond to a connection
  known to Bro's event engine.

- The new built-in function match_signatures() can be used in a policy
  script to send text directly into the signature engine (Robin Sommer).

- Correction: the 0.9a8 CHANGES states that the mail_script variable used
  for NOTICE_EMAIL defaults to mail_script.sh.  The correct value is instead
  "mail_notice.sh".

- The scripts rsh.bro and passwords.bro, and the passive-fingerprinting
  signatures policy/sigs/p0fsyn.osf were inadvertantly left out of the
  0.9a8 distribution.

- Added s2b (snort to bro) files into the distribution. (Jason Lee)

- Non-blocking packet capture under Linux has been fixed (Robin Sommer).

- Fixed printing of DNS replies, which used to work but was broken
  a number of months ago (Vern Paxson).

- The new script brolite-sigs separates out how signatures are configured
  in Bro Lite so the functionality can be enabled/disabled with a simple
  load statement (Roger Winslow).  That is, to use signatures with Bro
  lite, simply add "@load brolite-sigs".

- The new script variable enable_syslog (default T) controls whether
  alarm's are syslog'd (Robin Sommer).  As before, syslogs can only happen
  when Bro is reading from live network traffic (this should be changed
  at some point, to accommodate real-time Bro's that don't read the network
  but collect events from other sensors).  Previously, in that case syslog's
  always happened; now, you can turn them off using this variable.

- The new script variable expensive_profiling_multiple controls how
  often, when doing profiling, to perform more expensive forms of
  profiling, in particular, memory consumption profiling (Robin Sommer).
  If profiling_interval is set to 15 sec and expensive_profiling_multiple
  is set to 20, then expensive profiling will be done every 5 minutes
  (these are the defaults now in profiling.bro).  Also, the profiling_update
  event now includes a second argument, expensive: bool, which indicates
  whether the update corresponds to one of these expensive profiling
  intervals.

- First cut at parsing DNS AAAA replies (Scott Campbell).  This is quite
  incomplete - currently, the replies are turned into fake A record replies,
  due to the difficulty of dealing with IPv6 addresses if Bro wasn't built
  to analyze IPv6 traffic.

- software.bro has been tweaked to have a new control variable,
  "only_report_local" (default F).  If true, then only software versions
  for local addresses (as determined by is_local_addr()) will be
  reported.

- synflood.bro now has a script variable max_sources (default 100) that
  specifies the maximum number of sources to track for a given victim
  (Robin Sommer).

- Remote peers now negotiate their versions of the serialization format
  (Robin Sommer). If they don't agree then the connection is terminated.

- Generic UDP request/response processing has been moved into the new
  policy script udp-common.bro, which, unlike udp.bro, does *not* set the
  packet filter to capture all UDP traffic (Robin Sommer).  A number
  of UDP-based policy scripts have been modified to use udp-common.bro
  rather than udp.bro.

- When printing serialized/independent state, access times are now
  again included (Robin Sommer).

- Bro's implementation of timers has been switched (reverted) to using
  priority queues (Vern Paxson).

- The http-request.bro script variables skip_remote_sensitive_URIs and
  const sensitive_post_URIs are now exported so they can be accessed
  externally (Robin Sommer).

- Some new rootkit filenames have been added to ftp.bro and
  http-request.bro (Brian Tierney).  The plan is to eventually
  merge these lists so there's only one main list.

- trw.bro is now scoped as a module "TRW" (Brian Tierney).

- Better support of the '--disable-localpcap' flag to configure, and
  consolidated all the pcap checks in configure.in (Jason Lee).

- A bug in processing bare carriage-returns in Telnet input/output
  has been fixed (Vern Paxson).

- The Bro Lite bro.rc script has been tweaked to use the 'ax' flags
  instead of '-ax' (Jason Lee).

- A bug with reporting ICMP "ports" (i.e., type + code) has been fixed
  (Vern Paxson).

- Bug fix for excessively large RPC messages (Ruoming Pang).

- A bug with /0 subnet prefixes has been fixed (Robin Sommer).

- The function record_connection() now takes the file to write to
  as its first argument (Robin Sommer).

- remote.bro now tracks whether a given Destination is connected
  (Robin Sommer).

- mail_notice.sh is now installed as part of installing a distribution
  (Jason Lee).

- Fixed bug where the sort order for the test suite changed depending
  on locale. (Jason Lee)

- Bug fix for email_notice() when notice_action_filters not defined for
  given notice (Vern Paxson).

- The test suite test for rare-events fixed to not give false positives
  (Jason Lee).

- Date added for 0.9a8 release.


0.9a8 Wed Feb 16 17:09:34 PST 2005

- aux/rst/ contains the source for the "rst" tool used by Bro (via the
  policy script function terminate_connection() in conn.bro) to tear
  down established connections by forging RST packets.

- Bro's main event loop has been reworked (Robin Sommer).  This should
  (1) not cause any visible differences in most cases, (2) improve
  performance in some cases, (3) fixed problems running Bro without
  a network input (but still receiving asynchronous input from remote
  event sources).  There are some more changes coming to this soon.

- Passive OS fingerprinting has been added, based on Michal Zalewski's
  "p0f" tool (Holger Dreger).  Currently, it's limited to fingerprinting
  clients based on the initial SYNs they send.  To use it, define
  an event handler:

	OS_version_found(c: connection, host: addr, OS: OS_version)

  OS_version is a record containing a string $genre (e.g., "Solaris"),
  a string $detail (e.g., "2.0.27"), a count $dist (hop-count distance
  from monitor to host), and $match_type, which specifies via an
  enumerated type whether the match was direct from a signature,
  generic to the genre, or "fuzzy".

  The match is done against a passive fingerprinting signature file,
  which is specified by the variable passive_fingerprint_file.
  It defaults to "sigs/p0fsyn", which is found using $BROPATH
  and has an "osf" suffix added.

  You can restrict the matching to only be performed for hosts from
  particular subnets by adding those subnets to the variable
  generate_OS_version_event.  If it's empty (default), then all subnets
  are analyzed.

  Note, the passive fingerprinting should be integrated with the
  version-tracking in software.bro, but this hasn't been done yet.

- Support for IPv6 has been repaired and brought up to date.  Note, however,
  that inter-Bro communication currently only works over IPv4.

- Signature-matching is now off by default in brolite.bro.  If you want
  to use it, define use_signatures = T prior to @load'ing it.

- Notices are now tied to their corresponding connections (Scott Campbell).

- New backdoor detectors for IRC, SMTP, Gaobot (Scott Campbell).

- Signature matches now have a connection associated with them (Scott Campbell).

- Bro scripts that set initial timers (via "schedule" statements in a
  bro_init handler) but don't have any source of network input (trace
  files or live interfaces) now execute in real-time, with network_time
  set to the current time, rather than having their timers expire immediately.

- Default timeouts have been added to tables in trw.bro and http.bro, which
  have been found operationally to potentially grow very large (Scott Campbell).

- The new policy script large-conns.bro can be included in order to
  track the size of TCP connections (each direction is referred to
  as a "flow") using a secondary packet filter (Chema Gonzalez).
  This method is completely separate from Bro's usual size accounting,
  and offers the advantages that it tracks sizes even for connections
  that don't terminate (or for which Bro misses their establishment)
  and for connections with sizes > 4 GB.

  The interface is via the function:

	function estimate_flow_size_and_remove(cid: conn_id, orig: bool)

  If $orig=T, then an estimate of the size of the forward (originator)
  direction is returned.  If $orig=F, then the reverse (responder) direction
  is returned.  In both cases, what's returned is a "flow_size_est" record,
  which includes a flag, $have_est, indicating whether there was any
  estimate formed. If $have_est is T, then the record also includes
  an estimate in bounded by $lower (lower bound) and $upper (upper bound).
  The estimate also includes $num_inconsistent, which, if > 0, means that
  the estimates came from sequence numbers that were inconsistent, and
  thus something is wrong - perhaps packet drops by the secondary filter).
  Finally, calling the function causes the flow's record to be deleted.

- An RSH analyzer has been contributed by Manu (ManuX@rstack.org).
  It generates rsh_request and rsh_reply events, and the following notices:

	  DifferentRSH_Usernames
		Client and server username differ.

	  FailedRSH_Authentication
		Attempt to authenticate via RSH failed.

	  InteractiveRSH
		The RSH session appears to be interactive (multiple
		lines of user commands).

	  SensitiveRSH_Input
	  SensitiveRSH_Output
		RSH client input or server output match input_trouble/
		full_input_trouble or output_trouble/full_output_trouble.

- The new notice action NOTICE_EMAIL indicates that in addition to
  logging an alarm, it should also be sent out as email (Scott Campbell).
  By default, email is only sent if Bro is running on live traffic;
  you can override this via redef'ing the script variable mail_notification.
  Mail is sent using the script specified by the mail_script variable
  (default: "mail_script.sh", which is now included in the distribution,
  but at present is not installed), which must be in $PATH.  The mail
  is sent to the username specified in mail_dest (default: the local
  "bro" user, though you can change this to name@domain).

  Note that specifying email as a separate notice action may change
  in the future, to instead be an attribute that's associated with
  other notice actions.  For example, it may make sense to want
  to specify both NOTICE_ALARM_PER_CONN and NOTICE_EMAIL; currently,
  however, you can't do this.

- A similar notice action NOTICE_PAGE does the same thing as NOTICE_EMAIL
  except it send the mail to mail_page_dest (Scott Campbell).

- You can now use the attribute &rotate_size for file objects to
  specify the maximum file size in bytes (Robin Sommer). If the limit
  is reached, the file is rotated similiarly as is already done with
  &rotate (which, for consistency, has been renamed to &rotate_interval).

  For both &rotate_size and &rotate_interval, when they trigger they
  now generate events (rotate_size and rotate_interval, respectively;
  each takes the file as the sole argument) rather than invoking
  &postprocessor, which has been removed.

  There's also a new variable log_rotate_size to set a global size maximum.

  Related to log rotation are the following new built-in functions:

	rotate_file(f: file) closes the file, moves it to a temporary
	name, and opens a new one. It returns the new "rotate_info"
	record, which gives the temporary name and the open/close times.

	rotate_file_by_name(s: string): similar, but call by the name
	of the file rather than a Bro script value.  This is needed
	because some files are not represented by file objects but need
	to be rotated nevertheless (most importantly, the tcpdump save
	file and the dump files for dump_current_packet()). This function
	rotates the file with the given name.

  Finally, you can load the new policy script rotate-logs.bro to
  get default behavior of rotating all log files every hour.

- The new "@unload <script>" directive specifies that future @load's of
  <script> should be skipped.  This is useful for overriding analyzers
  loaded by scripts that pull in a bunch of analysis.  For example,

	@unload ntp
	@load mt

  would load all of the "mt" analyzers *except* ntp.bro.

- The new built-in function get_file_name(f: file): string returns
  the filename associated with a file (John McNicholas).

- The new built-in function get_contents_file(id: conn_id, direction: count)
  returns the contents file (set using set_contents_file()) for the given
  direction (John McNicholas).

- The new built-ins time_to_double() and double_to_time() convert between
  double values and time values (Robin Sommer).  The new built-in floor()
  returns the floor of a double value; this returned value is also a double.
  Thus, floor(-3.4) returns -4.0.

- Support for sending packets between Bro's (Robin Sommer).

- Bro now has a geneal mechanism internal for traversing policy scripts
  (Umesh Shankar).  Various script analyses can be specified using the
  new -z flag.

  Currently, the one supported form of analysis is "-z notice", which
  prints all of the different types of notices that the script you've
  loaded can generate.  For example, "bro -z notice ftp" will generate:

	  Found NOTICE: BackscatterSeen
	  Found NOTICE: FTP_PrivPort
	  Found NOTICE: FTP_BadPort
	  Found NOTICE: PortScan
	  Found NOTICE: FTP_ExcessiveFilename
	  Found NOTICE: ScanSummary
	  Found NOTICE: AddressDropped
	  Found NOTICE: DroppedPackets
	  Found NOTICE: SensitiveConnection
	  Found NOTICE: FTP_UnexpectedConn
	  Found NOTICE: SSH_Overflow
	  Found NOTICE: FTP_Sensitive
	  Found NOTICE: TerminatingConnection
	  Found NOTICE: PasswordGuessing
	  Found NOTICE: AddressDropIgnored
	  Found NOTICE: AddressScan

- The signature rule language now supports an "active" keyword,
  which can be set to "true" or "false", with the latter turning
  off the rule (Roger Winslow).  If set to false the signature will
  not be loaded into the rule matcher, otherwise it is.

- The signature rule language now supports meta data of the form
  ".MMM<whitespace>XXX", where MMM is arbitrary text which makes up the name
  of the meta data option and where XXX is arbitrary text up to the end
  of the current line (Roger Winslow).  The intent is that some forms of
  meta-data will be regularized/standardized in the future - information
  such as date modifed, category/class, weighting, etc.  For now, it
  provides a way to annotate rules with nominally more structure than just
  using comments (though it is currently treated the same, i.e., everything
  is ignored).

- The following meta data option names are now reserved: .version, .revision,
  .version-date, .revision-date, .date-created, .location

- The new enumerated type "transport_proto" is used to specify different
  types of transport protocols: "tcp", "udp", "icmp", and "unknown_transport".
  Associated with it are new built-in functions: is_udp_prot(), is_icmp_port(),
  get_conn_transport_proto, and get_port_transport_proto.  The latter two
  map a given connection and a given "port" value to their corresponding
  transport_proto value.

- A bunch of tuning (regular expressions for sensitive login sessions,
  scan detection thresholds, forbidden/hot usernames and filenames,
  sensitive URIs, "weird" actions) have been incorporated from
  operational configurations.

- Serious bugs in managing large numbers of files fixed.

- A serious bug with negative DNS TTL settings (and, more generally,
  with negative timer values) fixed.

- The traditional connection logging format is no longer supported.

- The SMTP analyzer's state machine processing has been modified to
  correctly deal with clients that (incorrectly) pipeline their commands
  (Ruoming Pang).

- A bug fixed in detecting SMTP relays for connections w/o message bodies
  (Ruoming Pang).

- A bunch of bugs in recording connection summaries for UDP flows
  have been fixed.

- A new script module, passwords.bro, generates PasswordExposed notices
  for activity (currently just rlogin/telnet logins) that expose passwords.

- A new script module, file-flush.bro, can be loaded to cause all log
  files to be flushed every file_flush_interval (default: 10) seconds.
  This is handy if you like to watch the files in real time.

- Zone transfers now generate a ZoneTransfer notice, unless the host
  making the request is in DNS::zone_transfers_okay.

- Bro's DNS cache (generated using -P and accessed using -F) is
  now kept in the .state/ subdirectory rather than in the user's
  home directory (Roger Winslow).

- Some changes to remote propagation of events/values and detection of
  state inconsistencies (Robin Sommer).

- A fix for avoiding delays on low-volume links for some systems for
  which it can take a long time to fill up the pcap buffer, and pcap doesn't
  return partial buffers (Robin Sommer).

- A bug in table expiration timers has been fixed (Robin Sommer).

- A bug in comparing subnets has been fixed.

- A bug in using a non-constant value for a &write_expire attribute
  has been fixed.

- A bug in using CONTENTS_BOTH for writing reassembled streams to
  files has been fixed (John McNicholas).

- A subtle but potentially damaging bug in fragment reassembly has
  been fixed.

- A bug with using local variables of vector types has been fixed.

- A bug with comparing strings has been fixed.

- Bro no longer generates the RST_with_data "weird", as with modern
  stacks it's no longer any sort of strange occurrence.

- Related to this, the signature rule matcher no longer matches
  against the payload of RST packets.  (Note, this is an imcompatibility
  with Snort.)

- Portmapper mappings are now written in the connection log in
  alphabetical order.

- The event engine variable frag_timeout now defaults to 5 minutes if you
  load frag.bro, and is accessed via redef rather than by defining the
  global directly.

- The interval that signatures.bro waits for until generating a signature
  summary can now be set using the new script variable sig_summary_interval,
  and a bug in generating the summaries has been fixed (Robin Sommer).

- The new script peer-status.bro generates periodic "update" events regarding
  a remote peer's status (Robin Sommer).  These take the form:

	type peer_status: record {
		res: bro_resources;
		stats: net_stats;
		current_time: time;
		cpu: double;		# average CPU load since last update
		default_filter: string;	# default capture filter
	};

- The bro_resources record returned by resource_usage() now includes
  three additional fields, $version (the version of Bro), $debug
  (T if Bro was compiled with debugging information), and $start_time
  (the time Bro began executing - clock time, not network time).

- The new built-in function same_object(o1: any, o2: any): bool
  returns true if its arguments refer to the same object, false
  otherwise.  This can be useful for comparing tables, for example
  in calls to table element expiration functions.

- The new built-in function bro_is_terminating(): bool returns true if
  Bro is done reading from its network input source(s) and is now
  in its final termination cleanup (Robin Sommer).

- A new built-in strftime() formats a timestamp, returning a string
  (Robin Sommer).

- A new built-in file_size() returns the size of the file with a
  given name (Robin Sommer - note: *not* a Bro file value).

- A potential deadlock with inter-Bro communication has been fixed
  (Robin Sommer).

- Bro now always forks a copy of itself when executing, as this
  can save considerable memory when using inter-Bro communication
  (Robin Sommer).

- The Bro interconnection protocol now includes explicit handshaking
  during session establishment to mark that a peer is ready (Robin Sommer).
  Implementing this includes a change in the wire protocol that is
  incompatible with the protocol used in the past.

- The TCP inactivity timer is now started whenever a connection
  transitions from a pre-establishment state (including "inactive")
  to some sort of established state (Robin Sommer).  Prior to this
  fix, connections for which a proper SYN handshake was not seen would
  not be timed out as inactive.

- The --disable-openssl configure option has been removed; now
  the only option is --with-openssl, and --with-openssl=no disables
  use of OpenSSL (Gregor Maier).

- A bug in invoking &expire_func functions has been fixed (Robin Sommer).

- A bug in logfile rotation has been fixed (Robin Sommer).

- A bug in recognizing negative floating point values has been fixed.

- worm.bro now suppresses the default signature action for worms
  it knows about, since it generates events for them (Robin Sommer).
  The list of worms detected via signatures now includes Bagle-BC.

- Signatures for known worms are now skipped when doing signature
  summaries and scan detection, if worm.bro is loaded (Robin Sommer).

- request_remote_events and request_remote_sync now implicitly
  do set_accept_state, too.

- Better error handling for SSL connections (Robin Sommer).

- Bug fixed which caused diagnostic messages to be lost when using
  inter-Bro communication (Robin Sommer).

- gcc 3.4 portability fixes (Brian Lindauer).

- Solaris portability fixes (Robin Sommer).

- The Bro distribution now includes and uses its own version of libpcap
  for portability reasons (Jason Lee).

- Some minor bug fixes to handling of tcpdump save files (Robin Sommer).

- Detection added for a (now quite old) SSHv1 overflow attack.

- A bug in skipping processing of connections for large chunks of
  data has been fixd (Chema Gonzalez).

- Some memory leaks fixed (Robin Sommer).

- fmt()'s "%d" format now accepts values of enum types.


0.9a7 Mon Nov  1 13:21:05 PST 2004

- New terminology:

	o We've found that the term "log" has been too overloaded,
	  sometimes meaning "something to record for audit purposes" and
	  other times meaning "something worthy of getting the operator's
	  attention right now, for example via syslog".

	  We are now using "log" to only refer to the first of these,
	  and refer to the second as "alarm".

	o We've found that "alert" (and the ALERT() function, etc) is
	  likewise confusing.  Some expect it to mean something alarm-worthy
	  (to use the new name from the previous item), while others
	  expect it to mean the output from a sensor, which might not
	  be worth getting the operator's attention.

	  We are now using "notice" to refer to what had previously
	  been called "alert".  So, for example, rather than call ALERT()
	  you call NOTICE() to enter something into the I've-seen-something-
	  maybe-it's-worth-an-alarm framework.  The first field associated
	  with such a call was $alert, which is now $note.  These notifications
	  are logged to notice.$BRO_LOG_SUFFIX rather than "alert."...

  These changes show up in many places in the policy scripts.  Some
  globals log_XXX are now alarm_XXX or notice_XXX.  However, uses of
  "log" that refer to "something to record for audit purposes" remain
  using that name.

- Bro now uses the "automake" suite of tools for its configuration
  (Jason Lee).  This includes major reworking of its Makefile's
  and configuration/installation scripts.

- Bro now flags event handlers that are never invoked (Umesh Shankar).
  This catches typos like:

	event bro_initt() { ... }

- The scripts directory now includes scripts for generating nightly
  reports (Roger Winslow).  Note that these are not presently a supported
  part of the public Bro distribution, but will become so in the
  not-too-distant future.

- policy/brolite.bro contains a first version of the "Bro Lite" configuration
  (Scott Campbell, Roger Winslow, et al).  This also is not presently
  supported for the public Bro distribution, but will become so.

- Major overhaul of the internal serialization framework (Robin Sommer).
  Some more changes are forthcoming, though not on this scale.

- Changed packet capture/drop statistics to (1) explicitly track received
  packets, (2) work better under Linux, (3) report link counts when available
  (Robin Sommer).  The net_stats record field "interface_drops" has been
  renamed "pkts_link".  It reports the number of packets captured by the
  NIC (if available), rather than the number dropped by the NIC (which never
  actually worked, anyway).

- The DNS analysis has been extended to deal with TSIG and to better deal
  with EDNS0 records, and its overall structure reworked somewhat
  (Scott Campbell & Roger Winslow).  The scripts in dns.bro are now
  inside "module DNS" scope.

- Improved logging of ICMP flows (Scott Campbell).  This includes the
  addition of a "len" field in icmp_conn.  Some significant additional
  changes/improvements to ICMP processing will be coming very soon.

- The Bro README and "quick start" documentation has been updated
  (Brian Tierney and Jim Rothfuss).

- Some significant state-holding problems in the presence of packet filtering
  or packet drops have been fixed.

- You can now instruct the signature engine to constrain the number of
  regular expressions it groups together into a single matcher using the
  new script variable "sig_max_group_size" (Robin Sommer).  This can result
  in significant memory savings for large sets of signatures.  It can
  increase the CPU processing required; however, in our testing so far
  this does not appear to be the case.

- The signature engine now provides better location information in its
  error messages (Robin Sommer).

- statistics.bro has been renamed profiling.bro.  Along with this change,
  the following script variables have been renamed:

	do_statistics => do_profiling
	statistics_file => profiling_file
	statistics_interval => profiling_interval

- A new script, stats.bro, can be included to generate light-weight
  running statistics on memory and CPU use.

- A new script, print-sig-states.bro, can be included to generate
  periodic dumps of signature-matching statistics (Robin Sommer).

- The connect_clear()/connect_ssl() built-ins have been replaced
  by a single connect() function (Robin Sommer).

- Remote Bro's (more generally, remote event sources) are now represented
  in policy scripts using the new "event_peer" type rather than as an
  addr/port pair, and get_event_source() has been renamed get_event_peer()
  (Robin Sommer).  The new function get_local_event_peer() returns an
  event_peer corresponding to the local Bro.

- The new script remote-print.bro can be used to print all events
  received from remote sources (Robin Sommer).

- When you call send_state(), an event finished_send_state() is now
  generated when all of the state has been sent (Robin Sommer).

- If you define a handler for the new print_hook() event, then it
  will be invoked every time a "print" statement executes (Robin Sommer).

- The -g flag no longer takes a directory as an argument, but always
  uses the .state directory (Robin Sommer).

- The new -I flag prints the value of a given identifier and exits
  (Robin Sommer).  This flag may go away in the future.  For now,
  its role is that it prints timestamps of things like table elements,
  which are no longer printed by a regular "print" statement.

- If a connection proceeds as SYN ->, <- RST, RST ->, this is now
  reported as a regular rejected ("REJ") connection, rather than
  one reset by the originator ("RSTO").

- The TCP analyzer now attempts to detect connections for which
  one side's packets are being filtered out.  It does this using
  two C++ variables (not presently accessible at the script level),

	const int max_initial_window = 4096;
	const int max_above_hole_without_any_acks = 4096;

  If more than max_initial_window data has been sent by one side and no
  ACKs have been seen by the other side, then it's assumed that no ACKs
  will ever show up, and the buffer for the data should be reclaimed
  immediately after it's reassembled.  max_above_hole_without_any_acks
  plays a similar role, but for the case when measurement drops have
  lead to Bro holding data for a sender for which it will never see
  an ACK (which would be an "ACK above hole"), so it should give up
  on buffering it.

- A portability bug in Bro's parser code has been fixed (Jason Lee).

- STARTTLS for SMTP now causes the SMTP session to be ignored, rather
  than generating a slew of error messages because the session becomes
  unparseable.

- Bro's "watchdog" handler is now less prone to crashing due to
  using non-reentrant library calls (Jason Lee).

- Better error message when reading an input file encounters an error.

- adtrace now prints its usage when invoked without the required
  arguments (Jason Lee).

- A bug in using recursive script functions has been fixed.

- A bunch of whitespace/code layout tweaks.


0.9a4 Wed Sep  8 17:33:54 PDT 2004

- The directory structure of the Bro distribution has changed (Jason Lee).
  The source code is now in a subdirectory, src/, and the scripts
  snort2bro (and snort2bro.cfg) and make-ftp-safe-vocabulary.awk have
  been moved into scripts/.

- "make install" has been revamped (Jason Lee).

- The format of the alert log file has changed.  Fields in it are
  colon-separated.  THIS WILL LIKELY CHANGE SOON.

- The policy for formatting signature matches has been revamped,
  including colon-separated fields in the signature log file
  (Roger Winslow).  THIS WILL LIKELY CHANGE SOON.

- The BRO_ID environment variable has been renamed BRO_LOG_SUFFIX.

- A new flag, -e, lets you specify Bro code to execute via the command
  line (Christian Kreibich).  So, for example,

	bro -r mytrace.tcpdump -e 'redef traditional_conn_format = T' tcp 

  will run tcp.bro on the trace "mytrace.tcpdump", but with
  traditional_conn_format redefined to be true.  Note that statements
  have an implicit ';' added to them for convenience.

- A new signature alert, "MultipleSigResponders", is generated if a
  host triggers the same signature on multiple responders.

- Bro now supports "packet profiling", which provides fairly fine-grained
  statistics on number of packets processed, volume, elapsed real/user/system
  time, and change in memory consumption (Holger Dreger).  Three variables
  control the output.  The double pkt_profile_freq controls the frequency
  of output.  The units in which it's interpreted depends on the setting
  of the pkt_profile_mode variable (which is of type pkt_profile_modes,
  an enum).  A value of PKT_PROFILE_MODE_SECS means that statistics
  are generated every pkt_profile_freq seconds; PKT_PROFILE_MODE_PKTS
  means every pkt_profile_freq packets; and PKT_PROFILE_MODE_BYTES, every
  pkt_profile_freq bytes.  The default (PKT_PROFILE_MODE_NONE) means
  to not generate packet profiling.

  Packet profiling is written to the new log file, pkt_profile_file.
  If you "@load pkt-profile", you can turn on packet profiling using
  some handy defaults.

- statistics.bro now reports on how many TCP connections are in
  <originator-state, responder-state> for the different TCP endpoint
  states (SYN sent, SYN ack'd, connection established, etc.).
  Contributed by Holger Dreger.

- tcp_content_delivery_ports_{orig,resp} are now table's of bool rather
  than set's (Ruoming Pang).  The semantics are that if you have a
  tcp_contents event handler, then if the orig/resp port is in the given
  table *and the yield value is T*, then the event will be invoked.  This
  allows you to now explicitly skip over some ports.

- The processing of default values in tables has been changed internally
  (Ruoming Pang).  It's possible this has introduced some subtle bugs
  (as some of these came up during testing).

- A serious bug in Base64 processing has been fixed (Ruoming Pang).

- The NetBIOS and SMB analyzers have been updated in minor ways
  (Ruoming Pang).

- statistics.bro now reports a "lag" figure indicating the elasped
  time between the last expired timer's target expiration time and
  the current packet timestamp (Robin Sommer).  Lag can grow if Bro
  is getting behind in timer expiration due to the setting of
  max_timer_expires.

- Bro's default filter is now "tcp or udp or icmp" rather than
  "tcp or udp".

- alert_info records now have an optional port associated with them
  (for example, to be used to describe scan activity).

- A bug has been fixed in which deleting a table element with an
  associated timer could crash Bro (Robin Sommer).

- A bug that would cause a crash for malformed EPASV directives
  has been fixed (Robin Sommer).

- A bug with inactivity timeouts not being generated for partial
  connections has been fixed (Robin Sommer).

- A bug in synflood.bro has been fixed (Robin Sommer).

- Some tuning adjustments to incremental expiration of table entries
  (Robin Sommer).

- Improved portability to Darwin (Christian Kreibich).

- alert_info records now have additional optional fields, "iconn"
  (associated ICMP connection), "dst" (destination address), and
  "p" (associated port).  The source_is_responder fields has been
  removed.

- The default packet filter now includes "icmp".

- Some memory allocation/free mismatches & minor leaks (Robin Sommer).

- Minor tweaks to ssl.bro (Robin Sommer).

- Bro now supports "null" link layers (Christian Kreibich).

- aux/adtrace contains a program that spits out MAC/IP information
  from traces (Holger Dreger).

- A bug in handling malformed RPCs has been fixed (Scott Campbell).

- The formatting of "weird" messages that have additional parameters
  has been changed to be more regularized with other "weird" messages.

- The new "weird" type "base64_illegal_encoding" takes the place of
  some previously unstructured Base64 "weird" errors.

- A tweak to ftp.bro will give it slightly more consistent results 
  for some forms of unusual traffic.


0.9a3 Wed Jul  7 22:06:26 PDT 2004

- Improved ICMP processing, including scan detection (Scott Campbell).

- ICMP "connections" are now considered unidirectional.

- Fixed broken VLAN support (integration of original patch was incomplete).

- Fixed a bug in erroneously generating additional "ContentGap"
  alerts after an initial one.

- Connection durations are now always reported as floating-point decimal,
  never in exponential notation.

- Removed unused time parameter from a bunch of internal calls.

- Fixed some compilation warnings.

- "make clean" now removes generated policy/*.bif.bro files (Christian
  Kreibich).


0.9a2 Fri Jun 11 00:07:04 PDT 2004

- NetBIOS analysis has been extended with a CIFS/SMB analyzer (Ruoming Pang).
  While this is incomplete, it has many important elements.  The corresponding
  events:

	smb_message(c: connection, is_orig: bool, cmd: string, msg: string)
	smb_com_tree_connect_andx(c: connection, path: string, service: string)
	smb_com_nt_create_andx(c: connection, name: string)
	smb_com_transaction(c: connection, is_orig: bool, subcmd: count,
				name: string, data: string)
	smb_com_transaction2(c: connection, is_orig: bool, subcmd: count,
				name: string, data: string)
	smb_com_read_andx(c: connection, is_orig: bool, data: string)
	smb_com_write_andx(c: connection, is_orig: bool, data: string)

  This analyzer is still experimental.

- Greater support for vectors (Umesh Shankar), much of it taken from
  the 'S' language.

  You can use a boolean vector as an index into another vector (providing
  both are the same length) and each 'T' value in the index extracts the
  corresponding element from the indexed vector.  For example, "x[x > 3]"
  returns a vector whose elements are those elements of x that are greater
  than, while if y is a vector of the same length as x then "y[x > 3]"
  extracts those elements of y that have the same position as the elements
  in x that are greater than 3.

  You can also use an arithmetic vector to index another vector.  Each
  element present in the index is extracted.  So, for example:

	global a: vector of count;
	global b: vector of string;

	a[1] = 3;
	a[2] = 3;
	a[3] = 1;

	b[1] = "foo";
	b[2] = "bar";
	b[3] = "bletch";

	print b[a];

  prints:

	[bletch, bletch, foo]

- The new built-ins any_set() and all_set() return true if for a given
  boolean vector any element is true or all of the elements is true
  (Umesh Shankar).  So, for example, "any_set(x < 0)" returns T if 
  an element of x is less than zero.

- The new built-in sort() takes a vector as an argument and sorts it
  *in place* (Umesh Shankar).  (The in-place semantics may change in the
  future.)  An optional second argument can be used to specify a
  function to call for comparing elements, and is required for non-arithmetic
  vectors.  For example, the following could be used to sort a vector
  of strings based solely on the length of the strings:

	function string_compare(a : string, b: string): int
		{
		local la = byte_len(a);
		local lb = byte_len(b);

		return (la < lb) ? -1 : ((lb > la) ? 1 : 0);
		}

- The new function order() has the same calling sequence as sort(),
  but instead of returning (and altering in place) the sorted vector,
  it returns a "vector of count" giving the *indices* that if used
  to index the vector will return it sorted.  So, for example,
  given two vectors x and y of the same length (but not necessarily
  of the same type),

	local x_sort_indices = order(x);
	x = x[x_sort_indices];
	y = y[x_sort_indices];

  will assign x to a sorted version of itself and also rearrange y such
  that elements of y that were paired with elements of x originally
  remain paired after the sorting.

- The ICMP analyzer now has a general notion of "context", i.e., the packet
  associated with ICMP status messages such as Unreachable or Time Exceeded
  (Ruoming Pang).  This changes the parameters to the icmp_unreachable
  event.  A new event, icmp_time_exceed, is now also available.

- The tcp_segment even has been replaced by a pair of new events (Ruoming Pang):

	tcp_packet(c: connection, is_orig: bool, flags: string, seq: count,
			ack: count, len: count, payload: string)

  is invoked for each TCP packet.  "flags" is a string containing "SFAPU"
  for the SYN/FIN/etc TCP flags.

	  tcp_contents(c: connection, is_orig: bool, seq: count,
			contents: string)

  is invoked for each chunk of the byte-stream that has been reassembled
  in sequence, providing it satisfies tcp_content_delivery_ports_{orig,resp},
  per the next item.

- You can specify the set of ports for which contents should be reassembled
  for the originator (responder, respectively) stream using the new sets
  tcp_content_delivery_ports_{orig,resp} (Ruoming Pang).  This can be
  useful for user-level stream analysis for protocols not known to Bro's
  event engine.  These controls may change to a "table of bool" in the future,
  in order to support an &default attribute.

- New built-in functions (Ruoming Pang):

	function interval_to_double(i: interval): double
		Converts a value of type "interval" to "double".

	function write_file(f: file, data: string): bool
		Writes the given string to the given file, returning
		F on error.

	function is_ascii(str: string): bool
		Returns T if the given string consists entirely of
		ASCII characters (i.e., in the range 0..127).

	function sqrt(x: double): double
		Returns the square-root of x, or -1 and a run-time error
		if x is < 0.

	function uuid_to_string(uuid: string): string
		Takes a UUID and returns its string representation, where
		UUID = Universal Unique Identifier as defined per
		http://www.opengroup.org/onlinepubs/9629399/apdxa.htm#tagcjh_20

	function string_to_ascii_hex(s: string): string
		Returns the ASCII hex representation of the given string.
		For example, string_to_ascii_hex("foo") returns "666f6f".

	function match_pattern(s: string, p:pattern): pattern_match_result
		Matches the given pattern against the given string, returning
		a record with three fields:

			matched: bool;	# T if a match was found, F otherwise
			str: string;	# portion of string that first matched
			off: count;	# 1-based offset where match starts

		For example,
			match_pattern("foobar", /o*[a-k]/)
		returns
			[matched=T, str=f, off=1]
		because the *first* match is for zero o's followed by an [a-k],
		while
			match_pattern("foobar", /o+[a-k]/)
		returns
			[matched=T, str=oob, off=2]

- Functions that terminate without returning a value when they were declared
  to do so now generate a run-time warning (Christian Kreibich).  Functions
  in the standard set of policy scripts that did this have been fixed.

- The new event non_dns_request(c: connection, msg: string) is generate
  to make the contents of malformed DNS requests available for analysis,
  with the assumption that these are actually some other protocol entirely
  (Ruoming Pang).

- If you redef truncate_http_URI to have a value >= 0, then any HTTP
  URIs generated by the event engine will be truncated to this length
  (Ruoming Pang).  This can be convenient when analyzing traffic that
  generates huge URIs (as do some automated attacks).

- "SEARCH" is now recognized as a standard HTTP method (Ruoming Pang).

- The new event connection_EOF(c: connection, is_orig: bool) is generated
  when one side of a connection closes (Ruoming Pang).

- synflood.bro and the corresponding event engine internals now works
  in terms of probabilities (0.0-1.0) instead of percentages (0-100)
  (Robin Sommer).  The script has had several tweaks, including using
  new_connection() rather than connection_attempt(), which gives it
  quicker response and broader coverage (it'll detect non-TCP flooding,
  too, so "synflood" is now a bit of a misnomer), at the cost of perhaps
  more CPU load.

- A signature for Witty has been added to policy/sigs/worm.sig (Ruoming Pang).

- Makefile now has a "test" target.  Currently this only works for internal
  development (we haven't put together a public test suite yet; that will
  take some time due to the need to make sure no sensitive information leaks).

- The built-in function generator now knows about "double" as a built-in type
  (Ruoming Pang).

- Some generated files have been removed from the Bro distribution since
  they're redundant (Ruoming Pang).

- A bug has been fixed in which contents files might not be correctly
  written upon termination of Bro (Ruoming Pang).

- A bug has been fixed in which UDP connections didn't generate
  new_connection events (Ruoming Pang).

- Support for the Linux "cooked capture" link layer (Ruoming Pang).

- BPF support has been factored into a separate class, BPF_Program, which
  makes for easier portability (Christian Kreibich).

- A serious low-level Dictionary bug has been fixed (Christian Kreibich).

- A bug that could cause Bro to crash if it receives an event from another
  Bro that it isn't able to process has been fixed (Christian Kreibich).

- A bug in set file descriptors non-blocking has been fixed
  (Christian Kreibich).

- A bug that could cause some error messages to generate crashes has
  been fixed.

- The global skip_event_tcp_option has been removed.


0.9a1 Mon Jun  7 01:33:00 PDT 2004

- 0.8a85 is the new STABLE release, to be updated only for bug (and
  portability) fixes.  0.9a1 is the new CURRENT release, which will have
  both bug fixes and new functionality.

- Support for FTP EPRT, EPSV (Holger Dreger).

- Change to timer management to recover memory more quickly (Robin Sommer).

- Tweaks to eliminate a number of compiler warnings (Robin Sommer).

- Statistics now report number of connections terminated due
  to inactivity (Robin Sommer).

- New Makefile target, pub-tar-no-doc builds a public tarball but
  without the (large) doc/ subdirectory.

- Bug fix for identd requests with illegal port numbers.

- The example of a scanning source to skip in scan.bro has been changed
  from an AltaVista robot (now obsolete) to a Google robot.

- Some previously fatal internal errors have now been turned into
  "internal warnings", which Bro is able to continue operating
  in their presence.


0.8a84 Wed May 26 23:33:39 PDT 2004

- autoconf tweaks for portability to Darwin (Christian Kreibich).

- Fixed subtle bug in chunked-IO reads (Christian Kreibich).

- Fixed bug for the "discarder" framework in which specifying an
  IP discarder would cause other TCP/UDP/ICMP discarders not being
  invoked (Christian Kreibich).

- Fatal bug in signature matching fixed (Robin Sommer).

- Missing member variable initialization fixed (Robin Sommer).  (Needed for
  compilation with new versions of gcc.)

- Makefile bug for "make install" fixed.

- Fixed bug that could lead to Bro crashing if an SMTP session
  had data sent after a RST.

- Removed some out-of-date SMTP analysis warning messages.


0.8a82 Tue Apr 27 11:53:24 PDT 2004

- Fixed inactivity timer loop when a packet arrives exactly when
  the timer is set to expire.


0.8a81 Mon Apr 26 22:46:37 PDT 2004

- A bunch of memory leaks fixed (Chema Gonzalez).

- A new HTTP analyzer variable, content_truncation_limit, controls
  how much of an HTTP request/reply contents element (i.e., what's
  passed to http_entity_data - this is *not* the entire content, but
  the next chunk's worth) is logged in the log file.  It defaults
  to 40 bytes.  Setting it to 0 means "log all of it".

- Fix to avoid crashing for malformed RPC requests.

- Improved OpenSSL auto-configuration (Robin Sommer).

- Fix for compiling without OpenSSL.

- A new built-in, double_to_count(), converts a value of type "double"
  to the corresponding "count" (Chema Gonzalez).  We should probably
  add floor(), ceil(), etc.

- Parameterization of trw.bro tweaked (Jaeyeon Jung).


0.8a79 Wed Mar 24 22:02:53 PST 2004

- Bro now has an SSL analyzer, written by Michael Kuhn and Benedikt Ostermaier,
  with further development by Scott Campbell.  It generates the following
  events:

	event process_X509_extensions(c: connection, ex: X509_extension)
	event ssl_X509_error(c: connection, err: int, err_string: string)
	event ssl_certificate(c: connection, cert: X509, is_server: bool)
	event ssl_certificate_seen(c: connection, is_server: bool)
	event ssl_conn_alert(c: connection, version: count, level: count,
				description: count)
	event ssl_conn_attempt(c: connection, version: int)
	event ssl_conn_established(c: connection, version: int,
				cipher_suite: count)
	event ssl_conn_reused(c: connection, session_id: sessionID)
	event ssl_conn_server_reply(c: connection, version: int)
	event ssl_conn_weak(name: string, c: connection)
	event ssl_session_insertion(c: connection, id: sessionID)

  Note, it still has a lot of rough edges; particularly, handling
  non-conformant input.  It also generates unnecessary ContentGap alerts
  due to the way it runs multiple analyzers (SSLv2 and SSLv3) on a single
  connection.  This will be fixed in the fairly near-term future.

- The manual has been updated with chapters on signatures (Robin Sommer)
  and using the interactive debugger (Umesh Shankar), along with a partial
  description of the new SSL analyzer (Michael Kuhn and Benedikt Ostermaier)
  and a number of updates to the documentation of built-in functions (Umesh
  Shankar), though this latter is still not complete since Umesh actually
  contributed this quite a while ago.

- Ruoming Pang has contributed a crude analyzer for DCE/RPC (used for Windows).
  It generates simple dce_rpc_request and dce_rpc_reply events.  It should
  not be considered stable.

- The traditional connection logging format (traditional_conn_format)
  is no longer the default.  The 0.8a70 release notes stated that this
  was the case but this time it really is :-).

- An experimental "vector" type has been added (Umesh Shankar).  A vector
  is an aggregate type.  For example:

	local suspect_hosts: vector of addr;

  You can randomly access elements starting with the first as 1, e.g.,

	suspect_hosts[1] = 10.0.0.8;

  and can also add elements at later postions even if there are gaps:

	suspect_hosts[31337] = 10.0.0.9;

  *The semantics and capabilities of vectors will be changing considerably.*

- Umesh Shankar has developed a framework for generating IDMEF messages.
  Currently it needs a modified version of libidmef, which is not included
  in this distribution.  Contact me or Umesh if you want a copy.

- A new attribute &synchronized causes the given global variable to
  be *synchronized* across concurrent instances of Bro (which are
  intercommunicating via remote.bro).  Any change made by one of them
  to the variable will be reflected (soon after) in the copy at
  the others.  A new event remote_state_inconsistency is generated
  if two instances both change the value before they're synchronized.
  (Contributed by Robin Sommer.)

- trw.bro implements a new scan detection algorithm, Threshold Random Walk
  (Jaeyeon Jung).  It's described in an upcoming IEEE S&P symposium paper.
  The analyzer generates two events:

	TRWAddressScan, # source flagged as scanner by TRW algorithm
	TRWScanSummary, # summary of scanning activities reported by TRW

  TRW is generally much more sensitive than Bro's regular detection algorithm.

- vlan.bro provides support for VLAN encapsulation.  More generally, Bro
  now has support for any sort of constant-offset encapsulation (Vinod
  Yegneswaran).  You specify the header size by redef'ing encap_hdr_size.
  You can also redef tunnel_port to be a UDP port which Bro treats as being
  the encapsulation (in the packet stream - not addressed to it) rather
  than all traffic.

- If you turn on statistics (i.e., via @load statistics) and also redef
  segment_profiling to T, then Bro will generate to the statistics file
  a trace of its "segment" processing.  A segment is a unit of internal
  execution.  Profiles look like:

	1058517499.615430 segment-processing-packet dt=0.000013 dmem=0
	1058517499.615430 segment-draining-events dt=0.000012 dmem=0
	1058517499.615671 segment-expiring-timers dt=0.000010 dmem=0
	1058517499.615671 segment-processing-packet dt=0.000010 dmem=0
	1058517499.615671 segment-draining-events dt=0.000012 dmem=0
	1058517499.615671 segment-policy/conn.bro:282 dt=0.000011 dmem=0
	1058517499.615671 segment-policy/conn.bro:253 dt=0.000012 dmem=0

  The first line states that at the given (packet) timestamp, the event
  engine processed a packet, taking 13 usec of CPU time to do so, and
  not consuming any memory (from the kernel's perspective; this is *not*
  fine-grained memory consumption).  The next lines indicate 12 usec were
  spent draining events and 10 usec expiring timers.  The last two lines
  indicate that the functions at lines 282 and 253 in conn.bro were
  executed, requiring 11 usec and 12 usec, respectively.

  Note #1: timings are just what rusage() reports, so not necessarily
  very accurate for small times.

  Note #2: there's a bug in tracking function line numbers that hasn't
  been ferreted out yet, so they're only roughly correct.

- The inactivity_timeout global has been split into tcp_inactivity_timeout/
  udp_inactivity_timeout/icmp_inactivity_timeout (Robin Sommer).  Using
  this, the default inactivity timeouts for UDP and ICMP have been changed
  from "no timeout" to 10 seconds.  This is needed because otherwise
  analyzing a stream of UDP or ICMP traffic generally gobbles up memory
  quickly and never recovers it; and there's seems little point in trying
  to consolidate long-lived-but-often-inactive UDP/ICMP streams.

- The new policy script cpu-adapt.bro is an extension to load-levels.bro
  (see CHANGES for 0.8a37 below) to adapt the packet filter based on the
  current CPU load. If the load is below cpu_lower_limit (default 40%),
  the load-level is decreased.  If it's above cpu_upper_limit (default
  90%), it's increased.  (Robin Sommer)
 
- The new policy script hand-over.bro can be used for a new running
  instance of Bro to smoothly take over operation from an old instance,
  i.e., it implements hand-over of state between two Bro instances when
  checkpointing (Robin Sommer). First, all persistent data (i.e. variables
  declared &persistent and connections for which make_connection_persistent()
  has been called) is transferred from the old instance to the new instance.
  Then the old instance terminates itself and the new one starts processing.

  The host from which we want to take over the state has to be added to
  remote_peers_clear (or remote_peers_ssl), setting hand_over to T. The
  host which is allowed to perform a hand-over with us has to be added
  with a port of 0/tcp and hand_over=T. An example for a handover between
  two instances on the same machine:

  @load hand-over
  redef remote_peers_clear += {
        [127.0.0.1, 47756/tcp] = [$hand_over = T],
        [127.0.0.1, 0/tcp] = [$hand_over = T]
        };

  (This interface may be subject to change in the future.)

- New script functions (Robin Sommer):

      function terminate()
          Terminates Bro via SIGTERM.

      function dump_config()
          Dumps Bro's full configuration into state_dir (one file per
          variable/type/function, etc.)

      function send_state(ip: addr, p: port)
          Send all of persistent state to the remote host.

      function set_accept_state(ip: addr, p: port, accept: bool)
          If accept is true, state sent by the given host will be
          accepted (default: false)

      function make_connection_persistent(c: connection)
          Declare the given connection state to be persistent (i.e.
          to be saved upon termination and exchanged by send_state).
          checkpoint.bro uses this to declare some services to be
          persistent by default.

      function is_local_interface(ip: addr): bool
          Returns true if the given address is assigned to a local interface.

- Printing of sets and tables now includes timestamps indicating when the
  element was added (Robin Sommer):

         ID did_ssh_version = {
            [129.187.20.9, F] = 1 @11/01-15:55,
            [212.144.77.26, T] = 2 @11/01-15:55,
            [141.84.116.26, T] = 10 @11/01-15:55,
            [217.232.245.249, T] = 1 @11/01-15:55,
            [217.235.217.149, T] = 1 @11/01-15:55,
            [129.187.39.13, F] = 2 @11/01-15:55,
            [129.187.208.139, F] = 1 @11/01-15:55,
            }

  The format may change in the future, and will probably be made an option.

- Similarly, you can print functions to get both a timestamp of the last
  time the given block was executed and a count of how often (Robin Sommer):

	  ID record_connection = record_connection
	  (@11/01-16:03 #6549)
	  {
	  id = c$id;
	  local_init = is_local_addr(id$orig_h);
	  local_addr = local_init ? id$orig_h : id$resp_h;
		    remote_addr = local_init ? id$resp_h : id$orig_h;
	  flags = local_init ? "L" : "";
	  if (remote_addr in neighbor_nets)
		  (@<never> #0)
		  flags = cat(flags, "U");

	  if ("" == flags)
		  (@11/01-16:03 #2110)
		  flags = "X";

	  is_tcp = is_tcp_port(id$orig_p);
	  ;
	  if (is_tcp)
		  (@11/01-16:03 #6549)
		  {
		  if (c$orig$state in conn_closed || c$resp$state in conn_closed
)
			  (@11/01-16:03 #4739)
			  duration = fmt("%.6g", c$duration);
		  else
			  (@11/01-16:03 #1810)
			  duration = "?";
	  [...]

- You can now specify numbers using hex constants, e.g., 0xabcd = 43981
  (Michael Kuhn and Benedikt Ostermaier).

- A new function, join_string_array(sep: string, a: string_array) concatenates
  strings in 'a' and inserts 'sep' between every two adjacent elements
  (Ruoming Pang).  E.g., join_string_array("", {"a", "b", "c"}) returns
  "a b c", and join_string_array("", a) is the same as cat_string_array(a).

- checkpoint.bro now makes some services persistent by default
  (Robin Sommer).

- The new_packet event now includes both the associated connection
  and a pkt_hdr describing the packet (Robin Sommer).

- The functions functions connect_ssl() and connect_clear() have been replaced
  by a single connect() function taking an additional parameter to
  differentiate the types (Robin Sommer).

- The new function stop_listening() unbinds the listen port (Robin Sommer).

- A new flag packet_filter_default says whether the Bro-level packet-filter
  will by default accept all or reject everything (Robin Sommer).

- Function calls can now be sent to remote Bro's, though there isn't yet
  an interface for accessing this from the script level (Robin Sommer).

- Bro now has an generalized internal framework for serializing objects
  and monitoring access to state (Robin Sommer).

- Better memory allocation accounting (Robin Sommer).

- A minor tweak to the output generated by statistics.bro.

- Improved localization of source code for functions in messages (but
  there are still some bug).

- Autoconf looks for -ltermcap (Robin Sommer).

- Fixes for bugs in the management of table expiration values (Chema Gonzalez).

- A bug in printing "void" values has been fixed (Chema Gonzalez).

- -t bug fixed (Chema Gonzalez).

- A bug has been fixed in which sometimes "expression value ignored"
  was erroneously generated.

- A bug with packet_contents and UDP packets with checksum errors
  has been fixed (Ruoming Pang).

- A memory leak in packet timestamp sorting via packet_sort_window
  has been fixed (Ruoming Pang).

- A bug has been fixed in expanding NULs when printing strings (Ruoming Pang).

- Bug fixes for extracting connection contents via contents.bro (Ruoming Pang).

- Bogus error message "Can't install default pcap filter" when using -F
  removed.


0.8a70 Sun Feb  8 14:19:45 PST 2004

- Bro has a new home page at

	http://www-nrg.ee.lbl.gov/bro.html

  It includes a "wish list" of Bro development projects:

	http://www-nrg.ee.lbl.gov/bro-wishlist.html

- The "match" expression has been completely overhauled (Umesh Shankar).
  It now has the syntax:

	match EXPR1 using EXPR2

  Its semantics are complicated, but it's very powerful (see its use for
  alert filtering below).  EXPR1 can have any type T.  EXPR2 must be of
  type "set[R]", where R is a record type.  R must have the following fields:

	$pred	- type is "function(T): bool".  This is the predicate
		  associated with the record.  It is passed in EXPR1's
		  value and returns true or false.

	$result	- can have any type T'.  This is the value to use when
		  if $pred returns true for EXPR1.

	$priority - type must be arithmetic (count, int, double).  This
		  is the priority associated with the match of EXPR1
		  if $pred returns true.
  
  The way the expression works is that EXPR1 is evaluated yielding a
  value V.  EXPR2 is then evaluated yielding a set of records whose
  type includes the above fields.  Bro then spins through each of the
  records in the set and tests whether its $pred predicate holds for V.
  If so, it records the given $result and the associated $priority.
  It then returns for the value of the entire expression the $result
  with the highest $priority.

  Here's an example.  The following script:

	global match_stuff = {
		[$pred = function(a: count): bool { return a > 5; },
		 $result = "it's big",
		 $priority = 2],

		[$pred = function(a: count): bool { return a > 15; },
		 $result = "it's really big",
		 $priority = 3],

		[$pred = function(a: count): bool { return T; },
		 $result = "default",
		 $priority = 0],
	};

	print match 0 using match_stuff;
	print match 10 using match_stuff;
	print match 20 using match_stuff;

  when executed will print:

	default
	it's big
	it's really big

  (Note that Bro actually will first evalute $priority before evaluating
  $pred, and if it already has a better (higher) priority result, it
  will not bother calling $pred.)

- There's a new syntax for designating function values (Umesh Shankar).
  It currently only works when associating a function value with a
  record field in a record constructor:

	[$foo(bad_guy: addr) = { launch_counter_DDOS(bad_guy); return 3.14; }]

  is equivalent to:

	[$foo = function(bad_guy: addr): double = {
			launch_counter_DDOS(bad_guy);
			return 3.14;
	}]

  The goal is to make such functions easier on the eye to express.
  The changes are (1) no "function" keywork necessary, (2) no function
  return type necessary (note, it is inferred from the "return" statement
  in the function body; eventually this will work for all functions, not
  just those in record constructors), (3) the '=' sign comes after the ')'
  rather than before the keyword "function".

  Given this syntax, we can rewrite the initialization of match_stuff
  in the example above as:

	global match_stuff = {
		[$pred(a: count) = { return a > 5; },
		 $result = "it's big",
		 $priority = 2],

		[$pred(a: count) = { return a > 15; },
		 $result = "it's really big",
		 $priority = 3],

		[$pred(a: count) = { return T; },
		 $result = "default",
		 $priority = 0],
	};

- The motivation behind these elaborate new mechanisms is to provide a
  powerful and streamlined way to filter alerts.  According, alert.bro
  now processes any alerts generated via ALERT() through a new global,
  alert_policy.  alert_policy's type is set[alert_policy_item], where
  alert_policy_item is:

	type alert_policy_item: record {
		result: AlertAction;
		pred: function(a: alert_info): bool;
		priority: count;
	};

  The idea is that you specify your alert post-filtering by redef'ing
  new elements into alert_policy.  For example, here are two post-filtering
  rules used at LBL to weed out uninteresting alerts:

	# Ignore connections marked as sensitive because they're
	# NTP to otherwise-sensitive hosts (first clause) or they happen
	# to involve 2766/tcp (Solaris Listen), which happens frequently
	# to ftp.ee.lbl.gov if Bro misses the PORT negotiation.
	[$pred(a: alert_info) =
		{
		return a$alert == SensitiveConnection &&
			(a$conn$id$resp_p == 123/udp || # NTP
			 a$msg == /Solaris listen service/);
		},
	 $result = ALERT_FILE,
	 $priority = 1],


	# Ignore sensitive URIs if the request was unsuccessful (code 404,
	# or not answered.)
	[$pred(a: alert_info) =
		{
		return a$alert == HTTP::HTTP_SensitiveURI &&
			a$msg == /.*((\(404)|(no reply)).*/;
		},
	 $result = ALERT_FILE,
	 $priority = 1],

  These rules are part of:

	redef alert_policy += {
		... these records and others ...
	};

  The idea behind them is to demote certain alerts that would ordinarily
  be syslog'd (i.e., the associated action is ALERT_LOG_ALWAYS) to instead
  just be recorded in the alert.$BRO_ID file.  Naturally, there are
  many other possibilities.  For example:

	[$pred(a: alert_info) = {
		if ( a$alert == FTP::FTP_Sensitive &&
		     a$msg == /.*crown_jewels.*/ )
			{
			system("page_the_duty_officer \"crown jewels theft!\"");
			return T;
			}
		else
			return F;
		},
	$result = ALERT_LOG_ALWAYS,
	$priority = 1000],

  would run the program page_the_duty_officer with the argument "crown
  jewels theft!" if an FTP_Sensitive alert was generated and the log message
  included the text "crown_jewels".  More generally, post-filtering needn't
  just be about deciding on how the alert is logged; the processing can
  run programs, update tables, etc., just like any other function call might.

- You can use the new function tally_alert_type in an alert_action_filters
  initialization in order to suppress immediate logging of an alert and
  instead have Bro generate a summary of how many times the given alert
  was seen when it exits.  You can use another new function, file_alert,
  to specify an alert_action_filters initialization that causes the alerts
  to just be written to the alert.$BRO_ID file but not otherwise logged.

  For example:

	redef alert_action_filters += {
		# Just summarize various packet capture glitches.
		[[RetransmissionInconsistency, ContentGap, DroppedPackets,
		  AckAboveHole]] =
			tally_alert_type,

		[RemoteWorm] = file_alert,
	};

  would specify that RetransmissionInconsistency (etc.) alerts should just
  be reported in the log file (log.$BRO_ID) as a total count, and
  RemoteWorm should only be put in the alert.$BRO_ID file, but not
  otherwise logged or counted.

  You could get the same functionality by writing alert_policy records,
  but they're quite a bit bulkier than the above.  Note that
  alert_action_filters entries take precedence over alert_policy
  records, but are evaluated *after* the "match" on alert_policy,
  so if you have predicates in alert_policy with side-effects (like the
  invocation of page_the_duty_officer in the example above), those
  will still happen.

- The alert_info record (which is used in calls to ALERT) now has
  slots for some more additional information:

	user: string;	# can hold an assocaited username
	filename: string;	# an associated filename
	method: string;	# associated HTTP method
	URL: string;	# associated URL
	n: count;	# any associated count/number/status code

  (These are all &optional, so you don't need to specify them if they're
  not appropriate.)  A number of ALERT's in the default policy scripts
  have been changed to include these.  The intent is to add more such
  information in the future.  Ideally, alert_policy records shouldn't
  be doing checks like "a$msg == /.*((\(404)|(no reply)).*/" but instead
  extracting the status code directly from a field of 'a' (which is an
  alert_info record).

- ALERT now fills in the '$id' field in the alert_info record with
  the $id of the associated connection, if the caller didn't suppy
  a $id but did supply a $conn.  Likewise, it will fill in $src with
  the $orig_h from $id (after first filling in $id).  The net result
  is that you can rely on $id and $src being set for any alert that
  has an associated connection.

- The HTTP analysis scripts (policy/http*.bro) have been converted to
  use the "module" facility, similar to how ftp.bro was converted for
  0.8a48.  This may require changing some of your own scripts, generally
  just to add "HTTP::" qualifiers.

- Now that the variables associated with FTP analysis are part of an
  "FTP" module, the "ftp_" prefix associated with:

	ftp_excessive_filename_len
	ftp_excessive_filename_trunc_len
	ftp_guest_ids
	ftp_hot_cmds
	ftp_hot_files
	ftp_hot_guest_files
	ftp_ignore_invalid_PORT
	ftp_ignore_privileged_PASVs
	ftp_log
	ftp_skip_hot

  has been removed, and these are now called:

	excessive_filename_len
	excessive_filename_trunc_len
	guest_ids
	hot_cmds
	hot_files
	hot_guest_files
	ignore_invalid_PORT
	ignore_privileged_PASVs
	log_file
	skip_hot

  To get to them from other scripts, you specify, for example,

	redef FTP::guest_ids = { .... };

  whereas before you had to use:

	redef FTP::ftp_guest_ids = { .... };

- The new connection logging format introduced in 0.8a57 is now the
  default, unless you redef the new variable "traditional_conn_format"
  to be T (Robin Sommer).  Connections using unidentified ephemeral
  ports now have a service of simply "other" rather than other-XXXX.
  The 'U' connection status flag has been removed (unless you're using
  traditional_conn_format).

- Tables can now be directly indexed by records, and indexing using records
  is no longer interchangeable with using a corresponding list of indices
  (Umesh Shankar).  This may require adjustments to existing policy
  scripts.

- Hostnames such as www.google.com now have type set[addr] rather than
  a funky internal list type.

- The new function dump_current_packet(file_name: string) dumps a copy of
  the current packet to the file with the given name, appending it if the
  file already exists (Robin Sommer).  The file is in tcpdump format.
  A handy use for this is in an event handler for signature_match(),
  to record packets that match given signatures.

- The event new_packet() is invoked for each new packet (Robin Sommer).
  It currently doesn't provide the packet contents but soon will in
  a fashion similar to secondary-filter.bro.

- "cf -f fmt" specifies a strtime() format.  -u specifics UTC time rather
  than local time (Mark Delow and Craig Leres).  cf now has a man page
  (Craig Leres).

- Two new variables, backdoor_ignore_local and backdoor_ignore_remote,
  can be used to specify backdoor signatures that should be ignored
  if the server is local/remote.

- A bug has been fixed in which a "next" executed in the final iteration
  of a for loop would mangle the subsequent processing of the outer
  statements (Chema Gonzalez).

- Bug fixes for MIME and Base64 processing (Ruoming Pang).

- pcap.bro now builds its filter in the opposite order (restrict_filters
  first), which can improve packet filtering performance (Robin Sommer).

- A bug in &default has been fixed.

- More SSL autoconf tweaks (Robin Sommer).

- Portability for different pcap_compile_nopcap() calling sequences
  (Chema Gonzalez).

- Some tweaks for a minor reduction in memory consumption.

- A memory leak for secondary packet filters has been fixed.

- The localization of error messages (what script line they correspond to)
  has been improved.


0.8a58 Tue Dec 16 08:55:47 PST 2003

- Compatibility with older versions of libpcap (Chema Gonzalez).


0.8a57 Tue Dec  9 10:14:30 PST 2003

- The format of Bro's connection summaries is changing.  The new format
  looks like

	  1069437569.904605 0.230644 1.2.3.4 5.6.7.8 http 59377 80 tcp 610 275 S3 L

  That is, <timestamp>, <duration>, <originator address>, <responder address>,
  <service>, <originator port>, <responder port>, <originator bytes>,
  <responder bytes>, <connection state>, <flags>.  (Robin Sommer)

  The script variable traditional_conn_format=T specifies to use the old
  format rather than this new one.  This is *currently* the default, but
  will change soon to default to F instead.  If you have comments on this
  new format, we'd like to hear them.

- The SigAction's available in signatures.bro have been extended (Robin Sommer).
  SIG_FILE_BUT_NO_SCAN is like SIG_FILE but without any horizontal/vertical
  processing; SIG_LOG_ONCE logs only an alert only the first time it occurs;
  SIG_LOG_PER_ORIG logs only the first instance of an alert generated by a
  particular originator; SIG_COUNT has been renamed SIG_COUNT_PER_RESP; and
  SIG_SUMMARY suppresses logging of individual alerts but generates a
  per-originator summary.

- A new -p option for snort2bro tells it to only process signatures that
  include matching on payload (Robin Sommer).

- You can now explicitly include or exclude particular SIDs when
  running snort2bro by specifying a configuration file via -c (Robin
  Sommer).  The format is simple, just "include" or "ignore" followed
  by the SID number:

	# sid-526 BAD TRAFFIC data in TCP SYN packet 
	ignore	526 

	# sid-623 matches a null-flags stealth scan.  Include it even
	# if we build with -p, since it doesn't tend to generate any
	# false positives.
	include	623

  The new "snort2bro.cfg" file gives examples (i.e., the above).

- Bro can now serialize functions and event handlers, meaning that these
  can be passed as values between Bro's and dumped using -g (Robin Sommer).
  One of the main goals in supporting this is to allow in situ alteration
  of the Bro's configuration (e.g., you can edit a function and change its
  functioning and have a running Bro pick up the change without having to
  stop and be restarted).  Such dynamic reconfiguration is experimentally
  supported via -g <dir> (see below).

- &persistent state is now stored in the *directory* given by state_dir
  (default: "./.state"), one file per variable, rather than a single file
  (Robin Sommer).

- Storing &persistent state to disk is now done incrementally: after writing
  each file, there's a delay of state_write_delay (default: 0.1 secs) before
  working on the next file (Robin Sommer). This may introduce small
  inconsistencies, but prevents load spikes that can lead to packet drops.
  Currently, there is no mechanism to incrementally store a single variable
  (like a large table), although there is already some framework in place
  to eventually support this.

- The *experimental* new -g <dir> option dumps the script-level configuration
  (excluding things defined in internal default scripts like bro.init)
  into the directory <dir>. These files may be printed with "bro -x <file>",
  or copied into the state_dir of a running Bro, which will then pick up
  the change if it has loaded checkpoint.bro.  (When picking up changes,
  event handlers are always added, while functions, types, and variables
  replace the current ones).

- Table values are now incrementally expired rather than all at once
  (Robin Sommer).  That is, if the expiration timer goes off and hundreds
  of values should now be expired, the work of doing so is spread over
  chunks of table_expire_size (default: 50) at a time, separated by a
  delay of table_expire_delay (default: 0.1 secs).  This change aims to
  prevent large processing spikes that can lead to packet drops.

- New built-ins sub() and gsub() act like awk's functions of the same
  name, changing substrings (either first, or all) that match a given
  regular expression to a given target string.  (Note, the calling sequence
  differs from the order used by awk.)

- The new auxiliary script aux/scripts/mvlog is a handy way to manage
  checkpointed logs.  See the script for documentation.

- The &expire_func function now takes two arguments.  The second is
  of type "any" and corresponds to the index(es) of the element being
  expired.  To access the individual indices, you use a new assignment form:

	[a, b, c] = index_val;

  (where index_val is the second argument of type "any").  This assigns
  a to the first index, b to the second, and c to the third.  NOTE: the
  use of "any" types here is *temporary* and will be changing in the
  future to a general "tuple" notion.  (Robin Sommer)

- scan.bro and conn.bro have been reworked to consume less memory and to
  support more flexible state expiration (Robin Sommer).

- The new builtin rescan_state() causes Bro to re-read any persistent
  data values (Robin Sommer).

- snort2bro now supports continued lines ("\<newline>") (Robin Sommer).

- The calling sequences of the software_version_found() and
  software_parse_error() events has changed, and a new event,
  software_unparsed_version_found(), is invoked for raw version
  strings (i.e., the version string prior to the event engine
  attempting to parse it into version/major/minor) (Robin Sommer).

- Software version tracking for clients now tracks all versions, not just
  the latest version (Robin Sommer).

- alert_info records now include an optional field event_src, which is the
  source of the event if it was received from an external Bro (Robin Sommer).

- Regular expressions now support {} iteration values of 0, and generate
  better error messages.

- Output generated by icmp.bro is now redirected into an "icmp" log file
  (Robin Sommer).

- autoconf tweaks for configuring OpenSSL on Linux (Ruoming Pang, Robin Sommer).
  Tested on RedHat (thanks to Anton Chuvakin), Debian, FreeBSD, Solaris.

- You can now turn off using OpenSSL even if the OS supports it, via
  configuring with --disable-openssl (Robin Sommer).

- Variable size computations (per global_sizes()) are now more accurate
  (Robin Sommer).

- A bug with combining file encryption and log rotation has been
  fixed (Robin Sommer).

- A problem tracking directionality in signatures fixed (Robin Sommer).

- Bro now continues running if DNS is not functioning (Robin Sommer).

- Rewriter memory use has been significantly reduced (Ruoming Pang).

- Some bugs with -A/-w interaction have been fixed (Ruoming Pang).


0.8a48 Tue Oct 21 15:56:13 PDT 2003

- There is now a mechanism in place for multiple Bro's to communicate with
  one another via sockets (Robin Sommer).  *This is quite experimental at
  this point* and may have significant bugs and/or need significant
  enhancements.

  By loading listen-clear.bro or listen-ssl.bro, an instance of Bro starts
  to listen on a TCP port.  The first of these listens for unencrypted
  connections and the second for connections encrypted via OpenSSL.  To
  connect to a listening Bro, you load remote-clear.bro or remote-ssl.bro.
  For this connection, you specify which events you want to receive by
  giving a regular expression (e.g.  "/http_*/" for all HTTP events),
  although only those events for which you have defined a local handler
  are actually requested.  Communication is uni-directional in the sense
  that for a certain connection the events go only from Bro-A to Bro-B but
  not from B to A (you could set up a second connection for the other
  direction, though).

  The OpenSSL certificate used to authorize remote  Bro's is specified in
  the script variable "ssl_ca_certificate" and the private key and certificate
  for the local Bro via "ssl_private_key".

  If Bro A connects to Bro B, by default it sends over its capture_filter.
  But Bro B uses it only if it has loaded remote-pcap.bro.  This is the
  beginning of structuring inter-Bro trust mechanisms.  It is done via two
  new events, remote_capture_filter(ip: addr, p: port, filter: string) and
  send_capture_filter(ip: addr, p: port, s: string) : bool.

  The implementation forks a second process which does the socket
  communication, so that the main process should not be affected too much.
  The two processes communicate via a pipe.

  You can call is_remote_event() to determine whether the event currently
  being handled was originated by a remote Bro (if T) or the local Bro
  (if F).

  If a connection with a remote Bro terminates (for whatever reason), Bro
  may try to reconnect automatically.

  A new function, get_event_source(), returns a record event_source
  describing the source that raised the last event.

  See doc/ssl.txt for an explanation of how to create the keys/certificates.   

- A fledgling Gnutella analyzer has been contributed (Mark Allman).
  It generates the following events:

	event gnutella_text_msg(c: connection, orig: bool, headers: string)
	event gnutella_binary_msg(c: connection, orig: bool, msg_type: count,
					ttl: count, hops: count, msg_len: count,
					payload: string, payload_len: count,
					trunc: bool, complete: bool)
	event gnutella_partial_binary_msg(c: connection, orig: bool,
						msg: string, len: count)
	event gnutella_establish(c: connection)
	event gnutella_not_establish(c: connection)
	event gnutella_http_notify(c: connection)

- Bro now supports a secondary channel for acquiring packets (Chema Gonzalez).
  You access it by redef'ing the new global "secondary_filters", adding
  table[string] of event(filter: string, pkt: pkt_hdr).  The string
  specifies a tcpdump filter; anytime a packet matches the filter
  (including packets that would *not* otherwise be matched by
  capture_filter), then the given event handler is invoked.

  For example,

	  redef secondary_filters += {
		  ["tcp[13] & 7 != 0"] = rst_syn_fin_flag,
	  }
  
  will invoke rst_syn_fin_flag() anytime a TCP packet is seen for
  which the SYN/FIN/RST bits are non-zero.  The event handler will
  be passed the string "tcp[13] & 7 != 0" (so it can tell which
  of possibly multiple filters matched) and a pkt_hdr value, which
  includes the IP header and, if present, the TCP, UDP or ICMP header.

  Another example, and what motivated the addition, is:

	  redef secondary_filters += {
		["ip[10:2] & 0xffc == 0x398"] = sampled_1_in_1024_packet,
	  }

  which will invoke sampled_1_in_1024_packet() any time the given
  10 bits in the IP checksum match the pattern 0x398.  If the checksum
  field is uniformly distributed then this roughly corresponds to
  1-in-1024 random sampling.  (Chema has also developed BPF additions
  to support true random sampling.)

  See policy/secondary-filter.bro for examples.

- Bro now does a much better job of keeping track of how much memory
  has been allocated for different structures (Robin Sommer).

  This includes more accurate computations for global_size().

  In addition, if you redef "statistics_interval" to be a non-zero time
  interval, then with that periodicity a summary of memory usage (including
  memory used by event engine components) is dumped to the file
  "statistics_file".  In addition, at this point a "do_statistics" event
  is generated.  You can also call the new built-in statistics_update()
  to generate memory statistics on demand.

  The above structure is likely to change in the future.  statistics_interval
  will probably go away, to be replaced by either explicit calls to
  statistics_update() (which you can do on a timer if you like by using
  "schedule"), or by a built-in function that returns a record of all
  the statistics, that you can then format however you want.

- A major memory leak in HTTP analysis has been fixed (Ruoming Pang).

- New attributes &rotate = <interval expression> and
  &postprocessor = <string expression> can be associated with a file
  variable in order to specify how often the file should be rotated to a
  new filename on disk, and, when rotation occurs, the name of a shell
  program to run on the now-older version as a postprocessor (Robin Sommer).

- Similarly, log_postprocessor and log_rotate_interval specify the default
  values for files.  Unless redef'd, these themselves default to the empty
  string (= no postprocessing) and 0.0 seconds (no rotation).  (Robin Sommer)

- A new attribute, &encrypt, applies to variables of "file" type.  It specifies
  that the version on disk should be encrypted, using either the key specified
  as the value of the attribute, or, if no value is specified, using the
  value of the new script variable log_encryption_key.  The key is an OpenSSL
  public key; it's used to then embed a Blowfish session key.  (Robin Sommer)

  A new utility, aux/bdcat/bdcat ("Bro decrypt cat") can be used to decrypt
  the files. 

- The internal structure of TCP analysis has been significantly altered.
  Previously, TCP_Endpoint tracked endpoint state and TCP_EndpointContents
  (derived from it) did stream reassembly.  These have now been separated;
  TCP_Endpoint still tracks endpoint state, but TCP_EndointContents has
  become TCP_Contents, and is only loosely coupled with TCP_Endpoint.
  The reason for this change is to lay the groundwork for (1) applying
  an application analyzer to a connection after several packets for
  the connection have already been read, and (2) applying *multiple*
  application analyzers to a single connection.

- Bro now supports the universal hashing if USE_UHASH is defined
  (Ruoming Pang).  Universal hashing is a lighter-weight alternative
  to MD5 hashing that retains the property of being very difficult
  for an attacker to guess.  It comes in two flavors, a 32-bit
  version (which you get with USE_UHASH) and a faster 16-bit version
  (which you get if you also define USE_NH).  Bro will likely switch
  to using these by default in the near future, as their performance
  gain over MD5 is significant.

- New built-in functions srand() and rand() provide access to the
  usual C-style random number seeding & generation (Chema Gonzalez).

- You can now specify server/client addresses to leave in the clear in
  IP address anonymization (via the built-in variables preserve_orig_addr
  and preserve_resp_addr). Correspondingly, the built-in function for
  IP anonymization now takes a parameter to specify the type of the address
  (orig, resp, or other), instead of the method of anonymization
  (seq, random, prefix-preserving).  (Ruoming Pang)

- Trace anonymization now has prelminary support for handling TCP options
  via the new event "tcp_option" (Ruoming Pang).  It is only invoked
  if skip_event_tcp_option is F (it defaults to T).

- A new event, tcp_segment, is similar to the event "packet_content"
  but provides more information: is_orig (directionality), sequence
  number, and acknowledgement number (Ruoming Pang).

- ./configure finds OpenSSL if it's in some standard location.  Otherwise,
  you may specify it --with-openssl=<path>.  If OpenSSL is not available,
  Bro compiles cleanly without and gives warnings if a script tries use SSL.
  (Robin Sommer)

- The internal links in manual/entire.html have been fixed so they
  now work (Chema Gonzalez).

- A new policy script, blaster.bro, detects instances of the W32.Blaster
  worm (Robin Sommer).

- Signature files (for input to the signature engine) now reside in
  policy/sigs/*.sig.  This directory is now on the default $BROPATH.

- sig.ex.ssl-worm.bro and sig.ex.web-rules.bro have been updated
  to reflect changes in keywords (Robin Sommer).  They've been
  renamed to ex.ssl-worm.sig and ex.web-rules.sig and reside
  in policy/sigs/, per the above.

- The module facility has been changed to have its scope limited to
  the current file plus any additional files that are automatically
  processed based on its name plus $BRO_PREFIXES.

- As an experiment, ftp.bro has been modified to a style that includes
  using "module FTP".  Likely other policy scripts will be converted
  in the near future, and their variable names changed accordingly
  (e.g., "ftp_hot_files" will become "FTP::hot_files").

- The new "match" expression has been modified to allow any yield type
  rather than just strings.  It is likely to change significantly again
  soon.

- Iterating over multi-dimensional tables/sets now works (Robin Sommer).
  For example:

    const remote_peers_ssl : table[addr, port] of Peer &redef;
    [...]
    for ( [ip, p] in remote_peers_ssl )
        connect_ssl(ip, p, remote_peers_ssl[ip, p]$retry); 

- Checkpointing of persistent state on SIGHUP now happens via bro.init
  (Robin Sommer).  Not tested.

- fmt() now supports %x for hexadecimal formatting (Chema Gonzalez).

- alert.bro logs the source for remote alerts; by redefining the new
  "event_source_description: string" locally, you can specify how a
  Bro is to refered to on the remote side.  (Robin Sommer)

- software.bro now tracks HTTP clients, too (Robin Sommer).  This
  will be extended in the near future.

- Some FreeBSD 5.X porting tweaks (Sergey Osokin).


0.8a37 Wed Sep  3 23:20:21 PDT 2003

- A new attribute, "&persistent", indicates that a global variable's
  state should persist on disk (Robin Sommer).  Currently, they
  reside in the file "state.bst".

  For example, given the following script:

	global a = 0 &persistent;

	event bro_init()
		{
		print ++a;
		}

  then every time you run it, a increases by one.

  You can dump the state file using "bro -x state.bst <scriptname>".
  (There's also a partially-implemented XML dumper which you invoke via
  "bro -X state.bst <scriptname>".)

  If you send Bro a HUP signal, then it will synchronize all persistent
  state to disk (checkpoint) at that point.  More generally, the policy
  script can do so at any time by calling the new built-in checkpoint_state().

  By including the new policy script "checkpoint", you can have Bro
  automatically checkpoint every time checkpoint_interval elapses
  (default 15 minutes).

- You can also record events to disk by loading the "capture-events"
  policy script, or calling the new capture_events() built-in (Robin Sommer).
  The events are written to the file events.bst.  You can dump these
  using bro -x (or -X), as above.  You can also replay them using
  "bro -R <file>".  By replaying them using the same policy script as
  generated them, you should get the same output.  For example:

	bro -r tracefile myscript capture-events

	...

	bro -R events.log myscript

- An experimental module facility has been added (Umesh Shankar).

  The module facility implements namespaces. Everything is in some namespace
  or other. The default namespace is called "GLOBAL" and is searched by
  default when doing name resolution. The scoping operator is "::" as in
  C++. You can only access things in the current namespace, things in the
  GLOBAL namespace, or things that have been explicitly exported from a
  different namespace. Exported variables and functions still require
  fully-qualified names. The syntax is as follows:

  module foo;  # Sets the current namespace to "foo"
  export {
	int i;
	int j;
  }
  int k;

  module bar;
  int i;

  foo::i = 1;
  bar::i = 2;
  print i;    # bar::i (since we're currently in module bar)
  j = 3;      # ERROR: j is exported, but the fully qualified name
              #        foo::j is required
  foo::k = 4; # ERROR: k is not exported

  The same goes for calling functions.

  One restriction currently in place is that variables not in the "GLOBAL"
  namespace can't shadow those in GLOBAL, so you can't have

    module GLOBAL;
    global i: int;

    module other_module;
    global i: int;

  It is a little confusing that the "global" declaration really only means
  that the variable i is global to the current module, not that it is truly
  global and thus visible everywhere (that would require that it be in
  GLOBAL, or if using the full name is ok, that it be exported).  Perhaps
  there will be a change to the syntax in the future to address this.

  The "module" statement cuts across @load commands, so that if you say:

	module foo;
	@load other_script;

  then other_script will be in module foo. Likewise if other_script changes
  to module bar, then the current module will be module bar even after
  other_script is done.  However, this functionality may change in the future
  if it proves problematic.

  The policy scripts in the Bro distribution have not yet been updated to
  use it, but there is a backward-compatibility feature so that existing
  scripts should work without modification. In particular, everything is
  put in GLOBAL by default.

- The hooks are now in place for communicating events between running
  Bro's.  An experimental implementation of doing so (written by Robin
  Sommer) will be integrated shortly.

- A side-effect of getting those hooks in place is that event handlers must
  now be directly specified (by naming them) rather than indirectly
  (for example, by indexing a table whose value yields an event handler).
  This may be fixed soon.

- An experimental "match" expression scans a list of predicates to find
  the first one that's both true and has the highest priority (Umesh Shankar).

  The general form is:

	match some_record on {
		[ priority, class_name, predicate ],
		...
	}

  where "predicate" is evaluated in the context of the value (and type) of
  some_record.  For example, if some_record has a field "num_alerts"
  then predicate could be "$num_alerts > 5".

  "priority" is a non-negative integer (i.e., of type "count"), and,
  for now, "class_name" is a string.

  For example,

	global c: conn_id;

	c = [ $orig_h = 0.0.0.0, $orig_p = 0/tcp,
	      $resp_h = 1.1.1.1, $resp_p = 1/tcp ];

	print match c on {
		[ 2, "emptyweb", $orig_h == 0.0.0.0 && $resp_p == 80/tcp ],
		[ 1, "emptyhost", $orig_h == 0.0.0.0 ],
		[ 0, "should not match", 1 == 0 ]
	};

  will print "emptyhost".

  The initial intent behind this is to eventually provide more flexible
  customization of alert processing, though it clearly has broader
  applicable.  *It is very likely that the semantics and perhaps the syntax
  of "match" will change in the near future.*

- Bro's packet filter is now computed via pcap.bro (which is automatically
  included via bro.init).  It uses two new built-ins:

	precompile_pcap_filter(id: PcapFilterID, s: string):  bool
	install_pcap_filter(id: PcapFilterID): bool

  The first is for precompiling a pcap filter so it can be installed
  or turned off dynamically. Associating an ID with the filter, you can
  then activate the filter by calling the second function (installing a
  new filter replaces the current one).  (Robin Sommer)

  Be default, pcap.bro is responsible for building a pcap string based on
  the capture/restrict_filters defined by the various analyzers. It compiles
  and installs this filter, so there is no observable difference in usage
  to the old implementation, except capture/restrict_filter are now *tables*
  rather than strings, and are written as plural rather than singular.
  So the analyzers need to define something like this:

	[finger.bro]
	redef capture_filters += { ["finger"] = "port finger" };

  This then allows "finger" to be used as the name for the corresponding
  filter element (see the next item).

- load-level.bro is an experimental policy script for allowing Bro to
  shed or add on load (in terms of which protocols it analyzes).  It
  provides three interface functions for setting the current loadlevel:

	# level is LoadLevel1, ..., LoadLevel10
	function set_load_level(level: PcapFilterID): bool

	function increase_load_level()
	function decrease_load_level()

  load-levels.bro defines ten different load levels (from 1 to 10, with 10
  being the one with the most load imposed on the system), which are
  configured by defining capture_load_level/restrict_load_levels: Example:

	redef capture_load_levels += {
		["dns"]          = LoadLevel1,
		["smtp"]         = LoadLevel2,
		["http-reply"]   = LoadLevel3,
		["http-request"] = LoadLevel8,
	};

  This means for example: "include the capture_filter associated with
  'http-reply' if the current load level is 3 or below".  There's a similar
  mechanism for restrict_filters:

	redef restrict_filters += {
		["cs-only"] = "net 131.159.0.0/16",
	};

	redef restrict_load_levels += {
		["cs-only"] = LoadLevel7,
	};

  This applies the given restrict_filter if the current load level is 7
  or *above*.

  The pcap filters belonging to the ten load levels are built and pre-compiled
  on startup. The three functions shown above just call install_pcap_filter()
  then.  (Robin Sommer)

- drop-adapt.bro tries to adjust the load level based on the current
  packet drop rate (Robin Sommer).

- synflood.bro is an experimental policy script for detecting SYN floods.
  It is not yet documented, other than brief comments in the script.
  (Robin Sommer)

- Subnet lookups now use Patricia trees instead of brute-force, which should
  make them significantly more efficient for large lists, e.g., local nets.
  (Robin Sommer)

- Due to the addition of modules, which use a C++-style "::" scope
  designator, you now need to use "0x" to introduce IPv6 address constants
  that use "::" to specify a series of empty octets.  For example, you
  used to be able to specify "deadbeef::cafe", but now this needs to be
  "0xdeadbeef::cafe".  Note that "1::2" can still be written without needing
  a "0x" prefix; it's just hex constants that start with letters that need
  the prefix.

- A new built-in, escape_string(), takes a string and returns a copy of
  it that uses escape sequences for any problematic characters (Robin Sommer).

- A number of low-level bug fixes and portability tweaks (Robin Sommer,
  Ruoming Pang, Christian Kreibich, Chema Gonzalez).

- A new timer, status_update_timer, fires for each ongoing connection
  every connection_status_update_interval seconds (default = 0 seconds,
  which means "don't fire).  (Robin Sommer)

- An additional Bro-level packet filter can filter/sample packets
  based on their src/dest ip/subnet (using a Patricia tree for
  efficiency; Robin Sommer). install_src_addr_filter(ip, flags, p) drops
  packets originating from ip with probability p (0..100) if none of the
  given TCP flags is set. install_src_net_filter, install_dst_addr_filter
  and install_dst_net_filter" work similarly.  The corresponding "uninstall_*"
  functions remove the filters again.

- The @if/@else/@endif mechanisms have been made more powerful (Robin Sommer).

- New configure option --enable-debug to compile without optimization
  (Robin Sommer).

- Small tweaks to the mpatrol support (Robin Sommer).

- SMTP is now one of the services which can use a small inactivity timeout
  in inactivity.bro (Robin Sommer).

- Alerts for signatures which are triggered by worms may be suppressed if
  we already know that the triggering host is indeed infected by a worm
  (Robin Sommer).

- Matches of a signature can now be counted per destination host
  (Robin Sommer).

- snort2bro now ignores sid-526 ("BAD TRAFFIC data in TCP SYN packet";
  Robin Sommer). Due to different semantics of Bro (matching stream-wise)
  and Snort (matching packet-wise) this signature generates a lot of
  false positives.


0.8a34 Sun Jul 13 09:11:32 PDT 2003

- The new "subnet" type corresponds to a CIDR prefix (Robin Sommer).
  You can use the '/' operator to convert an address to a subnet.  The
  "in" operator tests whether an address matches a subnet (e.g., "1.2.3.4
  in 1.2.255.255/16" yields T).

  You can index tables with index type "subnet" using addresses to retrieve
  the table entry with the longest-matching prefix for that address, and
  a number of tables/sets in the default scripts have been converted to
  this form.  So for example the local_16_nets and local_24_nets variables
  have been replaced by local_nets, which has type "set[subnet]",
  is_local_addr() now refers to it, and skip_scan_nets_{16,24} have
  likewise been consolidated into skip_scan_nets.

  One present deficiency is that subnets can't be used in tables/sets
  with multiple indices.  Fixing this is going to take some time ...
  Another deficiency is that the longest-match lookup is not very
  efficient.

  Caution: this feature has been only briefly tested with IPv6.

- ALERT now generates an event "alert_action" which includes the alert_info
  and AlertAction associated with an alert.  You can define your own handler
  to further customize alert processing.

- The "snort2bro" conversion utility has been extended to recognize some
  of the new Snort options (depth, distance, within; Robin Sommer).  For
  example:

      alert .... ( ... content:"|00 01 86 B8|";
		       content:"|00 00 00 02|"; distance:4; within:4;
		       content:"%x %x"; distance:16; within:256;
		       ... )

  is converted to:

      signature sid-1891 {
	  ...
	  payload /.*\x00\x01\x86\xB8.{4}\x00\x00\x00\x02.{16}.{0,251}%x %x/
	  }

  Note that not all of the new Snort options are supported yet.

- You can refer to script variables within signatures (Robin Sommer).
  For example, the following is now possible:

	signature sid-547 {
	  ip-proto == tcp
	  src-ip != local_nets
	  dst-ip == local_nets
	  dst-port == 21
	  event "FTP \"MKD  \" possible warez site"
	  tcp-state established
	  payload /[mM][kK][dD]  /
	  }

  This makes the signatures independent of the environment ("local_nets"
  in the above examle).  snort2bro now converts some well-known Snort
  variables into corresponding Bro variables, a number of which are listed
  in policy/site.bro.

- The default action for signature matches is now logging insted of only
  writing it to a file (Robin Sommer).

- You can now use the '^' and '$' regular expression operators inside
  subpatterns.  For example, /foo$/ | /^bar/ works now.

- You can now use "msec" for milliseconds (in addition to "usec" for
  microseconds, "sec" for seconds, etc).

- The log_file_name and open_log_file functions are now redef'able
  if you want to change their behavior.

- Bro now exits more cleanly upon exhaustion of memory (Robin Sommer).

- A bug was fixed for the case of getrlimit() returning RLIM_INFINITY for
  the maximum number of open files.

- Numerous additions of std:: scoping to address porting issues (Robin Sommer).

- gcc 3.X and Solaris portability fixes.

- A new event RemoteWorm is the complement of LocalWorm (Robin Sommer).

- A bug in which the FTP analyzer would complain about failing to be able
  to look up a connection has been fixed (Robin Sommer).

- You can now configure Bro using "--with-mpatrol" to activate MPatrol
  debugging hooks.  When built in this mode, -m dumps the leak table
  and -M shows unfreed parts of the heap.  These dumps can also be
  triggered using SIGUSR1 / SIGUSR2, respectively.  (Robin Sommmer)

- A script function get_contents_type() which returns the type of reassembling
  being done for a connection (none/orig-side/resp-side/both; Robin Sommer).

- A minor bugfix for the regular expression matcher (Robin Sommer).


0.8a32 Thu Jun 12 23:33:21 PDT 2003

- The low-level hash functions have been reimplemented to use HMAC-MD5 to
  counter the "algorithmic complexity attacks" discussed in the USENIX
  Security 2003 paper by Scott Crosby & Dan Wallach
  (http://www.cs.rice.edu/~scrosby/hash/) (Ruoming Pang and Vern Paxson)

- Ruoming Pang has made extensive changes to the anonymization/transformation
  framework.  A paper describing it will appear in SIGCOMM 2003, and will
  be included with the Bro distribution once the final copy is ready.

- Internal hash tables now resize incrementally, and more quickly.
  This makes a big difference in avoiding lengthy processing pauses when
  processing large volumes of traffic. (Craig Leres)

- gcc 3.1 and Linux portability tweaks.

- The calling sequence of http_request() has changed.  It now includes both
  the original URI and the escaped URI.  This was made explicit because
  we found that if only the original was passed in, it was too easy to forget
  to expand the escapes in it; and if only the escaped was passed in,
  some attacks could be missed. (Ruoming Pang)

- Signature rules can now refer to "finger" payload as well as HTTP and FTP
  (Robin Sommer).

- The signature engine now includes keywords "dst-ip", "dst-port",
  "ip-proto", "src-ip", "src-port".  (Robin Sommer)

- Packet sorting now defaults to off.

- The FTP analysis now attempts to track the current directory (Ruoming Pang).

- A number of scan detection additions have been added (Scott Campbell):

	activate_landmine_check = F
		if T, then access to more than landmine_thresh_trigger
		addresses in landmine_address (a set of addresses)
		constitutes a scan

	activate_priv_port_check = T
		if T, then inbound access to more than priv_scan_trigger
		privileged ports (i.e., ports < 1024) is considered a port
		scan.  You can exclude particular services from the count
		via troll_skip_service, which defaults to { smtp, ftp,
		ssh, 20/tcp, http }.

- The SMTP analysis now includes a new alert, HotEmailRecipient, which
  is triggered by mail sent to any of the addresses specified in
  the pattern hot_recipients (which defaults to a bare /@/).

- The new built-in cat_string_array_n() is like cat_string_array() except
  it takes starting and ending indices as additional arguments (Ruoming Pang).

- The new built-in sort_string_array() takes a string array and returns
  a sorted version (Ruoming Pang).  It currently uses an N^2 algorithm
  so shouldn't be used on large arrays.

- The new built-in subst() substitutes all instances of a given pattern
  in a given string with another string (Ruoming Pang).  For example,
  subst("fooboo", /oo/, "xxx") returns "fxxxbxxx".

- The new built-in cut_tail() trims characters from the end of a string
  (Ruoming Pang).  For example, cut_tail("fooboo", 2) returns "foob".

- sub_bytes() can now take a negative argument, which is with respect to
  the end of the string rather than the beginning (Ruoming Pang).

- The new built-in md5_hmac() returns an HMAC-MD5 hash of the given string
  (Ruoming Pang).  The HMAC secret key is generated from available entropy
  when Bro starts up, or it can be specified for repeatability using
  the new -K flag.

- The new built-in split_all() is like split() except that the returned
  array also includes the parts of the string that match the pattern
  used for the splitting (Ruoming Pang).

- The new built-in split_n() splits up to a given number of instances,
  optionally returning the parts matching the split pattern (Ruoming Pang).

- The new built-in split_complete() is the most general of the split functions
  (the others are written in terms of it).  It splits a given string
  with separators that appear in either a given pattern or a given set
  of strings, optionally including the text matching the separators in
  its return value, and optionally limiting the number of matches to
  a specified maximum.  (Ruoming Pang)

- The new built-in to_string_literal() returns an escaped version of a string
  suitable to feeding into Bro's parser.  For example, calling it on
  "foo\nbar" (where '\n' is an embedded newline) returns "foo\x0abar".
  (Ruoming Pang)

- subst_substring() has been renamed subst_string (Ruoming Pang).

- unescape_URI() no longer takes the connection as an argument (Ruoming Pang).

- config.guess and config.sub updated

- String escape expansion has been refined (Ruoming Pang) and some bugs
  fixed.  It now supports a format that's consistent with Bro string literals.

- Scanning of octal and hex escape sequences now stops after 3 or 2
  characters, respectively.  For example, "\0007" now expands to a
  NUL followed by the character '7'.  (Ruoming Pang)

- Bug fixes for handling of bare <CR> and <LF>'s when <CRLF> expected
  (Ruoming Pang), and associated "weird" events.

- A bug in signature matching reassembly has been fixed (Robin Sommer).

- A bug in reporting "bad connection size" for connection sizes > 2GB
  has been fixed.

- A bug in computing sizes for large partial connections has been fixed.

- A bug in delayed generation of connection closed events has been fixed.

- A framework has been added for compression of some "weird" events
  to only be generated once-per-connection (Ruoming Pang).

- Some of the "weird"'s generated by the HTTP analyzer have been regularized.

- Some memory management fixes.

- A performance problem in allocating List objects was fixed.

- The copyright dates have been updated (Craig Leres).


0.8a22 Wed Jan 15 16:47:18 PST 2003

- There is now a "conditional compilation" feature similar to C's preprocessor
  (contributed by Robin Sommer).  "@if (expr)" will include the remaining
  input lines up to "@endif" only if the given expression evaluates to true.
  "@ifdef (id)" will do so only if the given identifier has been defined,
  and "@ifndef (id)" only if it has not been defined.  There's currently
  no "@else" directive, and conditionals cannot be nested.  Both of these
  will be added in the future.

- New built-in functions (contributed by Ruoming Pang): parse_dotted_addr
  takes a string in A1.A2.A3.A4 form an returns a corresponding "addr" value.
  unescape_URI takes a URI that possibly has embedded escape sequences
  ("%61" for 'a') and expands the sequences.

- The URIs in HTTP requests are no longer automaticaly escaped.  Instead,
  they need to be manually escaped using unescape_URI.  As this is likely
  error-prone (users forgetting to do so), the plan is to change the
  calling sequence of http_request in the near future so that *both* the
  escaped and the unescaped URI are passed in.

- A number of g++ 3.0 porting tweaks have been added (thanks to Sean Irvine).

- The term "rule" has been systematically changed to "signature" (Robin Sommer).

- The functionality of monitoring packet drops has been moved into its
  own policy script, netstats.bro (Robin Sommer).

- A number of rewriter and location bugs have been fixed.

- Some missing HTTP "weird"'s have now been included in weird.bro.


0.8a21 Thu Nov 28 23:31:38 PST 2002

- A new mechanism will reorder packets within a window of packet_sort_window
  if due to their sequence numbers they appear to be out of order.  This
  can help a great deal when reading from multiple NICs.  (Contributed by
  Ruoming Pang.)

- A bug in regular-expression matching in which for example
  "2a1" == /[0-9]+/ would evaluate as true was fixed (Ruoming Pang).

- There's now a rewriter/anonymizer for FTP (Ruoming Pang).

- The rewriter/transformation framework now supports a notion of
  delaying transformation decisions until later (Ruoming Pang).

- An incompatibility with bison 1.75 has been identified.  The problem
  is that bison 1.75 generates incomplete location information for empty
  grammar productions (those for which the RHS is empty).  This will
  be fixed soon.

- Some bugs in the signature engine have been fixed (Robin Sommer).

- The sources no longer use the not-fully-portable hash_map template
  (Umesh Shankar).

- Some bugs with the debugger getting confused about source line number
  locations, and also with the -t option, have been fixed (Umesh Shankar).

- If a content gap occurs wholly inside an HTTP entity, then the analyzer
  will skip over the entity and continue processing any subseqeuent HTTP
  items, rather than giving up on processing the connection (Ruoming Pang).

- The following new built-in functions have been contributed by
  Ruoming Pang:

	function cat_string_array(a: string_array): string
	function split_all(str: string, re: pattern): string_array
	function strstr(big: string, little: string): count
	function subst_substring(s: string, from: string, to: string): string
	function int_to_count(n: int): count
	function fmt_ftp_port(a: addr, p: port): string


0.8a20 Sun Nov 17 20:09:31 PST 2002

- This is the first "public" Bro release (one accessible directly via
  the Web).  The README has been correspondingly updated.

- The user manual has been significantly expanded, and source for it is
  now included in the release.

- Some "active mapping" bug fixes (contributed by Umesh Shankar).

- The configuration script now checks for the presence of the necessary
  BIND libraries/headers, and also for bogus --enable options (contributed
  by Craig Leres).

- backdoor.bro now includes a tcpdump filter for detecting (some) KaZaA
  traffic.

- http-reply.bro now tracks the sizes of request/response entities.

- http-request.bro now treats an URL accessing variants of /etc/netconfig
  as sensitive.


0.8a18 Sun Oct 27 15:28:23 PST 2002

- Improvements to the performance of the SMTP analyzer (Ruoming Pang).
  A new function, skip_smtp_data, skips over the content of an SMTP
  message.

- If you're doing trace rewriting and specify -w but don't specify -A,
  then the rewritten traffic goes to the -w save file, along with the
  usual non-transformed traffic (Ruoming Pang).  If a connection is
  being transformed (which is specified by the new rewriting_smtp_trace
  and rewriting_http_trace globals), then only its transformed packets
  are written to the -w file; not the original packets.  This allows
  a potentially large degree of trace compression.  There's also a
  new variable omit_rewrite_place_holder, which directs that when rewriting
  a trace, packets that are placeholders for now-missing packets (because
  you've excised content) are *not* generated.  This diminishes the
  timing fidelity of the transformed trace, but also can save a large
  amount of space.

- SMTP relay analysis is now standalone from regular SMTP analysis
  (Ruoming Pang).

- Some memory management and error propagation fixes (Ruoming Pang and
  Vern Paxson).


0.8a16 Wed Oct 23 23:48:40 PDT 2002

- "--enable-brov6" was broken by some recent changes.  It now works again.

- Some "make distclean" tweaks.

- Error checking for "redef enum".


0.8a15 Tue Oct 22 00:02:51 PDT 2002

- Fixed Makefile bug which left out wide-spread dependencies on
  event.bif.netvar_h.


0.8a14 Mon Oct 21 01:16:46 PDT 2002

- The "add" statement has been implemented.  This allows you to add
  elements to sets.  A bunch of policy scripts that used to use
  "table [] of bool"'s for this purpose have been modified to instead
  use sets.

- You can now extend the elements of an enum using "redef enum foo += { ... }"
  where the additional names are listed in the braces (contributed by
  Umesh Shankar).  A number of policy scripts have been tweaked to use
  this for the (fairly) new Alert enum.  This allows you to create new
  Alert's without alert.bro having to know about them.

- Some bugs in identifying error locations have been fixed.

- -A now supports anonymizing IP addresses (contributed by Ruoming Pang).
  This includes four new functions: preserve_prefix, preserve_subnet, and
  preserve_net (which direct that the corresponding prefix/net/subnet
  not be anonymized) and anonymize_addr (which returns the anonymized
  version of the given address.

- Some bugs in HTTP -A have been fixed (thanks to Ruoming Pang).

- The beginnings of support for CIDR prefixes has been added by
  Ruoming Pang.

- Porting tweaks (use of map's rather than hash_map's, courtesy of
  Umesh Shankar; libedit comments fixed).

- http-detail.bro has gone away.

- Some more copyright and $Header -> $Id fixes.

- There is now a function string_cat() which concatenates two strings
  and returns the result (contributed by Ruoming Pang).  This function
  is useful in place of the regular cat() because the latter will
  expand escape sequences etc.  It probably shouldn't, and that may
  change in the future.


0.8a11 Sun Oct 13 10:53:07 PDT 2002

- The framework for defining built-in functions has been extended
  so it's also now the way to specify the interface to event handlers.
  See event.bif.  (Contributed by Ruoming Pang)

- A new policy script, http-abstract.bro, can be loaded when doing HTTP
  transformation (via http-rewriter.bro), contributed by Ruoming Pang.
  It trims HTTP text responses to the first "http_abstract_max_length" bytes.

- A new built-in, skip_http_entity_data, specifies that the entity
  associated with the current HTTP request/response should be skipped
  (Ruoming Pang).

- More changes have been made to the mechanisms for tracking source
  file locations in order to pinpoint error messages.  If you encounter
  problems, please let me know.

- If you try to configure but your resolve library lacks res_mkquery(),
  this is now a fatal error (Craig Leres).  Ideally, someone will modify
  DNS_Mgr.cc to work even if only blocking DNS is available ...

- In most (not all, yet) of the sources, the copyright notices have
  been updated, $Header has been changed to $Id, and config.h is included
  (Craig Leres).


0.8a10 Tue Oct  8 16:05:42 PDT 2002

- The way that Bro tracks the source-file location associated with
  different objects has been tweaked, with a resulting savings of
  about 10% in Bro's memory use and a smidgen of CPU time, too.

- Built-in functions now are better about identifying source-file locations
  associated with errors.

- The http.$BRO_ID log file format has changed to no longer track
  individual request streams.  (So it is now closer to how it used
  to operate in 0.7.)

- The autoconf setup has been tweaked so that you use:

	--enable-brov6              enable IPV6 processing
	--enable-activemapping      enable active mapping processing
	--enable-expire-dfa-states  enable DFA state expiration

  rather than --with-XXX as was the case in 0.8a9.


0.8a9 Mon Oct  7 10:15:12 PDT 2002

- A bunch of configuration/autoconf/portability tweaks (Craig Leres).
  These include compiling under Solaris.

  When running ../configure, you can now specify:

	--with-brov6              enable IPV6 processing
	--with-activemapping      enable active mapping processing
	--with-expire-dfa-states  enable DFA state expiration

  You no longer set these via editing the Makefile.

- Some bugs with HTTP 1.0 keep-alive connections fixed (Ruoming Pang).

- The "hf" suite of utilities is no longer distributed in aux/.  Instead,
  get it directly from ftp://ftp.ee.lbl.gov/hf.tar.gz.

- bro_logchk has been renamed bro-logchk.pl and is now distributed in aux/
  (it was supposed to be before, but was inadvertantly left out of the list
  of distribution files).


0.8a7 Fri Oct  4 22:24:30 PDT 2002

- HTTP reply analysis has been split into a number of policy script files:

	http-body.bro
	http-detail.bro
	http-entity.bro
	http-event.bro
	http-header.bro
	http-reply.bro
	http-rewriter.bro

  so you can readily trade off how detailed the HTTP processing is vs.
  the CPU/memory it consumes (contributed by Ruoming Pang).

- Bro now generates login_prompt events when the $TTYPROMPT environment
  variable is passed during a login session, in order to detect the
  recently publicized Solaris remote /bin/login exploit.

- Ruoming Pang has extended the framework for defining "rewriter" functions
  to now also serve as the way to define general built-in functions.

- bro.init has been rearranged to have a more logical structure (courtesy
  of Ruoming Pang).

- Craig Leres contributed a number of portability & autoconf tweaks.

- Craig Leres has extended nb_dns.{h,c} to support IPv6 DNS lookups.
  Bro does not yet take advantage of these extensions.

- The beginnings of portability to gcc 3.2 were added.  There unfortunately
  is more work to do here!

- The README has finally been updated to have more correspondence with
  the 0.8 release.


0.8a6 Wed Oct  2 18:58:12 PDT 2002

- Upgrade to autoconf 2.53 (Craig Leres).


0.8a5 Tue Oct  1 19:04:53 PDT 2002

- The regular expression matcher how has a mechanism to stop scanning when
  no further match is possible (Robin Sommer).  If you find problems with
  regular expression matching, especially if you're using EXPIRE_DFA_STATES,
  please let us know!

- Rule/signature files are now searched for using $BROPATH (Robin Sommer).
  In addition, you can define a list of signature files to incorporate
  using the new global "signature_files".  For example,

	redef signature_files += "web-stuff"

  will look for web-stuff and web-stuff.sig along $BROPATH.

- The tcp_deliver_undelivered variable is now tcp_match_undelivered and
  only applies to delivering otherwise-undelivered data to the signature
  matcher; such data is *not* delivered to any associated analyzer.
  (Robin Sommer)

- The framework for tracking version numbers now allows negative as
  well as positive versions (Robin Sommer).


0.8a4 Tue Oct  1 15:54:58 PDT 2002

- Support for extracting the contents of HTTP replies (and POST's),
  and for transforming/anonymizing HTTP traffic, contributed by Ruoming Pang.

- Some minor internal tweaks to the timer management to help track patterns
  of timer expiration.


0.8a3 Mon Sep 23 22:48:07 PDT 2002

- HTTP reply handling refined in policy scripts.

- New built-in functions to_int(), to_count() convert strings to
  integers/counts.

- Bug fixes for DNS_Mgr and IPv6.

- AckAboveHole alerts now ignored (just written to alert.$BRO_ID) by default.

- Packets belong to ignored connections (for example, partial connections if
  partial_connection_ok is false) are no longer recorded to the save file.

- Some minor formatting/naming tweaks.


0.8a1 Sat Sep 21 22:09:23 PDT 2002

- IPv6 support enabled if you build using -DBROv6.  Deficiencies: Bro
  doesn't yet look up hostnames for AAAA records; no handling of extension
  headers (if you have traces of these, please send them to me!); no
  handling of FTP PORT/PASV w/ IPv6 addresses (again, if you have traces,
  please send them!); DNS analyzer doesn't understand AAAA yet (again,
  please send me traces!); you have to change the capture_filter line
  in tcp.bro (as indicated in the script) in order to process TCP traffic,
  due to deficiencies in libpcap's support for IPv6.

- Bro is migrating towards a more structured way of handling log messages /
  alerts.  Analyzers now @load alert.bro, which has a function ALERT()
  for processing alerts.  Soon this function will provide a variety of
  filtering/processing hooks; expect changes.

- Bro now has an HTTP response analyzer (contributed by Ruoming Pang).
  The HTTP policy scripts have been split up into http.bro (just general
  definitions), http-request.bro (handles requests; loaded by http.bro),
  http-reply.bro (handles replies; you need to explicitly load this), and
  http-detail.bro (handles individual headers).  http-reply.bro will be
  undergoing some significant reworking in the near future; probably the
  scripts will be merged back into a single http.bro plus http-detail.bro.

- ssl-worm.bro contains a prototype policy script for detecting the
  Scalper SSL worm (contributed by Robin Sommer).  It uses the signature
  file sig.ex.ssl-worm.bro.  If someone has traces of Scalper in action
  to send us, that would be great.

- A new policy script, contents.bro, extracts the contents of each
  Bro connection into its own pair of files (one file for each
  direction).  Use in conjunction with -f or discarder_XXX() to
  extract specific connections.

- A new built-in function, strcmp(), returns the usual comparison between
  two strings (contributed by Robin Sommer).

- A new event, content_gap(), is generated when Bro detects that it is
  forced to skip over data in a reconstructed TCP stream because it is
  missing from the packet input.

- BIND8 is no longer included with the distribution.  If this causes you
  problems, let me know.

- aux/scripts/bro_logchk is a Perl script for parsing Bro HTTP & FTP logs
  (contributed by Jim Barlow).

- You can now compare addresses to see which is larger.  a < b means
  that in network order, the octets making up 'a' are ordered before
  those for 'b'.  E.g., 128.2.3.4 < 128.2.3.5 < 129.0.0.1.  Note that
  IPv4 addresses are all < IPv6 addresses (other than IPv4 addresses
  that are embedded in IPv6 addresses, e.g., ::128.2.3.4 < 128.2.3.5).

- Serious bug in TCP option handling fixed.

- Some bugs in CRLF handling fixed (courtesy Ruoming Pang).

- Bug in the implementation of &optional fixed.

- Bug in computing memory statistics when not reading packets (from
  an interface or the trace file) fixed.

- You can now include a trailing comma after the last item in an
  "enum" enumeration list.

- port-name.bro now maps 389/tcp to "ldap".

- A bug has been fixed in loading files multiple times


v0.7a175 Thu Aug 29 21:14:34 PDT 2002

- bro -s <file> reads in a *signature* file to search for regular expressions
  in packet payloads or TCP byte streams (written by Robin Sommer).
  See policy/rules.bro for an example of a policy script for processing
  the matches.

  Note that this feature is experimental, and will be evolving in the
  near future.

- The python script "snort2bro" reads in Snort signatures and translates
  them into Bro signature rules, suitable for processing using -s.

  An example of its operation is seen by running

	python snort2bro < sig.ex.web-rules.snort

  which, after reading in sig.ex.classification.config and
  sig.ex.reference.config, generates the output given in
  sig.ex.web-rules.bro, which is suitable to use as input to
  bro -s.

- bro -d invokes a gdb-like debugger (written by Umesh Shankar).  You can
  set breakpoints and watchpoints, examine tracebacks, print Bro expressions,
  and the like.  Type "help" for on-line help.

- bro -t <tracefile> turns on tracing of the policy script execution,
  written to the given file.

- Bro now includes an SMTP analyzer, which includes processing MIME
  message bodies (written by Ruoming Pang).  See smtp.bro and mime.bro
  for related policy scripts.  smtp.bro includes several experimental
  techniques for detecting mail relaying.

- You can now define enumerated types, such as

	type scanner_type: enum {
		SCANNER_STEALTH, SCANNER_HIGH_SPEED, SCANNER_AMBIGUOUS,
	};

  Enumerated types can be compared for equality with one another, and used
  as table indices, but cannot be converted to/from integers.

- bro -A <file> invokes an experimental, general trace transformation/
  anonymization framework (written by Ruoming Pang) which writes a modified
  tcpdump trace file from the input (which can be the network or another
  trace file) with potentially extensive modifications to the recorded
  packets.

  Transformers are built from .rw files (currently, {finger,ftp,ident,smtp}.rw),
  which are processed by the utility "rwcl" to generate both event engine
  analyzer components and rewriter policy scripts (for example, after
  configuring and building Bro, you'll find the scripts
  policy/{finger,ftp,ident,smtp}.rw.bro).

  See policy/smtp-rewriter.bro for an example of a policy script that
  performs transformation/anonymization.

- New built-ins:

	split(s: string, p: pattern): string_array;

	  takes a string and splits it into pieces at each occurrence of
	  the regular expression pattern p.  (The functionality is like
	  that in awk.)  It returns a string_array, which is a table[count]
	  of string that is indexed starting at 1, giving the different
	  pieces.

	  For example,

		split("foobar", /o/)

	  returns a 3-element table, for which [1] is the string "f",
	  [2] is the string "" (empty), and [3] is the string "bar".

	split1(s: string, p: pattern): string_array;

	  split1() does the same thing as split(), but only performs splitting
	  at the first occurrence, so it returns either a one-element table
	  (if the pattern doesn't appear in the string) or a two-element
	  table.  split1("foobar", /o/) returns a 2-element table for which
	  [1] is "f" and [2] is "obar".

	md5_hash(s: string): string

	  returns (in human-readable form) the MD5 hash of a given string.

	  So, for example,

		md5_hash("foobar")

	  yields "3858f62230ac3c915f300c664312c63f".

	to_addr(s: string): addr

	  takes a string representing an address in "dotted quad" format
	  and returns the correponding "addr" value.

	set_buf(f: file, buffered: bool)

	  sets the given file to have its writes buffered or unbuffered
	  depending on the value of "buffered".  It does not return a value.

	connection_exists: function(c: conn_id): bool

	  returns T if the given connection identifier corresponds to a
	  currently instantiated connection (one for which the event engine
	  has state), F otherwise.

	lookup_connection(c: conn_id): connection

	  returns the "connection" record associated with the given
	  connection identifier, or a fatal run-time error if there
	  isn't one.

	set_inactivity_timeout(c: conn_id, timeout: interval): interval

	  sets the inactivity timeout for the given connection to the
	  given interval, returning the old interval.

	  If the interval is non-zero, then when no packets have been
	  processed for a connection after that much time has elapsed,
	  the connection is deleted, and an "inactivity_timeout" event
	  generated.

	get_matcher_stats(): matcher_stats

	  used for gathering statistics about the signature matcher

	rewriting_trace(): bool

	  returns T if -A was specified (anonymize/rewrite a trace),
	  F otherwise.

- New events:

	connection_state_remove(c: connection);

	  Invoked when the event engine has removed the connection from
	  its state.

	connection_SYN_packet(c: connection, pkt: SYN_packet);

	  Invoked for each SYN/SYN-ACK packet.

	connection_timeout(c: connection);

	  Invoked when the event engine times out a connection - for
	  example, because the originator sent a SYN that was never
	  answered, so the connection was never established.

	connection_reused: event(c: connection);

	  Invoked when the event engine decides that a new SYN for
	  an existing connection reflects a reuse of the connection
	  four-tuple, rather than belonging to the existing connection.

- New globals:

	const ignore_checksums = F &redef;

	  If true, then the event engine does not verify checksums (and
	  hence will not discard packets with bad checksums).

	const tcp_deliver_undelivered = F &redef;

	  If true, then when the event engine closes a connection, if
	  that connection has a chunk of data not yet delivered to its
	  analyzer (which only happens if the data is above a sequence
	  hole, indicating either a packet filter glitch or a protocol
	  botch), then the undelivered data will at that point be delivered
	  to the connection's analyzer.

	const tcp_reassembler_ports_orig: set[port] = {} &redef;
	const tcp_reassembler_ports_resp: set[port] = {} &redef;

	  Sets of ports for which, if a connection has the corresponding
	  originator/responder port, then the event engine will reassemble
	  the byte stream of the connection.

	  Normally, the event engine reassembles byte streams for any
	  connection for which there's an analyzer, and otherwise doesn't.
	  These variables can be used to force reassembly for the originator
	  or responder side (respectively) of connections for which there
	  isn't otherwise an analyzer.  This is useful when doing signature
	  matching on reassembled byte streams, for protocols that are
	  not otherwise analyzed by the event engine.

	const table_expire_interval = 1 min &redef;

	  How often to check table entries to see whether they've expired
	  (see &read_expire, etc., below).

	const requires_trace_commitment = F;

	  If true, then when rewriting/anonymizing traces, nothing will
	  actually be written to the edited trace file unless you call:

		rewrite_commit_trace(c: connection, commit: bool, future: bool)

	  If "future" is true, then future rewritten packets will be
	  automatically commited; otherwise, writing them to the trace
	  file requires another explicit rewrite_commit_trace() call.

	const inactivity_timeout = 0 secs &redef;

	  As noted above, when a connection becomes inactive, time it out
	  after this interval.  If 0 secs, then don't time it out.

- An SSH analyzer extracts SSH client/server version information.  See
  ssh.bro for the related policy script.

- There's now a (very) simple TFTP analyzer available in tftp.bro.

- You can now set the global "frag_timeout" to an interval which controls
  how long fragments are kept before discarding them (contributed by Ashley
  Thomas).  If you don't set the global, or set it to 0.0 sec, then fragments
  are kept around indefinitely.

- An implementation of an experimental anti-evasion technique, "active
  mapping", has been written by Umesh Shankar.  It is not yet ready for
  general use, and isn't compiled in unless -DACTIVE_MAPPING.

- Four new attributes can now be associated with tables (implemented
  by Robin Sommer): &read_expire, &write_expire, and &create_expire
  will delete table entries after a given interval has elapsed since
  the table entry was last read, written, or created.  For example:

	global a: table[addr] of count &default=0 &create_expire = 5 sec;

  will delete each entry added to it 5 seconds after the entry was added,
  regardless of subsequent read/write activity to the element.

  &expire_func allows you to associate a function with the table such that
  whenever an entry expires, the function is invoked.  It's passed the
  value of the table entry (not the index - perhaps this should be changed),
  and returns an "interval" value.  If the interval is <= 0.0 seconds, then
  the table entry is immediately deleted.  Otherwise, it is deleted after
  the given interval has elapsed.

- When listing multiple attributes, you no longer separate them with
  commas.  For example, if you used to have:

	global a: table[string] of count &default=0, &redef;

  you now need to use:

	global a: table[string] of count &default=0 &redef;

- You can now construct records using

	[$field1 = <expression>, $field2 = <expression>, ...]

  Such record values can be assigned to other records providing that the
  target value's type includes all of the fields (same name and type)
  present in the record value, and that any missing fields have the
  &optional or &default attribute (see next item).

  You can also include a record value inside the record constructor, and
  all of its fields will be included in the constructed record value.

- Record fields can now be annotated with &optional, to indicate
  that the field needn't be present, or &default, which indicates
  a default value to provide if the field is missing.

- You can query as to whether a record has a value for a given field
  using the new "?$" operator.  So for example:

	type my_rec: record {
		num: count &default = 0;
		msg: string;	# mandatory, since no &optional/&default
	};

	global r: my_rec;

	r = [$msg = "hello"];

	print r?$num, r?$msg, r$num;

  will print "F, T, 0," because even though 'r' has a default value
  for $num (which shows up when printing r$num), that field is missing,
  hence r?$num is F.

- An experimental scheme has been added (by Umesh Shankar) for managing
  general attributes associated either with all values ("global attributes")
  or particular particular values.  This scheme is likely to change in
  the near future, and hence isn't explained here further.

- The DNS analysis now includes ADDL and AUTH records, and much richer
  policy script analysis (per policy/dns.bro).

- You can now "redef" a function or event handler to override its
  previous definition.  For a function, this looks like:

	redef log_hook = function(msg: string): bool
		{
		...
		}

  For an event handler, it's just the usual definition preceded by "redef.
  For example,

	redef event ack_above_hole(c: connection) { }

  would replace the default ack_above_hole handler with one that does nothing.

- HTTP server and HTTP proxy backdoor detectors have been added,
  generating http_signature_found and http_proxy_signature_found,
  respectively (contributed by Ruoming Pang).

- A KaZaA backdoor detector has been added, which generates
  kazaa_signature_found for likely KaZaA connections.

- The new policy scripts flag-irc.bro and flag-warez.bro provide
  hooks for defining site policies for detecting IRC and access
  to warez.

- portmapper.bro now tracks the services it sees, and the names are
  used in connection summaries rather than generic services like port-656.

- bro -C (or redef'ing the "ignore_checksums" global to T) instructs
  Bro to ignore any checksum errors and go ahead and analyze such packets.

- The (trivial) policy script print-globals.bro dumps out all of the policy
  script global variables and the amount of memory they consume.

- The policy script code-red.bro has been renamed worm.bro and generalized
  to detect Nimda as well as Code Red 1 & 2.

- A bunch of additional default sensitive URIs have been added to http.bro.
  http.bro also now doesn't report worm-related URIs.

- A bunch of less common portnames were removed from port-names.bro.

- Empty regular expressions are now allowed.

- The finger_request event now has a third parameter, the additional
  text after the username.

- More systematic handling of NULs and CRLF by the event engine.

- Hex escape sequences now must have exactly two hexadecimal characters.

- FYI - work has begun on significantly altering the way that policy
  scripts generate alerts.

- Work has begun (by Robin Sommer) on a general framework for tracking
  client/server versions.  See software.bro.

- Work has begun on a NETBIOS analyzer (see NetbiosSSN.cc).  Contributions
  (e.g., finishing it :-) welcome.

- Work has begun on migrating the internals to process IPv6 in addition
  to IPv4.

- A number of bug fixes, leaks, and memory allocation lint tweaks.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.7a90 Thu Sep 06 00:50:43 PDT 2001

- Better state management, especially for use in high-volume sites.  There
  are now a number of ways to track the resources consumed by Bro while it
  runs, and to reduce the amount of memory Bro consumes.  Adding

	@load print-resources

  will log a summary of the resources consumed by Bro when it exits, and

	@load reduce-memory

  will change a number of default values in order to significantly diminish
  the amount of memory Bro requires.

  Other ways to lower the required memory discussed in the next three items.

- The global "maintain_http_sessions" now controls whether http.bro tracks
  multiple HTTP connections between the same hosts as a single HTTP session.
  Doing so costs a considerable amount of state (so set to F for reducing
  the memory impact of HTTP analysis).

- The global "suppress_scan_checks", if true, turns off address and port
  scan detection.  (You can't achieve this by simply not @load'ing scan.bro,
  because it's loaded by some of the default policy scripts.)  Turning it
  off can save a lot of memory.

- Note, the ident.bro is also expensive in terms of state-holding.

- New library functions:

	- resource_usage() returns a record detailing real time,
	  CPU time, memory, other getrusage info, and the current
	  and maximum number of TCP/UDP/ICMP connections, and
	  timers and fragments.

	- val_size() returns the size in bytes needed to represent
	  a given value (which can be a record, a table, or a
	  simple constant, for example).  It's not fully accurate
	  but is in the right ballpark.

	- global_sizes() return a table mapping every global
	  variable to its size (per val_size()).  Useful for
	  tracking which ones are growing large over time.

- You can now control a number of timers related to Bro's connection
  management.  Setting them to lower values generally decreases the
  amount of state Bro has to keep (see reduce-memory.bro), though
  this can also make it easier for an attacker to evade detection:

	tcp_SYN_timeout: interval
		Controls how long Bro waits after seeing the
		beginning of a connection (whether due to a SYN
		or not; the timer is misnamed) before checking
		whether it elicited any reply.

	tcp_session_timer: interval
		After a connection has closed, wait this long for
		further activity before checking whether to time
		out its state.

	tcp_connection_linger: interval
		When checking a closed connection for further
		activity, Bro should consider it inactive if there
		hasn't been any for this long.  It also complains
		if the connection is reused before this much time
		has elapsed.

	tcp_attempt_delay: interval
		Bro waits this long upon seeing an initial SYN
		before timing out the connection attempt.

	tcp_close_delay: interval
		Upon seeing a normal connection close, Bro flushes
		state after this much time.

	tcp_reset_delay: interval
		Upon seeing a RST, Bro flushes state after this
		much time.

	tcp_partial_close_delay: interval
		Bro generates a connection_partial_close event this
		much time after one half of a partial connection
		closes, assuming there has been no subsequent
		activity.

	non_analyzed_lifetime: interval
		If a connection belongs to an application that you
		aren't analyzing, Bro times it out after this
		interval.  If 0 secs (default), then it doesn't
		time it out.

	dns_session_timeout: interval
	ntp_session_timeout: interval
	rpc_timeout: interval
		Bro waits this long before timing out a DNS/NTP/RPC
		request.

	max_timer_expires: count
		The maximum number of timers to expire after
		processing each new packet.  The value trades off
		spreading out the timer expiration load with
		possibly having to hold state longer.  A value of 0
		means "process all expired timers with each new
		packet".

- Two new flags control other facets of Bro's connection management,
  and thus state-holding:

	partial_connection_ok: bool
		Whether Bro should analyze connections for which
		it doesn't see the beginning, only the middle.
		This can be very expensive to do in the face of
		stealth-scanning, which looks like a bunch of
		partial connections.

		Note, the HTTP analyzer has been modified to
		now always skip partial connections.  This should
		instead be user controllable.

	tcp_SYN_ack_ok: bool
		If true, Bro instantiates connection state when
		it sees a SYN ack but not the initial SYN (even
		if partial_connection_ok is false).  The intent
		behind this knob (which is not well tested) is
		to allow you to filter out initial SYNs and only
		react to SYN acks.  This keeps Bro from holding
		state during SYN scans and SYN flooding, except
		for when the destination responds.

- Some other miscellaneous thresholds that you can now modify from your
  policy script:

	heartbeat_interval: count
		How often to generate net_stats_update() events.
		This timer really isn't needed any more, since
		you can use "schedule" to achieve the same effect.

	tcp_storm_thresh: count
		If Bro sees this many FINs/RSTs in a row, it
		flags them as a "storm".

	tcp_storm_interarrival_thresh: interval
		The FINs/RSTs must come with this much time or less
		between them.

- The state management for line-oriented applications like HTTP requests
  has been improved.

- The HTTP analyzer now expands %hex sequences.  If anyone has a Unicode
  expander to contribute, that'd be terrific.

- The Code Red detection is more robust (fewer false positives).

- A new redefinable variable, skip_services, lists applications that should
  not be analyzed for purposes of detecting scans.  (Default:  ident)
  The point of having this (poorly named) hook is so that code-red.bro
  can add "http" to it, to avoid reporting the zillions of Code Red scans
  that a site can see.

- Bro now matches regular expressions using lazy DFA evaluation.  The upshot
  of this is (1) it no longer maintains .bro-RE-cache.v1, (2) it starts
  up fast regardless of whether you've added new regular expressions, (3)
  you can afford to add lots of regular expressions, and (4) it's actually
  a bit faster.

- The list of "hot_ids" has been expanded with a number of other common
  root-privileged accounts.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.7a61 Fri Apr 06 14:51:47 PDT 2001

- An NTP analyzer has been added.  See policy/ntp.bro for a corresponding
  policy file that detects the newly discovered NTP remote buffer overflow
  attack.

- example-attacks/ is a new directory that contains trace files of attacks.
  Currently, there are just two to play with:

	bro -r example-attacks/ftp-site-exec.trace mt

  will run on a trace of a "site exec" overflow attack, and

	bro -r example-attacks/ntp-attack.trace mt ntp

  will run on an example of the NTP overflow.

- The doc/ directory includes the postscript and HTML versions of the
  first draft of the Bro manual.

- A new policy file, icmp.bro, has preliminary (and only partially developed)
  policy for analyzing ICMP.

- The file libpcap.bufsize.patch includes the patch necessary on some systems
  to increase the maximum libpcap buffer size.

- You can now use anonymous functions in &default expressions, so for
  example you can do:

	global foo: table[count] of string = {
		[1] = "1st", [2] = "2nd", [3] = "3rd",
	} &default = function(n: count): string { return fmt("%dth", n); };

  and then referring to foo[5] will yield "5th".

- There's now a "for" statement to iterate over the indices of a table
  or the members of a set:

	for ( i in foo )

  for the above "foo" will iterate with i assigned to 1, 2, and 3; *but
  not in general in that order*.

- The function contains_string() has been removed, and now you can instead
  use an expression like

	"bar" in "foobar"

  which will yield T.

- The scan detection now has a mechanism for attempting to detect SYN flooding
  backscatter and flagging it as different from a stealth scan.

- New event handlers:

	new_connection_contents()
		like new_connection(), but reassembles the
		stream so you can use set_content_files() to
		write it to a file

	udp_session_done()
		invoked when a UDP session (which is defined on
		a per-protocol basis; currently only for NTP)
		finishes.

	ntp_message()
		invoked for each NTP message

- UDP processing now does accounting for detecting scans.

- UDP processing now tracks numbers of requests/replies for sessions that
  support that notion.  The connections are annotated by udp_session_done()
  with "[m,n]" for "m" requests and "n" replies, providing either m or n > 1.

- New variable accessible from policy:

	watchdog_interval
		how often the watchdog should check for whether
		Bro is making internal progress

- A bunch of functions no longer have a first argument of the current time;
  get it instead from network_time() if you need it:

	authentication_accepted
	authentication_rejected
	conn_weird
	conn_weird_addl
	flow_weird
	net_weird

- A bunch of functions now return bool rather than int values:

	set_contents_file
	set_login_state
	set_record_packets
	skip_further_processing

- The variable "hot_dests" has been renamed to "hot_dsts".

- 111/tcp is now identified as "portmap" rather than "rpc".

- Connections flagged as hot for some types of characteristics are now
  annotated with the reason associated with the decision.  (I think a lot
  more of this is needed.)

- Portmapper dumps are annotated with the results of the mapping.  This
  will be streamlined in the future.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.7a48 Wed Sep 13 14:37:30 PDT 2000

- Changes between this release and v0.6 missing :-(

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.6 Wed Jul 21 17:02:50 PDT 1999

- Support for regular expressions added.  You specify lex-style regular
  expressions between '/'s, for example "/\/etc\/(passwd|shadow)/" has
  the type "pattern" and matches /etc/passwd or /etc/shadow (the slashes
  in the pattern need to be escaped or else they'd delimit the end of the
  pattern).  Pattern-matching is via the "in" operator, so for example:

	if ( filename in /\/etc\/(passwd|shadow)/ )
		sensitive_file_access(filename);

  or

	const sensitive_files = /\/etc\/(passwd|shadow)/;

	...

	if ( filename in sensitive_files )
		sensitive_file_access(filename);

  Presently the "in" operator requires that the entire left-hand side
  be matched by the pattern.  So, for example, if you want to find the
  string "eggdrop" anywhere inside the string "line", you would use

	if ( line in /.*eggdrop.*/ )

  If you leave off either of the .*'s, then eggdrop will only be matched
  at the beginning or end of the line.

  In the future, there will be mechanisms for specifying whether you
  want to match anywhere in a line, or anchored; accordingly, *the above
  syntax is subject to change*.

  Bro compiles regular expressions into DFAs for fast matching.  This can take
  quite a bit of time for complicated patterns.  Consequently, it maintains a
  cache of compiled regular expressions in $HOME/.bro-RE-cache-v1.  You can
  always safely remove this file; Bro will recreate/repopulate it as needed.
  It does not clean up unused entries from it, so if you change your patterns
  frequently, you will accumulated lots of old ones and should delete the
  file to garbage collect them.

- An rlogin analysis module has been added and the telnet analysis
  generalized to generic "login" analysis, with the following events:

	login_failure(c: connection, user: string, client_user: string,
			password: string, line: string)
		Generated on a failed attempt to log in.  client_user is
		the remote user name, if the login is via the rlogin
		protocol.
	login_success(c: connection, user: string, client_user: string,
					password: string, line: string)
		Generated on a successful attempt to log in.

	login_input_line(c: connection, line: string)
		Generated per line of input typed by the user.
	login_output_line(c: connection, line: string)
		Generated per line of output generated by the server.

	login_confused(c: connection, msg: string, line: string)
		Generated when a login dialog confuses the heuristic
		analyzer.  msg is a tag for the state mismatch that
		was unexpected, line is the corresponding dialog text.

	login_confused_text(c: connection, line: string)
		Once a connection is in the confused state, then this
		is generated for each subsequent line.

	login_terminal(c: connection, terminal: string)
		Generated if the terminal type associated with the
		connection is seen.

	login_display(c: connection, display: string)
		Generated if the display associated with the connection
		is seen.

	excessive_line(c: connection)
		Generated when the connection has produced an excessively
		long line.

  login_input_line() and login_output_line() are very powerful for
  detecting intrusions, when coupled with regular-expression matching.

  login_terminal() is used to detect backdoors that are triggered
  by the terminal environment variable.

- An ident analysis module has been added (port 113).  It generates
  ident_request, ident_reply, and ident_error events.  Port 113 used
  to be referred to as "auth"; now it's referred to as "ident".

- A new type of scan detection has been added, which is triggered
  by a remote host trying a large number of username/password
  combinations.  See the account_tried() function in scan.bro.

- The default search path for .bro files is now

	  .:priv-policy:policy:pub-policy:/usr/local/lib/bro

  where priv-policy/ is intended for private policy and pub-policy/
  for public policy.  The Bro alpha distribution ships with a
  sample set of pub-policy scripts.

- New built-ins:

	system(s: string): int
		executes the given shell command using system()
		and returns its status.

	set_contents_file(c: conn_id, direction: count, f: file)
		copies connection c's reassembled byte stream in
		either the originator-to-responder direction (if
		direction is CONTENTS_ORIG) or the responder-to-
		originator direction (CONTENTS_RESP) to the file f.

	reading_live_traffic(): bool
		returns true if Bro is running on live traffic (read
		from a network interface), false if it's reading from
		a save file.

	mkdir(f: string): bool
		creates the given directory, returning true if it
		was able to, false if not.

	get_orig_seq(c: conn_id): count;
		returns the highest sequence number sent by the
		originator of connection c.
	get_resp_seq(c: conn_id): count;
		same for c's responder.

- Additional new events (other than those related to the new analyzers):

	new_connection(c: connection)
		is generated whenever a new connection is seen.

	partial_connection(c: connection)
		is generated whenever a new partial connection (one
		that doesn't begin with a SYN handshake) is seen.

	pm_bad_port(r: connection, bad_p: count)
		is generated when a portmapper response contains
		a bad port number.

- Functions, tables and sets can now be assigned.  Assignment is
  made by reference to the underlying object.

- Bro no longer looks up identifiers using getservbyname() to see if they
  should be interpreted as port numbers, since this led to portability
  problems.  Instead, a number of constants are defined in bro.init:
  bgp, domain, finger, ftp, gopher, http, ident, rlogin, smtp, ssh and telnet,

- Bro now supports an arbitrary number of open files (not bound by
  the system's limit on file descriptors).

- There's now a finger_reply event to go with finger_request.

- A bunch more RPC service names have been added, thanks to Job de Haas
  and others.

- A bug has been fixed in the watchdog handling that caused it to
  sometimes expire after a period of network inactivity.

- The Bro paper in doc/ has been revised (it isn't quite up-to-date,
  but considerably closer than the USENIX version).

- There has been a large amount of reworking of the internals, both
  to Bro itself and in the policy scripts.  If you find something you're
  wondering about, feel free to send me mail asking about it.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.5 Sun Oct  4 00:19:35 PDT 1998

- Added Linux support.

- Major autoconf changes.

- Some tweaks to suppress g++ warnings.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.4 Fri Sep 11 00:08:04 PDT 1998

- The new aux/ directory has some utility programs and scripts.
  See aux/README for details.

- Documentation (though exceedingly limited) describing the connection
  log summaries generated by policy/tcp.bro now available in doc/conn-logs.

- The Telnet username recognition code has been heavily reworked and is
  now significantly more robust.

- A new policy file, policy/scan.bro, detects port and address scanning.

- Bro now detects the "Land" attack (a trivial addition to its spoof
  detection).

- The distribution now comes with BIND 8.1.2.

- A new flavor of "weird" event has been added, flow_weird, for unusual
  events associated with <src IP addr, dst IP addr> flows (i.e., a coarser
  granularity than TCP or UDP connections).

- Bro now reassembles fragments, checking for overlaps and consistency.
  Associated with fragments are the following flow_weird's:

	excessively_large_fragment
	excessively_small_fragment
	fragment_inconsistency
	fragment_overlap
	fragment_protocol_inconsistency
	fragment_size_inconsistency
	fragment_with_DF
	incompletely_captured_fragment

- The TCP stream reassembly code was rewritten to share functionality
  with the new fragment reassembly code.

- If a handler for it is present, then Bro will generate "packet_contents"
  events containing the contents of each packet it receives.  This is just
  for experimenting with, and, in particular, this event does *not* reflect
  TCP stream reassembly.

- The handling of "conn_weird" events in tcp.bro now has more options.
  In the weird_action table, WEIRD_LOG_ALWAYS means that the weird event
  should always be logged; WEIRD_LOG_PER_CONN means that it should be
  logged once per connection; and WEIRD_LOG_PER_ORIG that it should be
  launched once per originator IP address.

- The example hot.bro now includes provisions for flagging sensitive
  inbound and outbound services.

- Bro now limits the number of events processed when draining the event
  queue after processing a packet.  This makes Bro less prone to packet
  loss during high-speed scans.

- The line-oriented TCP endpoint class was split into TCP_EndpointLine,
  which supports the line-oriented functionality, and TCP_NVT (derived
  from TCP_EndpointLine), which implements the Network Virtual Terminal
  used by Telnet and FTP.

- The TCP_NVT class now understands the Telnet Environment option.

- Escape sequences are now '\' followed by 1 or more octal digits,
  instead of excatly three octal digits (which is error prone).

- If the watchdog timer expires, it now reports the number of events
  processed in the current batch of packets, as well as other timing
  information.

- Bro now should not report packet drops that occur after it has begun
  to exit (these can occur when draining the pending event queue takes
  a while).

- Bro now detects TCP acknowledgements that occur above a sequence
  hole, generating an ack_above_hole event.  Nominally, this indicates
  packet filter drops, but in fact some buggy TCPs manage to do this :-(.

- Fledgling HTTP support added.  An http_request event is generated when
  a new HTTP request is seen, and http_stats is generated when an HTTP
  connection terminates, giving (uninteresting to most people) statistics
  concerning the request(s).  A lot more is needed: parsing persistent
  connections and HTTP replies, for one.  This is just a start.  A stub
  for http_reply exists but these events are not presently generated.

- Ported to Linux (thanks to Pascal Bouchareine).

- A bug in to_lower() and to_upper() was fixed.

- The reporting for unexpected FTP connections now more directly
  identifies the corresponding FTP session.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.3 Fri Feb 13 19:59:39 PST 1998

- Timers are now implemented using calendar queues rather than priority
  queues (with thanks to Craig Leres).

- The new byte_len() built-in returns the length of a string interpreted
  as a set of bytes (including any NUL bytes, especially the final one
  for a typical string).

- The new sub_bytes() built-in extracts a subset of a string interpreted
  as a set of bytes (i.e., immune to any embedded NULs).

- Fixed bad interaction with the latest version of libpcap that on some
  BPF systems would cause Bro to exit any time a little bit of time went by
  without any traffic matching its filter.

- A bug with constant-folding of the ?: operator has been fixed.

- A new "conn_stats" event delivers statistical analysis (number of packets
  transmitted, retransmitted, out-of-order, replicated) of each connection.
  If you define a handler for it, then *no* other TCP processing is done.
  This was added for off-line analysis of traces with large numbers of
  connections in them.

- Some minor portability tweaks.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.2 Sat Oct 25 11:22:37 PDT 1997

- Added autoconf support, thanks to Scott Denton.

- Ported to FreeBSD, Solaris.

- Fixed a bug in which partial connections were ignored even for
  protocols (e.g., FTP) that can make use of partial dialogs.

- Included BIND version 8 sources directly in the distribution.

- Better usage() information (again thanks to Scott), -h and -v flags.

- README, CHANGES files created.