![]()
Publishing rights |
![]() Publishing rightsCopyright © Idonex AB 1998. All rights reserved. No part of this
document may be reproduced, in any form or by any means, without the written
permission of Idonex AB. Roxen and Pike are trademarks or registered
trademarks of Idonex AB. All product names are trademarks or registered
trademarks of their respective holders and recognized as such.
CreatorsProgrammingFredrik Noring, Johan Schön, Marcus Wellhardh
ManualAcknowledgementsMany thanks to Francesco Chemolli and Matthew Brookes and of course all the other testers. Thanks!LogView is a powerful logging and log analyzing tool that can be used by web site builders and administrators to view a lot of statistics about the accesses to the site. This information can be useful in a variety of ways, ranging from keeping track of the load on a site and what material is commonly accessed, to pointing out errors and flaws in the web site design. LogView is installed as a module in a Roxen Challenger web server. It performs two main tasks:
Once per day, LogView compiles the desired statistics from the logs and updates its internal databases. These updates does not significally slow down the server even on large sites where the logs can become equally large. LogView handles log files of gigabyte size without any performance problem, and it compresses its own log files to less than 1/4 of the size of an ordinary log file in the Common Log Format. LogView uses the term statistics group as a base for extracting statistic information from the logs. LogView can be configured to handle several different statistics groups, each one taking its data from an individually specified set of logs. Each group is also individually configured to use a number of functions to extract the corresponding kinds of statistics, such as file downloads and the names of visiting hosts, from the group. The complete set of functions available in LogView at the time of writing of this manual is presented in appendix A. To be useful the extracted statistics has to be presented to the user in a flexible and easily accessible way. LogView provides a graphical user interface accessed through a web browser to present the statistics and to interact with the user. This makes the information available from anywhere on the net. The statistic data can be shown in numerous ways: in table form, in different types of charts and diagrams, as good old portable text files, and it can even be exported to MS Excel format. LogView can also be set up to automatically create reports at regular intervals and deliver them by email. This is the user's part of the LogView manual. It contains information on different ways of using the LogView user interface to view the extracted statistics, and also on how to put LogView diagrams and tables on the user's own web pages on the server. LogView lets the user view the gathered statistics in an ordinary web browser. If LogView has been installed on the Roxen virtual server with the URL http://rhino.ceros.com/virtual/the default URL to the LogView access page is http://rhino.ceros.com/virtual/logview/However, the administrator may choose another location whithin the virtual file system. If the administrator has specified that access to the LogView pages should be password protected, the user will get a popup dialog where he has to log in before he can access the LogView page. On the top of all the LogView pages is located a button menu which helps the user to navigate between the four pages Statistics, Report, Advanced and Manual. The Manual button takes the user to a page containing a version of this manual, and the others provide different ways of looking at the gathered statistics. The main difference between the Statistics, Reports and Advanced pages lies in what possibilities the user has to affect the presentation. The Reports page contains a number of reports that has been prepared by the administrator, and is therefore the most static one - all the user needs to do is to select which report he wishes to see. The Statistics page is the normal way to look at the data, and provides basic possibilites to control what statistics to see and in which form, and in the Advanced page the user has even more freedom to control the details. These three levels of control possibilities makes it possible to provide quick-and-easy access for the users who either does not need full control or can have the administrator tailor a report for them, while the users that really need full control over the details can still get it. The statistics page provides an easily overviewed and almost self-explanatory way of looking at statistics gathered by all the functions that are enabled on the current statistics group. Selecting statistics groupIn the field labeled Group the current statistics group is shown. If several groups are defined, the user can select one of them in a list box. By default only one group is defined. It takes its data from the log of the Roxen virtual server that LogView is installed on, and the name of this group is the same as the name of the server. If no hits have been recieved since the creating of the selected statistics group, this is shown by the message "No items are logged". This should be remedied within one hour after the site has been accessed. Selecting period of timeIn the Period of time box the period of time for which to show statistics can be specified. The currently selected period of time is displayed within parantheses right next to the lable on the top of the box. The highlighted periods inside the box are the ones for which logged data is available, and clicking one of them simply selects that period of time. Initially only years and months are shown, but once a month is selected the box is expanded to also contain weeks and days. On the Statistics page, statistics can only be shown for a single year, month, week or day. To specify a more exact time interval, the user has to go to the Advanced page. Selecting kind of statisticsBelow the Period of time box, a number of available functions are listed in groups. To see statistics of a certain kind the user just has to select the corresponding function in the list. An overview over the functions can be found in appendix A. Selecting formatOnce the user has selected the kind of statistics to show, a report is displayed at the bottom of the page. This report is in a default format that is dependent of the kind of statistics that was selected. If the user is not pleased with this format, he can choose another one in the list box in the title of the report. The report page is used for viewing reports that has been prepared by the LogView administrator. The user just uses the list box to select the report he wants to view. In this page, the user can control all of LogViews capabilities for viewing statistics. Selecting groupAs was the case on the Statistics page, the statistics group, if more than one is available, can be selected in a list box. Selecting reportThe user can select the kind of report from all the kinds that are available for the selected statistics group in the first list box on the row. The list box labeled as is used to specify the granularity of the diagram or table. If days is selected here, a resulting bar chart will have one bar for each day for which data is available, and a table will accordingly have one row for each day. If instead month is selected, the bar chart will have one bar for each month, with a height that is the the sum of the all the values of the days of this month. The list box labeled by selects what should be printed for example as lables on the bars of a bar chart. To get a chart where each day has it's own bar and the lable Year-Week-Day (e g 1998-25-Tue) the user can select the report to be shown "as day, per week". A report that is "as day, per month", has the lable Year-Month-Day (e g 1998-Jun-16) on its bars instead. Trying to set the as value larger than the by value gives the following error message: LogView Error: unit cannot be greater than perSetting both values equal has an interesting effect when using the Sum feature, as we will se in the next section. Selecting presentationIn the first list box, the user can choose between Append or Sum. Append is the default, and gives the results we saw in the Report section. Using the Sum feature changes this behavior somewhat - instead of summing the data inside each day, month or year the data for several such units can be summed, to answer questions like "how many hits were processed in total between the 1998-06-02 and 1998-07-11?" However, to use this feature we have to use the trick to set the as and by time periods the same. There is also a possibility to set the maximum number of rows to be shown. The term rows seem to imply the use of a table, but the limitation works on all kinds of diagrams as well, and then limits the number of bars, pie slices or other corresponding items. Selecting from and to valuesThe user can specify the exact time period for which statistics should be shown by selecting the years, months and days for the beginning and end of the period. Selecting displayWhen the user has specified all of the above, he can start the display process by pressing one of the display buttons in the row at the bottom of the page. These can be divided into categories, of which only the members of the last one needs explaining. Tables:
If the administrator has configured LogView to permit this, the LogView diagrams or tables can be displayed on any page on the server. This is done by adding a <logview> tag to the page, with the correct parameters for showing the desired kind of report. The syntax of this tag is found in appendix B. A very handy way to automatically create the tag text is to go to the Advanced page, try out different parameters until the report is satisfactory, and then press tag display button and copy-and-paste the tag into the page. This is the administrator's part of the LogView manual. This part contains information of how to install and configure LogView, but to increase the understanding of these subjects, we start out with a deeper survey over the LogView system. Internal and external logsLogView handles two types of logs - logs that LogView collects itself, one for each virtual server it is installed to, and log files in Common log format that are created in other ways and imported to LogView. The first kind contains the most information and can therefore be used to create more complex statistics. All logs that LogView creates are compressed to save disk space. Statistics groupsEarlier in this manual, we mentioned the concept of statistics groups. Now is the time to dive deeper into this subject. A simple log analyzer might only collect separate statistics for a number of separate logs. This approach is not sufficient for large websites with many different virtual and physical servers, however. On such sites there is a need to show statistics extracted from data merged from the logs of several different servers. LogView solves this problem by not performing the statistics calculations directly on the logs, but instead on statistics groups that in turn can be connected to several different logs. To find out the total number of accesses to all the servers on the site, the administrator just has to create a statistics group with all the logs, and enable the function group Site activity, which contains the function Hits, on that statistics group. How this is done will be covered later, in the Configuration section. DatabasesLogView keeps a database for each function enabled on each statistics group. In these databases, LogView stores information that has been extracted from the logs by the functions. For example, if the function Most common browser is enabled on a group, a database is created to store the number of hits per used browser type. Inside the databases, the data is divided into different tables per a time unit, most often per day or hour. In this way, when the user requests a specific kind of statistics for a specific group and time interval, LogView can easily get the information needed to calculate the statistics from the database. The update operationOnce per day, or when the administrator uses the Update command (see the Configuration section), LogView performs an update operation. This has effects on both logs and databases. All imported external logs are checked, and the last hour's hits are appended to a corresponding internal LogView log. The logs created by LogView itself are not affected, since they are updated automatically after every access to their virtual server. The databases of the statistics groups are updated with the data gathered during the last hour, making the new data visible in the statistics.
At this point, LogView is already up and running with a standard configuration and can be reached at the default location /logview under the root of the virtual server, for example: http://virtual.server.name/logview However, it is a good idea to check if the default values need any modification. This is done by selecting the LogView option that has now appeared in the list of installed modules. In the following sections, the different options under the LogView module configuration are explained. Builtin variablesThis option is available for all Roxen modules. Detailed information about the different items here is found in the Roxen Administrator's Manual. Admin PasswordHere the password for LogView administration can be changed. By default, this value is the same as the password used for Roxen administration. Note: the number of the stars symbolizing the password is always four, regardless of the actual number of letters in the password. Admin user nameThe user name of the LogView administrator, by default the same as the Roxen administrator. Enable <logview> tagIf No is selected, the LogView tag will only work inside the LogView user interface pages. If Yes is selected, the tag can be used on any page on the virtual server, as described in the section Putting reports on other pages in the User's manual. The default value is No. Log PathSpecifies the directory where LogView will put its own logs. The default value is ../logs/new/logview.data/Note: Please use different directories for every LogView installation. Otherwise the state of the internal LogView files will become unclear and LogView will fail to operate correctly. Mount PathSpecifies the place in the virtual file system of the server where the LogView user interface can be accessed. The default value is /logview/ Viewer passwordSpecifies the password needed to access the user interface, if password protection is enabled. Note: the number of the stars symbolizing the password is always four, regardless of the actual number of letters in the password. By default, this value is empty. Viewer user nameIf this field contains a name, the user has to log in with this user name and the password specified above. Only one user can be specified. By default, this value is empty. The LogView administration interfaceAs mentioned earlier, LogView is installed with a default configuration which gives a functionality that could be quite enough for some applications. However, if this configuration needs to be changed, this is done via the LogView administration interface. This is a part of the LogView user interface that can only be accessed by the administrator. It is reached at the URL pointing at the subdirectory admin under the LogView directory itself, e g http://virtual.server.name/logview/admin The administrator has to log in with the name and password mentioned in the Installation section. When this is done the regular LogView user interface appears, with the addition of the extra menu items Event log and Configuration. The contents of these two pages are covered in the following sections. On the Event log page, two kinds of logs can be viewed: the Admin log and the Summary log. The Admin logThis is a log of events originating from the administration pages, such as the administrator logging in and configuration changes being made. The log also contains messages about any errors that have occured. The informational messages in the log are preceded by an information icon, while the warning and errors messages are preceded by warning and error icons respectively:
The Summary logThe Summary log is much more detailed and contains a summary of automatic LogView activities, such as importing logs and updating statistics databases. This log is mainly useful for tracking down errors. In this page, a number of aspects of LogView's behavior can be configured. All of these aspects have a section below. UpdateHere, the administrator can manually force an instant update of the statistics gathering, which otherwise takes place once an hour. This can be used for presenting really fresh statistics, or for testing purposes. Update status shows whether the update process is already running at the moment, and, if so, under which process number. LogsA Roxen wizard guides the administrator through the steps needed to configure the logs.
By default, LogView is configured with one log, the log of the virtual server that LogView is installed on. This log has the same name as the server. More logs can be added using the option Import logs below. Some of the attributes use Pike regular expressions, see Appendix C for the syntax. Statistics groupsThe statistics group are configured in another Roxen wizard.
A new group can be added by choosing that option and filling in the name. These logs are divided into: In the list, check the logs that should be included. Just select the function groups that are of interest. Import logsUsing this wizard, the administrator can specify existing logs in the Common log format to be imported into LogView.
Specify the full path to the log file. Specify a unique name for this log. Follow the same procedure as in the Logs section above. A new statistics group can be added by selecting that option and specifying the name. This is done as described in the section Statistics groups above Delete logsAfter selecting the log that is to be deleted, the administrator has the choice of keeping or deleting the log configuration. Selecting Keep will only delete the contents of the log, and LogView will continue to extract statistics from the log in the future. Selecting Delete will make LogView forget about the log altogether, as well as delete the log file. Rebuild/delete statistics groupAfter selecting the appropriate group, the administrator has the choice of only deleting the gathered data or the entire group, just as was the case with the logs in the previous section. Selecting Rebuild will only delete the data and then rebuild it from the logs. This could be used when a new log is added to the group, and the statistics for the past up to this point has to be recalculated. Selecting Delete will delete the group as well as the data. Report makerA report is a page containing LogView tags tailored by the LogView administrator. Here such pages can be created.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![]() |
Here the Advanced page in the LogView interface can be of great help - first try out different parameters to get the right look of the diagram or table, and then press the Tag button and copy-and-paste the LogView tag into the report. If the report should also be delivered by email, select this in the Report by email box. If report by email is chosen, some additional information has to be specified. Fill in the email address and edit the From and Subject fields if necessary. Also select how often and, if once a week, on what week day the report should be sent, in the Frequency box. Confirm the values as usual, and also choose whether a test mail should be sent right away. Delete reportIn the Delete report wizard the administrator can select a report and delete it. The first thing to do in case of a problem is to check LogView's error log files and, failing that, the main Roxen server's error log files. Both LogView and Roxen try to generate comprehensible reports on errors that occur. Specific problems:The installation of LogView failedThis is typically due to one of the following things:
LogView does not create any statisticsIf there's an error message in the logs like: ``Cannot decode free chunk'':
At the time of writing of this manual, Logview handles the following statistics functions, ordered in groups: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The LogView module provides a special RXML tag, <logview>, which provides an easy way of inserting statistics reports in ordinary RXML documents. The usage is simple:
<logview options>where the options are some of the following:
LogView options
LogView example
In LogView, Pike regular expressions are used to separate served files into different categories, like for example pages and non-pages, and also for specifying names of hosts to ignore in the statistics. Pike regular expressions are very powerful, as we will show in the following examples. Really, a Pike regexp is the same kind of regexp that is used by many UNIX tools, like egrep and awk, and the user who is experienced in this area may well stop reading right here, or jump directly to the reference chart to refresh his memory concerning the syntax.When constructing regexp to match the right strings, the characters are divided into normal characters (a - z, A - Z, 0 - 9) and special characters ( for example ".", "*", "(", ")", "|" and "&"). An normal character matches itself, and so a word built up by ordinary caracters matches itself. Also, a pattern matches a string if it matches any part of the string.
Sometimes we want to specify that the beginning or end of the string must match the pattern. This is done with the special characters ^ and $, respectively.
Now for the special characters "." and "*". A "." matches one occurance of any character and a character followed by a "*" maches any number, even zero, of consequent occcurances of the character. This letter might also be a ".", in which case zero or more of any letter is matched. Thus, ".*" gives the same effect as a single "*" character given in a filename at a UNIX or DOS prompt, which can be a bit confusing.
As can be noticed above, the first pattern also matched yyyyrhino.ceros.com, since "." is a special character. However, if we want to specify it to be matched as a normal character, we have to put the escape character, "\" in front of it.
Now we've covered the basic stuff about simple regexps. However,
there is also a possibility to use boolean functions to put together
several simple regexps to complex ones - for example, (regexp1) |
(regexp2) is a new regexp that matches a string if regexp1 or
regexp2 matches the string, and similiarly (regexp1) & (regexp2)
matches a string if both regexp1 and regexp2 matches the string.
This is the complete reference chart for the Pike regexp syntax, taken from the Pike manual.
Note that \ can be used to quote these characters in which case they match themselves, nothing else. Also note that when quoting these something in Pike you need two \ because Pike also uses this character for quoting. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||