Internet DRAFT - draft-hallam-http-logfile
draft-hallam-http-logfile
HTTP/1.1 200 OK
Date: Tue, 09 Apr 2002 00:15:48 GMT
Server: Apache/1.3.20 (Unix)
Last-Modified: Sat, 02 Mar 1996 13:57:53 GMT
ETag: "304c90-3841-313853e1"
Accept-Ranges: bytes
Content-Length: 14401
Connection: close
Content-Type: text/plain
INTERNET DRAFT Phillip M. Hallam-Baker, W3C
Expires in six months email: <hallam@w3.org>
Brian Behlendorf
email: <brian@organic.com>
21st February 1996
Extended Log File Format
<draft-hallam-http-logfile-00.txt>
Status of this Memo
This document is an Internet draft. Internet drafts are working
documents of the Internet Engineering Task Force (IETF), its areas
and its working groups. Note that other groups may also distribute
working information as Internet drafts.
Internet Drafts are draft documents valid for a maximum of six
months and can be updated, replaced or obsoleted by other documents
at any time. It is inappropriate to use Internet drafts as reference
material or to cite them as other than as "work in progress".
To learn the current status of any Internet draft please check the
"lid-abstracts.txt" listing contained in the Internet drafts shadow
directories on ftp.is.co.za (Africa), nic.nordu.net (Europe),
munnari.oz.au (Pacific Rim), ds.internic.net (US East coast) or
ftp.isi.edu (US West coast). Further information about the IETF can
be found at URL: http://www.cnri.reston.va.us/
Distribution of this document is unlimited. Please send comments to
the HTTP working group (HTTP-WG) of the Internet Engineering Task
Force (IETF) at < http://www.ics.uci.edu/pub/ietf/http/. This note
is also avaliable as a World Wide Web Consortium Working Draft
WD-logfile-960221, archived at
http://www.w3.org/pub/WWW/TR/WD-logfile-960221.html
Extended Log File Format
WD-logfile-960221
Extended Log File Format
W3C Working Draft _WD-logfile-960221_
This version:
http://www.w3.org/pub/WWW/TR/WD-logfile-960221.html
Latest version:
http://www.w3.org/pub/WWW/TR/WD-logfile.html
Authors:
Phillip M. Hallam-Baker <hallam@w3.org>
Phillip M. Hallam-Baker Page 1
Extended Log File Format
Brian Behlendorf <brian@organic.com>
------------------------------------------------------------------------------
Status of this document
This is a W3C Working Draft for review by W3C members and other
interested parties. It is a draft document and may be updated,
replaced or obsoleted by other documents at any time. It is
inappropriate to use W3C Working Drafts as reference material or to
cite them as other than "work in progress". A list of current W3C
working drafts can be found at: http://www.w3.org/pub/WWW/TR
Note: since working drafts are subject to frequent change, you are
advised to reference the above URL, rather than the URLs for working
drafts themselves.
Phillip M. Hallam-Baker Page 2
Extended Log File Format
Abstract
An improved format for Web server log files is presented. The format
is extensible, permitting a wider range of data to be captured. This
proposal is motivated by the need to capture a wider range of data
for demographic analysis and also the needs of proxy caches.
Introduction
Most Web servers offer the option to store logfiles in either the
common log format or a proprietary format. The common log file
format is supported by the majority of analysis tools but the
information about each server transaction is fixed. In many cases it
is desirable to record more information. Sites sensitive to personal
data issues may wish to omit the recording of certain data. In
addition ambiguities arise in analysing the common log file format
since field separator characters may in some cases occur within
fields. The extended log file format is designed to meet the
following needs:
* Permit control over the data recorded.
* Support needs of proxies, clients and servers in a common format
* Provide robust handling of character escaping issues
* Allow exchange of demographic data.
* Allow summary data to be expressed.
The log file format described permits customized logfiles to be
recorded in a format readable by generic analysis tools. A header
specifying the data types recorded is written out at the start of
each log.
This work is in part motivated by the need to support collection of
demographic data. This work is discussed at greater length in
companion drafts describing session identifier URIs [Hallam96a] and
more consistent proxy behaviour [Hallam96b].
Format
A extended log file contains a sequence of _lines_ containing ASCII
characters terminated by either the sequence CR or CRLF. Log file
generators should follow the line termination convention for the
platform on which they are executed. Analysers should accept either
form. Each line may contain either a _directive_ or an _entry_.
Entries consist of a sequence of _fields_ relating to a single HTTP
transaction. Fields are separated by whitespace, the use of tab
characters for this purpose is encouraged. If a field is unused in a
particular entry dash "-" marks the omitted field. Directives record
information about the logging process itself.
Phillip M. Hallam-Baker Page 3
Extended Log File Format
The following directives are defined:
Version: _<integer>_._<integer>_
The version of the extended log file format used. This draft
defines version 1.0.
Fields: [_<specifier>_...]
Specifies the fields recorded in the log.
Software: _string_
Identifies the software which generated the log.
Start-Date: _<date>_ _<time>_
The date and time at which the log was started.
End-Date:_<date>_ _<time>_
The date and time at which the log was finished.
Date:_<date>_ _<time>_
The date and time at which the entry was added.
Remark: _<text>_
Comment information. Data recorded in this field should be
ignored by analysis tools.
The directives Version and Fields are required and should preceed
all entries in the log. The Fields directive specifies the data
recorded in the fields of each entry.
Example
The following is an example file in the extended log format:
#Version: 1.0
#Date: 12-Jan-1996
#Fields: time cs-method cs-uri
00:34:23 GET /foo/bar.html
12:21:16 GET /foo/bar.html
12:45:52 GET /foo/bar.html
12:57:34 GET /foo/bar.html
Fields
The #Fields directive lists a sequence of _field identifiers_
specifying the information recorded in each entry. Field identifiers
may have one of the following forms:
_identifier_
Identifier relates to the transaction as a whole.
_prefix_-_identifier_
Phillip M. Hallam-Baker Page 4
Extended Log File Format
Identifier relates to information transfer between parties
defined by the value _prefix_.
_prefix_(_header_)
Identifies the value of the HTTP header field _header_ for
transfer between parties defined by the value _prefix_. Fields
specified in this manner always have the value <string>.
The following prefixes are defined:
cs
Client to Server.
sc
Server to Client.
sr
Server to Remote Server, this prefix is used by proxies.
rs
Remote Server to Server, this prefix is used by proxies.
x
Application specific identifier.
The identifier cs-method thus refers to the method in the request
sent by the client to the server while sc(Content-Type) refers to
the content type field of the reply.
Identifiers.
The following identifiers do not require a prefix
date
Date at which transaction completed, field has type <date>
time
Time at which transaction completed, field has type <time>
time-taken
Time taken for transaction to complete in seconds, field has
type <fixed>
bytes
bytes transfered, field has type <integer>
cached
Records wether a cache hit occured, field has type <integer> 0
indicates a cache miss.
The following identifiers require a prefix
ip
Phillip M. Hallam-Baker Page 5
Extended Log File Format
IP address and port, field has type <address>
dns
DNS name, field has type <name>
status
Status code, field has type <integer>
comment
Comment returned with status code, field has type <>
method
Method, field has type <name>
uri
URI, field has type <uri>
uri-stem
Stem protion alone of URI (omitting query), field has type <uri>
uri-query
Query portion alone of URI, field has type <uri>
host
DNS hostname used, field has type <name>
Special fields for log summaries.
Analysis tools may generate log summaries. A log summary lists the
number of operations performed on a URI but omits information
specific to a particular transaction.
The following field is mandatory and must preceed all others:
count
The number of entries for which the listed data, field has type
<>
The following fields may be used in place of time to allow
aggregation of log file entries over intervals of time.
time-from
Time at which sampling began, field has type <time>
time-to
Time at which sampling ended, field has type <time>
interval
Time over which sampling occurred in seconds, field has type
<integer>
Entries
Phillip M. Hallam-Baker Page 6
Extended Log File Format
This section describes the data formats for log file field entries.
These formats are chosen so as to avoid ambiguity, minimize the
difficulty of generation and parsing and provide for human
readability.
Each logfile entry consists of a sequence of fields separated by
whitespace and terminated by a CR or CRLF sequence. The meanings of
the fields are defined by a preceeding #Fields directive. If a field
is ommitted for a particular entry a single dash "-" is substituted.
Log file parsers should be tolerant of errors. If an entry contains
corrupt data or is terminated unexpectedly the parser should
resynchronize using the end of line marker and continue to parse the
following entries. Entries must not contain any ASCII control
characters.
<entry> = <field>* <end-of-line>
<field> = <integer> | <fixed> | <uri> | <date> | <time> | <string>
<digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
Integer
<integer> = <digit> +
Integers are represented as a sequence of digits.
Fixed Format Float
<float> = <digit> [. <digit>*]
URI
A URI as specified by RFC1738, relative URIs are specified by
RFC1808. URIs cannot by definition include whitespace or ASCII
control characters. Consequently no ambiguity arises from their use.
Date
<date> = <digit> <digit> <digit> <digit>
"-" <digit> <digit> "-" <digit> <digit>
Dates are recorded in the format YYYY-MM-DD where YYYY, MM and DD
stand for the numeric year, month and day respectively. This format
Phillip M. Hallam-Baker Page 7
Extended Log File Format
is chosen to assist collation using sort.
Time
<time> = <digit> <digit> ":" <digit> <digit>
[":" <digit> <digit> ["." <digit>*]
Times are recorded in the form HH:MM, HH:MM:SS or HH:MM:SS.S where
HH is the hour in 24 hour format, MM is minutes and SS is seconds.
String
<string> = '"' <schar>* '"'
<schar> = xchar | '"' '"'
Strings are output in quoted form. If a string contains a quotation
character the character is repeated. This format is unambiguous
since fields are by definition separated by whitespace.
No mechanism for incorporating control characters is defined.
Text
<text> = <char>*
The text field is used only by directives.
Name
<name> = <alpha> [ "." <alpha> * ]
DNS name.
Address
<name> = <integer> [ "." <integer> * ] [ ":" <integer> ]
Numeric IP address and optional port specifier.
Acknowledgements.
Phillip M. Hallam-Baker Page 8
Extended Log File Format
Robert Thau provided usefull advice and some code. John Mallery and
Roger Hurwitz helped develop many of the ideas.
Phillip M. Hallam-Baker
hallam@w3.org
World Wid Web Consortium
Cambridge MA
Brian Behlendorf
brian@organic.com
Organic Online
References.
[RFC1808]
R. Fielding _ Relative Uniform Resource Locators_, June 1995
[RFC1738]
T. Berners-Lee, L. Masinter, _ Uniform Resource Locators (URL)_,
December 1994
[Luotonen95]
A. luotonen, _ The Common Logfile Format_, 1995,
http://www.w3.org/pub/WWW/Daemon/User/Config/Logging.html
[Hallam96a]
P. M. Hallam-Baker _ Session Identification URI_ World Wide Web
Consortium Working Draft, WD-session-id.
[Hallam96b]
P. M. Hallam-Baker _ Notification for Proxy Caches_ World Wide
Web Consortium Working Draft, WD-proxy.
Phillip M. Hallam-Baker Page 9
