Apache 2.x remote logging

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache 2.x remote logging

Fredrik Steen

I'm in the process of building a web farm consisting of 10-20-30 something web
servers handling 300k of virtual hosts and have started to look at how to
handle logging for these hosts.

Do anyone have any good ideas how to handle that massive amount of log files?
Preferably using some sort of remote log collector.

--
 .''`. Fredrik Steen, [hidden email]
: :' : 2CD6 C838 BE77 795F 5EF1  3E5B DA91 EE7B A58E 164
`. `'  http://www.stone.nu/
  `--  http://www.debian.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

k18
Reply | Threaded
Open this post in threaded view
|

Re: Apache 2.x remote logging

k18
Alle 09:01, martedì 31 maggio 2005, Fredrik Steen ha scritto:
> I'm in the process of building a web farm consisting of 10-20-30 something
> web servers handling 300k of virtual hosts and have started to look at how
> to handle logging for these hosts.
>
> Do anyone have any good ideas how to handle that massive amount of log
> files? Preferably using some sort of remote log collector.
>

You can do it in 2 ways. With pipe in apache loggin or you can use two scripts
wich can help you in that you want.
However you can find more here:
http://slacksite.com/apache/logging.html


PS: Excuse me for my "bad" english... I'm italian


 --
 Email.it, the professional e-mail, gratis per te: http://www.email.it/f

 Sponsor:
 Occhialeria.it
* Scopri le migliori marche a prezzi imbattibili
 Clicca qui: <a href="http://adv.email.it/cgi-bin/foclick.cgi?mid‡9&d1-5">http://adv.email.it/cgi-bin/foclick.cgi?mid‡9&d1-5

Reply | Threaded
Open this post in threaded view
|

Re: Apache 2.x remote logging

Christian Hammers
In reply to this post by Fredrik Steen
Hello

On 2005-05-31 Fredrik Steen wrote:
> I'm in the process of building a web farm consisting of 10-20-30 something web
> servers handling 300k of virtual hosts and have started to look at how to
> handle logging for these hosts.

Don't know if it's performant enough but you could just use a script like
logger to do remote syslogging:
    CustomLog "| /usr/bin/logger -p local7.info -t www.example.com" ...

bye,

-christian-


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache 2.x remote logging

Fredrik Steen-2
In reply to this post by k18
k18 <[hidden email]> writes:

> Alle 09:01, martedì 31 maggio 2005, Fredrik Steen ha scritto:
>> I'm in the process of building a web farm consisting of 10-20-30 something
>> web servers handling 300k of virtual hosts and have started to look at how
>> to handle logging for these hosts.
>>
>> Do anyone have any good ideas how to handle that massive amount of log
>> files? Preferably using some sort of remote log collector.
>>
>
> You can do it in 2 ways. With pipe in apache loggin or you can use two scripts
> wich can help you in that you want.
> However you can find more here:
> http://slacksite.com/apache/logging.html

The log files will be huge from each web server and when merged sorted the
resulting log file will be enormous >20GB and then have to be split and stats
generated for each log file, don't think that will hold in the long run.

Tried merging 4 x 500MB access_log with mergelog[1] took 12 minutes on
standard machine (2.2Ghz 1G RAM (hw)Raid5 SCSI 10k disks)

Have anyone tested mod_log_spread with good result especially with the apache
2.x version?

> PS: Excuse me for my "bad" english... I'm italian

Same here I'm Swedish :)

[1] apt-cache show mergelog

--
 .''`. Fredrik Steen, [hidden email]
: :' : 2CD6 C838 BE77 795F 5EF1  3E5B DA91 EE7B A58E 164
`. `'  http://www.stone.nu/
  `--  http://www.debian.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache 2.x remote logging

Fredrik Steen
In reply to this post by k18
k18 <[hidden email]> writes:

> Alle 09:01, martedì 31 maggio 2005, Fredrik Steen ha scritto:
>> I'm in the process of building a web farm consisting of 10-20-30 something
>> web servers handling 300k of virtual hosts and have started to look at how
>> to handle logging for these hosts.
>>
>> Do anyone have any good ideas how to handle that massive amount of log
>> files? Preferably using some sort of remote log collector.
>>
>
> You can do it in 2 ways. With pipe in apache loggin or you can use two scripts
> wich can help you in that you want.
> However you can find more here:
> http://slacksite.com/apache/logging.html

The log files will be huge from each web server and when merged sorted the
resulting log file will be enormous >20GB and then have to be split and stats
generated for each log file, don't think that will hold in the long run.

Tried merging 4 x 500MB access_log with mergelog[1] took 12 minutes on
standard machine (2.2Ghz 1G RAM (hw)Raid5 SCSI 10k disks)

Have anyone tested mod_log_spread with good result especially with the apache
2.x version?

> PS: Excuse me for my "bad" english... I'm italian

Same here I'm Swedish :)

[1] apt-cache show mergelog

--
 .''`. Fredrik Steen, [hidden email]
: :' : 2CD6 C838 BE77 795F 5EF1  3E5B DA91 EE7B A58E 164
`. `'  http://www.stone.nu/
  `--  http://www.debian.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

VS: Apache 2.x remote logging

Tommi Pisto
In reply to this post by Christian Hammers
I found somewhere a perl script that does the logging separation.

Add to the httpd.conf
--- clip ---
    LogFormat "%V|%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-agent}i\""
    TransferLog |/usr/sbin/logger
--- clap ---

Add this script to the /usr/sbin/logger -file

--- clip ---
#!/usr/bin/perl -w

while($input = <STDIN>) {

    # Split string
    ($filename,$log_string) = split(/\|/,$input);

    # Output to file
    $file = "/home/www/logs/access_".$filename.".log";

    open(FILE_OUT,"+>>$file");
    print FILE_OUT $log_string;
    close(FILE_OUT);
}
--- clap ---

And now you will get all the virtual domain access logs to the separate
/home/www/logs/access_<domain name>.log -files

Best regards,

  Tommi Pisto  


-----Alkuperäinen viesti-----
Lähettäjä: Christian Hammers [mailto:[hidden email]]
Lähetetty: 31. toukokuuta 2005 16:00
Vastaanottaja: Fredrik Steen
Kopio: [hidden email]
Aihe: Re: Apache 2.x remote logging

Hello

On 2005-05-31 Fredrik Steen wrote:
> I'm in the process of building a web farm consisting of 10-20-30
> something web servers handling 300k of virtual hosts and have started
> to look at how to handle logging for these hosts.

Don't know if it's performant enough but you could just use a script like
logger to do remote syslogging:
    CustomLog "| /usr/bin/logger -p local7.info -t www.example.com" ...

bye,

-christian-


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Apache 2.x remote logging

Fredrik Steen
In reply to this post by Christian Hammers
Christian Hammers <[hidden email]> writes:

> Hello
>
> On 2005-05-31 Fredrik Steen wrote:
>> I'm in the process of building a web farm consisting of 10-20-30 something web
>> servers handling 300k of virtual hosts and have started to look at how to
>> handle logging for these hosts.
>
> Don't know if it's performant enough but you could just use a script like
> logger to do remote syslogging:
>     CustomLog "| /usr/bin/logger -p local7.info -t www.example.com" ...

Thanks, I tried that and got:
...
May 31 15:19:06 localhost last message repeated 3 times
...
entries in syslog. Changed logger to add a unique number for each entry and
those went away but I'm not sure if syslog will handle the flood of log
entries sent. Maybe someone have done this with good results?

--
 .''`. Fredrik Steen, [hidden email]
: :' : 2CD6 C838 BE77 795F 5EF1  3E5B DA91 EE7B A58E 164
`. `'  http://www.stone.nu/
  `--  http://www.debian.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: VS: Apache 2.x remote logging

Andreas John
In reply to this post by Tommi Pisto
Tommi Pisto schrieb:

> #!/usr/bin/perl -w
>
> while($input = <STDIN>) {
>
>     # Split string
>     ($filename,$log_string) = split(/\|/,$input);
>
>     # Output to file
>     $file = "/home/www/logs/access_".$filename.".log";
>
>     open(FILE_OUT,"+>>$file");
>     print FILE_OUT $log_string;
>     close(FILE_OUT);
> }

This won't go remote? It is similar to apt-cache search vlogger. But
Vlogger can write into a (remote) mysql DB.

rgds,
Andreas


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache 2.x remote logging

Christian Hammers
In reply to this post by Fredrik Steen


On 2005-05-31 Fredrik Steen wrote:
> May 31 15:19:06 localhost last message repeated 3 times
> ...
> entries in syslog. Changed logger to add a unique number for each entry and
> those went away but I'm not sure if syslog will handle the flood of log
> entries sent. Maybe someone have done this with good results?

Try syslog-ng or another more sophisticated daemon which supports permanent
TCP connections and buffersize options (to reduce loss) and splitting the log
to different files according to a configurable pattern.

At least I hope that a recent syslog daemon should be the most optimized
program for receiving log files over the net (although AFAIK the standard
/usr/bin/logger program does not speak TCP).

bye,

-christian-


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Apache 2.x remote logging

Christian Kujau-6
In reply to this post by Fredrik Steen
Fredrik Steen wrote:
> Do anyone have any good ideas how to handle that massive amount of log files?
> Preferably using some sort of remote log collector.

on apache1.3 we have libapache-mod-log-sql, to be found here:
   ftp://ftp.gplhost.com/debian
we even got the author of modlogan [1] to write a plugin, so we're loggin
directly into an mysql-db, and generate stats out of the db. pretty cool,
pretty fast - but not available for apache2 :-(

[1] http://jan.kneschke.de/projects/modlogan/
--
BOFH excuse #445:

Browser's cookie is corrupted -- someone's been nibbling on it.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]