Subject: I need help to understand a log in mysql db - msg#00040

Previous Message by Date: click to view message preview

Re: How to convert syslog-ng logs into xml format

Hi, Padmanabhan, Rajeesh (GE Healthcare) <Rajeesh.Padmanabhan@xxxxxx> [20070211 16:36:58 +0530]: > > I have a syslog-ng clients & server. Clients should be able to transfer > logs in xml format to server. & server stores in a local location(xml > file). Idea is to upload this to web & backup, which can read data in > xml fomat. > I think you might have missed the key point behind my reply...you still are yet to give a reason why you need it in an XML format. If its for 'archiving' and 'retrieval' then you have just given two reasons *not* to use XML. As I mentioned, XML best suited for exporting and importing data[1] and not much else...you want live parsing of data and XML is the last thing you want to be using. Archiving logs is best done in raw flat text files, to make the data suitable for access via a web frontend you should consider some like pre-processing[1] the data and dumping it into an SQL database. This is the perfect storage format for accessing the data. Meanwhile you archive with flat textfiles. To restore the data you simply replay it back through the SQL pre-processor thingy. The way, when I get around to doing it, is to use SQL triggers to post-process the pre-processed data. So the 'pre-processing' stage would involve breaking it out into columns the main information[2] and then a post-processing trigger would (for example) pull out the SMTP message ID for an SMTP transaction and the to/from fields to dump things into an SMTP specific table. Seriously, you have not given a reason to use XML yet and I strongly recommend you go back to the drawing board and have a rethink. Do not feel bad about this, we all make this mistake, we are trying to help you to avoid it :) Cheers Alex [1] break out the timestamp, host IP, process, pid, log level, etc into separate columns [2] there are already piping scripts available for syslog-ng out there to do this > Could you please explain me to create a template that generate logs in > xml format. > > Cheers! > Rajeesh > > -----Original Message----- > From: syslog-ng-bounces@xxxxxxxxxxxxxxxx > [mailto:syslog-ng-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Alexander > Clouter > Sent: Sunday, February 11, 2007 4:21 PM > To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] How to convert syslog-ng logs into xml format > > Hi, > > Padmanabhan, Rajeesh (GE Healthcare) <Rajeesh.Padmanabhan@xxxxxx> > [20070211 15:40:27 +0530]: > > > > Hi All, > > > > I need to convert syslog-ng logs into xml format. Could you please > > help me to do the same. If someone can send me a sample format, would > > be great. > > > Oh dear...sounds like someone caught 'xml fever', it happens to all of > us at some stage and we usually recover from the illness when its too > late and the project is already using it... > > XML is something thats abused more often than its used properly. Its > useful if you have two rather different systems that have no common data > exchange mechanism between them and so XML is used as a 'common' format. > Bear in mind that this means is at both ends you have to get programming > tech-monkeys to write scripts to convert the data to XML and then back > from XML into another format. In situations like this you do not care > that it could take two weeks of solid CPU crunching to import the data, > just as long as it gets there... > > Now as you have not given much detail about why you need the files in an > XML format or what you plan on doing with the files the follow format is > probably > 'suitable': > > <?xml version="1.0" encoding="UTF-8"?> > <syslog> > <message>first syslog message</message> > <message>second syslog message</message> > ... > </syslog> > > I'm finding it hard to think of a situation where XML and syslog data > would be a good idea given that its rather easy to parse and store > syslog data already. People want to archive and easily search (usually > with nothing more than grep) their syslog logs and in an XML format > searching your logs *will* slow down by at least a factor of 1000. > > A far more suitable format, if flat text files is not good enough, is a > SQL database; probably the only real useful application of an SQL > database[1]. > > I'm unsure how to help you recover from your XML fever, it was difficult > for me to grow out of. I learn from my mistakes and unfortunately for > me when you base a project around XML the mistakes (for me at least) do > not have an effect till much later on. There is nothing on the Internet > as suggested reading but I am happy to discuss this off list if you want > to. > > I did some digging around to try to see if there was anything commercial > doing this and found[2] something and was shocked at how they were doing > it...its a bad schema and seems to be a solution to something that is > not actually a problem. > > *Everything* that deals with logging data should support syslog built > logfiles (you might need to use syslog-ng with its template() command to > structure the logfile to be in a format that the import engine expects > though), if it does not you should look for some other software. My gut > feelings tell me this is an internal homebrew project though.. > > I hope the 'fever' breaks soon :) > > Cheers > > Alex > > [1] thats another rant of mine though :) [2] > http://www.kiwisyslog.com/kb/idx/4/125/article/ > > _______________________________________________ > syslog-ng maillist - syslog-ng@xxxxxxxxxxxxxxxx > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > > > > _______________________________________________ > syslog-ng maillist - syslog-ng@xxxxxxxxxxxxxxxx > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > > _______________________________________________ syslog-ng maillist - syslog-ng@xxxxxxxxxxxxxxxx https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

Next Message by Date: click to view message preview

Re: log_msg_size() not used for syslog-ng internal messages

* Wolfram Schlich <lists@xxxxxxxxxxxxxxxxxxx> [2007-02-08 13:06]: > Hi, > > it seems that syslog-ng does not use log_msg_size() for its own > generated messages, because "Log statistics" messages are cut at > exactly 1024 characters, even if 'log_msg_size(32768)' is set > in syslog-ng.conf. > The used syslog-ng version is 2.0.2. Balazs, can you confirm my observation? -- Regards, Wolfram Schlich <wschlich@xxxxxxxxxx> Gentoo Linux * http://dev.gentoo.org/~wschlich/ _______________________________________________ syslog-ng maillist - syslog-ng@xxxxxxxxxxxxxxxx https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

Previous Message by Thread: click to view message preview

How to convert syslog-ng logs into xml format

Hi All,   I need to convert syslog-ng logs into xml format. Could you please help me to do the same. If someone can send me a sample format, would be great.   Thanks in advance.   -Rajeesh _______________________________________________ syslog-ng maillist - syslog-ng@xxxxxxxxxxxxxxxx https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

Next Message by Thread: click to view message preview

usertty endutent fix

Here's a fix for the usertty destination. Without it, you get one message to a user tty, then nothing, because the utmp database is positioned at the end, so future getutents return "no more entries." -- Bryan Henderson San Jose, California --- syslog-ng-2.0.2+20070202/src/afuser.c 2007-02-01 23:11:48.000000000 +0000 +++ syslog-ng/src/afuser.c 2007-02-12 00:28:32.000000000 +0000 @@ -87,6 +140,7 @@ } } } + endutent(); if ((path_flags & PF_FLOW_CTL_OFF) == 0) log_msg_ack(msg); log_msg_unref(msg); _______________________________________________ syslog-ng maillist - syslog-ng@xxxxxxxxxxxxxxxx https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html