Incease allowable size of uploads

With the new diagnostics file structure for capturing system files and device health, the resulting file is often too large to be uploaded with the current size restriction.  It must be realized that if the system has been up for a while before the problem occurs, the file will already be large but often the problem condition will generate a lot of repeating error messages which quickly inflate the size the the file. 

This restriction makes it difficult for the volunteer helpers and must also increase the difficulties of the LT support team.  While we can tell the person with the issue to upload the offending file to an file sharing website, many people do not have an account setup on such a site and requesting them to do so for a single use seems to a real imposition.

Increasing the upload size restriction is an option, but I can't help feeling that that could potentially increase LimeTech costs.  I decided I wanted to understand the problem better, as to why some are so big, and what can be done.  I've looked through the diagnostics files I had access to, and found that in ALL cases it is an exploding syslog that is the problem.  If anyone knows of other files that break the size limit, please speak up.  I found one docker.log of 400K, but that compresses to 4K, practically nothing.  To be a problem, the syslog has to be larger than something like 15MB!  And from experience with syslogs, I know that there's nothing interesting after the first few megabytes.  In every case I know of, when errors start repeating, it's ALWAYS the very first errors that you are interested in.  It's interesting to know it's repeating, but you don't need many of them to know that.  So if we were to truncate syslogs to 10MB, I can't imagine anyone anywhere that would complain!  If some one does, because they enjoy examining multiple megabytes of repeating error messages, please speak up!  I thought of just splitting off huge syslogs from the diagnostics, into syslog.zip and diagnostics.zip, but that introduces confusion, and mistakes in following instructions, additional support hassles.  I really think the simplest and cleanest solution is detect syslog files over 10MB and truncate them to 10MB.  Of course, the huge syslog files may not be 'syslog', they may be 'syslog.1', 'syslog.2', etc.

  if size of syslog_file > 10MB

      tail --lines=200 /var/log/syslog_file >/tmp/diagnostics/logs/tail.txt

      truncate -s 10M /var/log/syslog_file

  fi

And build the diagnostics file as usual.  You even have the last 200 lines, just in case they're interesting.  This isn't likely to cut the syslog file immediately after a line break, but who cares, it will be a broken line 10MB out!

Because they're text files, syslog files typically compress to about 11%.  Huge syslogs are mostly repeating errors which compress much more, to about 1%.

I personally have not had a problem but a new user has a problem with his system and his diagnostic file was too big to upload.  See this posting in his thread:

    http://lime-technology.com/forum/index.php?topic=48058.msg462346#msg462346

In fact, from one of his posts, I am not sure he has yet purchased the software but I have the feeling that if he can't resolve this issue, he is going to find another solution.  Now, how does one tell a Noob to run a shell script?  It is hard to tell where in the log that the problem occurred.  If there are 30MB of repeating errors, that is not of much use to someone who can actually interpret system logs.  And I keep reading that it usually the last few entries in the log BEFORE the repeating errors start that are the most important clues as to what is happening. 

I just downloaded my diagnostics file for my MEDIA server that has been up for 33 days and it is 59KB.  The syslog is 186KB which is compressed to 26KB.  The upload limit is 192KB.  Perhaps, the size limit for .zip files could be increased while retaining the current limits for other file types.  (I fully realize that there are persons who would upload 4MB photos of monitor screens (and server builds) if there weren't some restrictions!) 

PS---  My calculations says that the problem would begin before the system log gets to be about 1.3MB in size. 

I couldn't quite tell from your post what your opinion was.  You seemed to be agreeing with me on some points, including the fact that we don't need the extra megabytes of error messages, but you still prefer increasing the upload limit over truncating the useless megabytes?

On the size calculations, I think you are extrapolating from your own syslog and its current compression percentage.  Typical syslogs appear to have roughly 10% to 20% of repetition, resulting in about 11% to your 14% compression.  But when you add megabytes of repeating errors, the repetition approaches 99% and the compression drops to close to 1%, about 10 times the compression, meaning the syslog can be 10 times as big.  I've seen a 12MB syslog piece that compresses to 150KB.  Truncating to 10MB should be fine, but if others wish, truncating at 5MB is fine too.

Actually, I was hoping for some input from LimeTech on their feelings about this problem/issue.  Perhaps it is a simple matter to increase file size with a minimum of additional costs.  Perhaps, the file size increase could be restricted to just .zip files.  I know they do limit the extensions that they allow.

I see that you are now in a discussion with the OP in his thread about the file size.  Perhaps, he can give you a better picture of what is going on.  Maybe, you should ask him in a PM to e-mail you and attach the file as an attachment. 

He's been very helpful, quickly answered and provided the critical numbers.  This time it was the docker.log that blew it up, 133MB worth.  Assuming that's also in /var/log (can't check myself just now), then that probably filled the folder, blocking further logging.  So the same truncate technique should probably be applied to any logging file over 5MB in that folder, with the possible nice side effect that requesting Diagnostics could clear up a lot of room there, perhaps 120MB or more.

I'm not in favor of increasing the limit, because we're trying to control abuse.  Besides, I don't think it helps here, because if a log file blows up, it's quickly into multi-megabytes.  Increasing from 192 to 350 probably won't help with any of the larger ones.  The diagnostics files are probably going to be under 100KB or over 1MB.  And who wants a 10MB (or larger) syslog?  No one wants to examine the last 9MB of that, plus it's almost certain to not be useful.  Chopping them makes them all fit within the limits.

Based on the above, I've created a feature request.

I am having issues with my machine, and i can't easily upload the diagnostics file, it just happens that Frank1940 was one of the folks trying to help me as well.  I just found this researching before i posted a question in how to post larger diagnostics files.

Just upload it to pastebin and link it?